[LU-2559] test: lustre-rsync-test test 1 - setfattr <file path>: Operation not supported

• Type: Bug
• Resolution: Won't Fix
• Priority: Blocker
• Fix Version/s: Lustre 2.5.0
• Affects Version/s: Lustre 2.3.0, Lustre 2.4.0, Lustre 2.1.3
• Labels: LB

Environment

Server: 2.1.3-1nasS, centos 6.3, 2.6.32_279.2.1.el6
Client: 2.3.0-2nasC, sles11sp2, 3.0.42_0.7.3
 mds: service337
 oss: service361, service362
clients: service331, service332
Git source: https://github.com/jlan/lustre-nas

• Severity: 3
• Assignee: Bob Glossman (Inactive)
• Reporter: Jay Lan (Inactive)

Description

== lustre-rsync-test test 1: Simple Replication ====================================================== 23:39:05 (1357025945)
lustre-MDT0000: Registered changelog user cl1
Replication #1
Lustre filesystem: lustre
MDT device: lustre-MDT0000
Source: /mnt/nbp0-1
Target: /var/acc-sm/target
Target: /var/acc-sm/target2
Statuslog: /var/acc-sm/lustre_rsync.log
Changelog registration: cl1
Starting changelog record: 0
Clear changelog after use: no
Errors: 0
lustre_rsync took 0 seconds
Changelog records consumed: 20
setfattr: /mnt/nbp0-1/d0.lustre-rsync-test/d1/file5: Operation not supported
Replication #2
Replication of operation failed(-17): 20 SLINK (4) [0x200000400:0xe:0x0] [0x200000400:0x3:0x0] link3
Lustre filesystem: lustre
MDT device: lustre-MDT0000
Source: /mnt/nbp0-1
Target: /var/acc-sm/target
Target: /var/acc-sm/target2
Statuslog: /var/acc-sm/lustre_rsync.log
Changelog registration: cl1
Starting changelog record: 20
Clear changelog after use: no
Errors: 1
lustre_rsync took 0 seconds
Changelog records consumed: 4
/var/acc-sm/target/d0.lustre-rsync-test/d1/file5: user.foo: No such attribute
/var/acc-sm/target2/d0.lustre-rsync-test/d1/file5: user.foo: No such attribute
lustre-rsync-test test_1: @@@@@@ FAIL: Error in replicating xattrs.
Trace dump:
= /usr/lib64/lustre/tests/test-framework.sh:3643:error_noexit()
= /usr/lib64/lustre/tests/test-framework.sh:3665:error()
= /usr/lib64/lustre/tests/lustre-rsync-test.sh:193:test_1()
= /usr/lib64/lustre/tests/test-framework.sh:3907:run_one()
= /usr/lib64/lustre/tests/test-framework.sh:3937:run_one_logged()
= /usr/lib64/lustre/tests/test-framework.sh:3808:run_test()
= /usr/lib64/lustre/tests/lustre-rsync-test.sh:205:main()
Dumping lctl log to /var/acc-sm/test_logs/lustre-rsync-test.test_1.*.1357025946.log
FAIL 1 (3s)

Comments

[Peter Jones]

ok then close away!

[Bob Glossman (Inactive)]

Peter
Sure looks like our test setup takes care of it. In a full run from 12/19 of master client on 2.1 server lustre-rsync-test passed fine.

Looking in the log of the provisioning phase, I see:

21:01:04:MDS_MOUNT_OPTS=${MDS_MOUNT_OPTS:-"-o user_xattr,acl"}

Later on those opts show up explicitly on the command line of the MDS mount command:

21:01:38:CMD: client-20-ib mkdir -p /mnt/mds1; mount -t lustre -o user_xattr,acl /dev/lvm-MDS/P1 /mnt/mds1

I think there's nothing needed to be done here.

[Peter Jones]

Bob

You should be able to find recent interop test runs to see whether we are handling this already. If not then please could you open a TT ticket outlining the changes that we need to make to avoid this unnecessary noise in our test results.

Thanks

Peter

[Bob Glossman (Inactive)]

Peter, Pretty sure this will impact any newer version client trying to run on 2.1 servers. All MDS_MOUNT_OPTS are blank by default in standard configs. What I'm not sure about is if our test environment does something to work around this during interop tests. Since it can easily be masked entirely by just setting environment variables, our test setups may already do so on Toro. If we routinely do new client on old server during our regular interop testing, I don't see how we could have avoided seeing the problem without such setup.

[Jay Lan (Inactive)]

Yes, Bob, it is OK to close it if you do not need to make any change. Thanks.

[Peter Jones]

Bob

Before you close the ticket - will this issue affect 2.4 clients trying to interoperate with 2.1 servers? If so, we should land a change to master to avoid this creating noise in our interop testing for 2.4

Peter

[Bob Glossman (Inactive)]

Thanks for the confirmation, Jay.
Is it OK then for me to go ahead and close this bug? Is it sufficient to know that setting extra cfg or environment variables is needed when interoperating with 2.3 clients on 2.1 servers?

[Jay Lan (Inactive)]

Bob, you are right again!

After adding user_xattr to MDS_MOUNT_OPTS, it worked!

I did not include lustre-rsync-test when I converted my test scripts from acc-sm to auster until recently. That explains why I did not get hit by the problem earlier.

[Bob Glossman (Inactive)]

flags reported by mount differ from those shown in /proc/mounts. Here are both.

2.3 client:

# mount
centos2:/lustre on /mnt/lustre type lustre (rw,user_xattr,flock)

# cat /proc/mounts
192.168.0.36@tcp:/lustre /mnt/lustre lustre rw,relatime,flock 0 0

2.1 server:

# mount
/dev/sdb on /mnt/mds1 type lustre (rw)

# cat /proc/mounts
/dev/sdb /mnt/mds1 lustre ro 0 0

I think I see where you are going with this. Looks like MGS/MDS gets mounted on the server without user_xattr set. Looking at the test scripts I see a difference between 2.1 and 2.3. 2.1 has MDS_MOUNT_OPTS explicitly defined with user_xattr in it in cfg/local.sh, 2.3 has empty opts.

I'm guessing that in the 2.3 timeframe server mounts always do user_xattr by default and no longer require explicit flags. This messes up when using script & cfg files from 2.3 on 2.1 servers.

Jay, can you check this out by adding explicit
MDS_MOUNT_OPTS="-o user_xattr,acl"
to your cfg file on clients?
If you just copied or modified the cfg/local.sh from the build these are set empty.
Setting MDS_MOUNT_OPTS as an environment variable should work too.

Try the test with this change.
You can do the failing test alone with:

auster -rv lustre-rsync-test --only 1

[nasf (Inactive)]

Can you perform "mount" on both 2.1 server and 2.3 client to check (and paste out) which flags have been enabled when you met the issues "Lustre: Disabling user_xattr feature because it is not supported on the server"? Thanks.