https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2559 Lustre <p>== lustre-rsync-test test 1: Simple Replication ====================================================== 23:39:05 (1357025945)<br/> lustre-MDT0000: Registered changelog user cl1<br/> Replication #1<br/> Lustre filesystem: lustre<br/> MDT device: lustre-MDT0000<br/> Source: /mnt/nbp0-1<br/> Target: /var/acc-sm/target<br/> Target: /var/acc-sm/target2<br/> Statuslog: /var/acc-sm/lustre_rsync.log<br/> Changelog registration: cl1<br/> Starting changelog record: 0<br/> Clear changelog after use: no<br/> Errors: 0<br/> lustre_rsync took 0 seconds<br/> Changelog records consumed: 20<br/> setfattr: /mnt/nbp0-1/d0.lustre-rsync-test/d1/file5: Operation not supported<br/> Replication #2<br/> Replication of operation failed(-17): 20 SLINK (4) <span class="error">&#91;0x200000400:0xe:0x0&#93;</span> <span class="error">&#91;0x200000400:0x3:0x0&#93;</span> link3<br/> Lustre filesystem: lustre<br/> MDT device: lustre-MDT0000<br/> Source: /mnt/nbp0-1<br/> Target: /var/acc-sm/target<br/> Target: /var/acc-sm/target2<br/> Statuslog: /var/acc-sm/lustre_rsync.log<br/> Changelog registration: cl1<br/> Starting changelog record: 20<br/> Clear changelog after use: no<br/> Errors: 1<br/> lustre_rsync took 0 seconds<br/> Changelog records consumed: 4<br/> /var/acc-sm/target/d0.lustre-rsync-test/d1/file5: user.foo: No such attribute<br/> /var/acc-sm/target2/d0.lustre-rsync-test/d1/file5: user.foo: No such attribute<br/> lustre-rsync-test test_1: @@@@@@ FAIL: Error in replicating xattrs.<br/> Trace dump:<br/> = /usr/lib64/lustre/tests/test-framework.sh:3643:error_noexit()<br/> = /usr/lib64/lustre/tests/test-framework.sh:3665:error()<br/> = /usr/lib64/lustre/tests/lustre-rsync-test.sh:193:test_1()<br/> = /usr/lib64/lustre/tests/test-framework.sh:3907:run_one()<br/> = /usr/lib64/lustre/tests/test-framework.sh:3937:run_one_logged()<br/> = /usr/lib64/lustre/tests/test-framework.sh:3808:run_test()<br/> = /usr/lib64/lustre/tests/lustre-rsync-test.sh:205:main()<br/> Dumping lctl log to /var/acc-sm/test_logs/lustre-rsync-test.test_1.*.1357025946.log<br/> FAIL 1 (3s)</p> Server: 2.1.3-1nasS, centos 6.3, 2.6.32_279.2.1.el6<br/> Client: 2.3.0-2nasC, sles11sp2, 3.0.42_0.7.3<br/> &nbsp;mds: service337<br/> &nbsp;oss: service361, service362<br/> clients: service331, service332<br/> Git source: <a href="https://github.com/jlan/lustre-nas">https://github.com/jlan/lustre-nas</a> LU-2559

test: lustre-rsync-test test 1 - setfattr <file path>: Operation not supported

Bug Blocker Resolved Won't Fix Bob Glossman Jay Lan LB Wed, 2 Jan 2013 18:32:06 +0000 Tue, 15 Oct 2013 22:04:35 +0000 Fri, 11 Jan 2013 16:18:45 +0000 Lustre 2.3.0 Lustre 2.4.0 Lustre 2.1.3 Lustre 2.5.0 0 4 <p>Bob</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>So far I haven't been able to reproduce the failure. I have been trying to vary the client side only. It will take me a bit longer to bring up a 2.1 server in case it's the server side that causes the problem. I will keep trying to reproduce it.</p> <p>Looks like this is an interop problem between 2.3 clients and 2.1 servers. The key fact is the following error that shows up in client dmesg at mount time:</p> <p>Lustre: Disabling user_xattr feature because it is not supported on the server</p> <p>This happens with any 2.3 or 2.3+ client, sles11 or centos.<br/> Once the client is mounted with user_xattr disabled, any setfattr command attempted by the client will fail.</p> <p>Not sure exactly why the 2.3 client thinks the 2.1 server is incapable of doing user_xattr.</p> <p>nasf, adding you to the watcher list as Peter asked. This bug seems to be due to version interop problems with connect flags. Do you know anything in the 2.3 timeframe that might be related? It was suggested you might know or have worked in this area. Just looking at the 2.1 vs. 2.3 code I haven't been able to spot anything obvious, all the flag definitions and so forth look compatible.</p> <p>I saw this error again when testing the SUSE version of lustre-2.1.3 client for sles11sp2. Note that my servers are still 2.1.3.</p> <p>Jay, are you sure you have a 2.1.3 client for sles11 sp2? I think the standard prebuilt rpms for sles11 are for sp1, not sp2. I haven't been able to build any client 2.1 for sp2. I've only succeeded in building client 2.3 and master for sp2.</p> <p>you could do lctl lustre_build_version on the client to double check.</p> <p>Bob, forget about the 2.1.3 client I mentioned in earlier post on Jan 10. That was a beta version from SUSE. Since Peter said Intel would not support that version, it is irrelevant.</p> <p>But I did experience that problem in 2.3.0 though as I first reported.</p> <p>Can you perform "mount" on both 2.1 server and 2.3 client to check (and paste out) which flags have been enabled when you met the issues "Lustre: Disabling user_xattr feature because it is not supported on the server"? Thanks.</p> <p>flags reported by mount differ from those shown in /proc/mounts. Here are both.</p> <p>2.3 client:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># mount centos2:/lustre on /mnt/lustre type lustre (rw,user_xattr,flock) # cat /proc/mounts 192.168.0.36@tcp:/lustre /mnt/lustre lustre rw,relatime,flock 0 0 </pre> </div></div> <p>2.1 server:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># mount /dev/sdb on /mnt/mds1 type lustre (rw) # cat /proc/mounts /dev/sdb /mnt/mds1 lustre ro 0 0 </pre> </div></div> <p>I think I see where you are going with this. Looks like MGS/MDS gets mounted on the server without user_xattr set. Looking at the test scripts I see a difference between 2.1 and 2.3. 2.1 has MDS_MOUNT_OPTS explicitly defined with user_xattr in it in cfg/local.sh, 2.3 has empty opts.</p> <p>I'm guessing that in the 2.3 timeframe server mounts always do user_xattr by default and no longer require explicit flags. This messes up when using script &amp; cfg files from 2.3 on 2.1 servers.</p> <p>Jay, can you check this out by adding explicit<br/> MDS_MOUNT_OPTS="-o user_xattr,acl"<br/> to your cfg file on clients?<br/> If you just copied or modified the cfg/local.sh from the build these are set empty.<br/> Setting MDS_MOUNT_OPTS as an environment variable should work too.</p> <p>Try the test with this change.<br/> You can do the failing test alone with:</p> <p>auster -rv lustre-rsync-test --only 1</p> <p>Bob, you are right again! <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>After adding user_xattr to MDS_MOUNT_OPTS, it worked!</p> <p>I did not include lustre-rsync-test when I converted my test scripts from acc-sm to auster until recently. That explains why I did not get hit by the problem earlier.</p> <p>Thanks for the confirmation, Jay.<br/> Is it OK then for me to go ahead and close this bug? Is it sufficient to know that setting extra cfg or environment variables is needed when interoperating with 2.3 clients on 2.1 servers?</p> <p>Bob</p> <p>Before you close the ticket - will this issue affect 2.4 clients trying to interoperate with 2.1 servers? If so, we should land a change to master to avoid this creating noise in our interop testing for 2.4</p> <p>Peter</p> <p>Yes, Bob, it is OK to close it if you do not need to make any change. Thanks.</p> <p>Peter, Pretty sure this will impact any newer version client trying to run on 2.1 servers. All MDS_MOUNT_OPTS are blank by default in standard configs. What I'm not sure about is if our test environment does something to work around this during interop tests. Since it can easily be masked entirely by just setting environment variables, our test setups may already do so on Toro. If we routinely do new client on old server during our regular interop testing, I don't see how we could have avoided seeing the problem without such setup.</p> <p>Bob</p> <p>You should be able to find recent interop test runs to see whether we are handling this already. If not then please could you open a TT ticket outlining the changes that we need to make to avoid this unnecessary noise in our test results.</p> <p>Thanks</p> <p>Peter</p> <p>Peter<br/> Sure looks like our test setup takes care of it. In a full run from 12/19 of master client on 2.1 server lustre-rsync-test passed fine.</p> <p>Looking in the log of the provisioning phase, I see:</p> <p>21:01:04:MDS_MOUNT_OPTS=${MDS_MOUNT_OPTS:-"-o user_xattr,acl"}</p> <p>Later on those opts show up explicitly on the command line of the MDS mount command:</p> <p>21:01:38:CMD: client-20-ib mkdir -p /mnt/mds1; mount -t lustre -o user_xattr,acl /dev/lvm-MDS/P1 /mnt/mds1</p> <p>I think there's nothing needed to be done here.</p> <p>ok then close away! <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> Development Rank 1|hzvegv: Rank (Obsolete) 5985 Severity