https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4016 Lustre <p>Test results at <a href="https://maloo.whamcloud.com/test_sessions/eb9d697c-2658-11e3-8d26-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/eb9d697c-2658-11e3-8d26-52540035b04c</a></p> <p>metadata-updates fails with "attributes check failed". Node logs don't exist, but the suite_log shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Checking file(s) attributes ... c08: /lustre/scratch/d0.metadata-updates/c07/testfile [ uid gid size mode ] expected : 0 0 1024 0644 ; got : root 0 1024 0644 c08: Wrong file attributes c09: /lustre/scratch/d0.metadata-updates/c07/testfile [ uid gid size mode ] expected : 0 0 1024 0644 ; got : root 0 1024 0644 c09: Wrong file attributes pdsh@c10: c08: ssh exited with exit code 56 c07: /lustre/scratch/d0.metadata-updates/c07/testfile [ uid gid size mode ] expected : 0 0 1024 0644 ; got : root 0 1024 0644 c07: Wrong file attributes pdsh@c10: c09: ssh exited with exit code 56 pdsh@c10: c07: ssh exited with exit code 56 c10: /lustre/scratch/d0.metadata-updates/c07/testfile [ uid gid size mode ] expected : 0 0 1024 0644 ; got : root 0 1024 0644 c10: Wrong file attributes pdsh@c10: c10: ssh exited with exit code 56 metadata-updates : @@@@@@ FAIL: attributes check failed Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4264:error_noexit() = /usr/lib64/lustre/tests/metadata-updates.sh:227:main() </pre> </div></div> <p>Although this was run during HSM testing, the copytool was not started and HSM was not enabled on the MDS, but all HSM attributes were in the test configuration files.</p> Build # 1687<br/> OpenSFS cluster with combined MGS/MDS, single OSS with two OSTs, four clients; one agent (c07), one with robinhood/db running (c08) and just running as Lustre clients (c09, c10) LU-4016

metadata-updates failure: attributes check failed

Bug Minor Resolved Fixed James Nunez James Nunez Thu, 26 Sep 2013 22:06:07 +0000 Tue, 31 Dec 2013 15:45:22 +0000 Fri, 20 Dec 2013 15:14:09 +0000 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.4.2 Lustre 2.5.1 0 7 <p>Another similar failure while HSM was enabled: <a href="https://maloo.whamcloud.com/test_sessions/277b4f38-2779-11e3-8a10-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/277b4f38-2779-11e3-8a10-52540035b04c</a></p> <p>There are additional failures, after truncate and chmod:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Checking file(s) attributes ... c10: /lustre/scratch/d0.metadata-updates/c08/testfile [ uid gid size mode ] expected : 0 0 1024 0644 ; got : root 0 1024 0644 c10: Wrong file attributes c08: /lustre/scratch/d0.metadata-updates/c08/testfile [ uid gid size mode ] expected : 0 0 1024 0644 ; got : root 0 1024 0644 c08: Wrong file attributes pdsh@c10: c10: ssh exited with exit code 56 c09: /lustre/scratch/d0.metadata-updates/c08/testfile [ uid gid size mode ] expected : 0 0 1024 0644 ; got : root 0 1024 0644 c09: Wrong file attributes pdsh@c10: c08: ssh exited with exit code 56 pdsh@c10: c09: ssh exited with exit code 56 metadata-updates : @@@@@@ FAIL: attributes check failed Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4264:error_noexit() = /usr/lib64/lustre/tests/metadata-updates.sh:227:main() Dumping lctl log to /tmp/test_logs/2013-09-26/124150/metadata-updates..*.1380231630.log Part 2. file(s) attributes modification. Performing chmod 00222 ... Checking file(s) attributes ... c10: /lustre/scratch/d0.metadata-updates/c08/testfile [ uid gid size mode ] expected : 0 0 1024 0222 ; got : root 0 1024 0222 c10: Wrong file attributes pdsh@c10: c10: ssh exited with exit code 56 c09: /lustre/scratch/d0.metadata-updates/c08/testfile [ uid gid size mode ] expected : 0 0 1024 0222 ; got : root 0 1024 0222 c09: Wrong file attributes c08: /lustre/scratch/d0.metadata-updates/c08/testfile [ uid gid size mode ] expected : 0 0 1024 0222 ; got : root 0 1024 0222 c08: Wrong file attributes pdsh@c10: c08: ssh exited with exit code 56 pdsh@c10: c09: ssh exited with exit code 56 metadata-updates : @@@@@@ FAIL: wrong attributes after chmod Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4264:error_noexit() = /usr/lib64/lustre/tests/metadata-updates.sh:232:main() Dumping lctl log to /tmp/test_logs/2013-09-26/124150/metadata-updates..*.1380231632.log Changing atime, mtime ... Checking atime, mtime ... Part 3. truncate file(s) to 0 size, check new file size. Truncating file(s) ... Checking file(s) attributes ... c10: /lustre/scratch/d0.metadata-updates/c08/testfile [ uid gid size mode ] expected : 0 0 0 0222 ; got : root 0 0 0222 c10: Wrong file attributes pdsh@c10: c10: ssh exited with exit code 56 c08: /lustre/scratch/d0.metadata-updates/c08/testfile [ uid gid size mode ] expected : 0 0 0 0222 ; got : root 0 0 0222 c08: Wrong file attributes pdsh@c10: c08: ssh exited with exit code 56 c09: /lustre/scratch/d0.metadata-updates/c08/testfile [ uid gid size mode ] expected : 0 0 0 0222 ; got : root 0 0 0222 c09: Wrong file attributes pdsh@c10: c09: ssh exited with exit code 56 metadata-updates : @@@@@@ FAIL: wrong attributes after truncate Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4264:error_noexit() = /usr/lib64/lustre/tests/metadata-updates.sh:240:main() Dumping lctl log to /tmp/test_logs/2013-09-26/124150/metadata-updates..*.1380231634.log </pre> </div></div> <p>Another failure at <a href="https://maloo.whamcloud.com/test_sessions/4574cfe8-35e1-11e3-b051-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/4574cfe8-35e1-11e3-b051-52540035b04c</a></p> <p>I've been seeing this on rhel and sles - I think since I started testing with 1.8.5. I observed it on 2.4.1 today. Infiniband cluster, two clients, one MDS, two OSS with one OST each. Upcall is default, nsswitch point to files nis.</p> <p>I've never looked at it much - I just excluded the failing tests. It looks like stat is reporting %U, instead of %u. Can't say why. I looked for a bug in coreutils, but didn't see anything obvious. I've never observed this behavior outside of running acceptance.</p> <p> for HOST in ${HOSTS//,/ } ; do<br/> TESTFILE=$TESTDIR/\\\$HOST/$FILE;<br/> + echo \\\stat -c \\\"%u %g %s 0%a\\\" \\\$TESTFILE<br/> tmp=\\\$(stat -c \\\"%u %g %s 0%a\\\" \\\$TESTFILE);<br/> echo \\\"\\\$TESTFILE [ uid gid size mode ] expected : $attr ; got : \\\$tmp \\\";</p> <p>n006: stat -c root %g %s 0%a /mnt/accfs/d0.metadata-updates/n006/testfile<br/> n007: stat -c root %g %s 0%a /mnt/accfs/d0.metadata-updates/n006/testfile</p> <p>So our %u was eaten by pdsh:</p> <ol> <li>pdsh -w n006,n007 echo '%u'<br/> n007: root<br/> n006: root</li> <li>pdsh -L<br/> 3 modules loaded:</li> </ol> <p>Module: rcmd/exec<br/> Author: Mark Grondona &lt;mgrondona@llnl.gov&gt;<br/> Descr: arbitrary command rcmd connect method<br/> Active: yes</p> <p>Module: misc/netgroup<br/> Author: Mark Grondona &lt;mgrondona@llnl.gov&gt;<br/> Descr: Target netgroups from pdsh<br/> Active: yes<br/> Options:<br/> -g groupname target hosts in netgroup "groupname"<br/> -X groupname exclude hosts in netgroup "groupname"</p> <p>Module: rcmd/ssh<br/> Author: Jim Garlick &lt;garlick@llnl.gov&gt;<br/> Descr: ssh based rcmd connect method<br/> Active: yes</p> <p>Less clear is what we can do about it. I do not see an option to pdsh to prohibit this processing. Escaping it as %%u works for me - but would presumably break configurations it currently works for.</p> <p>This is a old bug that has been around since the 1.6 days. I thought I was the only one that saw this problem. The way I work around it is with the below patch.</p> <p>diff -urp metadata-updates.sh.orig metadata-updates.sh<br/> &#8212; metadata-updates.sh.orig 2013-05-16 09:54:56.000000000 -0400<br/> +++ metadata-updates.sh 2013-05-20 13:33:15.000000000 -0400<br/> @@ -32,7 +32,7 @@<br/> NEW_ATIME="2001-01-01 GMT"<br/> NEW_MTIME="2005-05-05 GMT"</p> <p>-test_UID=$(id -u)<br/> +test_UID=$(id -u -n)<br/> test_GID=$(id -g)</p> <p> SUMFILE=$TESTDIR/mdsum<br/> @@ -106,7 +106,7 @@<br/> do_nodesv $NODES_TO_USE "set $TRACE;<br/> for HOST in ${HOSTS//,/ } ; do<br/> TESTFILE=$TESTDIR/\\\$HOST/$FILE;</p> <ul class="alternate" type="square"> <li>tmp=\\\$(stat -c \\\"%u %g %s 0%a\\\" \\\$TESTFILE);<br/> + tmp=\\\$(stat -c \\\"%U %g %s 0%a\\\" \\\$TESTFILE);<br/> echo \\\"\\\$TESTFILE [ uid gid size mode ] expected : $attr ; got : \\\$tmp \\\";<br/> if [ x\\\"\\\$tmp\\\" != x\\\"$attr\\\" ] ; then<br/> echo \\\"Wrong file attributes\\\";</li> </ul> <p>That works. I assumed that test_UID needed to be numerical, but it's not actually used anywhere else. I think we should rename it (because it's not a UID anymore) and change GID, too, for consistency.</p> <p>Mind if I submit a patch with those changes, based on your idea?</p> <p>Sure go right ahead <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p><a href="http://review.whamcloud.com/8052" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8052</a></p> <p>Against master, applies cleanly to 2.4.1 and works in my environment.</p> <p>Landed to master.</p> <p>Can we cherry pick it against 2.4.1. We will later need it cherry picked against 2.5.1</p> <p>The patch should apply to 2.4 and 2.5 with no problems. I'll request that it be cherry-picked against 2.4 and 2.5.</p> <p>Reopened to apply patch to 2.5 and 2.4.</p> <p>2.4 patch: <a href="http://review.whamcloud.com/#/c/8333/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8333/</a><br/> 2.5 patch: <a href="http://review.whamcloud.com/#/c/8332/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8332/</a></p> <p>Patch <a href="http://review.whamcloud.com/8052" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8052</a> was cherry-picked to Lustre b2_4 branch for 2.4.2.</p> <p>Landed for 2.4.2 and 2.6. Will land for 2.5.1 shortly</p> Development Rank 1|hzw467: Rank (Obsolete) 10782 Severity