https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5810 Lustre <p>This issue was created by maloo for John Hammond &lt;john.hammond@intel.com&gt;</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>-----============= acceptance-small: sanity ============----- Sat Oct 25 17:04:33 UTC 2014 Running: bash /usr/lib64/lustre/tests/sanity.sh == sanity test complete, duration -o sec == 17:04:34 (1414256674) CMD: shadow-23vm10.shadow.whamcloud.com,shadow-23vm9 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/bin:/bin:/usr/sbin:/sbin::/sbin:/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh check_config_client /mnt/lustre shadow-23vm9: Checking config lustre mounted on /mnt/lustre shadow-23vm10: Checking config lustre mounted on /mnt/lustre Checking servers environments CMD: shadow-23vm11 running=\$(grep -c /mnt/ost1' ' /proc/mounts); mpts=\$(mount | grep -c /mnt/ost1' '); if [ \$running -ne \$mpts ]; then echo \$(hostname) env are INSANE!; exit 1; fi ... CMD: shadow-23vm12 lctl get_param -n timeout Using TIMEOUT=20 CMD: shadow-23vm12 lctl dl | grep ' IN osc ' 2&gt;/dev/null | wc -l CMD: shadow-23vm10.shadow.whamcloud.com lctl dl | grep ' IN osc ' 2&gt;/dev/null | wc -l disable quota as required CMD: shadow-23vm11,shadow-23vm12,shadow-23vm8,shadow-23vm9 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bin:/usr/bin:/bin:/usr/sbin:/sbin::/sbin:/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh set_default_debug \"vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck\" \"all -lnet -lnd -pinger\" 4 CMD: shadow-23vm11,shadow-23vm12,shadow-23vm8 /usr/sbin/lctl set_param osd-ldiskfs.track_declares_assert=1 || true osd-ldiskfs.track_declares_assert=1 osd-ldiskfs.track_declares_assert=1 osd-ldiskfs.track_declares_assert=1 rm: cannot remove `/mnt/lustre/d0.tar-shadow-23vm5/etc/init.d/rc3.d': Directory not empty status script Total(sec) E(xcluded) S(low) ------------------------------------------------------------------------------------ test-framework exiting on error </pre> </div></div> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/6ad49578-5c8e-11e4-b08a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6ad49578-5c8e-11e4-b08a-5254006e85c2</a>.</p> LU-5810

sanity: rm: cannot remove `/mnt/lustre/d0.tar-shadow-23vm5/etc/init.d/rc3.d': Directory not empty

Bug Minor Resolved Cannot Reproduce WC Triage Maloo Mon, 27 Oct 2014 17:18:03 +0000 Fri, 11 Sep 2020 23:54:44 +0000 Fri, 11 Sep 2020 23:54:44 +0000 0 5 <p>It is strange that there is a <tt>shadow-23vm5</tt> directory that is not empty, yet according to the config for the test session, the nodes listed are <tt>shadow-23vm<span class="error">&#91;8-12&#93;</span></tt>. This implies that the shadow-23vm5 node mounted the wrong filesystem for some reason and proceeded to write there.</p> <p>A similar configuration problem appeared in <a href="https://jira.whamcloud.com/browse/LU-5064" title="sanity-scrub test_13: ls should fail" class="issue-link" data-issue-key="LU-5064"><del>LU-5064</del></a> and <a href="https://jira.whamcloud.com/browse/LU-5076" title="Test failure on test suite conf-sanity, subtest test_46a test failed to respond and timed out" class="issue-link" data-issue-key="LU-5076"><del>LU-5076</del></a>, and possibly others, where nodes that were not part of the test configuration were accessing the filesystem.</p> <p>They have been marked duplicates of TEI-1993.</p> <p>After researched, I doubt this is a problem where we cross mount.<br/> The file /mnt/lustre/d0.tar&#45;$hostname comes from a script run_tar.sh, this is only called from recovery&#45;[mds|double|random]-scale.sh. I have checked report around the time of this report (up to a few days) and can not find any shadow&#45;23vm5 report running recovery&#45;[mds|double|random]&#45;scale tests. It's possible that someone/somescript that accidently login to shadow&#45;23vm5 and run the test and left it behind.</p> <p>Can you please check if shadow-23vm5 is reserved for some user job, or if it is some stuck or forgotten process that is still running there? That also happened with <a href="https://jira.whamcloud.com/browse/LU-5064" title="sanity-scrub test_13: ls should fail" class="issue-link" data-issue-key="LU-5064"><del>LU-5064</del></a>. </p> <p>shadow-23vm5 has always been in autotest. at the time around the failure shadow-23vm5 wasn't running recover-mds-scale.</p> <p>I've experienced a similar problem on the OpenSFS cluster; the test framework can't remove a directory from a previous test, not another node/VM. If you think this is a different problem, I can open a new ticket for this. </p> <p>Results are at <a href="https://testing.hpdd.intel.com/test_sessions/f13ba544-8618-11e4-ac52-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/f13ba544-8618-11e4-ac52-5254006e85c2</a></p> <p>replay-dual had several tests fail including 22a and 22c. When replay-vbr starts up, no tests run due to the remove at the top of the script. This remove fails with</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rm: cannot remove `/lustre/scratch/d22a.replay-dual': Directory not empty rm: cannot remove `/lustre/scratch/d22c.replay-dual': Directory not empty status script Total(sec) E(xcluded) S(low) ------------------------------------------------------------------------------------ test-framework exiting on error </pre> </div></div> <p>replay-vbr is marked as FAIL with 0/0 subtests passed.</p> <p>Then insanity starts running and runs 16 tests. The test suite is marked as FAIL with no subtest actually failing, but the remove during the test cleanup must have triggered the failure. In the test logs, we see:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== insanity test complete, duration 2157 sec == 14:50:03 (1418770203) rm: cannot remove `/lustre/scratch/d22a.replay-dual': Directory not empty rm: cannot remove `/lustre/scratch/d22c.replay-dual': Directory not empty insanity : @@@@@@ FAIL: remove sub-test dirs failed Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4665:error_noexit() = /usr/lib64/lustre/tests/test-framework.sh:4696:error() = /usr/lib64/lustre/tests/test-framework.sh:4210:check_and_cleanup_lustre() = /usr/lib64/lustre/tests/insanity.sh:781:main() Dumping lctl log to /tmp/test_logs/2014-12-15/220919/insanity..*.1418770204.log </pre> </div></div> <p>Debug patch for this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LU-5810 tests: add client hostname to lctl mark Improve debug messages to include the originating hostname. Signed-off-by: Andreas Dilger &lt;andreas.dilger@intel.com&gt; Change-Id: I441bf8294c38135276a5a0f0853dbebf4358c563 </pre> </div></div> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13113/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13113/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5810" title="sanity: rm: cannot remove `/mnt/lustre/d0.tar-shadow-23vm5/etc/init.d/rc3.d&#39;: Directory not empty" class="issue-link" data-issue-key="LU-5810"><del>LU-5810</del></a> tests: add client hostname to lctl mark<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9c4156e6fc146a198bb342e28eb246f1076889bd</p> <p>James Nunez (james.a.nunez@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/20894" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20894</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-5810" title="sanity: rm: cannot remove `/mnt/lustre/d0.tar-shadow-23vm5/etc/init.d/rc3.d&#39;: Directory not empty" class="issue-link" data-issue-key="LU-5810"><del>LU-5810</del></a> tests: add client hostname to lctl mark"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: dc25382d26a409c12b79d0c9a82ba5a0fa7c521c</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/20894/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20894/</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-5810" title="sanity: rm: cannot remove `/mnt/lustre/d0.tar-shadow-23vm5/etc/init.d/rc3.d&#39;: Directory not empty" class="issue-link" data-issue-key="LU-5810"><del>LU-5810</del></a> tests: add client hostname to lctl mark"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d700bd76aadb7b3ae8fda27dec1d58723b9b95fe</p> Duplicate LU-5064 Development Rank 1|hzwzjr: Rank (Obsolete) 16296 Severity