https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10553 Lustre <p>This issue was created by maloo for Cliff White &lt;cliff.white@intel.com&gt;</p> <p>This issue relates to the following test suite run: </p> <p>On multiple runs, we see permission errors when cleaning up the test, the files showing in the error report appear to be artifacts from previous (replay-*) tests. <br/> Examples:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">== sanity-pfl test complete, duration 777 sec ======================================================== 02:29:35 (1516357775) rm: cannot remove <span class="code-quote">'/mnt/lustre/d23b.replay-dual'</span>: Directory not empty .... == sanity-pfl test complete, duration 773 sec ======================================================== 00:38:53 (1516437533) rm: cannot remove <span class="code-quote">'/mnt/lustre/f4h.replay-vbr'</span>: Operation not permitted sanity-pfl : @@@@@@ FAIL: remove sub-test dirs failed ... == sanity-pfl test complete, duration 773 sec ======================================================== 12:42:04 (1516567324) rm: cannot remove <span class="code-quote">'/mnt/lustre/f4h.replay-vbr'</span>: Operation not permitted sanity-pfl : @@@@@@ FAIL: remove sub-test dirs failed ... </pre> </div></div> LU-10553

d23b.replay-dual: Directory not empty, FAIL: remove sub-test dirs failed

Bug Minor Open Unresolved WC Triage Maloo easy tests Tue, 23 Jan 2018 20:54:02 +0000 Mon, 27 Mar 2023 04:12:55 +0000 Lustre 2.11.0 0 6 <p>Typically, scripts like sanity.sh will clean up test files at the start to avoid issues like this. Also, most test scripts should only be accessing files that they created, so there may be some cleanup work needed in these sanity-pfl tests. </p> <p>I started to open a new ticket until I saw this ticket. Here is just a little more detail on what we see in Maloo for these failed test sessions. </p> <p>Lustre test suites fail because “rm: cannot remove '/mnt/lustre/f4h.replay-vbr': Operation not permitted”</p> <p>We have many cases of a Lustre test suite have a FAIL status, but, when you look at all the subtests, all of the subtests PASS. By looking at the end of the suite_log for the failed test suite, you will see an error when the suite tries to clean up the file system. For example (<a href="https://testing.hpdd.intel.com/test_sets/14461086-0359-11e8-bd00-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/14461086-0359-11e8-bd00-52540065bddc</a> ), a recent insanity test suite failure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== insanity test complete, duration 1255 sec ========================================================= 19:55:20 (1517025320) rm: cannot remove '/mnt/lustre/f4h.replay-vbr': Operation not permitted insanity : @@@@@@ FAIL: remove sub-test dirs failed Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:5336:error() = /usr/lib64/lustre/tests/test-framework.sh:4830:check_and_cleanup_lustre() </pre> </div></div> <p>In the example above, we can’t clean up (rm) the files in the file system because a file remains. Yet, I don’t know why we would get an “Operation not permitted” when trying to delete a file. When one test suite completes and another starts, there should not be any tasks running from previous test suites. The solution may be related/similar to the patch for <a href="https://jira.whamcloud.com/browse/LU-6609" title="recovery-small test_26a : FAIL: remove sub-test dirs failed" class="issue-link" data-issue-key="LU-6609"><del>LU-6609</del></a>; <a href="https://review.whamcloud.com/#/c/14843" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/14843</a>. </p> <p>In the same test session referenced above, sanity-quota cannot clean up the file system and sanity-pfl, lustre-rsync-test, metadata-updates, ost-pools, mds-survey, performance-sanity, parallel-scale, large-scale, and obdfilter-survey fails due to the f4h.replay-vbr file. </p> <p>Looking at the replay-vbr results, we see that replay-vbr test 4h did fail and, looking at relay-vbr suite_log, we see that replay-vbr, couldn’t remove that file:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== replay-vbr test complete, duration 890 sec ======================================================== 19:34:23 (1517024063) replay-vbr: FAIL: test_1b trevis-7vm9 not evicted replay-vbr: FAIL: test_2b trevis-7vm9 not evicted replay-vbr: FAIL: test_3b trevis-7vm9 not evicted replay-vbr: FAIL: test_4c trevis-7vm9 not evicted replay-vbr: FAIL: test_4d trevis-7vm9 not evicted replay-vbr: FAIL: test_4f trevis-7vm9 not evicted replay-vbr: FAIL: test_4h trevis-7vm9 not evicted replay-vbr: FAIL: test_5b trevis-7vm9 not evicted replay-vbr: FAIL: test_5c trevis-7vm9 not evicted replay-vbr: FAIL: test_6c trevis-7vm9 not evicted replay-vbr: FAIL: test_6d trevis-7vm9 not evicted replay-vbr: FAIL: test_7a Test 7a.1 failed replay-vbr: FAIL: test_7b Test 7b.1 failed replay-vbr: FAIL: test_7c Test 7c.1 failed replay-vbr: FAIL: test_7e Test 7e.1 failed replay-vbr: FAIL: test_7f Test 7f.1 failed replay-vbr: FAIL: test_7h Test 7h.1 failed replay-vbr: FAIL: test_7i Test 7i.1 failed replay-vbr: FAIL: test_10b trevis-7vm9:/mnt/lustre not evicted replay-vbr: FAIL: test_12a test_12a failed with 4 rm: cannot remove '/mnt/lustre/f4h.replay-vbr': Operation not permitted replay-vbr : @@@@@@ FAIL: remove sub-test dirs failed Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:5336:error() = /usr/lib64/lustre/tests/test-framework.sh:4830:check_and_cleanup_lustre() </pre> </div></div> <p>Lustre test suites fail because “rm: cannot remove '/mnt/lustre/d81d.replay-single': Directory not empty”</p> <p>We have many cases of a Lustre test suite FAIL testing, but, when you look at all the subtests, all of the subtests PASS. By looking at the end of the suite_log for the failed test suite, you will see an error when the suite tries to clean up the file system. For example (<a href="https://testing.hpdd.intel.com/test_sets/294da78c-0363-11e8-a10a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/294da78c-0363-11e8-a10a-52540065bddc</a>), a recent insanity test suite failure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== insanity test complete, duration 2289 sec ========================================================= 20:59:08 (1517029148) rm: cannot remove '/mnt/lustre/d81d.replay-single': Directory not empty insanity : @@@@@@ FAIL: remove sub-test dirs failed Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:5336:error() = /usr/lib64/lustre/tests/test-framework.sh:4830:check_and_cleanup_lustre() </pre> </div></div> <p>Looking at replay-single, test 81d does fail and we get the same error message when trying to clean up the file system at the end of the test suite:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== replay-single test complete, duration 7320 sec ==================================================== 20:20:35 (1517026835) replay-single: FAIL: test_0c File exists and it shouldn't replay-single: FAIL: test_44c unliked after fail abort replay-single: FAIL: test_80d /usr/bin/lfs getstripe -M /mnt/lustre/d80d.replay-single/remote_dir failed replay-single: FAIL: test_81d rmdir failed replay-single: FAIL: test_120 dir-0 still exists rm: cannot remove '/mnt/lustre/d81d.replay-single': Directory not empty replay-single : @@@@@@ FAIL: remove sub-test dirs failed </pre> </div></div> <p>In the same test session referenced above, recovery-small, replay-ost-single, replay-dual, replay-vbr, sanity-quota, sanity-pfl, lustre-rsync-test, metadata-updates, ost-pools, mds-survey, performance-sanity, parallel-scale, large-scale, and obdfilter-survey all are unable to remove the f4h.replay-vbr file and some of those tests fail solely due to this. </p> <p>After replay-dual test 23b failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: onyx-40vm2 mkdir /mnt/lustre2/d23b.replay-dual/remote_dir onyx-40vm2: mkdir: cannot create directory `/mnt/lustre2/d23b.replay-dual/remote_dir': File exists replay-dual test_23b: @@@@@@ FAIL: Remote creation failed 1 </pre> </div></div> <p>The following test suites failed with:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rm: cannot remove `/mnt/lustre/d23b.replay-dual': Directory not empty </pre> </div></div> <p><a href="https://testing.hpdd.intel.com/test_sessions/b7b66042-d0a6-4a0a-8c96-d3dc4110333f" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/b7b66042-d0a6-4a0a-8c96-d3dc4110333f</a></p> <p>I guess we need to update the test suite to reboot (reformat?) in such cases, so at least we only have one test script failing instead of a whole series.</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36747" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36747</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10553" title="d23b.replay-dual: Directory not empty, FAIL: remove sub-test dirs failed" class="issue-link" data-issue-key="LU-10553">LU-10553</a> tests: create and cleanup test specific working dir<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b3a63eb54d40d414be2d03303ce63ab4832440cc</p> <p>I don't think this is a problem from the test-framework.sh not trying to delete the test directories, but rather a defect in Lustre/DNE where the directory simply <b>cannot</b> be deleted because it has a file in it that is not visible on the client for some reason.</p> Duplicate LU-15710 Related LU-9827 Development Rank 1|hzzrkn: Rank (Obsolete) 9223372036854775807 Severity