https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10729 Lustre <p>test fails on failover config: hosts != failoverhosts</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">== replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 =============== 19:53:39 (1519674819) fail_loc=0x1701 fail_loc=0 Failing mds1 on fre909 Stopping /mnt/lustre-mds1 (opts:) on fre909 pdsh@fre913: fre909: ssh exited with exit code 1 Failing mds2 on fre910 Stopping /mnt/lustre-mds2 (opts:) on fre910 pdsh@fre913: fre910: ssh exited with exit code 1 reboot facets: mds1 Failover mds1 to fre910 ... fre915: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1168028 6688 1062924 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1168028 6272 1063340 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 4607356 34096 4306876 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 4607356 34096 4306876 1% /mnt/lustre[OST:1] lustre-OST0002_UUID 4607356 34100 4306872 1% /mnt/lustre[OST:2] lustre-OST0003_UUID 4607356 34100 4306872 1% /mnt/lustre[OST:3] filesystem_summary: 18429424 136392 17227496 1% /mnt/lustre fre914: mkdir: cannot create directory <span class="code-quote">'/mnt/lustre2/d23d.replay-dual/remote_dir'</span>: File exists pdsh@fre913: fre914: ssh exited with exit code 1 replay-dual test_23d: @@@@@@ FAIL: Remote creation failed 1 </pre> </div></div> LU-10729

replay-dual test_23d: FAIL: Remote creation failed 1 : mkdir: cannot create directory': File exists

Bug Critical Resolved Fixed Alex Zhuravlev Elena Gryaznova Tue, 27 Feb 2018 14:20:06 +0000 Wed, 29 Nov 2023 21:24:45 +0000 Tue, 11 Apr 2023 20:22:30 +0000 Lustre 2.11.0 Lustre 2.10.4 Lustre 2.10.5 Lustre 2.10.7 Lustre 2.12.1 Lustre 2.14.0 Lustre 2.12.6 Lustre 2.16.0 0 8 <p>+1 on b2_10</p> <p><a href="https://testing.hpdd.intel.com/test_sets/9a12c54c-2eab-11e8-b3c6-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9a12c54c-2eab-11e8-b3c6-52540065bddc</a></p> <p>When this test fails, we see that several test suites that run after replay-dual fail because they can't clean up this sub test directory; the tests don't even start to run or the test suite fails on clean up after all tests are run. </p> <p>From the end of the replay-vbr test_log at <a href="https://testing.whamcloud.com/test_sets/95dc7668-98ef-11e8-a9f7-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/95dc7668-98ef-11e8-a9f7-52540065bddc</a>, we see</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rm: cannot remove '/mnt/lustre/d23d.replay-dual': Directory not empty Trace dump: = /usr/lib64/lustre/tests/replay-vbr.sh:31:main() replay-vbr: FAIL: test-framework exiting on error </pre> </div></div> <p>I'm gong to attribute these failures to this ticket.</p> <p>the llog cancel for a previous distributed transaction race with the update from rmdir ../remote_dir and if that llog write wins then rmdir's update is on hold (until the reply for llog write).<br/> I think the easiest fix is just to wait a bit before rmdir to let that cancel write to get processed.</p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45623" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45623</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10729" title="replay-dual test_23d: FAIL: Remote creation failed 1 : mkdir: cannot create directory&#39;: File exists" class="issue-link" data-issue-key="LU-10729"><del>LU-10729</del></a> tests: replay-dual/23d to wait<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f06970ed6d6e839b82b9623c1ae007164e3d5d50</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/45623/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/45623/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10729" title="replay-dual test_23d: FAIL: Remote creation failed 1 : mkdir: cannot create directory&#39;: File exists" class="issue-link" data-issue-key="LU-10729"><del>LU-10729</del></a> tests: replay-dual/23d to wait<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 63a19f6f666b9d18fede66ce8bcd2d799b5e0fa7</p> <p>Landed for 2.16</p> <p>I also see this fail intermittently on replay-dual test_22d.</p> <p>"Xing Huang &lt;hxing@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52343" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52343</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10729" title="replay-dual test_23d: FAIL: Remote creation failed 1 : mkdir: cannot create directory&#39;: File exists" class="issue-link" data-issue-key="LU-10729"><del>LU-10729</del></a> tests: replay-dual/22d to wait<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bfa32d30813269f74c721eef5aca8930f40230e8</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52343/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52343/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10729" title="replay-dual test_23d: FAIL: Remote creation failed 1 : mkdir: cannot create directory&#39;: File exists" class="issue-link" data-issue-key="LU-10729"><del>LU-10729</del></a> tests: replay-dual/22d to wait<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c0973b9fd64036adb3991615e76a97c6aa0b384e</p> Related LU-6006 LU-17102 Development Rank 1|hzztev: Rank (Obsolete) 9223372036854775807 Severity