https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16065 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/d5f0d54d-a199-4071-9e41-e86f87f40229" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/d5f0d54d-a199-4071-9e41-e86f87f40229</a></p> <p>test_81a failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rm remote dir failed </pre> </div></div> <p>env: DNE zfs<br/> replay-single test 81a/b/f/g all failed with this error</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 9499.983850] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 21:34:23 \(1658093663\) [ 9500.318282] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 21:34:23 (1658093663) [ 9501.082644] Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701 [ 9501.537124] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 9501.538189] Lustre: Skipped 1 previous similar message [ 9501.539108] LustreError: 202414:0:(ldlm_lib.c:3218:target_send_reply_msg()) @@@ dropping reply req@00000000b254ad1d x1738627440004288/t377957122055(0) o1000-&gt;lustre-MDT0001-mdtlov_UUID@10.240.26.120@tcp:421/0 lens 1488/4320 e 0 to 0 dl 1658093671 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 9527.585438] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2&gt;/dev/null [ 9528.133003] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2&gt;/dev/null [ 9529.436192] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-77vm5.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9529.804116] Lustre: DEBUG MARKER: onyx-77vm5.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 4 [ 9533.839594] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2&gt;/dev/null [ 9533.858943] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2&gt;/dev/null [ 9534.426529] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2&gt;/dev/null [ 9534.437882] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2&gt;/dev/null [ 9535.718546] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-77vm2.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9535.725722] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-77vm1.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9536.131130] Lustre: DEBUG MARKER: onyx-77vm1.onyx.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9536.148517] Lustre: DEBUG MARKER: onyx-77vm2.onyx.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9536.625978] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9536.637584] Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9537.052563] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9537.059777] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9538.811568] Lustre: DEBUG MARKER: /usr/sbin/lctl mark replay-single test_81a: @@@@@@ FAIL: rm remote dir failed [ 9539.151269] Lustre: DEBUG MARKER: replay-single test_81a: @@@@@@ FAIL: rm remote dir failed </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> replay-single test_81a - rm remote dir failed</p> LU-16065

replay-single test_81a: rm remote dir failed

Bug Minor Open Unresolved WC Triage Maloo Tue, 2 Aug 2022 15:35:38 +0000 Wed, 5 Jul 2023 10:07:11 +0000 Lustre 2.15.1 0 3 <p>It looks like this same subtest has been failing for a long time (back to 2021-01-01 at least), and was attributed to <a href="https://jira.whamcloud.com/browse/LU-6864" title="DNE3: Support multiple modify RPCs in flight for MDT-MDT connection" class="issue-link" data-issue-key="LU-6864"><del>LU-6864</del></a>. However, that ticket is closed and was more of a feature patch, so better to use this one going forward.</p> <p>Is this still true?</p> <p>This subtest failed after a long series of other subtest failures started by test_70b failing with <a href="https://jira.whamcloud.com/browse/LU-10616" title="replay-single test_70b fails with &#39;rundbench load on &lt;hostname(s)&gt; failed!&#39; " class="issue-link" data-issue-key="LU-10616">LU-10616</a>. I suspect it is just fallout from that issue (incomplete cleanup, etc.).</p> <p>There have only been 3 failures of this subtest in the past 6 months:</p> <p><a href="https://testing.whamcloud.com/search?horizon=15552000&amp;status%5B%5D=FAIL&amp;test_set_script_id=f6a12204-32c3-11e0-a61c-52540025f9ae&amp;sub_test_script_id=fc2b3a3e-32c3-11e0-a61c-52540025f9ae&amp;source=sub_tests#redirect" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/search?horizon=15552000&amp;status%5B%5D=FAIL&amp;test_set_script_id=f6a12204-32c3-11e0-a61c-52540025f9ae&amp;sub_test_script_id=fc2b3a3e-32c3-11e0-a61c-52540025f9ae&amp;source=sub_tests#redirect</a></p> <p>Two of them look like stand-alone failures on review patches:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre></pre> </div></div> <p>rm: cannot remove '/mnt/lustre/d81a.replay-single': Directory not empty<br/> replay-single test_81a: @@@@@@ FAIL: rmdir failed </p> {norformat} <p>and one is also likely fallout after <tt>test_70b</tt> failed.</p> Related LU-10616 LU-6864 LU-16336 LU-15335 Development Rank 1|i02w07: Rank (Obsolete) 9223372036854775807 Severity