https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-17385 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;adilger@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/e80cc085-ac08-4f47-b354-22551a7da132" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/e80cc085-ac08-4f47-b354-22551a7da132</a></p> <p>test_26a failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(7) only 3 of 4 MDTs are in completed </pre> </div></div> <p>Test session details:<br/> clients: <a href="https://build.whamcloud.com/job/lustre-master-patchless/840" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-master-patchless/840</a> - 4.18.0-425.10.1.el8_7.x86_64<br/> servers: <a href="https://build.whamcloud.com/job/lustre-master-patchless/840" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-master-patchless/840</a> - 4.18.0-425.10.1.el8_7.x86_64</p> <p>&lt;&lt;Please provide additional information about the failure here&gt;&gt;</p> <p>First started on 2023-12-20 for full runs, may be related to recent patch landing. </p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity-lfsck test_26a - (7) only 3 of 4 MDTs are in completed</p> LU-17385

sanity-lfsck test_26a: only 3 of 4 MDTs are in completed

Bug Major Resolved Fixed Alexander Zarochentsev Maloo always_except Thu, 21 Dec 2023 18:56:17 +0000 Sun, 4 Feb 2024 14:24:50 +0000 Sun, 4 Feb 2024 14:24:50 +0000 Lustre 2.16.0 Lustre 2.16.0 0 7 <p>Hongchao</p> <p>This seems to have started failing only very recently. Can you identify which change introduced this issue?</p> <p>Thanks</p> <p>Peter</p> <p>The LFSCK_NOTIFY request is failed to be processed by MDT, but it is strange that other LFSCK_NOTIFY request succeed</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00100000:1.0:1703075647.451663:0:1223156:0:(service.c:2333:ptlrpc_server_handle_request()) Handling RPC req@00000000a373e95a pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_003:lustre-MDT0000-mdtlov_UUID+5:1744723:x1785766154025664:12345-10.240.26.106@tcp:1101:lctl.0 00000100:00100000:0.0:1703075647.451664:0:1215944:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.240.26.106@tcp, seq: 1159 00000100:00100000:0.0:1703075647.451667:0:1215944:0:(service.c:2333:ptlrpc_server_handle_request()) Handling RPC req@00000000617d7544 pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_001:lustre-MDT0000-mdtlov_UUID+5:1744723:x1785766154025792:12345-10.240.26.106@tcp:1101:lctl.0 00000100:00100000:1.0:1703075647.451692:0:1223156:0:(service.c:2382:ptlrpc_server_handle_request()) Handled RPC req@00000000a373e95a pname:cluuid+ref:pid:xid:nid:opc:job mdt_out00_003:lustre-MDT0000-mdtlov_UUID+5:1744723:x1785766154025664:12345-10.240.26.106@tcp:1101:lctl.0 Request processed in 29us (98us total) trans 0 rc -95/-95 00000100:00100000:1.0:1703075647.451695:0:1223156:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.240.26.106@tcp, seq: 1158 00100000:10000000:0.0:1703075647.451784:0:1215944:0:(lfsck_lib.c:2707:lfsck_load_one_trace_file()) lustre-MDT0003-osd: unlink lfsck sub trace file lfsck_namespace_01: rc = 0 </pre> </div></div> <p>the request (xid= x1785766154025664) failed with -95 immediately, but the similar request (xid = x1785766154025792) succeeded</p> <p>will create a debug patch to collect the logs</p> <p>"Hongchao Zhang &lt;hongchao@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53530" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53530</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17385" title="sanity-lfsck test_26a: only 3 of 4 MDTs are in completed " class="issue-link" data-issue-key="LU-17385"><del>EX-8860</del></a> lfsck: debug patch<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 17251801b1cf5516132edebd6677e2f34fcbc61c</p> <p>It looks like this test failure was introduced by patch <a href="https://review.whamcloud.com/50998" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/50998</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-16826" title="MDS nodes panicked running lfsck repair create lost objects: (osd_handler.c:6260:osd_index_declare_ea_insert()) ASSERTION( fid != ((void *)0) ) failed" class="issue-link" data-issue-key="LU-16826"><del>LU-16826</del></a> tests: lfsck to repair a dangling remote entry</tt>" landing on 2023-12-20 which added sanity-lfsck.sh test_23d, but used:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Test-Parameters: trivial testlist=sanity-lfsck ... env=ONLY=23d </pre> </div></div> <p>so it is likely leaving the filesystem in a bad state after test_23d finished and this causes test_24 and test_26a to also fail.</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53544" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53544</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17385" title="sanity-lfsck test_26a: only 3 of 4 MDTs are in completed " class="issue-link" data-issue-key="LU-17385"><del>LU-17385</del></a> tests: add sanity-lfsck/24 debugging<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 94f62d0d5bea764b3b0287662384a524283dd419</p> <p>This is failing 22/62 runs since the <a href="https://jira.whamcloud.com/browse/LU-16826" title="MDS nodes panicked running lfsck repair create lost objects: (osd_handler.c:6260:osd_index_declare_ea_insert()) ASSERTION( fid != ((void *)0) ) failed" class="issue-link" data-issue-key="LU-16826"><del>LU-16826</del></a> test case landed.  I don't see anything obvious in the test logs, like an MDT reconnecting in test_26/test_27 after it was stopped/started in test_23d, so I added some more debugging to see why this is failing.</p> <p>I'll also push a revert of the patch that added test_23d and confirm that this stops the problem from being hit, and we'll have it ready if there is no quick solution.</p> <p><del>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch:</del> <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53545" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53545</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-17385" title="sanity-lfsck test_26a: only 3 of 4 MDTs are in completed " class="issue-link" data-issue-key="LU-17385"><del>LU-17385</del></a> revert: <a href="https://jira.whamcloud.com/browse/LU-16826" title="MDS nodes panicked running lfsck repair create lost objects: (osd_handler.c:6260:osd_index_declare_ea_insert()) ASSERTION( fid != ((void *)0) ) failed" class="issue-link" data-issue-key="LU-16826"><del>LU-16826</del></a> tests: lfsck to repair a dangling remote entry</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: fb6c848ef816ecb17f02ac461c2352ced320c593</del></p> <p>&gt; It looks like this test failure was introduced by patch <a href="https://review.whamcloud.com/50998" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/50998</a> "<a href="https://jira.whamcloud.com/browse/LU-16826" title="MDS nodes panicked running lfsck repair create lost objects: (osd_handler.c:6260:osd_index_declare_ea_insert()) ASSERTION( fid != ((void *)0) ) failed" class="issue-link" data-issue-key="LU-16826"><del>LU-16826</del></a> tests: lfsck to repair a dangling remote entry" landing on 2023-12-20 which added sanity-lfsck.sh test_23c, but used:</p> <p>no, the patch adds test_23d, test_23c is an old one with similar name:</p> <p>run_test 23c "LFSCK can repair dangling name entry (3)<br/> run_test 23d "LFSCK can repair a dangling name entry to a remote object</p> <p>Sorry, a typo on my part, and fixed in my comment. It is the new test that landed which caused the problem. </p> <p>It looks like MDT0000 has "finished" the LFSCK run, but with an error:</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>status: partial flags: incomplete</pre> </div></div> <p>It isn't clear from the test output why it is "partial". It doesn't look like waiting longer (600s) in <tt>wait_all_targets_blocked()</tt> is better than using "<tt>&#45;w</tt>" in this case, because the LFSCK threads are all finished, but with an error. </p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53544/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53544/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17385" title="sanity-lfsck test_26a: only 3 of 4 MDTs are in completed " class="issue-link" data-issue-key="LU-17385"><del>LU-17385</del></a> tests: always_except sanity-lfsck/24<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 955e38051765609fe3a661035c0fab2cfca733ce</p> <p>The LFSCK command used in test_23d to start layout LFSCK on MDT0000 uses option "-o", which will broadcast LFSCK to all MDTs</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>int jt_lfsck_start(int argc, char **argv) { ... case 'o': start.ls_flags |= LPF_ALL_TGT | LPF_BROADCAST | LPF_OST_ORPHAN; break; ... } </pre> </div></div> <p>the LFSCK command used in test_24 to start namespace LFSCK is also sent to all MDTs, but some MDT could not complete<br/> the previous layout LFSCK yet, then it will find the two LFSCKs are different and return -EOPNOTSUPP</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>int lfsck_start(const struct lu_env *env, struct dt_device *key, struct lfsck_start_param *lsp) { ... if (!thread_is_init(thread) &amp;&amp; !thread_is_stopped(thread)) { rc = -EALREADY; if (unlikely(start == NULL)) { spin_unlock(&amp;lfsck-&gt;li_lock); GOTO(out, rc); } while (start-&gt;ls_active != 0) { if (!(type &amp; start-&gt;ls_active)) { type &lt;&lt;= 1; continue; } com = __lfsck_component_find(lfsck, type, &amp;lfsck-&gt;li_list_scan); if (com == NULL) com = __lfsck_component_find(lfsck, type, &amp;lfsck-&gt;li_list_double_scan); if (com == NULL) { rc = -EOPNOTSUPP; &lt;--------- return with error -EOPNOTSUPP break; } ... } ... } ... } </pre> </div></div> <p>the corresponding logs</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000020:00000001:1.0:1703243105.771867:0:217993:0:(tgt_handler.c:1621:tgt_handle_lfsck_notify()) Process entered 00100000:00000001:0.0:1703243105.771867:0:228721:0:(lfsck_lib.c:3489:lfsck_in_notify()) Process entered 00100000:00000001:1.0:1703243105.771868:0:217993:0:(lfsck_lib.c:3489:lfsck_in_notify()) Process entered 00100000:00000001:0.0:1703243105.771868:0:228721:0:(lfsck_lib.c:3104:lfsck_start()) Process entered 00100000:00000001:1.0:1703243105.771869:0:217993:0:(lfsck_lib.c:3104:lfsck_start()) Process entered 00100000:00000001:1.0:1703243105.771873:0:217993:0:(lfsck_bookmark.c:107:lfsck_bookmark_store()) Process entered 00080000:00000001:1.0:1703243105.771874:0:217993:0:(osd_handler.c:1912:osd_trans_create()) Process entered 00080000:00000010:1.0:1703243105.771877:0:217993:0:(osd_handler.c:1927:osd_trans_create()) kmalloced '(oh)': 288 at 00000000a80bb82f. 00100000:00000001:0.0:1703243105.771880:0:228721:0:(lfsck_lib.c:3174:lfsck_start()) Process leaving via out (rc=18446744073709551521 : -95 : 0xffffffffffffffa1) 00100000:00000001:0.0:1703243105.771886:0:228721:0:(lfsck_lib.c:3546:lfsck_in_notify()) Process leaving (rc=18446744073709551521 : -95 : ffffffffffffffa1) 00000020:00000001:0.0:1703243105.771887:0:228721:0:(tgt_handler.c:1629:tgt_handle_lfsck_notify()) Process leaving (rc=18446744073709551521 : -95 : ffffffffffffffa1) 00080000:00000001:1.0:1703243105.771888:0:217993:0:(osd_handler.c:1955:osd_trans_create()) Process leaving (rc=18446619811807463424 : -124261902088192 : ffff8efc05778800) 00010000:00000040:0.0:1703243105.771889:0:228721:0:(ldlm_lib.c:3238:target_committed_to_req()) last_committed 0, transno 0, xid 1785973155960256 00010000:00000001:0.0:1703243105.771890:0:228721:0:(ldlm_lib.c:3307:target_send_reply()) Process entered 00010000:00000200:0.0:1703243105.771892:0:228721:0:(ldlm_lib.c:3295:target_send_reply_msg()) @@@ sending reply req@00000000c24bd5b0 x1785973155960256/t0(0) o1101-&gt;lustre-MDT0000-mdtlov_UUID@10.240.38.25@tcp:111/0 lens 320/224 e 0 to 0 dl 1703243116 ref 1 fl Interpret:/200/0 rc -95/0 job:'lctl.0' uid:0 gid:0 </pre> </div></div> <p>The sanity-lfsck test_23d is now skipped, so any patch that is fixing it needs to remove the <tt>always_except</tt> line. </p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=hongchao.zhang" class="user-hover" rel="hongchao.zhang">hongchao.zhang</a>, thanks for the analysis, I am submitting a patch to test the idea.</p> <p>"Alexander Zarochentsev &lt;alexander.zarochentsev@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53591" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53591</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17385" title="sanity-lfsck test_26a: only 3 of 4 MDTs are in completed " class="issue-link" data-issue-key="LU-17385"><del>LU-17385</del></a> tests: sanity-lfsck 23d fix and enable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2f4f656947703d8b44a7ea49a8b2c84020591307</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53591/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53591/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17385" title="sanity-lfsck test_26a: only 3 of 4 MDTs are in completed " class="issue-link" data-issue-key="LU-17385"><del>LU-17385</del></a> tests: sanity-lfsck 23d fix and enable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 028ed64d90cfdeb908fb5574aacf2f71c259e2c2</p> <p>Merged for 2.16</p> Related LU-16826 Development Rank 1|i045mv: Rank (Obsolete) 9223372036854775807 Severity