https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8886 Lustre <p>It is found that the LFSCK failed just after resuming from the last checkpoint. The log is as following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00100000:10000000:8.0:1480550565.633225:0:8251:0:(lfsck_lib.c:2573:lfsck_post_generic()) soaked-MDT0000-osd: waiting for assistant to do lfsck_layout post, rc = -61 00100000:10000000:8.0:1480550565.633247:0:8254:0:(lfsck_engine.c:1781:lfsck_assistant_engine()) soaked-MDT0000-osd: LFSCK assistant unknown status: rc = 0 00100000:10000000:8.0:1480550565.633252:0:8254:0:(lfsck_engine.c:1805:lfsck_assistant_engine()) soaked-MDT0000-osd: LFSCK assistant sync before exit 00100000:10000000:25.0:1480550565.633853:0:8253:0:(lfsck_engine.c:1805:lfsck_assistant_engine()) soaked-MDT0000-osd: LFSCK assistant sync before exit 00100000:10000000:8.0:1480550565.636178:0:8254:0:(lfsck_engine.c:1811:lfsck_assistant_engine()) soaked-MDT0000-osd: LFSCK assistant synced before exit: rc = 0 00100000:10000000:8.0:1480550565.636190:0:8254:0:(lfsck_engine.c:1838:lfsck_assistant_engine()) soaked-MDT0000-osd: lfsck_namespace LFSCK assistant thread exit: rc = 0 00100000:10000000:25.0:1480550565.650715:0:8253:0:(lfsck_engine.c:1811:lfsck_assistant_engine()) soaked-MDT0000-osd: LFSCK assistant synced before exit: rc = 0 00100000:10000000:25.0:1480550565.650725:0:8253:0:(lfsck_engine.c:1838:lfsck_assistant_engine()) soaked-MDT0000-osd: lfsck_layout LFSCK assistant thread exit: rc = -61 00100000:10000000:24.0:1480550565.650761:0:8251:0:(lfsck_lib.c:2585:lfsck_post_generic()) soaked-MDT0000-osd: the assistant has done lfsck_layout post, rc = -61 00100000:10000000:24.0:1480550565.650810:0:8251:0:(lfsck_layout.c:4680:lfsck_layout_master_post()) soaked-MDT0000-osd: layout LFSCK master post done: rc = 0 00100000:10000000:24.0:1480550565.650813:0:8251:0:(lfsck_lib.c:2573:lfsck_post_generic()) soaked-MDT0000-osd: waiting for assistant to do lfsck_namespace post, rc = -61 00100000:10000000:24.0:1480550565.650815:0:8251:0:(lfsck_lib.c:2585:lfsck_post_generic()) soaked-MDT0000-osd: the assistant has done lfsck_namespace post, rc = -61 </pre> </div></div> LU-8886

LFSCK failed to resume from the last checkpoint

Bug Minor Resolved Fixed nasf nasf Thu, 1 Dec 2016 11:21:50 +0000 Fri, 23 Dec 2016 05:37:02 +0000 Fri, 23 Dec 2016 05:37:02 +0000 Lustre 2.10.0 0 4 <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/24056" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/24056</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8886" title="LFSCK failed to resume from the last checkpoint" class="issue-link" data-issue-key="LU-8886"><del>LU-8886</del></a> lfsck: handle -ENODATA for the end of iteration<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f6a09af4d58f7d49e0988a9a8e595b1b152d972b</p> <p>Tried the patch, does not appear to be working - first time lfsk started the node crashed</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Dec 2 09:56:17 lola-8 kernel: LustreError: 14454:0:(lfsck_namespace.c:4492:lfsck_namespace_double_scan()) ASSERTION( list_empty(&amp;lad-&gt;lad_req_list) ) failed: Dec 2 09:56:17 lola-8 kernel: LustreError: 14454:0:(lfsck_namespace.c:4492:lfsck_namespace_double_scan()) LBUG Dec 2 09:56:17 lola-8 kernel: Pid: 14454, comm: lfsck Dec 2 09:56:17 lola-8 kernel: Dec 2 09:56:17 lola-8 kernel: Call Trace: Dec 2 09:56:17 lola-8 kernel: [&lt;ffffffffa081e875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Dec 2 09:56:17 lola-8 kernel: [&lt;ffffffffa081e9bf&gt;] lbug_with_loc+0x3f/0x90 [libcfs] Dec 2 09:56:17 lola-8 kernel: [&lt;ffffffffa11d1c2e&gt;] lfsck_namespace_double_scan+0xee/0x120 [lfsck] Dec 2 09:56:17 lola-8 kernel: [&lt;ffffffffa11cee40&gt;] lfsck_master_engine+0x590/0x1460 [lfsck] Dec 2 09:56:17 lola-8 kernel: [&lt;ffffffff81067650&gt;] ? default_wake_function+0x0/0x20 Dec 2 09:56:17 lola-8 kernel: [&lt;ffffffffa11ce8b0&gt;] ? lfsck_master_engine+0x0/0x1460 [lfsck] Dec 2 09:56:17 lola-8 kernel: [&lt;ffffffff810a138e&gt;] kthread+0x9e/0xc0 Dec 2 09:56:17 lola-8 kernel: [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 Dec 2 09:56:17 lola-8 kernel: [&lt;ffffffff810a12f0&gt;] ? kthread+0x0/0xc0 Dec 2 09:56:17 lola-8 kernel: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>The new failure looks like <a href="https://jira.whamcloud.com/browse/LU-8647" title="lfsck_namespace_double_scan()) ASSERTION( list_empty(&amp;lad-&gt;lad_req_list) ) failed" class="issue-link" data-issue-key="LU-8647"><del>LU-8647</del></a>. Have you applied the patch <a href="https://jira.hpdd.intel.com/browse/LU-8647" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-8647</a> in the test?</p> <p>No, has that patch landed on master? If not, why not?</p> <p><a href="https://jira.whamcloud.com/browse/LU-8647" title="lfsck_namespace_double_scan()) ASSERTION( list_empty(&amp;lad-&gt;lad_req_list) ) failed" class="issue-link" data-issue-key="LU-8647"><del>LU-8647</del></a> landed to master in October under the ticket number <a href="https://jira.whamcloud.com/browse/LU-8569" title="Sharded DNE directory full of files that don&#39;t exist" class="issue-link" data-issue-key="LU-8569"><del>LU-8569</del></a> at <a href="https://review.whamcloud.com/#/c/22723/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/22723/</a><br/> (... that's what is stated in the ticket <a href="https://jira.whamcloud.com/browse/LU-8647" title="lfsck_namespace_double_scan()) ASSERTION( list_empty(&amp;lad-&gt;lad_req_list) ) failed" class="issue-link" data-issue-key="LU-8647"><del>LU-8647</del></a>)</p> <blockquote> <p>No, has that patch landed on master? If not, why not?</p></blockquote> <p>Cliff,</p> <p>As James mentioned, such patch has already been in master. But I am not sure whether the build your testing on Lola contains such patch or not.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/24056/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/24056/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8886" title="LFSCK failed to resume from the last checkpoint" class="issue-link" data-issue-key="LU-8886"><del>LU-8886</del></a> lfsck: handle -ENODATA for the end of iteration<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8ff4556055edf496a0d23cf35fd0d63619143363</p> Development Rank 1|hzyx4v: Rank (Obsolete) 9223372036854775807 Severity