https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10419 Lustre <p>We do OSS failover, trigger LFSCK:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"></pre> </div></div> <p>lctl lfsck_start -M soaked-MDT0000 -s 1000 -t all -A{code]</p> <p>The lfsck start hangs, lfsck is not started, the clients wedge in state 'comp' the entire system wedges. I have dumped Lustre Logs from all MDS, attached. I have crash-dumped all the MDT nodes and the dumps are available on Spirit. lfsck_layout is unkillable. </p> Soak performance cluster - Lustre version=2.10.2_4_gb151f34 LU-10419

LFSCK fails to start, hangs systems.

Bug Critical Resolved Fixed nasf Cliff White soak Wed, 20 Dec 2017 19:17:45 +0000 Wed, 1 Aug 2018 21:13:47 +0000 Thu, 14 Jun 2018 03:55:08 +0000 Lustre 2.11.0 Lustre 2.10.2 Lustre 2.10.3 Lustre 2.12.0 Lustre 2.10.5 0 5 <p>Assigning this to Fan Yong so that it is in his queue when he returns from vacation.</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30768" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30768</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10419" title="LFSCK fails to start, hangs systems. " class="issue-link" data-issue-key="LU-10419"><del>LU-10419</del></a> lfsck: no delay for notify RPC<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d8827a8ce44db121f80223dc7189e32f5bf3fd45</p> <p>We are hitting this issue on b2_10.3 RC1 - need to patch ported over there</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30831" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30831</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10419" title="LFSCK fails to start, hangs systems. " class="issue-link" data-issue-key="LU-10419"><del>LU-10419</del></a> lfsck: no delay for notify RPC<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: e20f5d402a8a6547544b33d4890b6910e7cf9f95</p> <p>The patch for master is way out of date, uses and old kernel and contains old bugs (<a href="https://jira.whamcloud.com/browse/LU-10459" title="LBUG o2iblnd_cb.c:991:kiblnd_check_sends_locked()) ASSERTION( conn-&gt;ibc_nsends_posted &lt;= conn-&gt;ibc_queue_depth ) failed:" class="issue-link" data-issue-key="LU-10459"><del>LU-10459</del></a>) Can you move the patch to the tip of current master, so that it is testable? </p> <p><a href="https://review.whamcloud.com/#/c/30768/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/30768/</a> set 2 is against the latest master, here is the Jenkins build:<br/> <a href="https://build.hpdd.intel.com/job/lustre-reviews/53764/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-reviews/53764/</a> </p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30768/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30768/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10419" title="LFSCK fails to start, hangs systems. " class="issue-link" data-issue-key="LU-10419"><del>LU-10419</del></a> lfsck: no delay for notify RPC<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 39816213632cf9083530f1a8b644459d13e3c980</p> <p>Landed for 2.11</p> <p>Seeing this again on DNE-enable system. version=2.10.57_58_gf24340c<br/> I can crash dump systems if desired</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=cliffw" class="user-hover" rel="cliffw">cliffw</a>,</p> <p>Where can I get related logs?</p> <p>Thanks!</p> <p>Logs are on spirit /scratch/logs/syslogs and /scratch/logs/console. The crash dumps are in /scratch/dumps on spirit. </p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31475" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31475</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10419" title="LFSCK fails to start, hangs systems. " class="issue-link" data-issue-key="LU-10419"><del>LU-10419</del></a> lfsck: skip dead target<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: be9f2eedf5039fa6308460aca6a84daa6b8003b1</p> <p>With the current patch, lfsck does not stop. Currently also having mount timeouts. I have crashed dumped soak-8 while lfsck was hanging, logs are available on spirit.<br/> /scratch/dumps/soak-8.spirit.hpdd.intel.com/10.10.1.108-2018-03-06-19:16:47</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31475/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31475/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10419" title="LFSCK fails to start, hangs systems. " class="issue-link" data-issue-key="LU-10419"><del>LU-10419</del></a> lfsck: skip dead target<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 012834c5e7c7be50ff117cee4ac473d7fee4294d</p> <p>Landed for 2.11</p> <p>Oleg Drokin (oleg.drokin@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31600" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31600</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-10419" title="LFSCK fails to start, hangs systems. " class="issue-link" data-issue-key="LU-10419"><del>LU-10419</del></a> lfsck: skip dead target"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1387fa1c012dfdf5eb4f90efeb06edd45788064f</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31600/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31600/</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-10419" title="LFSCK fails to start, hangs systems. " class="issue-link" data-issue-key="LU-10419"><del>LU-10419</del></a> lfsck: skip dead target"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9ba637b8949b1b8a5f2506e654a9b62d5c0cc245</p> <blockquote> <p>With the current patch, lfsck does not stop. Currently also having mount timeouts. I have crashed dumped soak-8 while lfsck was hanging, logs are available on spirit.<br/> /scratch/dumps/soak-8.spirit.hpdd.intel.com/10.10.1.108-2018-03-06-19:16:47</p> </blockquote> <p>The LFSCK master engine was blocked when sending OUT_ATTR_GET RPC to MDT2 that may be offline or in recovery. We expect the lfsck_stop() can wakeup the blocked LFSCK engines and make them to exit, but we only single (SIGINT) the LFSCK assistant engines, forget to do that for the LFSCK master engine.</p> <p> </p> <p>So the trouble is not related with the patch <a href="https://review.whamcloud.com/31475/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31475/.</a></p> <p>I will make another patch to notify the master engine when lfsck_stop().</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31627" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31627</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10419" title="LFSCK fails to start, hangs systems. " class="issue-link" data-issue-key="LU-10419"><del>LU-10419</del></a> lfsck: single master engine when stop<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e3e7d1a41711cfb0a12b941a88bf8c0bf3b4cc89</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31627/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31627/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10419" title="LFSCK fails to start, hangs systems. " class="issue-link" data-issue-key="LU-10419"><del>LU-10419</del></a> lfsck: signal master engine when stop<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1ece380412efd5dba2a8c345830f0456a4922301</p> <p>John L. Hammond (jhammond@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/30831/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30831/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10419" title="LFSCK fails to start, hangs systems. " class="issue-link" data-issue-key="LU-10419"><del>LU-10419</del></a> lfsck: no delay for notify RPC<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 9fef9ad10b26a4338c22105e66308ead5408173e</p> Duplicate LU-10036 Development Rank 1|hzzpz3: Rank (Obsolete) 9223372036854775807 Severity