https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5848 Lustre <p>This issue was created by maloo for Nathaniel Clark &lt;nathaniel.l.clark@intel.com&gt;</p> <p>This issue relates to the following test suite run:<br/> <a href="https://testing.hpdd.intel.com/test_sets/81b27534-6234-11e4-8055-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/81b27534-6234-11e4-8055-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/9037c036-5d26-11e4-ae10-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9037c036-5d26-11e4-ae10-5254006e85c2</a></p> <p>The sub-test test_18e failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(4) MDS4 is not the expected 'completed' </pre> </div></div> <p>Please provide additional information about the failure here.</p> <p>Info required for matching: sanity-lfsck 18e</p> LU-5848

sanity-lfsck test_18e: MDS is not the expected 'completed'

Bug Critical Resolved Fixed nasf Maloo Mon, 3 Nov 2014 16:02:18 +0000 Tue, 7 Jul 2015 13:19:55 +0000 Tue, 7 Jul 2015 13:19:55 +0000 Lustre 2.7.0 Lustre 2.8.0 0 6 <p>According to the test_logs, the LFSCK status on the MDT4 is 'scanning-phase2', not the expected "completed". So means the LFSCK on the MDT4 should have moved to the second-stage scanning. Normally, before the second-stage scanning start, the LFSCK assistant thread will generate the LFSCK log "LFSCK assistant phase2 scan start", but I cannot find such log in the MDT4 debug_log, that is abnormal. The most possible reason is that the LFSCK assistant thread was blocked inside the dt_sync() before generating such LFSCK log message. Currently, we do not know what happened during the dt_sync(). I will make a debug patch to verify that.</p> <p>Here is the debug patch:<br/> <a href="http://review.whamcloud.com/#/c/12573/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12573/</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12573/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12573/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5848" title="sanity-lfsck test_18e: MDS is not the expected &#39;completed&#39; " class="issue-link" data-issue-key="LU-5848"><del>LU-5848</del></a> debug: more debug log for dt_sync<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ec37d78caf6cbd8118cfe1ff012c828950356e7a</p> <p>Landed to master for 2.7.0</p> <p>+1 at "https://testing.hpdd.intel.com/test_sets/f720aa60-c121-11e4-b948-5254006e85c2". May be there is something new to learn from the added debug logs ?</p> <p>According to the log, the layout LFSCK on MDT1 was NOT fall into dt_sync(), instead, the layout LFSCK assistant thread did not arrived dt_sync() point yet. Two possible cases:<br/> 1) The layout LFSCK assistant thread blocked inside lfsck_assistant_notify_others().<br/> 2) There are dead lock between the master engine and assistant thread.</p> <p>Unfortunately, we have not the kernel stack trace for all the failure instances. We cannot know exactly what happened. Is there any way to get the stack trace?</p> <p>Hello Nasf, after I tried to further+in-deep debug my own failure case, I agree with you that there is not much of interest (than the expected but missing log entries as you pointed!) in the Lustre debug logs.<br/> You may want to add an "echo t &gt; /proc/sysrq-trigger" cmd, sent to all nodes/MDS running MMDT facets, in sanity-lfsck/test_18e upon error.</p> <p>As I remembered, Maloo will collect the kernel stack trace automatically. But I do not know why it does not now.</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13950" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13950</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5848" title="sanity-lfsck test_18e: MDS is not the expected &#39;completed&#39; " class="issue-link" data-issue-key="LU-5848"><del>LU-5848</del></a> lfsck: debug log for sanity-lfsck test_18e<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 249163ca2d2e605be1d7617ca1b0a778ccb05e5e</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13950/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13950/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5848" title="sanity-lfsck test_18e: MDS is not the expected &#39;completed&#39; " class="issue-link" data-issue-key="LU-5848"><del>LU-5848</del></a> lfsck: debug log for sanity-lfsck test_18e<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: bb597d302fe0259ce60691677a6e79c3ff19bbb2</p> Development Rank 1|hzx01b: Rank (Obsolete) 16378 Severity