https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5023 Lustre <p>This issue was created by maloo for Bob Glossman &lt;bob.glossman@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/4d306b04-d1ed-11e3-a9d1-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/4d306b04-d1ed-11e3-a9d1-52540035b04c</a>.</p> <p>searching in maloo I see 8 similar failures reported starting 5/1/2014<br/> suspect something happened then to start causing these failures<br/> Don't think it's specific to the mod under test.</p> <p>The sub-test test_12 failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>Info required for matching: sanity-lfsck 12</p> LU-5023

sanity-lfsck test_12: OST lfsck_stop hung

Bug Minor Resolved Fixed nasf Maloo Wed, 7 May 2014 18:17:30 +0000 Tue, 3 Feb 2015 18:18:52 +0000 Tue, 3 Feb 2015 18:18:52 +0000 Lustre 2.7.0 0 7 <p>There appears to be some deadlock on ost:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>01:01:32:INFO: task ll_ost00_004:12219 blocked for more than 120 seconds. 01:01:32: Not tainted 2.6.32-431.11.2.el6_lustre.g324fa81.x86_64 #1 01:01:32:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 01:01:32:ll_ost00_004 D 0000000000000000 0 12219 2 0x00000080 01:01:32: ffff880079891bb0 0000000000000046 0000000000000000 ffff88006c70facd 01:01:32: ffffffffa08c3b3d ffffffffa08c3b3b ffff880079891cc0 ffff88006c710000 01:01:32: ffff88007d7fe5f8 ffff880079891fd8 000000000000fbc8 ffff88007d7fe5f8 01:01:32:Call Trace: 01:01:32: [&lt;ffffffff81529b0e&gt;] __mutex_lock_slowpath+0x13e/0x180 01:01:32: [&lt;ffffffffa0864143&gt;] ? null_alloc_rs+0xf3/0x390 [ptlrpc] 01:01:32: [&lt;ffffffff815299ab&gt;] mutex_lock+0x2b/0x50 01:01:32: [&lt;ffffffffa0dd9cff&gt;] lfsck_stop+0x12f/0x410 [lfsck] 01:01:32: [&lt;ffffffffa0827505&gt;] ? lustre_msg_buf+0x55/0x60 [ptlrpc] 01:01:32: [&lt;ffffffffa084f086&gt;] ? __req_capsule_get+0x166/0x6e0 [ptlrpc] 01:01:32: [&lt;ffffffffa0dde214&gt;] lfsck_in_notify+0x444/0x5f0 [lfsck] 01:01:32: [&lt;ffffffffa0882852&gt;] tgt_handle_lfsck_notify+0x62/0x150 [ptlrpc] 01:01:32: [&lt;ffffffffa088a93c&gt;] tgt_request_handle+0x23c/0xac0 [ptlrpc] 01:01:32: [&lt;ffffffffa0839b3a&gt;] ptlrpc_main+0xd1a/0x1980 [ptlrpc] 01:01:32: [&lt;ffffffffa0838e20&gt;] ? ptlrpc_main+0x0/0x1980 [ptlrpc] 01:01:32: [&lt;ffffffff8109aee6&gt;] kthread+0x96/0xa0 01:01:33: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 01:01:33: [&lt;ffffffff8109ae50&gt;] ? kthread+0x0/0xa0 01:01:33: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>LFSCK patches that have landed around that time:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit a8c9796fddaeda0572a64cedb471b2eead78c506 CommitDate: Tue Apr 29 17:22:17 2014 +0000 LU-4941 lfsck: check LOV EA header properly Reviewed-on: http://review.whamcloud.com/10045 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 63e7b15a1baa1aae03dfdc965e0777cf754ff29a CommitDate: Wed Apr 30 03:41:14 2014 +0000 LU-4895 lfsck: not create object to repair dangling by default Reviewed-on: http://review.whamcloud.com/9989 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 1dbba329174e6c7f7712f01fc4e44c44400fbc92 CommitDate: Wed Apr 30 03:42:00 2014 +0000 LU-4556 tests: speed up sanity-lfsck and sanity-scrub tests Reviewed-on: http://review.whamcloud.com/9704 </pre> </div></div> <p>Not sure if any of these are the cause, but likely candidates. Jodi thought that this hadn't been hit since the initial group of failures, so it is possible that a later patch may have fixed it?</p> <p>It cannot say that there is deadlock on the OST. The real case is probably like that:</p> <p>1) The LFSCK thread on the OST is doing something but it is blocked.<br/> 2) At that time, the first lfsck_stop RPC from the MDT comes in, the RPC handler (on the OST) holds the lfsck::li_mutex to signal the LFSCK thread, then wait until the LFSCK thread stopped.<br/> 3) Because the LFSCK thread is blocked on some unknown event, it cannot respond the stop signal, so the first stop RPC is blocked there.<br/> 4) Then the subsequent lfsck_stop RPC handlers will be blocked at the lfsck::li_mutex by the first lfsck_stop RPC handler.</p> <p>So we need to find out what the LFSCK thread was doing at that time. Unfortunately, the current log was not enough to indicate that. I will make a debug patch to collect more logs.</p> <p>The debug patch for more logs:</p> <p><a href="http://review.whamcloud.com/10276" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10276</a></p> <p>more:<br/> <a href="https://maloo.whamcloud.com/test_sets/2eada7a0-d4c7-11e3-808a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2eada7a0-d4c7-11e3-808a-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/465c7ec4-d33e-11e3-a102-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/465c7ec4-d33e-11e3-a102-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/542c478e-d75b-11e3-9d16-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/542c478e-d75b-11e3-9d16-52540035b04c</a></p> <p>&gt; <a href="https://maloo.whamcloud.com/test_sets/2eada7a0-d4c7-11e3-808a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2eada7a0-d4c7-11e3-808a-52540035b04c</a><br/> &gt; <a href="https://maloo.whamcloud.com/test_sets/465c7ec4-d33e-11e3-a102-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/465c7ec4-d33e-11e3-a102-52540035b04c</a><br/> &gt; <a href="https://maloo.whamcloud.com/test_sets/542c478e-d75b-11e3-9d16-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/542c478e-d75b-11e3-9d16-52540035b04c</a></p> <p>In above failure cases, the test_11b has failed because improperly FID seq checking and error handler. I have updated the patch 10276 to resolve that.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/10276/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10276/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5023" title="sanity-lfsck test_12: OST lfsck_stop hung" class="issue-link" data-issue-key="LU-5023"><del>LU-5023</del></a> tests: check FID seq properly for sanity-lfsck t_11b<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 89a53691076ba45ef415de511ed840a0e3cbdd2c</p> <p>Landed for 2.7</p> Related LU-5049 Development Rank 1|hzwm1z: Rank (Obsolete) 13903 Severity