https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6230 Lustre <p>For open case, the client side open handling thread may hit error after the MDT grant the open. Under the such case, the client should send close RPC to the MDT as cleanup; otherwise, the open handle on the MDT will be leaked there until the client umount or evicted.</p> <p>If the LFSCK marks LU_OBJECT_HEARD_BANSHEE on the MDT-object that is opened by others for repairing some inconsistency, such as repairing multiple-referenced OST-object, because the leaked open handle still references the MDT-object, then it will block the subsequent threads that want to locate such object via FID.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>23:07:57:INFO: task mdt00_000:6380 blocked for more than 120 seconds. 23:07:57: Not tainted 2.6.32-504.8.1.el6_lustre.g0ef66b1.x86_64 #1 23:07:57:mdt00_000 D 0000000000000001 0 6380 2 0x00000080 23:07:57:Call Trace: 23:07:57: [&lt;ffffffffa05f62af&gt;] ? lu_object_find_try+0x9f/0x260 [obdclass] 23:07:57: [&lt;ffffffffa05f64ad&gt;] lu_object_find_at+0x3d/0xe0 [obdclass] 23:07:57: [&lt;ffffffffa05f6566&gt;] lu_object_find+0x16/0x20 [obdclass] 23:07:57: [&lt;ffffffffa0ebe056&gt;] mdt_object_find+0x56/0x170 [mdt] 23:07:57: [&lt;ffffffffa0ef5407&gt;] mdt_reint_open+0x1527/0x2c70 [mdt] 23:07:57: [&lt;ffffffffa0edd0cd&gt;] mdt_reint_rec+0x5d/0x200 [mdt] 23:07:57: [&lt;ffffffffa0ec123b&gt;] mdt_reint_internal+0x4cb/0x7a0 [mdt] 23:07:57: [&lt;ffffffffa0ec1706&gt;] mdt_intent_reint+0x1f6/0x430 [mdt] 23:07:57: [&lt;ffffffffa0ebfcf4&gt;] mdt_intent_policy+0x494/0xce0 [mdt] 23:07:57: [&lt;ffffffffa07c24f9&gt;] ldlm_lock_enqueue+0x129/0x9d0 [ptlrpc] 23:07:57: [&lt;ffffffffa07ee48b&gt;] ldlm_handle_enqueue0+0x51b/0x13f0 [ptlrpc] 23:07:57: [&lt;ffffffffa086e951&gt;] tgt_enqueue+0x61/0x230 [ptlrpc] 23:07:57: [&lt;ffffffffa086f59e&gt;] tgt_request_handle+0x8be/0x1000 [ptlrpc] 23:07:57: [&lt;ffffffffa081f5c1&gt;] ptlrpc_main+0xe41/0x1960 [ptlrpc] 23:07:57: [&lt;ffffffff8109e66e&gt;] kthread+0x9e/0xc0 23:07:57: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 </pre> </div></div> LU-6230

open handle leak

Bug Blocker Resolved Fixed nasf nasf Tue, 10 Feb 2015 09:58:14 +0000 Thu, 26 Feb 2015 19:12:14 +0000 Thu, 26 Feb 2015 19:12:14 +0000 Lustre 2.7.0 Lustre 2.7.0 0 8 <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13709" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13709</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6230" title="open handle leak" class="issue-link" data-issue-key="LU-6230"><del>LU-6230</del></a> llite: cleanup open handle for client open failure<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 307f1432afc920155b8b800841a626fcfe858bc8</p> <p>In fact, the <a href="https://jira.whamcloud.com/browse/LU-5791" title="LFSCK 5: use bottom object for consistency verification" class="issue-link" data-issue-key="LU-5791"><del>LU-5791</del></a> patch <a href="http://review.whamcloud.com/#/c/13392/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13392/</a> (that has been landed to master already) depends on this patch, otherwise the sanity-lfsck will get failure.</p> <p>Fan Yong, as much as I want to land this patch to fix the current testing problem, it is bad that a poorly-behaving client can cause the MDS to deadlock. That seems like a bug in the MDS code that it is blocked on a client open reference, even if the client is doing the wrong thing. I'd expect the MDS to not care at all whether the file is open and/or unlinked forever, since this is a normal use case and not even an error. At worst the MDS should evict such a client if there is a clear problem.</p> <p>We found this issue during sanity-lfsck test_17 failure. There are some failure instances in <a href="https://jira.whamcloud.com/browse/LU-6727" title="parallel-scale test mdtestfpp hung" class="issue-link" data-issue-key="LU-6727"><del>LU-6727</del></a>.<br/> The root reason for such failure is that when the LFSCK repairing multiple referenced OST-object, it will set the un-recognized MDT-object as LU_OBJECT_HEARD_BANSHEE to make such MDT-object to reload OSP-object after the LOV EA refreshed. But at that time, some others may still reference such wrong MDT-object (in sanity-lfsck test_17, it is the leaked open handle). Then all the subsequent object locating against such MDT-object will be blocked there.</p> <p>Generally, asking other to release the reference is NOT the right solution, because it is normal that someone reference the MDT-object for very time, such as open().</p> <p>The right solution is NOT set LU_OBJECT_HEARD_BANSHEE on the MDT-object, instead, we should make the LOD-object to re-attach the OSP-object(s) after the LOV EA refreshed.</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13848" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13848</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6230" title="open handle leak" class="issue-link" data-issue-key="LU-6230"><del>LU-6230</del></a> lfsck: reload OSP-object via set LOV EA on LOD-object<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3d020e94a3bdb74f4db956f551bac058eaf18a44</p> <p>&gt; For open case, the client side open handling thread may hit error after the MDT grant the open. Under the such case, the client should send close RPC to the MDT as cleanup; otherwise, the open handle on the MDT will be leaked there until the client umount or evicted. In further, if someone unlinked the file, but because the open handle holds the reference on such file/object, then it will block the subsequent threads that want to locate such object via FID.</p> <p>Is this description correct? If there is an open handle then mod_count should remain positive and so the object will not be destroyed. So LU_OBJECT_HEARD_BANSHEE will not be set because of the unlink. Or am I missing something? Would you please offer a test case that reproduces the behavior in the description?</p> <p>&gt;&gt; For open case, the client side open handling thread may hit error after the MDT grant the open. Under the such case, the client should send close RPC to the MDT as cleanup; otherwise, the open handle on the MDT will be leaked there until the client umount or evicted. In further, if someone unlinked the file, but because the open handle holds the reference on such file/object, then it will block the subsequent threads that want to locate such object via FID.<br/> &gt; Is this description correct? If there is an open handle then mod_count should remain positive and so the object will not be destroyed. So LU_OBJECT_HEARD_BANSHEE will not be set because of the unlink. Or am I missing something? Would you please offer a test case that reproduces the behavior in the description?</p> <p>Sorry, some misguide in the issue description. It should be the LFSCK set LU_OBJECT_HEARD_BANSHEE on the MDT-object to repair multiple-referenced OST-object, but such MDT-object is still referenced by the leaked open handle. I have updated the issue description.</p> <p>As for the test case, sanity-lfsck test_17 can be used to verify that. All patches based on current master failed for this issue.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13848/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13848/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6230" title="open handle leak" class="issue-link" data-issue-key="LU-6230"><del>LU-6230</del></a> lfsck: reload OSP-object via set LOV EA on LOD-object<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7c82a9c81d03dec059132dddafd0bdde188b321d</p> <p>Patch landed to Master. Additional work will land under <a href="https://jira.whamcloud.com/browse/LU-6301" title="open handle leak" class="issue-link" data-issue-key="LU-6301"><del>LU-6301</del></a></p> Duplicate LU-6272 Related LU-6301 Development Rank 1|hzx62n: Rank (Obsolete) 17440 Severity