https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4179 Lustre <p>We have the crash dumps. But requires analysis by US personal only.</p> <p>LustreError: 6425:0:(mdt_open.c:1690:mdt_reint_open()) ASSERTION( !lustre_handle_is_used(&amp;lhc-&gt;mlh_reg_lh) ) failed: ^M<br/> LustreError: 6425:0:(mdt_open.c:1690:mdt_reint_open()) LBUG^M<br/> 4&gt;Pid: 6425, comm: mdt01_002^M<br/> &lt;4&gt;^M<br/> &lt;4&gt;Call Trace:^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa041f895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa041fe97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0ca7553&gt;&#93;</span> mdt_reint_open+0x1973/0x20c0 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0ca832c&gt;&#93;</span> mdt_reconstruct_open+0x68c/0xc30 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa072d6a6&gt;&#93;</span> ? __req_capsule_get+0x166/0x700 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa07061ae&gt;&#93;</span> ? lustre_pack_reply_flags+0xae/0x1f0 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c9b195&gt;&#93;</span> mdt_reconstruct+0x45/0x120 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c76cfb&gt;&#93;</span> mdt_reint_internal+0x6bb/0x780 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c7708d&gt;&#93;</span> mdt_intent_reint+0x1ed/0x520 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c74f3e&gt;&#93;</span> mdt_intent_policy+0x39e/0x720 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06bd7e1&gt;&#93;</span> ldlm_lock_enqueue+0x361/0x8d0 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06e424f&gt;&#93;</span> ldlm_handle_enqueue0+0x4ef/0x10b0 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c753c6&gt;&#93;</span> mdt_enqueue+0x46/0xe0 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c7bab7&gt;&#93;</span> mdt_handle_common+0x647/0x16d0 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0706c0c&gt;&#93;</span> ? lustre_msg_get_transno+0x8c/0x100 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0cb5295&gt;&#93;</span> mds_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0716428&gt;&#93;</span> ptlrpc_server_handle_request+0x398/0xc60 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa04205de&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0431dbf&gt;&#93;</span> ? lc_watchdog_touch+0x6f/0x170 <span class="error">&#91;libcfs&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa070d789&gt;&#93;</span> ? ptlrpc_wait_event+0xa9/0x290 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810557f3&gt;&#93;</span> ? __wake_up+0x53/0x70^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa07177be&gt;&#93;</span> ptlrpc_main+0xace/0x1700 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0716cf0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0716cf0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0716cf0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20^M<br/> &lt;4&gt;^M<br/> &lt;0&gt;Kernel panic - not syncing: LBUG^M<br/> &lt;4&gt;Pid: 6425, comm: mdt01_002 Tainted: G --------------- T 2.6.32-358.6.2.el6.20130607.x86_64.lustre240 #1^M<br/> &lt;4&gt;Call Trace:^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8153e8da&gt;&#93;</span> ? panic+0xa7/0x190^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa041feeb&gt;&#93;</span> ? lbug_with_loc+0x9b/0xb0 <span class="error">&#91;libcfs&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0ca7553&gt;&#93;</span> ? mdt_reint_open+0x1973/0x20c0 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0ca832c&gt;&#93;</span> ? mdt_reconstruct_open+0x68c/0xc30 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa072d6a6&gt;&#93;</span> ? __req_capsule_get+0x166/0x700 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa07061ae&gt;&#93;</span> ? lustre_pack_reply_flags+0xae/0x1f0 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c9b195&gt;&#93;</span> ? mdt_reconstruct+0x45/0x120 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c76cfb&gt;&#93;</span> ? mdt_reint_internal+0x6bb/0x780 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c7708d&gt;&#93;</span> ? mdt_intent_reint+0x1ed/0x520 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c74f3e&gt;&#93;</span> ? mdt_intent_policy+0x39e/0x720 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06bd7e1&gt;&#93;</span> ? ldlm_lock_enqueue+0x361/0x8d0 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06e424f&gt;&#93;</span> ? ldlm_handle_enqueue0+0x4ef/0x10b0 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c753c6&gt;&#93;</span> ? mdt_enqueue+0x46/0xe0 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c7bab7&gt;&#93;</span> ? mdt_handle_common+0x647/0x16d0 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0706c0c&gt;&#93;</span> ? lustre_msg_get_transno+0x8c/0x100 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0cb5295&gt;&#93;</span> ? mds_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0716428&gt;&#93;</span> ? ptlrpc_server_handle_request+0x398/0xc60 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa04205de&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0431dbf&gt;&#93;</span> ? lc_watchdog_touch+0x6f/0x170 <span class="error">&#91;libcfs&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa070d789&gt;&#93;</span> ? ptlrpc_wait_event+0xa9/0x290 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810557f3&gt;&#93;</span> ? __wake_up+0x53/0x70^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa07177be&gt;&#93;</span> ? ptlrpc_main+0xace/0x1700 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0716cf0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> ? child_rip+0xa/0x20^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0716cf0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0716cf0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20^M</p> our source is at <a href="git://github.com/jlan/lustre-nas.git">git://github.com/jlan/lustre-nas.git</a> LU-4179

LBUG ASSERTION( !lustre_handle_is_used(&lhc->mlh_reg_lh) ) failed:

Bug Critical Resolved Fixed Di Wang Mahmoud Hanafi server Tue, 29 Oct 2013 17:56:57 +0000 Tue, 31 Dec 2013 15:41:09 +0000 Tue, 3 Dec 2013 19:16:53 +0000 Lustre 2.4.0 Lustre 2.4.1 Lustre 2.6.0 Lustre 2.4.2 Lustre 2.5.1 0 7 <p>Correction Effected version should be 2.4 not 2.1.5</p> <p>Keith is a US citizen and so should be ok to look over the crash dump.</p> <p>I have sent email asking for the steps to access the crashdump. </p> <p>Can you confirm both Lustre Server and Client Version? </p> <p>clients are 2.1.5 and a few 2.3<br/> Servers are 2.4.0-3nas </p> <p>See git://github.com/jlan/lustre-nas.git for our source tree.</p> <p>I'll will try to get you the crash dumps.</p> <p>If you are looking for source at github,<br/> look for nas-2.1.5 branch for the 2.1.5 code and nas-2.4.0-1 branch for the 2.4.0-3nasS code.</p> <p>Is DNE being used?</p> <p>How often are you hitting this issue? Can you say when it started? </p> <p>There is no easy answer from the crashdump and it is in a complicated area of the code. I have unwound the stack and have the struct mdt_thread_info *info being passed into mdt_reint_open. I am going to see if I can find the mdt_lock_handle tomorrow and continue to dig into this issue. </p> <p>I was able to extract a good Lustre Debug log from the crash. The system was busy with service.c:1079:ptlrpc_update_export_timer() during the time of the error. I don't know if this is important or not yet. </p> <p>We not using DNE.</p> <p>We are hitting this bug at least once a day. So this is number one on our priority list to be fixed.</p> <p>I have attached a possible fix from Di Wang. Please test it and report back. </p> <p>If you get an LASSERT with this patch applied please send a fresh crashdump. </p> <p>A patch against master can be tracked here: <a href="http://review.whamcloud.com/8142" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8142</a> </p> <p>I applied the patch against 2.4.0 and built. Hit build errors:</p> <p>/usr/src/redhat/BUILD/lustre-2.4.0/lustre/mdt/mdt_open.c: In function 'mdt_reint_open':<br/> /usr/src/redhat/BUILD/lustre-2.4.0/lustre/mdt/mdt_open.c:1741: error: invalid storage class for function 'mdt_mfd_closed'<br/> cc1: warnings being treated as errors<br/> /usr/src/redhat/BUILD/lustre-2.4.0/lustre/mdt/mdt_open.c:1740: error: ISO C90 forbids mixed declarations and code<br/> /usr/src/redhat/BUILD/lustre-2.4.0/lustre/mdt/mdt_open.c:2004: error: expected declaration or statement at end of input<br/> make<span class="error">&#91;8&#93;</span>: *** <span class="error">&#91;/usr/src/redhat/BUILD/lustre-2.4.0/lustre/mdt/mdt_open.o&#93;</span> Error 1<br/> make<span class="error">&#91;7&#93;</span>: *** <span class="error">&#91;/usr/src/redhat/BUILD/lustre-2.4.0/lustre/mdt&#93;</span> Error 2<br/> make<span class="error">&#91;7&#93;</span>: *** Waiting for unfinished jobs....<br/> make<span class="error">&#91;6&#93;</span>: *** <span class="error">&#91;/usr/src/redhat/BUILD/lustre-2.4.0/lustre&#93;</span> Error 2<br/> make<span class="error">&#91;5&#93;</span>: *** <span class="error">&#91;_module_/usr/src/redhat/BUILD/lustre-2.4.0&#93;</span> Error 2</p> <p>Sorry Jay, Let me work with this patch some more. We see this with the Master build as well. </p> <p>Hmm, the patch post on <a href="http://review.whamcloud.com/8142" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8142</a> missing a "}"</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/lustre/mdt/mdt_open.c b/lustre/mdt/mdt_open.c index b4057a0..570d07c 100644 --- a/lustre/mdt/mdt_open.c +++ b/lustre/mdt/mdt_open.c @@ -1841,17 +1841,18 @@ int mdt_reint_open(struct mdt_thread_info *info, struct mdt_lock_handle *lhc) } } - LASSERT(!lustre_handle_is_used(&amp;lhc-&gt;mlh_reg_lh)); - - /* get openlock if this is not replay and if a client requested it */ - if (!req_is_replay(req)) { - rc = mdt_object_open_lock(info, child, lhc, &amp;ibits); - if (rc != 0) - GOTO(out_child_unlock, result = rc); - else if (create_flags &amp; MDS_OPEN_LOCK) - mdt_set_disposition(info, ldlm_rep, DISP_OPEN_LOCK); + if (lustre_handle_is_used(&amp;lhc-&gt;mlh_reg_lh)) { + LASSERT((lustre_msg_get_flags(req-&gt;rq_reqmsg) &amp; MSG_RESENT)); + } else { + /* get openlock if this is not replay and if a client requested it */ + if (!req_is_replay(req)) { + rc = mdt_object_open_lock(info, child, lhc, &amp;ibits); + if (rc != 0) + GOTO(out_child_unlock, result = rc); + else if (create_flags &amp; MDS_OPEN_LOCK) + mdt_set_disposition(info, ldlm_rep, DISP_OPEN_LOCK); + } } - /* Try to open it now. */ rc = mdt_finish_open(info, parent, child, create_flags, created, ldlm_rep); </pre> </div></div> <p>But please hold off a bit to try, since I need revisit these code a bit, Thanks!</p> <p>I just updated the patch <a href="http://review.whamcloud.com/#/c/8142/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8142/</a> , please try again. Thanks.</p> <p>Hi Di,</p> <p>The new patch can not be applied to 2.4.0.</p> <p>Your new patch contains this line:<br/> GOTO(out_child_unlock, result = rc);<br/> but my code does not have out_child_unlock label.<br/> There is "out_child" label in the routine, but I can not tell what other changes need to be made.</p> <p>Could you write a patch that is applicable on top of 2.4.0?</p> <p>Thanks!</p> <p><a href="http://review.whamcloud.com/8145" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8145</a> Here is the one for b2_4, and the patch should be able to applied to 2.4.0 as well, please try.</p> <p>Jay</p> <p>Just to be clear - please only use the patch in a test environment until we have completed our validation</p> <p>Thanks</p> <p>Peter</p> <p>What is the risk with this patch. We can't fully test the patch using test env because we didn't isolate trigger. We would need to deploy it on production to fully test it.</p> <p>IMHO, the patch itself is fine, but we can not say it is "safe', until it went through all of our validation processes, i.e. reviewing, passing internal tests and landing, as peter mentioned.</p> <p><a href="http://review.whamcloud.com/8145" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8145</a> has the wrong tag "based on master), actually that patch is based on b2_4, so I resubmit a new one <a href="http://review.whamcloud.com/8173" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8173</a> please track this one.</p> <p>Hmm, I had no problem patching and building based on<br/> <a href="http://review.whamcloud.com/8145" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8145</a> in my b2_4 build environment.</p> <p>I compared one in 8145 with one in 8173, the patches seemed to be identical to me?</p> <p>Yes, they are same, but just some tag issue. If you just pull out the patch and build yourself, you can ignore it.</p> <p>Lustre client: <a href="http://build.whamcloud.com/job/lustre-b2_4/57/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/57/</a><br/> Lustre server: <a href="http://build.whamcloud.com/job/lustre-b2_3/41/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/41/</a> (2.3.0)</p> <p>racer test also hit this LBUG:<br/> <a href="https://maloo.whamcloud.com/test_sets/d1007e84-54b0-11e3-9029-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/d1007e84-54b0-11e3-9029-52540035b04c</a></p> <p>Patched applied and haven't see this bug.<br/> Note: NASA BUILD 2.4.0.4-1nasS</p> <p>Please close</p> <p>Great - thanks Mahmoud!</p> Related LU-3607 Development Epic server Rank 1|hzw77r: Rank (Obsolete) 11313 Severity