https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4403 Lustre <p>&lt;0&gt;LustreError: 5766:0:(ldlm_lock.c:851:ldlm_lock_decref_internal_nolock()) ASSERTION( lock-&gt;l_readers &gt; 0 ) failed: ^M<br/> &lt;0&gt;LustreError: 5766:0:(ldlm_lock.c:851:ldlm_lock_decref_internal_nolock()) LBUG^M<br/> &lt;4&gt;Pid: 5766, comm: mdt00_020^M<br/> &lt;4&gt;^M<br/> &lt;4&gt;Call Trace:^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0414895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0414e97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06b03b2&gt;&#93;</span> ldlm_lock_decref_internal_nolock+0xd2/0x180 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06b4aad&gt;&#93;</span> ldlm_lock_decref_internal+0x4d/0xaa0 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa054a315&gt;&#93;</span> ? class_handle2object+0x95/0x190 <span class="error">&#91;obdclass&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06b5f69&gt;&#93;</span> ldlm_lock_decref+0x39/0x90 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0dd74a3&gt;&#93;</span> mdt_save_lock+0x63/0x300 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06fd900&gt;&#93;</span> ? lustre_swab_ldlm_reply+0x0/0x40 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0dd779c&gt;&#93;</span> mdt_object_unlock+0x5c/0x160 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0e05a4c&gt;&#93;</span> mdt_object_open_unlock+0xac/0x110 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0e0c9b4&gt;&#93;</span> mdt_reint_open+0xdd4/0x20e0 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0e0e34c&gt;&#93;</span> mdt_reconstruct_open+0x68c/0xc30 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa07226a6&gt;&#93;</span> ? __req_capsule_get+0x166/0x700 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06fb1ae&gt;&#93;</span> ? lustre_pack_reply_flags+0xae/0x1f0 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0e01195&gt;&#93;</span> mdt_reconstruct+0x45/0x120 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0ddccfb&gt;&#93;</span> mdt_reint_internal+0x6bb/0x780 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0ddd08d&gt;&#93;</span> mdt_intent_reint+0x1ed/0x520 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0ddaf3e&gt;&#93;</span> mdt_intent_policy+0x39e/0x720 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06b27e1&gt;&#93;</span> ldlm_lock_enqueue+0x361/0x8d0 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06d924f&gt;&#93;</span> ldlm_handle_enqueue0+0x4ef/0x10b0 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0ddb3c6&gt;&#93;</span> mdt_enqueue+0x46/0xe0 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0de1ab7&gt;&#93;</span> mdt_handle_common+0x647/0x16d0 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0e1b2b5&gt;&#93;</span> mds_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa070b428&gt;&#93;</span> ptlrpc_server_handle_request+0x398/0xc60 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa04155de&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0426dbf&gt;&#93;</span> ? lc_watchdog_touch+0x6f/0x170 <span class="error">&#91;libcfs&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0702789&gt;&#93;</span> ? ptlrpc_wait_event+0xa9/0x290 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810557f3&gt;&#93;</span> ? __wake_up+0x53/0x70^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa070c7be&gt;&#93;</span> ptlrpc_main+0xace/0x1700 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa070bcf0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa070bcf0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa070bcf0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20^M<br/> &lt;4&gt;^M</p> LU-4403

ASSERTION( lock->l_readers > 0 )

Bug Major Resolved Fixed Jinshan Xiong Mahmoud Hanafi llnl mn4 Fri, 20 Dec 2013 04:33:41 +0000 Mon, 4 Aug 2014 23:57:07 +0000 Sat, 8 Feb 2014 05:32:12 +0000 Lustre 2.4.1 Lustre 2.6.0 Lustre 2.5.2 0 10 <p>Mike</p> <p>Do you think that this might be related to <a href="https://jira.whamcloud.com/browse/LU-4135" title="mdt_save_lock() is broken" class="issue-link" data-issue-key="LU-4135"><del>LU-4135</del></a>?</p> <p>Peter</p> <p>IMHO, the fix in <a href="https://jira.whamcloud.com/browse/LU-4135" title="mdt_save_lock() is broken" class="issue-link" data-issue-key="LU-4135"><del>LU-4135</del></a> only fix the problem in master, which does not exist in b2_4. So this should be different issue, I will take a look now.</p> <p>It seems this is caused by the race between mdt_intent_fixup_resend and mdt_object_unlock, i.e. mdt_intent_fixup_resend might return a released lock here. Here is the patch <a href="http://review.whamcloud.com/8642" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8642</a></p> <p>Thanks Di!</p> <p><a href="http://review.whamcloud.com/8680" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8680</a> patch for master.</p> <p>We just hit the same problem half an hour ago. The MDS runs the lustre server with patch #8642 included.</p> <p>Jay</p> <p>How long has the patch been applied? Is it possible to ascertain yet whether the frequency of occurrence has been altered since it was applied?</p> <p>Peter</p> <p>Jay,</p> <p>Same stack trace? If not, please post here. Are there any other console error messages? </p> <p>WangDi</p> <p>Here is the stack trace.</p> <p>bp7-mds1 login: Lustre: MGS: haven't heard from client fd1923ac-e3da-a3a1-46c2-d0613e7a86a3 (at 10.151.0.150@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880696019800, cur 1388538333 expire 1388538183 last 1388538106^M<br/> Lustre: Skipped 6 previous similar messages^M<br/> CONSOLE MARK 201312311730<br/> <span class="error">&#91;-- MARK -- Tue Dec 31 18:00:00 2013&#93;</span>^M<br/> CONSOLE MARK 201312311830<br/> <span class="error">&#91;-- MARK -- Tue Dec 31 19:00:00 2013&#93;</span>^M<br/> LustreError: 5827:0:(ldlm_lock.c:851:ldlm_lock_decref_internal_nolock()) ASSERTION( lock-&gt;l_readers &gt; 0 ) failed: ^M<br/> LustreError: 5827:0:(ldlm_lock.c:851:ldlm_lock_decref_internal_nolock()) LBUG^M<br/> Pid: 5827, comm: mdt01_041^M<br/> ^M<br/> Call Trace:^M<br/> <span class="error">&#91;&lt;ffffffffa0414895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa0414e97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa06b03b2&gt;&#93;</span> ldlm_lock_decref_internal_nolock+0xd2/0x180 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa06b4aad&gt;&#93;</span> ldlm_lock_decref_internal+0x4d/0xaa0 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa054a315&gt;&#93;</span> ? class_handle2object+0x95/0x190 <span class="error">&#91;obdclass&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa06b5f69&gt;&#93;</span> ldlm_lock_decref+0x39/0x90 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa0dd74a3&gt;&#93;</span> mdt_save_lock+0x63/0x300 <span class="error">&#91;mdt&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa06fd900&gt;&#93;</span> ? lustre_swab_ldlm_reply+0x0/0x40 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa0dd779c&gt;&#93;</span> mdt_object_unlock+0x5c/0x160 <span class="error">&#91;mdt&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa0e05a5c&gt;&#93;</span> mdt_object_open_unlock+0xac/0x110 <span class="error">&#91;mdt&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa0e0c9c4&gt;&#93;</span> mdt_reint_open+0xdd4/0x20e0 <span class="error">&#91;mdt&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa0e0e35c&gt;&#93;</span> mdt_reconstruct_open+0x68c/0xc30 <span class="error">&#91;mdt&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa07226a6&gt;&#93;</span> ? __req_capsule_get+0x166/0x700 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa06fb1ae&gt;&#93;</span> ? lustre_pack_reply_flags+0xae/0x1f0 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa0e011a5&gt;&#93;</span> mdt_reconstruct+0x45/0x120 <span class="error">&#91;mdt&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa0ddcd0b&gt;&#93;</span> mdt_reint_internal+0x6bb/0x780 <span class="error">&#91;mdt&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa0ddd09d&gt;&#93;</span> mdt_intent_reint+0x1ed/0x520 <span class="error">&#91;mdt&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa0ddaf4e&gt;&#93;</span> mdt_intent_policy+0x39e/0x720 <span class="error">&#91;mdt&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa06b27e1&gt;&#93;</span> ldlm_lock_enqueue+0x361/0x8d0 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> ^M<br/> Entering kdb (current=0xffff88067f709540, pid 5827) on processor 12 Oops: (null)^M</p> <p>One thing to note: when the MDS hits this lbug several OSS will crash here is their bt.<br/> crash&gt; bt<br/> PID: 12506 TASK: ffff880efef7b540 CPU: 11 COMMAND: "ll_ost_io03_077"<br/> #0 <span class="error">&#91;ffff880efef7d150&#93;</span> machine_kexec at ffffffff81035d2b<br/> #1 <span class="error">&#91;ffff880efef7d1b0&#93;</span> crash_kexec at ffffffff810c0412<br/> #2 <span class="error">&#91;ffff880efef7d280&#93;</span> kdb_kdump_check at ffffffff81285507<br/> #3 <span class="error">&#91;ffff880efef7d290&#93;</span> kdb_main_loop at ffffffff812886f7<br/> #4 <span class="error">&#91;ffff880efef7d3a0&#93;</span> kdb_save_running at ffffffff8128285e<br/> #5 <span class="error">&#91;ffff880efef7d3b0&#93;</span> kdba_main_loop at ffffffff81463518<br/> #6 <span class="error">&#91;ffff880efef7d3f0&#93;</span> kdb at ffffffff812859f6<br/> #7 <span class="error">&#91;ffff880efef7d460&#93;</span> kdba_entry at ffffffff81462e37<br/> #8 <span class="error">&#91;ffff880efef7d470&#93;</span> notifier_call_chain at ffffffff81544b85<br/> #9 <span class="error">&#91;ffff880efef7d4b0&#93;</span> atomic_notifier_call_chain at ffffffff81544bea<br/> #10 <span class="error">&#91;ffff880efef7d4c0&#93;</span> notify_die at ffffffff8109c15e<br/> #11 <span class="error">&#91;ffff880efef7d4f0&#93;</span> __die at ffffffff81542a52<br/> #12 <span class="error">&#91;ffff880efef7d520&#93;</span> die at ffffffff8100f288<br/> #13 <span class="error">&#91;ffff880efef7d550&#93;</span> do_general_protection at ffffffff81542632<br/> #14 <span class="error">&#91;ffff880efef7d580&#93;</span> general_protection at ffffffff81541dc5<br/> <span class="error">&#91;exception RIP: lqe64_hash_keycmp+12&#93;</span><br/> RIP: ffffffffa0bf8f2c RSP: ffff880efef7d630 RFLAGS: 00010206<br/> RAX: 000000000000114b RBX: ffff880fd044f680 RCX: 0000000000000000<br/> RDX: 0000000000000000 RSI: 5a5a5a5a5a5a5a5a RDI: ffff880f11a5ef60<br/> RBP: ffff880efef7d630 R8: 0000000000000003 R9: 0000000000000001<br/> R10: 0000000000000001 R11: 0000000000000000 R12: ffff880efef7d6c0<br/> R13: ffff880f11a5ef60 R14: 0000000000000000 R15: 5a5a5a5a5a5a5a5a<br/> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018<br/> #15 <span class="error">&#91;ffff880efef7d638&#93;</span> cfs_hash_bd_lookup_intent at ffffffffa04269e5 <span class="error">&#91;libcfs&#93;</span><br/> #16 <span class="error">&#91;ffff880efef7d688&#93;</span> cfs_hash_dual_bd_lookup_locked at ffffffffa0426b67 <span class="error">&#91;libcfs&#93;</span><br/> #17 <span class="error">&#91;ffff880efef7d6b8&#93;</span> cfs_hash_lookup at ffffffffa0427da4 <span class="error">&#91;libcfs&#93;</span><br/> #18 <span class="error">&#91;ffff880efef7d708&#93;</span> lqe_locate at ffffffffa0bf94f7 <span class="error">&#91;lquota&#93;</span><br/> #19 <span class="error">&#91;ffff880efef7d748&#93;</span> qsd_op_begin at ffffffffa0c0c2bf <span class="error">&#91;lquota&#93;</span><br/> #20 <span class="error">&#91;ffff880efef7d7e8&#93;</span> osd_declare_qid at ffffffffa0d0ac76 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> #21 <span class="error">&#91;ffff880efef7d848&#93;</span> osd_declare_inode_qid at ffffffffa0d0b03e <span class="error">&#91;osd_ldiskfs&#93;</span><br/> #22 <span class="error">&#91;ffff880efef7d8a8&#93;</span> osd_declare_write_commit at ffffffffa0cfc3d1 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> #23 <span class="error">&#91;ffff880efef7d948&#93;</span> ofd_commitrw_write at ffffffffa0dc1a93 <span class="error">&#91;ofd&#93;</span><br/> #24 <span class="error">&#91;ffff880efef7d9d8&#93;</span> ofd_commitrw at ffffffffa0dc4abd <span class="error">&#91;ofd&#93;</span><br/> #25 <span class="error">&#91;ffff880efef7da68&#93;</span> obd_commitrw at ffffffffa0d791d8 <span class="error">&#91;ost&#93;</span><br/> #26 <span class="error">&#91;ffff880efef7dae8&#93;</span> ost_brw_write at ffffffffa0d831d1 <span class="error">&#91;ost&#93;</span><br/> #27 <span class="error">&#91;ffff880efef7dc78&#93;</span> ost_handle at ffffffffa0d8942b <span class="error">&#91;ost&#93;</span><br/> #28 <span class="error">&#91;ffff880efef7ddb8&#93;</span> ptlrpc_server_handle_request at ffffffffa0707428 <span class="error">&#91;ptlrpc&#93;</span><br/> #29 <span class="error">&#91;ffff880efef7deb8&#93;</span> ptlrpc_main at ffffffffa07087be <span class="error">&#91;ptlrpc&#93;</span><br/> #30 <span class="error">&#91;ffff880efef7df48&#93;</span> kernel_thread at ffffffff8100c0ca</p> <p>hmm, I think this OSS crash is a different issue, you probably need open a new ticket for this.</p> <p>Hmm, Does your lustre version includes the fix from <a href="https://jira.whamcloud.com/browse/LU-4179" title="LBUG ASSERTION( !lustre_handle_is_used(&amp;lhc-&gt;mlh_reg_lh) ) failed:" class="issue-link" data-issue-key="LU-4179"><del>LU-4179</del></a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> LU-4179 mdt: skip open lock enqueue during resent Skip open lock enqueue, if the open lock has been acquired(mdt_intent_fixup_resent) during resent. Signed-off-by: wang di &lt;di.wang@intel.com&gt; Change-Id: I625ca438e28520416ee2af884d0a9f9e6f21cf2e Reviewed-on: http://review.whamcloud.com/8173 Tested-by: Jenkins Tested-by: Maloo &lt;hpdd-maloo@intel.com&gt; Reviewed-by: Oleg Drokin &lt;oleg.drokin@intel.com&gt; </pre> </div></div> <p>and </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> LU-3273 mdt: Hold med_open_lock before walking med_open_head Fixed a bug where during replay mdt_mfd_open() calls mdt_handle2mfd() without acquiring the med_open_lock. We now take the med_open_lock before traversing med_open_head list. This bug was noticed during the analysis of LU-3233. Signed-off-by: Swapnil Pimpale &lt;spimpale@ddn.com&gt; Change-Id: Ib879f65d41d35f266897e8961dac78e6c4f0d9ec Reviewed-on: http://review.whamcloud.com/7272 Tested-by: Hudson Tested-by: Maloo &lt;whamcloud.maloo@gmail.com&gt; Reviewed-by: John L. Hammond &lt;john.hammond@intel.com&gt; Reviewed-by: Andreas Dilger &lt;andreas.dilger@intel.com&gt; </pre> </div></div> <p>Thanks.</p> <p>Peter, the patch was installed Saturday 12/28 afternoon. I do not know how frequent it crashed before. Mahmoud would know better.</p> <p>Di, the source is at <a href="https://github.com/jlan/lustre-nas/commits/nas-2.4.0-1" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jlan/lustre-nas/commits/nas-2.4.0-1</a>.<br/> The relevant branch is nas-2.4.0-1. The build included commits up to <a href="https://jira.whamcloud.com/browse/LU-4403" title="ASSERTION( lock-&gt;l_readers &gt; 0 )" class="issue-link" data-issue-key="LU-4403"><del>LU-4403</del></a>.</p> <p>We do have <a href="https://jira.whamcloud.com/browse/LU-4179" title="LBUG ASSERTION( !lustre_handle_is_used(&amp;lhc-&gt;mlh_reg_lh) ) failed:" class="issue-link" data-issue-key="LU-4179"><del>LU-4179</del></a>, but not <a href="https://jira.whamcloud.com/browse/LU-3273" title="mdt_mfd_open() may call mdt_handle2mfd() w/o holding med_open_lock" class="issue-link" data-issue-key="LU-3273"><del>LU-3273</del></a>.</p> <p>Mahmoud, Could you please trace to the source code which line this "<span class="error">&#91;&lt;ffffffffa0e05a5c&gt;&#93;</span> mdt_object_open_unlock+0xac/0x110 <span class="error">&#91;mdt&#93;</span>" refers to?</p> <p>FILE: mdt_open.c<br/> int mdt_reint_open(struct mdt_thread_info *info, struct mdt_lock_handle *lhc)<br/> {<br/> .<br/> .<br/> out_child:<br/> mdt_object_open_unlock(info, child, lhc, ibits, result); &lt;+++++++++++++++++++++++++++++++ THIS LINE 1733<br/> mdt_object_put(info-&gt;mti_env, child);<br/> out_parent:<br/> mdt_object_unlock_put(info, parent, lh, result || !created);<br/> out:<br/> if (result)<br/> lustre_msg_set_transno(req-&gt;rq_repmsg, 0);<br/> return result;<br/> }</p> <p>crash&gt; bt -l<br/> PID: 5707 TASK: ffff880824046080 CPU: 16 COMMAND: "mdt00_005"<br/> #0 <span class="error">&#91;ffff88068a5a74c8&#93;</span> machine_kexec at ffffffff81035d2b<br/> /usr/src/debug/kernel-lustre240-2.6.32-358.6.2.el6/linux-2.6.32-358.6.2.el6.20130607.x86_64/arch/x86/kernel/machine_kexec_64.c: 336<br/> #1 <span class="error">&#91;ffff88068a5a7528&#93;</span> crash_kexec at ffffffff810c0412<br/> /usr/src/debug/kernel-lustre240-2.6.32-358.6.2.el6/linux-2.6.32-358.6.2.el6.20130607.x86_64/kernel/kexec.c: 1121<br/> #2 <span class="error">&#91;ffff88068a5a75f8&#93;</span> kdb_kdump_check at ffffffff81285507<br/> /usr/src/debug/kernel-lustre240-2.6.32-358.6.2.el6/linux-2.6.32-358.6.2.el6.20130607.x86_64/kdb/kdbmain.c: 1214<br/> #3 <span class="error">&#91;ffff88068a5a7608&#93;</span> kdb_main_loop at ffffffff812886f7<br/> /usr/src/debug/kernel-lustre240-2.6.32-358.6.2.el6/linux-2.6.32-358.6.2.el6.20130607.x86_64/kdb/kdbmain.c: 1322<br/> #4 <span class="error">&#91;ffff88068a5a7718&#93;</span> kdb_save_running at ffffffff8128285e<br/> /usr/src/debug/kernel-lustre240-2.6.32-358.6.2.el6/linux-2.6.32-358.6.2.el6.20130607.x86_64/kdb/kdbsupport.c: 798<br/> #5 <span class="error">&#91;ffff88068a5a7728&#93;</span> kdba_main_loop at ffffffff81463518<br/> /usr/src/debug/kernel-lustre240-2.6.32-358.6.2.el6/linux-2.6.32-358.6.2.el6.20130607.x86_64/arch/x86/kdb/kdba_support.c: 980<br/> #6 <span class="error">&#91;ffff88068a5a7768&#93;</span> kdb at ffffffff812859f6<br/> /usr/src/debug/kernel-lustre240-2.6.32-358.6.2.el6/linux-2.6.32-358.6.2.el6.20130607.x86_64/kdb/kdbmain.c: 2165<br/> #7 <span class="error">&#91;ffff88068a5a77d8&#93;</span> panic at ffffffff8153e8fb<br/> /usr/src/debug/kernel-lustre240-2.6.32-358.6.2.el6/linux-2.6.32-358.6.2.el6.20130607.x86_64/kernel/panic.c: 117<br/> #8 <span class="error">&#91;ffff88068a5a7858&#93;</span> lbug_with_loc at ffffffffa0414eeb <span class="error">&#91;libcfs&#93;</span><br/> #9 <span class="error">&#91;ffff88068a5a7878&#93;</span> ldlm_lock_decref_internal_nolock at ffffffffa06b03b2 <span class="error">&#91;ptlrpc&#93;</span><br/> #10 <span class="error">&#91;ffff88068a5a7898&#93;</span> ldlm_lock_decref_internal at ffffffffa06b4aad <span class="error">&#91;ptlrpc&#93;</span><br/> #11 <span class="error">&#91;ffff88068a5a78f8&#93;</span> ldlm_lock_decref at ffffffffa06b5f69 <span class="error">&#91;ptlrpc&#93;</span><br/> #12 <span class="error">&#91;ffff88068a5a7928&#93;</span> mdt_save_lock at ffffffffa0c6d4a3 <span class="error">&#91;mdt&#93;</span><br/> /usr/src/debug/lustre-2.4.0/lustre/mdt/mdt_handler.c: 2711<br/> #13 <span class="error">&#91;ffff88068a5a7978&#93;</span> mdt_object_unlock at ffffffffa0c6d79c <span class="error">&#91;mdt&#93;</span><br/> /usr/src/debug/lustre-2.4.0/lustre/mdt/mdt_handler.c: 2737<br/> #14 <span class="error">&#91;ffff88068a5a79a8&#93;</span> mdt_object_open_unlock at ffffffffa0c9ba5c <span class="error">&#91;mdt&#93;</span><br/> /usr/src/debug/lustre-2.4.0/lustre/mdt/mdt_open.c: 1270<br/> #15 <span class="error">&#91;ffff88068a5a79f8&#93;</span> mdt_reint_open at ffffffffa0ca29c4 <span class="error">&#91;mdt&#93;</span><br/> /usr/src/debug/lustre-2.4.0/lustre/mdt/mdt_open.c: 1734<br/> #16 <span class="error">&#91;ffff88068a5a7ae8&#93;</span> mdt_reconstruct_open at ffffffffa0ca435c <span class="error">&#91;mdt&#93;</span><br/> /usr/src/debug/lustre-2.4.0/libcfs/include/libcfs/libcfs_debug.h: 211<br/> #17 <span class="error">&#91;ffff88068a5a7b78&#93;</span> mdt_reconstruct at ffffffffa0c971a5 <span class="error">&#91;mdt&#93;</span><br/> /usr/src/debug/lustre-2.4.0/libcfs/include/libcfs/libcfs_debug.h: 211<br/> #18 <span class="error">&#91;ffff88068a5a7b98&#93;</span> mdt_reint_internal at ffffffffa0c72d0b <span class="error">&#91;mdt&#93;</span><br/> /usr/src/debug/lustre-2.4.0/libcfs/include/libcfs/libcfs_debug.h: 211<br/> #19 <span class="error">&#91;ffff88068a5a7bd8&#93;</span> mdt_intent_reint at ffffffffa0c7309d <span class="error">&#91;mdt&#93;</span><br/> /usr/src/debug/lustre-2.4.0/lustre/mdt/mdt_handler.c: 3828<br/> #20 <span class="error">&#91;ffff88068a5a7c28&#93;</span> mdt_intent_policy at ffffffffa0c70f4e <span class="error">&#91;mdt&#93;</span><br/> /usr/src/debug/lustre-2.4.0/libcfs/include/libcfs/libcfs_debug.h: 211<br/> #21 <span class="error">&#91;ffff88068a5a7c68&#93;</span> ldlm_lock_enqueue at ffffffffa06b27e1 <span class="error">&#91;ptlrpc&#93;</span><br/> #22 <span class="error">&#91;ffff88068a5a7cc8&#93;</span> ldlm_handle_enqueue0 at ffffffffa06d924f <span class="error">&#91;ptlrpc&#93;</span><br/> #23 <span class="error">&#91;ffff88068a5a7d38&#93;</span> mdt_enqueue at ffffffffa0c713d6 <span class="error">&#91;mdt&#93;</span><br/> /usr/src/debug/lustre-2.4.0/lustre/mdt/mdt_handler.c: 2283<br/> #24 <span class="error">&#91;ffff88068a5a7d58&#93;</span> mdt_handle_common at ffffffffa0c78787 <span class="error">&#91;mdt&#93;</span><br/> /usr/src/debug/lustre-2.4.0/lustre/mdt/mdt_handler.c: 3040<br/> #25 <span class="error">&#91;ffff88068a5a7da8&#93;</span> mds_regular_handle at ffffffffa0cb12c5 <span class="error">&#91;mdt&#93;</span><br/> /usr/src/debug/lustre-2.4.0/lustre/mdt/out_handler.c: 1031<br/> #26 <span class="error">&#91;ffff88068a5a7db8&#93;</span> ptlrpc_server_handle_request at ffffffffa070b428 <span class="error">&#91;ptlrpc&#93;</span><br/> #27 <span class="error">&#91;ffff88068a5a7eb8&#93;</span> ptlrpc_main at ffffffffa070c7be <span class="error">&#91;ptlrpc&#93;</span><br/> #28 <span class="error">&#91;ffff88068a5a7f48&#93;</span> kernel_thread at ffffffff8100c0ca<br/> /usr/src/debug////////kernel-lustre240-2.6.32-358.6.2.el6/linux-2.6.32-358.6.2.el6.20130607.x86_64/arch/x86/kernel/entry_64.S: 1213</p> <p>Thanks! Mahmoud. It seems in 2.4, MDS will enqueue the open lock anyway, and if the client does not require the open lock, MDT will release the lock later. It is different as b2_1, which only enqueue the open lock if the client requires it. (probably brought in by layout lock patch?). Anyway, this change will add some "local" open lock to the export hash list, which will trigger this problem. I will post a patch soon. </p> <p>Ok, I just updated the patch.</p> <p>b2_4 <a href="http://review.whamcloud.com/8642" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8642</a><br/> master <a href="http://review.whamcloud.com/8680" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8680</a></p> <p>Di, is <a href="https://jira.whamcloud.com/browse/LU-3273" title="mdt_mfd_open() may call mdt_handle2mfd() w/o holding med_open_lock" class="issue-link" data-issue-key="LU-3273"><del>LU-3273</del></a> related to this problem? </p> <p>Jay, probably no, the patch here should be enough for this problem. Btw: Did you create a new ticket for the crash you find on OSS?</p> <p>We installed the latest version of the patch. It still crashed 2 times after the update. We now know what is causing this. A user is using hdf5 to do parallel writes to 1 file from ~2000 process. I am working on getting a reproducer.</p> <p>The new crashes were exactly the same.</p> <p>Interesting. Hmm, if you can reproduce this on a test system and collecting some debug log when crash happened, it would be very helpful. Thanks.</p> <p>Which debug log would you like for me to collect. Since the system drops to kdb as soon as we hit the LBUG we need a way to collect those logs.</p> <p>You can disable panic_on_lbug on MDS (lctl set_param panic_on_lbug=0), then if LBUG happens, the system will dump the debug log somewhere automatically (you can see this in console message). And also if you can set debug level to -1 (lctl set_param debug=-1) and debug size to 20 (lctl set_param debug_size=30) on MDS, that would make sure the debug log include enough information we need during LBUG, but this parameter changes(debug and debug_size) will slow down your system, please be aware of this.</p> <p>Hi Mahmoud,</p> <p>will you please tell me what's the tip of your branch in your comment at:</p> <p>"Mahmoud Hanafi added a comment - 02/Jan/14 9:05 AM"</p> <p>Jinshan</p> <p>Just a quick update, will you try the patch below to see if I have some good luck here:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[jinxiong@intel nasa]$ git diff diff --git a/lustre/mdt/mdt_open.c b/lustre/mdt/mdt_open.c index 545507f..f2a23ee 100644 --- a/lustre/mdt/mdt_open.c +++ b/lustre/mdt/mdt_open.c @@ -1437,7 +1437,7 @@ <span class="code-object">int</span> mdt_reint_open(struct mdt_thread_info *info, struct mdt_lock_handle *lhc) struct lu_fid *child_fid = &amp;info-&gt;mti_tmp_fid1; struct md_attr *ma = &amp;info-&gt;mti_attr; __u64 create_flags = info-&gt;mti_spec.sp_cr_flags; - __u64 ibits; + __u64 ibits = 0; struct mdt_reint_record *rr = &amp;info-&gt;mti_rr; struct lu_name *lname; <span class="code-object">int</span> result, rc; </pre> </div></div> <p>obviously ibits is not initialized in this case and it caused the lock in @lhc is dropped which we don't hold any references.</p> <p>Ah, good catch. Jinshan. Really missed this.</p> <p>I have attached mdt_thread_info structure from the dump. It may help...</p> <p>Mahmoud: please try the update patch <a href="http://review.whamcloud.com/#/c/8642/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8642/</a></p> <p>patch <a href="http://review.whamcloud.com/6511" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6511</a> already fixed this problem. Worth trying this alone if you guys have a chance.</p> <p>drop the priority as there is no response from customer meanwhile I believe we've found the root cause of this issue.</p> <p>We had #8642 patch set 5 installed on Jan 8th. Yesterday morning the mds crashed (patch set 4) and booted up with patch set 5. </p> <p>Today early morning the mds crashed again; however, it was caused by another bug in OSS and the OSS crash brought down the mds. So, we have patch set 5 running for &gt; 1 day without hitting this problem. We will let it soak more time.</p> <p>thanks for the update Jay and good luck with patch set 5.</p> <p>Patch set 5 didn't fix the issue. We just hit this bug again.</p> <p>LustreError: 45299:0:(ldlm_lock.c:851:ldlm_lock_decref_internal_nolock()) ASSERTION( lock-&gt;l_readers &gt; 0 ) failed: <br/> LustreError: 45299:0:(ldlm_lock.c:851:ldlm_lock_decref_internal_nolock()) LBUG<br/> Pid: 45299, comm: mdt02_087</p> <p>PID: 20719 TASK: ffff880368864aa0 CPU: 24 COMMAND: "mdt01_059"<br/> #0 <span class="error">&#91;ffff88036c5394c8&#93;</span> machine_kexec at ffffffff81035e8b<br/> #1 <span class="error">&#91;ffff88036c539528&#93;</span> crash_kexec at ffffffff810c0492<br/> #2 <span class="error">&#91;ffff88036c5395f8&#93;</span> kdb_kdump_check at ffffffff812858d7<br/> #3 <span class="error">&#91;ffff88036c539608&#93;</span> kdb_main_loop at ffffffff81288ac7<br/> #4 <span class="error">&#91;ffff88036c539718&#93;</span> kdb_save_running at ffffffff81282c2e<br/> #5 <span class="error">&#91;ffff88036c539728&#93;</span> kdba_main_loop at ffffffff81463988<br/> #6 <span class="error">&#91;ffff88036c539768&#93;</span> kdb at ffffffff81285dc6<br/> #7 <span class="error">&#91;ffff88036c5397d8&#93;</span> panic at ffffffff8153efbf<br/> #8 <span class="error">&#91;ffff88036c539858&#93;</span> lbug_with_loc at ffffffffa045deeb <span class="error">&#91;libcfs&#93;</span><br/> #9 <span class="error">&#91;ffff88036c539878&#93;</span> ldlm_lock_decref_internal_nolock at ffffffffa0706402 <span class="error">&#91;ptlrpc&#93;</span><br/> #10 <span class="error">&#91;ffff88036c539898&#93;</span> ldlm_lock_decref_internal at ffffffffa070aafd <span class="error">&#91;ptlrpc&#93;</span><br/> #11 <span class="error">&#91;ffff88036c5398f8&#93;</span> ldlm_lock_decref at ffffffffa070bfb9 <span class="error">&#91;ptlrpc&#93;</span><br/> #12 <span class="error">&#91;ffff88036c539928&#93;</span> mdt_save_lock at ffffffffa0e3c483 <span class="error">&#91;mdt&#93;</span><br/> #13 <span class="error">&#91;ffff88036c539978&#93;</span> mdt_object_unlock at ffffffffa0e3c77c <span class="error">&#91;mdt&#93;</span><br/> #14 <span class="error">&#91;ffff88036c5399a8&#93;</span> mdt_object_open_unlock at ffffffffa0e6acfc <span class="error">&#91;mdt&#93;</span><br/> #15 <span class="error">&#91;ffff88036c5399f8&#93;</span> mdt_reint_open at ffffffffa0e71d14 <span class="error">&#91;mdt&#93;</span><br/> #16 <span class="error">&#91;ffff88036c539ae8&#93;</span> mdt_reconstruct_open at ffffffffa0e736ac <span class="error">&#91;mdt&#93;</span><br/> #17 <span class="error">&#91;ffff88036c539b78&#93;</span> mdt_reconstruct at ffffffffa0e66445 <span class="error">&#91;mdt&#93;</span><br/> #18 <span class="error">&#91;ffff88036c539b98&#93;</span> mdt_reint_internal at ffffffffa0e41cfb <span class="error">&#91;mdt&#93;</span><br/> #19 <span class="error">&#91;ffff88036c539bd8&#93;</span> mdt_intent_reint at ffffffffa0e42090 <span class="error">&#91;mdt&#93;</span><br/> #20 <span class="error">&#91;ffff88036c539c28&#93;</span> mdt_intent_policy at ffffffffa0e3ff3e <span class="error">&#91;mdt&#93;</span><br/> #21 <span class="error">&#91;ffff88036c539c68&#93;</span> ldlm_lock_enqueue at ffffffffa0708831 <span class="error">&#91;ptlrpc&#93;</span><br/> #22 <span class="error">&#91;ffff88036c539cc8&#93;</span> ldlm_handle_enqueue0 at ffffffffa072f1ef <span class="error">&#91;ptlrpc&#93;</span><br/> #23 <span class="error">&#91;ffff88036c539d38&#93;</span> mdt_enqueue at ffffffffa0e403c6 <span class="error">&#91;mdt&#93;</span><br/> #24 <span class="error">&#91;ffff88036c539d58&#93;</span> mdt_handle_common at ffffffffa0e46ad7 <span class="error">&#91;mdt&#93;</span><br/> #25 <span class="error">&#91;ffff88036c539da8&#93;</span> mds_regular_handle at ffffffffa0e80615 <span class="error">&#91;mdt&#93;</span><br/> #26 <span class="error">&#91;ffff88036c539db8&#93;</span> ptlrpc_server_handle_request at ffffffffa07613c8 <span class="error">&#91;ptlrpc&#93;</span><br/> #27 <span class="error">&#91;ffff88036c539eb8&#93;</span> ptlrpc_main at ffffffffa076275e <span class="error">&#91;ptlrpc&#93;</span><br/> #28 <span class="error">&#91;ffff88036c539f48&#93;</span> kernel_thread at ffffffff8100c0ca</p> <p>Can you share me the following info:<br/> 1. Comparing to the previous patches, did it last longer after applying patch set 5?<br/> 2. what's the tip of the source tree you're running?</p> <p>Jinshan</p> <p>My bad. The patch set #5 was in my nas-2.4.0-1 branch, but not<br/> in nas-2.4.1 branch. It was an earlier patch in nas-2.4.1 branch.</p> <p>We just upgraded our server to 2.4.1 yesterday.</p> <p>Patch landed for 2.6</p> <p>just fyi, we were hit on 9th feb...i've attached the mds.log. <br/> this is on lustre 2.4.2, haven't patched yet.</p> <p>Patch for b2_5 at <a href="http://review.whamcloud.com/#/c/9779/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9779/</a></p> Development Rank 1|hzwbpj: Rank (Obsolete) 12085 Severity