https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5314 Lustre <p>Our MDS hit LBUG and crashed this evening. Here are the /var/log/messages:</p> <p>Jul 9 18:40:22 meerkat-mds-10-1 kernel: Lustre: meerkat-MDT0000: Client e27741dc-f76c-ea5a-c426-4c6b5e86a758 (at 198.202.118.120@tcp) reconnecting<br/> Jul 9 18:40:24 meerkat-mds-10-1 kernel: Lustre: meerkat-MDT0000: Client 42a919ae-9df5-e771-0e9a-7ee82fdc33d9 (at 198.202.119.106@tcp) reconnecting<br/> Jul 9 18:40:24 meerkat-mds-10-1 kernel: Lustre: Skipped 1 previous similar message<br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: Lustre: 3457:0:(mdt_handler.c:1338:mdt_getattr_name_lock()) Although resent, but still not get child lockparent:<span class="error">&#91;0x200003fb3:0xaa9f:0x0&#93;</span> child:<span class="error">&#91;0x200003fb3:0xb42e:0x0&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: LustreError: 3457:0:(mdt_handler.c:3568:mdt_intent_lock_replace()) ASSERTION( lustre_msg_get_flags(req-&gt;rq_reqmsg) &amp; 0x0002 ) failed: <br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: LustreError: 3457:0:(mdt_handler.c:3568:mdt_intent_lock_replace()) LBUG<br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: Pid: 3457, comm: mdt03_003<br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: Call Trace:<br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa02e7895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa02e7e97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa0c9c3b1&gt;&#93;</span> mdt_intent_lock_replace+0x391/0x400 <span class="error">&#91;mdt&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa0cb34c6&gt;&#93;</span> mdt_intent_getattr+0x3b6/0x490 <span class="error">&#91;mdt&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa0c9ff1e&gt;&#93;</span> mdt_intent_policy+0x39e/0x720 <span class="error">&#91;mdt&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa0575831&gt;&#93;</span> ldlm_lock_enqueue+0x361/0x8d0 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa059c1cf&gt;&#93;</span> ldlm_handle_enqueue0+0x4ef/0x10b0 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa0ca03a6&gt;&#93;</span> mdt_enqueue+0x46/0xe0 <span class="error">&#91;mdt&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa0ca6a97&gt;&#93;</span> mdt_handle_common+0x647/0x16d0 <span class="error">&#91;mdt&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa05beb8c&gt;&#93;</span> ? lustre_msg_get_transno+0x8c/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa0ce06a5&gt;&#93;</span> mds_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa05ce3a8&gt;&#93;</span> ptlrpc_server_handle_request+0x398/0xc60 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa02e85de&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa02f9d9f&gt;&#93;</span> ? lc_watchdog_touch+0x6f/0x170 <span class="error">&#91;libcfs&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa05c5709&gt;&#93;</span> ? ptlrpc_wait_event+0xa9/0x290 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffff81063990&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa05cf73e&gt;&#93;</span> ptlrpc_main+0xace/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa05cec70&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20<br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa05cec70&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffffa05cec70&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Jul 9 18:40:29 meerkat-mds-10-1 kernel: </p> Linux meerkat-mds-10-1.local 2.6.32-358.23.2.el6_lustre.x86_64 #1 SMP Thu Dec 19 19:57:45 PST 2013 x86_64 x86_64 x86_64 GNU/Linux<br/> LU-5314

Lustre 2.4.2 MDS hit LBUG and crash

Bug Major Resolved Cannot Reproduce Niu Yawei Haisong Cai sdsc Thu, 10 Jul 2014 02:51:49 +0000 Fri, 23 Sep 2022 01:14:13 +0000 Wed, 19 Jul 2017 13:20:54 +0000 Lustre 2.4.3 0 6 <p>Could you please take a look at this Niu?<br/> Thanks,<br/> ~ jfc.</p> <p>mdt_getattr_name_lock() cleared the MSG_RESENT flag:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> CWARN(<span class="code-quote">"Although resent, but still not get child lock"</span> <span class="code-quote">"parent:"</span>DFID<span class="code-quote">" child:"</span>DFID<span class="code-quote">"\n"</span>, PFID(mdt_object_fid(parent)), PFID(mdt_object_fid(child))); lustre_msg_clear_flags(req-&gt;rq_reqmsg, MSG_RESENT); LDLM_LOCK_PUT(lock); GOTO(relock, 0); </pre> </div></div> <p>That'll trigger the LASSERT on MSG_RESENT in mdt_intent_lock_replace():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (new_lock-&gt;l_export == req-&gt;rq_export) { /* * Already gave <span class="code-keyword">this</span> to the client, which means that we * reconstructed a reply. */ LASSERT(lustre_msg_get_flags(req-&gt;rq_reqmsg) &amp; MSG_RESENT); lh-&gt;mlh_reg_lh.cookie = 0; RETURN(ELDLM_LOCK_REPLACED); } </pre> </div></div> <p>This part of code has been heavily changed in <a href="https://jira.whamcloud.com/browse/LU-2827" title="mdt_intent_fixup_resent() cannot find the proper lock in hash" class="issue-link" data-issue-key="LU-2827"><del>LU-2827</del></a>, the RESENT flag won't be cleared now, so the LASSERT won't be triggered anymore. The backport of <a href="https://jira.whamcloud.com/browse/LU-2827" title="mdt_intent_fixup_resent() cannot find the proper lock in hash" class="issue-link" data-issue-key="LU-2827"><del>LU-2827</del></a> is on <a href="http://review.whamcloud.com/#/c/10902/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10902/</a> .</p> <p>Hi Niu,</p> <p>The same MDS has thrown FID errors like below and caused clients to hang. Do you think they are related?</p> <p>Jul 4 20:51:33 meerkat-mds-10-1 kernel: LustreError: 19626:0:(osd_oi.c:655:osd_oi_insert()) meerkat-MDT0000: the FID <span class="error">&#91;0x2000030d7:0x220:0x0&#93;</span> is used by two objects: 647048266/1095373013 647048267/1095373014<br/> Jul 4 20:51:47 meerkat-mds-10-1 kernel: LustreError: 3655:0:(osd_oi.c:655:osd_oi_insert()) meerkat-MDT0000: the FID <span class="error">&#91;0x2000025ff:0x2c9a:0x0&#93;</span> is used by two objects: 647048307/1095373111 647048308/1095373112<br/> Jul 4 20:56:54 meerkat-mds-10-1 kernel: LustreError: 19257:0:(osd_oi.c:655:osd_oi_insert()) meerkat-MDT0000: the FID <span class="error">&#91;0x200003094:0x20b5:0x0&#93;</span> is used by two objects: 647048393/1095374047 647048397/1095374048<br/> Jul 5 07:28:22 meerkat-mds-10-1 kernel: LustreError: 3657:0:(osd_oi.c:655:osd_oi_insert()) meerkat-MDT0000: the FID <span class="error">&#91;0x200001f67:0x3e3d:0x0&#93;</span> is used by two objects: 647560781/1095604748 647560782/1095604749<br/> Jul 5 17:25:56 meerkat-mds-10-1 kernel: LustreError: 3637:0:(osd_oi.c:655:osd_oi_insert()) meerkat-MDT0000: the FID <span class="error">&#91;0x200002547:0x457:0x0&#93;</span> is used by two objects: 647561074/3949023701 647561076/3949023703<br/> Jul 5 17:25:56 meerkat-mds-10-1 kernel: LustreError: 3534:0:(osd_oi.c:655:osd_oi_insert()) meerkat-MDT0000: the FID <span class="error">&#91;0x2000020c8:0xe6f:0x0&#93;</span> is used by two objects: 647561077/3949023704 647561079/3949023706<br/> Jul 5 17:25:57 meerkat-mds-10-1 kernel: LustreError: 3670:0:(osd_oi.c:655:osd_oi_insert()) meerkat-MDT0000: the FID <span class="error">&#91;0x200001ac1:0x4037:0x0&#93;</span> is used by two objects: 647561031/3949023726 647561045/3949023727<br/> Jul 5 17:35:42 meerkat-mds-10-1 kernel: LustreError: 3685:0:(osd_oi.c:655:osd_oi_insert()) meerkat-MDT0000: the FID <span class="error">&#91;0x2000020a1:0x26f6:0x0&#93;</span> is used by two objects: 647561130/3949042621 647561131/3949042622<br/> Jul 6 00:00:05 meerkat-mds-10-1 kernel: LustreError: 3645:0:(osd_oi.c:655:osd_oi_insert()) meerkat-MDT0000: the FID <span class="error">&#91;0x200001a0e:0x35a8:0x0&#93;</span> is used by two objects: 647561202/3949603984 647561206/3949603985</p> <p>Hello Niu,<br/> You may have missed that but in fact there are regression issues with my b2_4 back-port (<a href="http://review.whamcloud.com/#/c/10902/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10902/</a>) of <a href="https://jira.whamcloud.com/browse/LU-2827" title="mdt_intent_fixup_resent() cannot find the proper lock in hash" class="issue-link" data-issue-key="LU-2827"><del>LU-2827</del></a> changes. And I checked that the b2_5 version (<a href="http://review.whamcloud.com/#/c/10492/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10492/</a>) is ok and that it will land soon now.</p> <p>This issue is fixed on newer releases and SDSC upgraded some time back</p> Related LU-2827 LU-11457 Development End date Thu, 9 Oct 2014 02:51:49 +0000 Rank 1|hzwr1z: Rank (Obsolete) 14850 Severity Start date Thu, 10 Jul 2014 02:51:49 +0000