https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7173 Lustre <p>While running a simulated user work load our MDS crashed due to the following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;3&gt;[ 5913.638287] LustreError: 15340:0:(ldlm_lock.c:371:ldlm_lock_destroy_internal()) ### lock still on resource ns: mdt-atlas1-MDT0000_UUID lock: ffff883fd37b3700/0xb9db2cef5d1e41 39 lrc: 3/0,0 mode: CR/CR res: [0x200252cc7:0x3:0x0].0 bits 0x8 rrc: 2 type: IBT flags: 0x50000000000000 nid: 9310@gni100 remote: 0xd3f59509a44985ef expref: 60 pid: 15340 timeout: 0 lvb_type: 3 &lt;0&gt;[ 5913.674996] LustreError: 15340:0:(ldlm_lock.c:372:ldlm_lock_destroy_internal()) LBUG &lt;4&gt;[ 5913.683915] Pid: 15340, comm: mdt03_052 &lt;4&gt;[ 5913.688327] &lt;4&gt;[ 5913.688327] Call Trace: &lt;4&gt;[ 5913.692965] [&lt;ffffffffa0430895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] &lt;4&gt;[ 5913.700886] [&lt;ffffffffa0430e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] &lt;4&gt;[ 5913.707968] [&lt;ffffffffa0707871&gt;] ldlm_lock_destroy_internal+0x251/0x2c0 [ptlrpc] &lt;4&gt;[ 5913.716573] [&lt;ffffffffa07092b5&gt;] ldlm_lock_destroy+0x35/0x130 [ptlrpc] &lt;4&gt;[ 5913.724118] [&lt;ffffffffa070a311&gt;] ldlm_lock_enqueue+0x161/0x980 [ptlrpc] &lt;4&gt;[ 5913.731760] [&lt;ffffffffa0733e9b&gt;] ldlm_handle_enqueue0+0x51b/0x10c0 [ptlrpc] &lt;4&gt;[ 5913.739797] [&lt;ffffffffa0d6f1a6&gt;] mdt_enqueue+0x46/0xe0 [mdt] &lt;4&gt;[ 5913.746362] [&lt;ffffffffa0d7401a&gt;] mdt_handle_common+0x52a/0x1470 [mdt] &lt;4&gt;[ 5913.753804] [&lt;ffffffffa0db0615&gt;] mds_regular_handle+0x15/0x20 [mdt] &lt;4&gt;[ 5913.761064] [&lt;ffffffffa0762f55&gt;] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] &lt;4&gt;[ 5913.769866] [&lt;ffffffffa0442785&gt;] ? lc_watchdog_touch+0x65/0x170 [libcfs] &lt;4&gt;[ 5913.777632] [&lt;ffffffffa075b929&gt;] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] &lt;4&gt;[ 5913.785377] [&lt;ffffffffa07656dd&gt;] ptlrpc_main+0xaed/0x1930 [ptlrpc] &lt;4&gt;[ 5913.792557] [&lt;ffffffffa0764bf0&gt;] ? ptlrpc_main+0x0/0x1930 [ptlrpc] &lt;4&gt;[ 5913.799689] [&lt;ffffffff8109e78e&gt;] kthread+0x9e/0xc0 &lt;4&gt;[ 5913.805270] [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 &lt;4&gt;[ 5913.810950] [&lt;ffffffff8109e6f0&gt;] ? kthread+0x0/0xc0 &lt;4&gt;[ 5913.816622] [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> Clients running Lustre-2.7.57 plus patches using an lustre 2.5.4 server back end. LU-7173

ldlm_lock_destroy_internal() LBUG encountered during 2.8 large scale testing

Bug Major Resolved Fixed Zhenyu Xu James A Simmons patch Wed, 16 Sep 2015 18:17:42 +0000 Fri, 2 Dec 2022 18:16:16 +0000 Mon, 1 Feb 2016 12:39:57 +0000 Lustre 2.8.0 Lustre 2.8.0 0 15 <p>Bobijam</p> <p>Could you please look into this issue?</p> <p>Thanks</p> <p>Peter</p> <p>Have you run this similar workload with 2.5.4 clients without problem? This is a server-side bug and it <em>shouldn't</em> be possible for the client to induce this kind of failure regardless of what it is doing. The only thing I can think of is that the 2.7.57 client is causing the MDS to follow some error handling path that isn't exercised by 2.5 clients.</p> <p>Never seen this problem with the cray 2.5 clients or the lustre 2.7 clients we run with. This only happened with the pre-2.8 clients.</p> <p>Ugh. Your ftp server died on me so I uploaded the vmcore files and dmesgs at <a href="http://www.infradead.org/~jsimmons/2015-09-15.tgz" class="external-link" target="_blank" rel="nofollow noopener">http://www.infradead.org/~jsimmons/2015-09-15.tgz</a></p> <p>git commit 5517eab06eb99e4ecb66be251a10e70c37547610 added layout lock handling, esp. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">diff --git a/lustre/ldlm/ldlm_inodebits.c b/lustre/ldlm/ldlm_inodebits.c index e10a654..b68dd59 100644 --- a/lustre/ldlm/ldlm_inodebits.c +++ b/lustre/ldlm/ldlm_inodebits.c @@ -190,8 +190,12 @@ <span class="code-object">int</span> ldlm_process_inodebits_lock(struct ldlm_lock *lock, __u 64 *flags, LASSERT(cfs_list_empty(&amp;res-&gt;lr_converting)); check_res_locked(res); - <span class="code-keyword">if</span> (!first_enq) { - LASSERT(work_list != NULL); + <span class="code-comment">/* (*flags &amp; LDLM_FL_BLOCK_NOWAIT) is <span class="code-keyword">for</span> layout lock right now. */</span> + <span class="code-keyword">if</span> (!first_enq || (*flags &amp; LDLM_FL_BLOCK_NOWAIT)) { + *err = ELDLM_LOCK_ABORTED; + <span class="code-keyword">if</span> (*flags &amp; LDLM_FL_BLOCK_NOWAIT) + *err = ELDLM_LOCK_WOULDBLOCK; + rc = ldlm_inodebits_compat_queue(&amp;res-&gt;lr_granted, lock, NULL); <span class="code-keyword">if</span> (!rc) RETURN(LDLM_ITER_STOP); @@ -201,6 +205,8 @@ <span class="code-object">int</span> ldlm_process_inodebits_lock(struct ldlm_lock *lock, __u6 4 *flags, ldlm_resource_unlink_lock(lock); ldlm_grant_lock(lock, work_list); + + *err = ELDLM_OK; RETURN(LDLM_ITER_CONTINUE); } </pre> </div></div> <p>and in ldlm_lock_enqueue() </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* policies are not executed on the client or during replay */</span> <span class="code-keyword">if</span> ((*flags &amp; (LDLM_FL_HAS_INTENT|LDLM_FL_REPLAY)) == LDLM_FL_HAS_INTENT &amp;&amp; !local &amp;&amp; ns-&gt;ns_policy) { rc = ns-&gt;ns_policy(ns, lockp, cookie, lock-&gt;l_req_mode, *flags, NULL); <span class="code-keyword">if</span> (rc == ELDLM_LOCK_REPLACED) { .... } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (rc != ELDLM_OK || (rc == ELDLM_OK &amp;&amp; (*flags &amp; LDLM_FL_INTENT_ONLY))) { ldlm_lock_destroy(lock); RETURN(rc); } } </pre> </div></div> <p>I suspect that some intent_only lock has been granted and ldlm_lock_destroy() takes non-granted lock as its precondition.</p> <p>two of the vmcore-dmesg.txt shows that problematic locks are granted (mode: CR/CR) LAYOUT locks (lvb_type: 3 -&gt;LVB_T_LAYOUT)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;3&gt;[ 5913.638287] LustreError: 15340:0:(ldlm_lock.c:371:ldlm_lock_destroy_intern al()) ### lock still on resource ns: mdt-atlas1-MDT0000_UUID lock: ffff883fd37b3 700/0xb9db2cef5d1e4139 lrc: 3/0,0 mode: CR/CR res: [0x200252cc7:0x3:0x0].0 bits 0x8 rrc: 2 type: IBT flags: 0x50000000000000 nid: 9310@gni100 remote: 0xd3f59509 a44985ef expref: 60 pid: 15340 timeout: 0 lvb_type: 3 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;3&gt;[ 969.928837] LustreError: 16016:0:(ldlm_lock.c:371:ldlm_lock_destroy_intern al()) ### lock still on resource ns: mdt-atlas1-MDT0000_UUID lock: ffff881fbaae9 9c0/0xcc3ee5780a4e2eea lrc: 3/0,0 mode: CR/CR res: [0x20024d1ef:0x2e:0x0].0 bits 0x8 rrc: 1 type: IBT flags: 0x50000000000000 nid: 91@gni100 remote: 0xb81bd76a7 d6191aa expref: 57 pid: 16016 timeout: 0 lvb_type: 3 </pre> </div></div> <p>Hi James,</p> <p>Is it easy to reproduce?</p> <p>Not so easy to reproduce. We saw it only during our test shot on Titan running the 2.8 clients with a 2.5.4 server back end. It only showed up when we ran our test harness. After the first couple of runs the problem stopped showing up <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/> I have never seen it at smaller scales.</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/16497" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16497</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7173" title="ldlm_lock_destroy_internal() LBUG encountered during 2.8 large scale testing" class="issue-link" data-issue-key="LU-7173"><del>LU-7173</del></a> ldlm: do not grant intent-only lock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9702522581eaf7ad309e8c91931d571b9cad0b7b</p> <p>The patch looks good but it will not be until Oct 13th that we get it test it out.</p> <p>We have been a bunch of testing of this patch at various scales before our Titan test shot and we are seeing very large meta data performance improvement with this patch.</p> <p>Results with patch</p> <p> Directory creation : 37714.322 11084.963 26771.564 9055.694<br/> Directory stat : 58904.478 26689.126 48141.812 9343.208<br/> Directory removal : 34290.774 12088.699 22193.915 6303.121<br/> File creation : 4937.247 43.594 4020.814 1064.732<br/> File stat : 796.591 124.390 476.913 149.635<br/> File read : 30091.505 19214.287 26420.152 3238.929<br/> File removal : 4577.256 1845.895 4072.557 591.356</p> <p>Results before patch:</p> <p> Directory creation: 5657.386 4416.307 5086.521 376.908<br/> Directory stat : 74770.631 72558.713 73347.269 544.856<br/> Directory removal : 7990.972 6751.102 7465.554 382.254<br/> File creation : 6867.935 6227.613 6558.801 201.736<br/> File read : 28801.818 27562.517 28219.511 412.373<br/> File removal : 5889.433 4705.258 5566.932 314.018</p> <p>Well, there is an improvement for directory creation and removal performance with the patch, but the directory stat and file creation and removal performance is down and the performance results are much more variable between runs.</p> <p>who said it is about intent_only? is there a recovery? is it DNE?<br/> I have a similar dump and this is intent_getattr resend, due to some reason getattr created a new lock and the previous update lock is to be destroyed due to LOCK_REPLACED.</p> <p>I had a discussion about this patch with Oleg. The patch only is suppose to touch the recovery path but we saw this LBUG during an application run. We will see if this patch resolves our problem on our Oct 27 test shot.</p> <p>I had a lengthly discussion with Vitaly about what appears to be the same bug they are hitting (much closer to master, though).</p> <p>Sicne there's no recovery at play here, we believe the intent_only flag might be a red herring and instead the problem is the other one:</p> <p>ldlm_lock_enqueue:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> rc = ns-&gt;ns_policy(ns, lockp, cookie, lock-&gt;l_req_mode, *flags, NULL); <span class="code-keyword">if</span> (rc == ELDLM_LOCK_REPLACED) { ... } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (rc != ELDLM_OK || (rc == ELDLM_OK &amp;&amp; (*flags &amp; LDLM_FL_INTENT_ONLY))) { ldlm_lock_destroy(lock); RETURN(rc); } </pre> </div></div> <p>This is correctly identified by Bobijam, but we believe the case here is that of rc != ELDLM_OK (which is supported by crashdump data from this other affected site). (for the record, rc is -2, or -ENOENT).<br/> The other bit of info we gathered from there is the request is a RESEND (but not replay).</p> <p>So imagine we have this situation - an intent request comes in (getattr in this case), it is handled, and then a reply is sent (along with a lock), but the reply is then lost.<br/> Client resend the request, mdt fixup code finds the old lock and substitutes it in.<br/> Now the policy function fails and returns rc that's not OK. This will prompt us to call the lock_destroy on the already granted lock and trigger this assertion.</p> <p>The mystery was - what might cause a successful request to fail on retry, esp. with ENOENT with a lock in place, no less.<br/> We have noticed that the request at the crash point did not contain reply buffers, so that means it failed super early on in processing, in body_unpack to be precise. This function does a lookup and could return -2 if the object is not there.<br/> The request is also a getattr_by_fid.</p> <p>So it appears the sequence of events is roughly:<br/> a file is opened.<br/> the file is unlinked<br/> openhandle holder does fstat and causes getattr_by_fid request<br/> reply is lost<br/> the file is closed. // How this could happen while the getattr did not finish is not very clear to be sure. Perhaps that was not fstat, but some other way to get at the file and raced with unlink?<br/> getattr_by_fid is resent - the object is no longer there, mdt_unpack_req_pack_rep-&gt;mdt_body_unpack-&gt;mdt_object_init-&gt;...-&gt;osd_object_init() returns -ENOENT.</p> <p>I would like to verify this theory on the ORNL crashdump, but unfortunately it does not contain the kernel vmlinux and lustre modules with symbols, so I cannot.<br/> Can we have that additional data please?</p> <p>The idea for a fix is to additionally check if the failed request is a resend (with a granted lock) and in that case we need to "soft cancel" it - i.e. to send a blocking ast.</p> <p>This is needed because it's possible the other reply made it to the client while we are working here, so it knows about the lock and if we cancel it outright - there's a bit of state inconsistency.<br/> If the client does not know about the lock, it'll respond with a n error and the lock would be canceled imediatelly.<br/> In any case there's no need to wait for the lock to be canceled here and we then can proceed with returning the error to the client.</p> <p>So, I got the symbos information from James and the picture got a bit clouded.</p> <p>What I see similar: it's a resend, there is rc -2 returned from policy function, there's no repbuffers created hinting at a very early on failure.<br/> Differences: The request is ldlm layout intent.</p> <p>Now the really strange part:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; p *(( struct mdt_thread_info *)0xffff8840296cb000)-&gt;mti_pill $9 = { rc_req = 0xffff884045bd3000, rc_fmt = 0xffffffffa080c9a0 &lt;RQF_MDS_REINT_SETXATTR&gt;, rc_loc = RCL_SERVER, rc_area = {{4294967295, 4294967295, 4294967295, 4294967295, 4294967295, 4294967295, 4294967295, 4294967295, 4294967295}, {4294967295, 4294967295, 4294967295, 4294967295, 4294967295, 4294967295, 4294967295, 4294967295, 4294967295}} } </pre> </div></div> <p>So somehow this request got wrong request format assigned, which is really weird.<br/> Both crashes have this same wrong request format too, so it's not a purely coincidence.</p> <p>So I think it would be an interesting experiment to perform a forced resend for an IT_LAYOUT intent and see if it happens to fail every time. First with 2.5.4 on both ends (easier to organize) and then 2.5.4 on server and master on client.</p> <p>I envision a manual test to be something like:<br/> dd if=/dev/urandom of=/mnt/lustre/file1 bs=1024<br/> press ^Z to pause that process<br/> cp /etc/passwd /mnt/lustre/file2<br/> lfs swap_layouts /mnt/lustre/file1 /mnt/lustre/file2<br/> This will invalidate the layout lock, so next write activity from paused dd would trigger intent lock.</p> <p>Then set OBD_FAIL_LDLM_REPLY (0x30c, or really 0x8000030c to make it one time) as fail_loc on MDS.<br/> Unpause the dd (with fg) and make sure the fail_loc did trigger on MDS (this fail_loc does not work properly in master so need to make sure it works in 2.5.4).</p> <p>If it crahes - then we know it's every time such a resend happens and it should be easy to figure it out afterwards, if not we'll need some more investigations.</p> <p>In any case I do not have time for this right now as I am departing for a conference tomorrow.</p> <p>James, if you have time to perform an experiment like this, that would be great as a first step, as I think Bobijam is busy with another conference in the next few days.</p> <p>I just tried your test using a 2.8 client &lt;-&gt; 2.8 server. No crash happened. I will work the admin to setup a 2.8 client to test against our 2.5 server setup. I did see time outs on the clients and MDS side.</p> <p>due to <a href="https://jira.whamcloud.com/browse/LU-5604" title="Lots of FAIL_ID checking are lost" class="issue-link" data-issue-key="LU-5604"><del>LU-5604</del></a>, OBD_FAIL_LDLM_REPLY does not work, you have to use OBD_FAIL_MDS_LDLM_REPLY_NET</p> <p>Vitaly Fertman (vitaly.fertman@seagate.com) uploaded a new patch: <a href="http://review.whamcloud.com/17501" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17501</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7173" title="ldlm_lock_destroy_internal() LBUG encountered during 2.8 large scale testing" class="issue-link" data-issue-key="LU-7173"><del>LU-7173</del></a> mdt: intent vs unlink race<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c38c254ccad0e1a4b3a5eab99a6d3e09fd6f021f</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17501/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17501/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7173" title="ldlm_lock_destroy_internal() LBUG encountered during 2.8 large scale testing" class="issue-link" data-issue-key="LU-7173"><del>LU-7173</del></a> mdt: intent vs unlink race<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9b520c36ddff687c51bea026d78380eb85981971</p> <p>Another instance found for interop : 2.7.1 Server/EL6.7 Client<br/> Server: 2.7.1, b2_7_fe/34<br/> Client: master, build# 3303, RHEL 6.7<br/> <a href="https://testing.hpdd.intel.com/test_sets/1dbf619c-bb05-11e5-9137-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1dbf619c-bb05-11e5-9137-5254006e85c2</a></p> <p>Another instance found for interop : 2.5.5 Server/EL6.7 Client<br/> Server: 2.5.5, b2_5_fe/62<br/> Client: master, build# 3303, RHEL 6.7<br/> <a href="https://testing.hpdd.intel.com/test_sets/0ac277da-bb25-11e5-861c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0ac277da-bb25-11e5-861c-5254006e85c2</a></p> <p>Another instance found for interop : 2.5.5 Server/EL7 Client<br/> Server: 2.5.5, b2_5_fe/62<br/> Client: master, build# 3303, RHEL 7<br/> <a href="https://testing.hpdd.intel.com/test_sets/76574e5a-bb0a-11e5-87b4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/76574e5a-bb0a-11e5-87b4-5254006e85c2</a></p> <p>It looks like this fix already landed to master for 2.8</p> Duplicate Related LU-6334 LU-7535 Development Rank 1|hzxntz: Rank (Obsolete) 9223372036854775807 Severity