https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3504 Lustre <p>Grove's MDS had some trouble yesterday when it got into a situation where all available CPUs were spinning in lock_res_and_lock, presumably on the same ldlm_lock. The system was crashed to gather a core dump. Upon Lustre starting back up, it went through recovery and then got back into the same state where all cores were spinning in lock_res_and_lock. Rebooting again, and bringing lustre up with the "abort_recov" option got the system back into a usable state.</p> <p>The core dump captured show all CPUs spinning with one of the following back traces:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 25091 TASK: ... CPU: 0 COMMAND: "mdt00_033" ... --- &lt;IRQ STACK&gt; --- #17 [...] ret_from_intr [exception RIP: _spin_lock+30] #18 [...] lock_res_and_lock+0x30 at ... [ptlrpc] #19 [...] ldlm_handle_enqueue0+0x907 at ... [ptlrpc] #20 [...] mdt_enqueue+0x46 at ... [mdt] #21 [...] mdt_handle_common+0x648 at ... [mdt] #22 [...] mds_regular_handle+0x15 at ... [mdt] #23 [...] ptlrpc_server_handle_request+0x398 at ... [ptlrpc] #24 [...] ptlrpc_main+0xace at ... [ptlrpc] #25 [...] child_rip+0xa PID: 25291 TASK: ... CPU:1 COMMAND: "mdt00_088" ... --- &lt;NMI exception stack&gt; --- #6 [...] _spin_lock+0x1e #7 [...] lock_res_and_lock+0x30 at ... [ptlrpc] #8 [...] ldlm_lock_enqueue+0x11d at ... [ptlrpc] #9 [...] ldlm_handle_enqueue+0x4ef at ... [ptlrpc] #10 [...] mdt_enqueue+0x46 at ... [mdt] #11 [...] mdt_handle_common+0x648 at ... [mdt] #12 [...] mds_regular_handle+0x15 at ... [mdt] #13 [...] ptlrpc_server_handle_request+0x398 at ... [ptlrpc] #14 [...] ptlrpc_main+0xace at ... [ptlrpc] #15 [...] child_rip+0xa </pre> </div></div> LU-3504

MDS: All cores spinning on ldlm lock in lock_res_and_lock

Bug Major Resolved Fixed Bruno Faccini Prakash Surya sequoia zfs Tue, 25 Jun 2013 18:27:19 +0000 Fri, 21 Mar 2014 19:52:54 +0000 Thu, 29 Aug 2013 13:49:40 +0000 Lustre 2.4.0 0 14 <p>Was there also any backtrace dump in the syslog/console before the forced crash-dump ??<br/> Is it possible to get the syslog (and console/conman log if available) for the time period covering the day of the crash ?? Also a "foreach bt" output from crash tool running over the crash-dump could be useful.</p> <p>Bruno, this system is on the SCF so I'm not able to copy anything directly to you. If you have any specific things you'd like me to look at, let me know; but any logs, backtraces, etc have to be manually copied over by hand so that's not an option for bulk requests (i.e. "foreach bt" from crash).</p> <p>I don't see any stack traces in the console logs other than the "CPU soft lockup" messages for the threads spinning on the lock that I mentioned in the description of this ticket.</p> <p>I'm looking through the back traces and I do see one thread that looks like it might be sleeping while holding the lock the other threads are spinning on. I need to walk the code to verify this thread is holding the lock, but here is it's back trace:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> PID: 24974 TASK: ... CPU: 19 COMMAND: "mdt02_010" #0 [...] schedule #1 [...] __cond_resched+0x2a at ... #2 [...] _cond_resched+0x30 at ... #3 [...] __kmalloc+0x130 at ... #4 [...] cfs_alloc+0x30 at ... [libcfs] #5 [...] cfs_hash_create+0x151 at ... [libcfs] #6 [...] ldlm_init_flock_export+0x54 at ... [ptlrpc] #7 [...] ldlm_process_flock_lock+0x1432 at ... [ptlrpc] #8 [...] ldlm_lock_enqueue+0x405 at ... [ptlrpc] #9 [...] ldlm_handle_enqueue0+0x4ef at ... [ptlrpc] #10 [...] mdt_enqueue+0x46 at ... [mdt] #11 [...] mdt_handle_common+0x648 at ... [mdt] #12 [...] mdt_regular_handle+0x15 at ... [mdt] #13 [...] ptlrpc_server_handle_request+0x398 at ... [ptlrpc] #14 [...] ptlrpc_main+0xace at ... [ptlrpc] #15 [...] child_rip+0xa at ... </pre> </div></div> <p>Sure, this one looks guilty !! BTW, can you check if it is in this state since a long time ? We can get an idea about that if you check its "ps -l 24794" output against the latest ones ("ps -l | head").</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; ps -l 24974 [1016382628073301] [RU] PID: 24974 TASK: ... CPU: 19 COMMAND: "mdt02_010" crash&gt; ps -l | head -n1 [1016382635871136] [RU] PID: 25198 TASK: ... CPU: 6 COMMAND: "mdt02_058" </pre> </div></div> <p>It looks like we don't have <tt>CONFIG_DEBUG_SPINLOCK_SLEEP</tt> enabled on this system, so the kernel *<b>would not</b>* have complained if the thread did in fact sleep while holding the lock.</p> <p>I think this change may have introduced this issue.. <a href="http://review.whamcloud.com/2240" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2240</a></p> <p>Anyway, thread "mdt02_010", has been scheduled since less than a second before crash so it may not be the responsible of the suspected dead-lock. Unless the code it runs only forces re-sched()s with lock held ... I need to investigate in this direction.</p> <p>Also, what make you think change #2240 could be the reason ?</p> <p>I have proposed a fix here: <a href="http://review.whamcloud.com/6790" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6790</a></p> <blockquote> <p>Also, what make you think change #2240 could be the reason ?</p></blockquote> <p>Because of this change:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>-static inline void ldlm_flock_blocking_link(struct ldlm_lock *req, - struct ldlm_lock *lock) +static inline int ldlm_flock_blocking_link(struct ldlm_lock *req, + struct ldlm_lock *lock) { + int rc = 0; + /* For server only */ if (req-&gt;l_export == NULL) - return; + return 0; + + if (unlikely(req-&gt;l_export-&gt;exp_flock_hash == NULL)) { + rc = ldlm_init_flock_export(req-&gt;l_export); + if (rc) + goto error; + } - LASSERT(cfs_list_empty(&amp;req-&gt;l_flock_waitq)); - cfs_write_lock(&amp;req-&gt;l_export-&gt;exp_flock_wait_lock); + LASSERT(cfs_hlist_unhashed(&amp;req-&gt;l_exp_flock_hash)); req-&gt;l_policy_data.l_flock.blocking_owner = lock-&gt;l_policy_data.l_flock.owner; req-&gt;l_policy_data.l_flock.blocking_export = - class_export_get(lock-&gt;l_export); - - cfs_list_add_tail(&amp;req-&gt;l_flock_waitq, - &amp;req-&gt;l_export-&gt;exp_flock_wait_list); - cfs_write_unlock(&amp;req-&gt;l_export-&gt;exp_flock_wait_lock); + lock-&gt;l_export; + req-&gt;l_policy_data.l_flock.blocking_refs = 0; + + cfs_hash_add(req-&gt;l_export-&gt;exp_flock_hash, + &amp;req-&gt;l_policy_data.l_flock.owner, + &amp;req-&gt;l_exp_flock_hash); +error: + return rc; } </pre> </div></div> <p>Prior to <a href="http://review.whamcloud.com/2240" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2240</a> , ldlm_flock_blocking_link() didn't make any allocations, now it might via ldlm_init_flock_export(). If the ldlm lock is held at this point as I <em>think</em> it is, that is bad news. I'm a little weary of the added call to cfs_hash_add also, but I haven't checked the code to determine if that might schedule.</p> <p>Prakash,<br/> I agree about this possible dead-lock scenario, but did you check that all spinning threads do it on the same resource spin-lock that's the one currently owned by PID 24974 ?<br/> Also I put a comment on you patch because I think the FLock hashing stuff needs to only occur for MDT exports that needs it.</p> <blockquote> <p>I agree about this possible dead-lock scenario, but did you check that all spinning threads do it on the same resource spin-lock that's the one currently owned by PID 24974 ?</p></blockquote> <p>No, I'm not certain of this, but circumstantial evidence supports the claim. I took a bit of time trying to find the ldlm lock out of the stack for one of the spinning threads, but was unsuccessful. I could look again if you think it would be time well spent, but even if I found the lock for each spinning thread I'm doubtful it would give us useful information without CONFIG_DEBUG_SPINLOCK enabled. If I did find the lock for each spinning thread, what information would you like be to gather from it?</p> <p>Is MDS node arch x86 ? If yes, RDI register of the NMI'ed spinning threads should be &amp;((struct ldlm_resource *)-&gt;lr_lock). And then quick&amp;dirty way to find if it is used/locked by PID 24974 it is a bit tricky but you will need to 1st find what pointer/address is stored in location <span class="error">&#91;stack-address where (ldlm_handle_enqueue0+0x4ef) return-address is stored - 0x8&#93;</span>, substract -0x40 to it and then get/dump this new stack address/memory-location to get the ldlm_lock address and see if its l_resource field matches ...</p> <p>If not x86 arch I need to work on it !!...</p> <p>It is x86_64, I'll see what I can come up with.</p> <p>Prakash, I worked on getting backtraces out of crash dumps a bit this year. If you are interested in trying a crash module to do that then you can find it at <a href="https://github.com/jhammond/xbt" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jhammond/xbt</a>. If it works for you then great, if not then we don't officially support it and I never commented on this issue.</p> <p>John, Thanks for the pointer. I'll have to give that a look when I get some time. Some of the other projects on your page look interesting as well..</p> <p>Here's what I got out of the crash dump so far..</p> <p>All of the spinning threads have this address in the RDI register <tt>0xffff880c74a0bad8</tt></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> crash&gt; p -x *(spinlock_t *)0xffff880c74a0bad8 $3 = { raw_lock = { slock = 0x5065504c } } # This is interesting. The x86_64 implementation uses ticket spin locks # such that the spin lock is split into two halves, "next" and "owner". # See: http://lwn.net/Articles/267968 # Let's subtract the two to determine how many threads are waiting on # the lock: crash&gt; p 0x5065-0x504c $20 = 25 # This aligns well, we have 24 threads spinning on all this lock (all 24 # cores of the system) and one thread holding the lock. crash&gt; struct -xo ldlm_resource struct ldlm_resource { ... [0x18] spinlock_t lr_lock; # offset is 0x18 bytes into ldlm_resource ... } crash&gt; p -x *((struct ldlm_resource *)(0xffff880c74a0bad8-0x18)) ... # looks like a valid structure, good :) crash&gt; p -x 0xffff880c74a0bad8-0x18 $30 = 0xfff880c74a0bac0 # The address of the ldlm_resource </pre> </div></div> <p>I still need to examine the thread making the kmalloc call and match it up (or not) with this spinning thread.</p> <p>After looking at the offending code I began wondering why these are spin locks in the first place. Typically, spin locks are used to protect tiny performance critical sections of the code. Using them more broadly as this code does can lead to issues exactly like this where a function is used improperly because the executing context is unclear. So my question is why aren't these locks a mutex? Is this just historical?</p> <p>Hello Brian and all,<br/> I agree with you, the fact that a spin-lock is used instead of mutex/semaphore is questionable but it is more a design subject that must be addressed/evaluated in a separate ticket/task.<br/> Concerning the fix for this ticket's problem, I made a negative comment on Prakash one (<a href="http://review.whamcloud.com/6790" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6790</a>) because I think it will move the FLock hashing stuff (introduced in <a href="http://review.whamcloud.com/2240" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2240</a>) in a place where the allocation will occur unnecessary (OSS/MGS, and for MDS/MDT exports where there is no FLock), thus I pushed 2 patches because I think there could be 2 ways to fix it :</p> <p> _ <a href="http://review.whamcloud.com/6843" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6843</a>, is in the same direction than Prakash one but just try to ensure that FLock hashing stuff will only occur when necessary, and whith no resource-&gt;lr_lock held for sure !</p> <p> _ <a href="http://review.whamcloud.com/6844" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6844</a>, does not change the current FLock involved algorithms but just ensure that the hashing stuff kmem allocation will occur atomically.</p> <p>But actually both patches experienced strange auto-tests failures (like in sanity-hsm !) that I need to investigate ...</p> <p>Both patches passed the tests finally !!... Need reviewers to give their feed-back now. Even if both patches are 2 different ways to fix original problem for this ticket, we may also decide that both patches are complementary since one ensures that FLock hashing stuff will occur without holding lr_lock which is not bad and the other will ensure that all layers allocating hashing data structures will do it atomically which is also a good idea !!</p> <p>Apparently there is an anternative fix from Xyratex for this issue: <a href="http://review.whamcloud.com/#/c/5471/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/5471/</a> I wonder if it's better?</p> <p>Because as I already commented negatively in Prakash's change #6790 for this same <a href="https://jira.whamcloud.com/browse/LU-3504" title="MDS: All cores spinning on ldlm lock in lock_res_and_lock" class="issue-link" data-issue-key="LU-3504"><del>LU-3504</del></a>, doing FLock related hashing stuff allocation in ldlm_init_export() will cause it to be allocated for all types of exports when it is only required for MDTs and then not for all clients/exports. It is why I decided to do it during 1st FLock on export.</p> <p>If you agree on this, may be I/you should also comment on this in Change #5471/<a href="https://jira.whamcloud.com/browse/LU-2835" title="mds crash, cfs_hash_bd_del_locked()) ASSERTION( bd-&gt;bd_bucket-&gt;hsb_count &gt; 0 ) failed" class="issue-link" data-issue-key="LU-2835"><del>LU-2835</del></a> ??</p> <p>Liang negatively commented in <a href="http://review.whamcloud.com/6844" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6844</a> because as he wrote "Please don't force cfs_hash to use atomic allocator, lustre can have very large hash table and can have hundreds of thousands cfs_hash instances.". So if, according to his experience with hashing, #6844 is not a good idea finally, I need to succeed with #6843 ...</p> <p>resolved by <a href="https://jira.whamcloud.com/browse/LU-2835" title="mds crash, cfs_hash_bd_del_locked()) ASSERTION( bd-&gt;bd_bucket-&gt;hsb_count &gt; 0 ) failed" class="issue-link" data-issue-key="LU-2835"><del>LU-2835</del></a>. export is checked in ldlm_init_flock_export().</p> <p>It is true and I read too quick change #5471 to miss that MDT-export only is checked, but again don't you think allocation will still occur even for exports/clients that will not/never use FLocks then ?</p> <p>In fact we have the choice between an unconditional, but may be not necessary, allocation in change #5471 and allocation on first need in #6843 ... Each having respectively a memory and a cpu/test cost.</p> <p>Hello Prakash,<br/> Seems that after Liang's feed-back/veto on my 2nd fix proposal (change #6844) I will have to abandon it, but what do you think about your original patch proposal (change #6790) ??</p> <p>After Liang's feedback, I agree, #6844 is unnecessary and incorrect. I think, in order to fix this issue, we just need to land one of either: #5471, #6790, or #6843. All three of those patches address the same issue in slightly different ways, so as long as one of those is landed, I believe this will be fixed.</p> <p>Change, 5471 has landed to master. Should this ticket now be closed and the other patches abandoned? Or are the other patches also still needed?</p> <p>AFAIK, this issue is fixed as of <a href="http://review.whamcloud.com/#/c/5471/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/5471/</a> landing (no other patches needed).</p> <p>Actually, I'm reopening this. I see it's marked to be fixed for 2.5.0 and 2.4.1. The patch that landed is for master which should resolve 2.5.0, but we still need to backport the patch to fix 2.4.1.</p> <p>Reducing from 2.5 blocker as the master patch landed, but leaving open until it is ported to 2.4.1.</p> <p>Patch <a href="http://review.whamcloud.com/6843" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6843</a> has just been abandonned in benefit of change <a href="http://review.whamcloud.com/#/c/5471/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/5471/</a> from <a href="https://jira.whamcloud.com/browse/LU-2835" title="mds crash, cfs_hash_bd_del_locked()) ASSERTION( bd-&gt;bd_bucket-&gt;hsb_count &gt; 0 ) failed" class="issue-link" data-issue-key="LU-2835"><del>LU-2835</del></a>, as per reviewers/gatekeeper decision.</p> <p>So can we close this issue as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-2835" title="mds crash, cfs_hash_bd_del_locked()) ASSERTION( bd-&gt;bd_bucket-&gt;hsb_count &gt; 0 ) failed" class="issue-link" data-issue-key="LU-2835"><del>LU-2835</del></a>?</p> <p>I also think there is nothing else to do with this ticket, but Prakash, do you also agree with Peter's suggestion ?</p> <p>If this fix doesn't need to be backported to 2.4, then yes, it can be closed.</p> <p>I verified that the <a href="http://review.whamcloud.com/#/c/5471/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/5471/</a> patch/change for <a href="https://jira.whamcloud.com/browse/LU-2835" title="mds crash, cfs_hash_bd_del_locked()) ASSERTION( bd-&gt;bd_bucket-&gt;hsb_count &gt; 0 ) failed" class="issue-link" data-issue-key="LU-2835"><del>LU-2835</del></a> is in origin/b2_4 branch already.<br/> So can we close ?</p> Related LU-2835 LU-4801 Development Rank 1|hzvtz3: Rank (Obsolete) 8822 Severity