https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1088 Lustre <p>While investigating <a href="https://jira.whamcloud.com/browse/LU-1087" title="mdt thread spinning out of control" class="issue-link" data-issue-key="LU-1087"><del>LU-1087</del></a>, the ll_mgs_* threads suddenly went nuts and shot the load through the roof, to the point where the node is almost completely unresponsive, and a "top" that I had running is only able to redraw ever minute or so.</p> <p>The console is mostly unresponsive, but it did respond to a sysreq-l, so I can see that they are all in a backtrace similar to this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Call Trace: [&lt;ffffffffa06da060&gt;] lock_res_and_lock+0x30/0x40 [ptlrpc] [&lt;ffffffffa06deca3&gt;] ldlm_lock_enqueue+0x453/0x7e0 [ptlrpc] [&lt;ffffffffa06fd206&gt;] ldlm_handle_enqueue0+0x406/0xd70 [ptlrpc] [&lt;ffffffffa06fdbd6&gt;] ldlm_handle_enqueue+0x66/0x70 [ptlrpc] [&lt;ffffffffa06fdbe0&gt;] ? ldlm_server_completion_ast+0x0/0x590 [ptlrpc] [&lt;ffffffffa06fe170&gt;] ? ldlm_server_blocking_ast+0x0/0x740 [ptlrpc] [&lt;ffffffffa0b55245&gt;] mgs_handle+0x545/0x1350 [mgs] [&lt;ffffffffa04933f1&gt;] ? libcfs_debug_vmsg1+0x41/0x50 [libcfs] [&lt;ffffffffa04933f1&gt;] ? libcfs_debug_vmsg1+0x41/0x50 [libcfs] [&lt;ffffffffa0723181&gt;] ptlrpc_main+0xcd1/0x1690 [ptlrpc] [&lt;ffffffffa07224b0&gt;] ? ptlrpc_main+0x0/0x1690 [ptlrpc] [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 [&lt;ffffffffa07224b0&gt;] ? ptlrpc_main+0x0/0x1690 [ptlrpc] [&lt;ffffffffa07224b0&gt;] ? ptlrpc_main+0x0/0x1690 [ptlrpc] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 </pre> </div></div> lustre 2.1.0-21chaos (github.com/chaos/lustre) LU-1088

mgs threads go nuts

Bug Critical Resolved Fixed Lai Siyao Christopher Morrone Thu, 9 Feb 2012 20:24:01 +0000 Mon, 4 Jun 2012 02:21:41 +0000 Mon, 4 Jun 2012 02:21:41 +0000 Lustre 2.3.0 0 6 <p>We gave up on waiting for the node to recover on its own. We forced a crash dump and are rebooting now.</p> <p>MDS went through recovery and seems happy for the moment.</p> <p>once you meet this problem again, please grab all the traces and attach them here.</p> <p>Alex, we have a crash dump. If you want backtraces from all tasks, we'll get you that. No need to wait for another instance.</p> <p>Attach "foreach bt" from momus mds.</p> <p>Added Alex as a watcher so he is aware of Chris's answer</p> <p>From the stack trace:<br/> PID: 21012 TASK: ffff880812dd7500 CPU: 8 COMMAND: "ll_mgs_22"<br/> #0 <span class="error">&#91;ffff88043f407e90&#93;</span> crash_nmi_callback at ffffffff81029746<br/> #1 <span class="error">&#91;ffff88043f407ea0&#93;</span> notifier_call_chain at ffffffff814f3eb5<br/> #2 <span class="error">&#91;ffff88043f407ee0&#93;</span> atomic_notifier_call_chain at ffffffff814f3f1a<br/> #3 <span class="error">&#91;ffff88043f407ef0&#93;</span> notify_die at ffffffff81096a6e<br/> #4 <span class="error">&#91;ffff88043f407f20&#93;</span> do_nmi at ffffffff814f1b33<br/> #5 <span class="error">&#91;ffff88043f407f50&#93;</span> nmi at ffffffff814f1440<br/> <span class="error">&#91;exception RIP: vsnprintf+779&#93;</span><br/> RIP: ffffffff81276c2b RSP: ffff880812da5810 RFLAGS: 00000206<br/> RAX: ffff880812da5a40 RBX: ffff8805aa88a529 RCX: 0000000000000002<br/> RDX: ffffffffa076a2fc RSI: ffff8805aa88b000 RDI: ffff8805aa88a529<br/> RBP: ffff880812da58a0 R8: 0000000000000073 R9: 0000000000000000<br/> R10: 0000000000000001 R11: 00000000000000dc R12: ffffffffa075b626<br/> R13: ffffffffa075b624 R14: ffff880812da5980 R15: ffff8805aa88b000<br/> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018<br/> &#8212; &lt;NMI exception stack&gt; &#8212;<br/> #6 <span class="error">&#91;ffff880812da5810&#93;</span> vsnprintf at ffffffff81276c2b<br/> #7 <span class="error">&#91;ffff880812da5868&#93;</span> cfs_set_ptldebug_header at ffffffffa048919b <span class="error">&#91;libcfs&#93;</span><br/> #8 <span class="error">&#91;ffff880812da58a8&#93;</span> libcfs_debug_vmsg2 at ffffffffa0492b13 <span class="error">&#91;libcfs&#93;</span><br/> #9 <span class="error">&#91;ffff880812da5a08&#93;</span> libcfs_debug_vmsg1 at ffffffffa04933f1 <span class="error">&#91;libcfs&#93;</span><br/> #10 <span class="error">&#91;ffff880812da5a68&#93;</span> ldlm_lock_dump at ffffffffa06da97e <span class="error">&#91;ptlrpc&#93;</span><br/> #11 <span class="error">&#91;ffff880812da5aa8&#93;</span> ldlm_resource_dump at ffffffffa06e0ae3 <span class="error">&#91;ptlrpc&#93;</span><br/> #12 <span class="error">&#91;ffff880812da5b08&#93;</span> ldlm_granted_list_add_lock at ffffffffa06dd280 <span class="error">&#91;ptlrpc&#93;</span><br/> #13 <span class="error">&#91;ffff880812da5b48&#93;</span> ldlm_grant_lock at ffffffffa06dd83c <span class="error">&#91;ptlrpc&#93;</span><br/> #14 <span class="error">&#91;ffff880812da5ba8&#93;</span> ldlm_process_plain_lock at ffffffffa06edc1a <span class="error">&#91;ptlrpc&#93;</span><br/> #15 <span class="error">&#91;ffff880812da5c48&#93;</span> ldlm_lock_enqueue at ffffffffa06deb6d <span class="error">&#91;ptlrpc&#93;</span><br/> #16 <span class="error">&#91;ffff880812da5cb8&#93;</span> ldlm_handle_enqueue0 at ffffffffa06fd206 <span class="error">&#91;ptlrpc&#93;</span><br/> #17 <span class="error">&#91;ffff880812da5d28&#93;</span> ldlm_handle_enqueue at ffffffffa06fdbd6 <span class="error">&#91;ptlrpc&#93;</span><br/> #18 <span class="error">&#91;ffff880812da5d68&#93;</span> mgs_handle at ffffffffa0b55245 <span class="error">&#91;mgs&#93;</span><br/> #19 <span class="error">&#91;ffff880812da5df8&#93;</span> ptlrpc_main at ffffffffa0723181 <span class="error">&#91;ptlrpc&#93;</span><br/> #20 <span class="error">&#91;ffff880812da5f48&#93;</span> kernel_thread at ffffffff8100c14a</p> <p>So I imagine you have a lot of clients (tens of thousands?), and once all of them somehow got disconnected, they all come rushing to reconnect back and get their config lock too (all on the same resource)..<br/> Now in the middle of the locking we have this ldlm_resource_dump() call that would print all locks on the resource, which would take quite a while I can imagine, and everybody else is spinning on the resource spinlock meanwhile.</p> <p>That ldlm_resource_dump is D_IFO which is probably bad idea and should be D_DLMTRACE too, what is the lustre debug level you are running at?<br/> I wonder if we should just change ldlm_resource_dump() to only print the individual locks if the refcount is below some threshold, as otherwise it is just too much info I suspect.</p> <p>Yes, I had recently added D_INFO to get a look at what the spinning mdt thread was doing for <a href="https://jira.whamcloud.com/browse/LU-1087" title="mdt thread spinning out of control" class="issue-link" data-issue-key="LU-1087"><del>LU-1087</del></a>.</p> <p>We have a few thousand clients. Most of them should not be disconnected. There are on the order of a couple hundred that might reboot and reconnect at any time (BGP nodes).</p> <p>I think that we definitely need ldlm_resource_dump changed. I certainly accept that performance is reduced when I enable higher logging levels, but I don't expect a denial-of-service attack. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>It looks okay to use RCU for resource lock dump, and compared to ldlm_lock_debug() ldlm_lock_dump() is inefficient, I'll replace it the former one.</p> <p>Review is on <a href="http://review.whamcloud.com/#change,2250" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2250</a>.</p> <p>Landed for 2.3</p> Development Rank 1|hzv6nz: Rank (Obsolete) 4614 Severity