Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-1087] mdt thread spinning out of control https://jira.whamcloud.com/browse/LU-1087 Lustre <p>On one of our filesystems that was upgraded to lustre 2.1 last week, the MDS has one thread that appears to be stick spinning:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>13711 root 20 0 0 0 0 R 99.9 0.0 6289:28 mdt_253 </pre> </div></div> <p>Using sysreq (since crash can't trace a live process):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2012-02-09 16:18:39 Call Trace: 2012-02-09 16:18:39 [&lt;ffffffffa06de440&gt;] ? ldlm_cancel_locks_for_export_cb+0x0/0xd0 [ptlrpc] 2012-02-09 16:18:39 [&lt;ffffffffa06de440&gt;] ? ldlm_cancel_locks_for_export_cb+0x0/0xd0 [ptlrpc] 2012-02-09 16:18:39 [&lt;ffffffffa0498f46&gt;] cfs_hash_for_each_empty+0xc6/0x170 [libcfs] 2012-02-09 16:18:39 [&lt;ffffffffa06db15f&gt;] ldlm_cancel_locks_for_export+0x1f/0x30 [ptlrpc] 2012-02-09 16:18:39 [&lt;ffffffffa06e7c94&gt;] server_disconnect_export+0x64/0x170 [ptlrpc] 2012-02-09 16:18:39 [&lt;ffffffffa0c50dcb&gt;] mdt_obd_disconnect+0x4b/0xd50 [mdt] 2012-02-09 16:18:39 [&lt;ffffffffa071681e&gt;] ? lustre_pack_reply_flags+0xae/0x1b0 [ptlrpc] 2012-02-09 16:18:39 [&lt;ffffffffa0716931&gt;] ? lustre_pack_reply+0x11/0x20 [ptlrpc] 2012-02-09 16:18:39 [&lt;ffffffffa06e4d4a&gt;] target_handle_disconnect+0x16a/0x3a0 [ptlrpc] 2012-02-09 16:18:39 [&lt;ffffffffa0c4a805&gt;] mdt_disconnect+0x35/0x100 [mdt] 2012-02-09 16:18:39 [&lt;ffffffffa0c4674d&gt;] mdt_handle_common+0x73d/0x12b0 [mdt] 2012-02-09 16:18:39 [&lt;ffffffffa0717334&gt;] ? lustre_msg_get_transno+0x54/0x90 [ptlrpc] 2012-02-09 16:18:39 [&lt;ffffffffa0c47395&gt;] mdt_regular_handle+0x15/0x20 [mdt] 2012-02-09 16:18:39 [&lt;ffffffffa0723181&gt;] ptlrpc_main+0xcd1/0x1690 [ptlrpc] 2012-02-09 16:18:39 [&lt;ffffffffa07224b0&gt;] ? ptlrpc_main+0x0/0x1690 [ptlrpc] 2012-02-09 16:18:39 [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 2012-02-09 16:18:39 [&lt;ffffffffa07224b0&gt;] ? ptlrpc_main+0x0/0x1690 [ptlrpc] 2012-02-09 16:18:39 [&lt;ffffffffa07224b0&gt;] ? ptlrpc_main+0x0/0x1690 [ptlrpc] 2012-02-09 16:18:39 [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 </pre> </div></div> lustre 2.1.0-21chaos (github.com/chaos/lustre) LU-1087 mdt thread spinning out of control Bug Critical Resolved Duplicate Lai Siyao Christopher Morrone Thu, 9 Feb 2012 19:31:43 +0000 Fri, 13 Jul 2012 15:11:22 +0000 Tue, 5 Jun 2012 06:04:06 +0000 Lustre 2.1.0 0 5 <p>Excerpt from the log for the spinning thread, showing it repeatedly trying to free the same hash:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00010000:00000040:15.0:1328835885.126894:0:13711:0:(ldlm_resource.c:1102:ldlm_resource_getref()) getref res: ffff88082dc5fa80 count: 2 00010000:00000040:15.0:1328835885.126895:0:13711:0:(ldlm_resource.c:1141:ldlm_resource_putref()) putref res: ffff88082dc5fa80 count: 1 00000001:00000040:15.0:1328835885.126925:0:13711:0:(hash.c:1687:cfs_hash_for_each_empty()) Try to empty hash: 60009e44-9f98-9, loop: 178108 00010000:00000040:15.0:1328835885.126957:0:13711:0:(ldlm_resource.c:1102:ldlm_resource_getref()) getref res: ffff88082dc5fa80 count: 2 00010000:00000040:15.0:1328835885.126958:0:13711:0:(ldlm_resource.c:1141:ldlm_resource_putref()) putref res: ffff88082dc5fa80 count: 1 00000001:00000040:15.0:1328835885.126988:0:13711:0:(hash.c:1687:cfs_hash_for_each_empty()) Try to empty hash: 60009e44-9f98-9, loop: 178109 00010000:00000040:15.0:1328835885.127020:0:13711:0:(ldlm_resource.c:1102:ldlm_resource_getref()) getref res: ffff88082dc5fa80 count: 2 00010000:00000040:15.0:1328835885.127022:0:13711:0:(ldlm_resource.c:1141:ldlm_resource_putref()) putref res: ffff88082dc5fa80 count: 1 00000001:00000040:15.0:1328835885.127051:0:13711:0:(hash.c:1687:cfs_hash_for_each_empty()) Try to empty hash: 60009e44-9f98-9, loop: 178110 </pre> </div></div> <p>Lai</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Hi Chris, did you see any error messages about ldlm lock before this happened?</p> <p>We saw all manner of timeouts, and reconnects, slow service warnings and just generally a lot of log traffic. There are certainly lock callback timer expirations and other ldlm stuff happening in there.</p> <p>Chris, your log shows you've enabled D_INFO log, could you enable D_DLMTRACE and check which lock is in the dead loop (ldlm_cancel_lock_for_export_cb() should print lock information)?</p> <p>If it happens again I will do that. Unfortunately D_INFO caused <a href="https://jira.whamcloud.com/browse/LU-1088" title="mgs threads go nuts" class="issue-link" data-issue-key="LU-1088"><del>LU-1088</del></a>, so I was not able to get any further debugging information.</p> <p>I've posted a patch for review: <a href="http://review.whamcloud.com/#change,3028" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3028</a></p> <p>This is believed to be a duplicate of <a href="https://jira.whamcloud.com/browse/LU-1428" title="MDT servrice threads spinning in cfs_hash_for_each_relax()" class="issue-link" data-issue-key="LU-1428"><del>LU-1428</del></a></p> Duplicate LU-1428 Development Rank 1|hzvhdr: Rank (Obsolete) 6465 Severity