https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7640 Lustre <p>MDS reported stuck mdt threads and dump stack trace</p> <p>&lt;code&gt;<br/> Jan 7 20:02:53 nbp8-mds1 kernel: LNet: Service thread pid 16286 was inactive for 464.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:<br/> Jan 7 20:02:53 nbp8-mds1 kernel: LNet: Skipped 4 previous similar messages<br/> Jan 7 20:02:57 nbp8-mds1 kernel: Pid: 16286, comm: mdt02_020<br/> Jan 7 20:02:57 nbp8-mds1 kernel: <br/> Jan 7 20:02:57 nbp8-mds1 kernel: Call Trace:<br/> Jan 7 20:02:57 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa04eee01&gt;&#93;</span> ? libcfs_debug_msg+0x41/0x50 <span class="error">&#91;libcfs&#93;</span><br/> Jan 7 20:02:57 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa078af70&gt;&#93;</span> ? ldlm_expired_completion_wait+0x0/0x360 <span class="error">&#91;ptlrpc&#93;</span><br/> Jan 7 20:02:57 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa078f835&gt;&#93;</span> ldlm_completion_ast+0x545/0x920 <span class="error">&#91;ptlrpc&#93;</span><br/> Jan 7 20:02:57 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffff81061fe0&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> Jan 7 20:02:57 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa078ef00&gt;&#93;</span> ldlm_cli_enqueue_local+0x1f0/0x5e0 <span class="error">&#91;ptlrpc&#93;</span><br/> Jan 7 20:02:57 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa078f2f0&gt;&#93;</span> ? ldlm_completion_ast+0x0/0x920 <span class="error">&#91;ptlrpc&#93;</span><br/> Jan 7 20:02:57 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa0e72de0&gt;&#93;</span> ? mdt_blocking_ast+0x0/0x2a0 <span class="error">&#91;mdt&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa0e7cc06&gt;&#93;</span> mdt_object_lock0+0x1b6/0xb30 <span class="error">&#91;mdt&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa0e72de0&gt;&#93;</span> ? mdt_blocking_ast+0x0/0x2a0 <span class="error">&#91;mdt&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa078f2f0&gt;&#93;</span> ? ldlm_completion_ast+0x0/0x920 <span class="error">&#91;ptlrpc&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa0e7d644&gt;&#93;</span> mdt_object_lock+0x14/0x20 <span class="error">&#91;mdt&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa0e85b8e&gt;&#93;</span> mdt_getattr_name_lock+0x8fe/0x19d0 <span class="error">&#91;mdt&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa07df766&gt;&#93;</span> ? __req_capsule_get+0x166/0x710 <span class="error">&#91;ptlrpc&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa07ba7b4&gt;&#93;</span> ? lustre_msg_get_flags+0x34/0xb0 <span class="error">&#91;ptlrpc&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa0e86ef9&gt;&#93;</span> mdt_intent_getattr+0x299/0x480 <span class="error">&#91;mdt&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa0e75c3e&gt;&#93;</span> mdt_intent_policy+0x3ae/0x770 <span class="error">&#91;mdt&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa076f2c5&gt;&#93;</span> ldlm_lock_enqueue+0x135/0x980 <span class="error">&#91;ptlrpc&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa0798ebb&gt;&#93;</span> ldlm_handle_enqueue0+0x51b/0x10c0 <span class="error">&#91;ptlrpc&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa0e76106&gt;&#93;</span> mdt_enqueue+0x46/0xe0 <span class="error">&#91;mdt&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa0e7aada&gt;&#93;</span> mdt_handle_common+0x52a/0x1470 <span class="error">&#91;mdt&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa0eb74a5&gt;&#93;</span> mds_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa07c80c5&gt;&#93;</span> ptlrpc_server_handle_request+0x385/0xc00 <span class="error">&#91;ptlrpc&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa04f08d5&gt;&#93;</span> ? lc_watchdog_touch+0x65/0x170 <span class="error">&#91;libcfs&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa07c0a69&gt;&#93;</span> ? ptlrpc_wait_event+0xa9/0x2d0 <span class="error">&#91;ptlrpc&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa07ca89d&gt;&#93;</span> ptlrpc_main+0xafd/0x1780 <span class="error">&#91;ptlrpc&#93;</span><br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffff8100c20a&gt;&#93;</span> child_rip+0xa/0x20<br/> Jan 7 20:03:00 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffffa07c9da0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1780 <span class="error">&#91;ptlrpc&#93;</span><br/> Jan 7 20:03:01 nbp8-mds1 kernel: <span class="error">&#91;&lt;ffffffff8100c200&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Jan 7 20:03:01 nbp8-mds1 kernel: <br/> Jan 7 20:03:01 nbp8-mds1 kernel: LustreError: dumping log to /tmp/lustre-log.1452225773.16286<br/> &lt;code&gt;</p> <p>I am attaching /var/log/messages and lustre debug dump.</p> <p>The mds need to be rebooted to clear up the error state.</p> LU-7640

stuck mdt thread required reboot of mds

Bug Critical Resolved Duplicate Zhenyu Xu Mahmoud Hanafi Fri, 8 Jan 2016 06:46:16 +0000 Wed, 26 Apr 2017 14:45:56 +0000 Thu, 4 Feb 2016 22:19:34 +0000 Lustre 2.5.3 0 5 <p>it's dup of <a href="https://jira.whamcloud.com/browse/LU-7372" title="replay-dual test_26: test failed to respond and timed out" class="issue-link" data-issue-key="LU-7372"><del>LU-7372</del></a>, and there is patch at <a href="http://review.whamcloud.com/17853" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17853</a></p> <p>How did you think this is a dup of <a href="https://jira.whamcloud.com/browse/LU-7372" title="replay-dual test_26: test failed to respond and timed out" class="issue-link" data-issue-key="LU-7372"><del>LU-7372</del></a>? The stack trace do not look alike.</p> <p>The thread is waiting for a lock get granted or cancelled (ldlm_completion_ast()), and that never happens. And #17853 has fix about ldlm_expired_completion_wait() returning -ETIMEDOUT other than 0, so that the thread won't stuck.</p> <p>Thank you Zhenyu!<br/> After a series of problems last night to the mds/mgs of one of our lustre filesystems, it was upgraded to run with <a href="https://jira.whamcloud.com/browse/LU-7372" title="replay-dual test_26: test failed to respond and timed out" class="issue-link" data-issue-key="LU-7372"><del>LU-7372</del></a> patch. We will see <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>We had a crash after the patch was applied.</p> <p>what's the crash backtrace?</p> <p>duplicate of <a href="https://jira.whamcloud.com/browse/LU-7692" title="LNet: Service thread Hung" class="issue-link" data-issue-key="LU-7692"><del>LU-7692</del></a></p> <p>Just to clarify, recent versions of <a href="http://review.whamcloud.com/17853" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17853</a> no longer contain the change to ldlm_expired_completion_wait() mentioned above and this should not be considered a duplicate of <a href="https://jira.whamcloud.com/browse/LU-7372" title="replay-dual test_26: test failed to respond and timed out" class="issue-link" data-issue-key="LU-7372"><del>LU-7372</del></a>.</p> Related LU-7372 Development Rank 1|hzxxkn: Rank (Obsolete) 9223372036854775807 Severity