https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LUDOC-323 Lustre Documentation <p>On MDT LNet service thread hung Dumped stack trace</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">an 20 13:42:10 nbp8-mds1 kernel: LNet: Service thread pid 17862 was inactive <span class="code-keyword">for</span> 424.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace <span class="code-keyword">for</span> debugging purposes: Jan 20 13:42:10 nbp8-mds1 kernel: Pid: 17862, comm: mdt00_057 Jan 20 13:42:14 nbp8-mds1 kernel: Jan 20 13:42:14 nbp8-mds1 kernel: Call Trace: Jan 20 13:42:14 nbp8-mds1 kernel: [&lt;ffffffff8155a9c2&gt;] schedule_timeout+0x192/0x2e0 Jan 20 13:42:14 nbp8-mds1 kernel: [&lt;ffffffff81083300&gt;] ? process_timeout+0x0/0x10 Jan 20 13:42:14 nbp8-mds1 kernel: [&lt;ffffffffa078af70&gt;] ? ldlm_expired_completion_wait+0x0/0x360 [ptlrpc] Jan 20 13:42:14 nbp8-mds1 kernel: [&lt;ffffffffa078f7a1&gt;] ldlm_completion_ast+0x4b1/0x920 [ptlrpc] Jan 20 13:42:14 nbp8-mds1 kernel: [&lt;ffffffff81061fe0&gt;] ? default_wake_function+0x0/0x20 Jan 20 13:42:14 nbp8-mds1 kernel: [&lt;ffffffffa078ef00&gt;] ldlm_cli_enqueue_local+0x1f0/0x5e0 [ptlrpc] Jan 20 13:42:14 nbp8-mds1 kernel: [&lt;ffffffffa078f2f0&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Jan 20 13:42:14 nbp8-mds1 kernel: [&lt;ffffffffa0e72de0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e7cde4&gt;] mdt_object_lock0+0x394/0xb30 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e72de0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa078f2f0&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e7d644&gt;] mdt_object_lock+0x14/0x20 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e7d801&gt;] mdt_object_find_lock+0x61/0x170 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0ea982c&gt;] mdt_reint_open+0x88c/0x21a0 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa04fb5d6&gt;] ? upcall_cache_get_entry+0x296/0x880 [libcfs] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0652ef0&gt;] ? lu_ucred+0x20/0x30 [obdclass] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e71935&gt;] ? mdt_ucred+0x15/0x20 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e8e51c&gt;] ? mdt_root_squash+0x2c/0x3f0 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa07df766&gt;] ? __req_capsule_get+0x166/0x710 [ptlrpc] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e92481&gt;] mdt_reint_rec+0x41/0xe0 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e77ed3&gt;] mdt_reint_internal+0x4c3/0x780 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e7845e&gt;] mdt_intent_reint+0x1ee/0x410 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e75c3e&gt;] mdt_intent_policy+0x3ae/0x770 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa076f2c5&gt;] ldlm_lock_enqueue+0x135/0x980 [ptlrpc] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0798ebb&gt;] ldlm_handle_enqueue0+0x51b/0x10c0 [ptlrpc] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e76106&gt;] mdt_enqueue+0x46/0xe0 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e7aada&gt;] mdt_handle_common+0x52a/0x1470 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0eb74a5&gt;] mds_regular_handle+0x15/0x20 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa07c80c5&gt;] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa04f08d5&gt;] ? lc_watchdog_touch+0x65/0x170 [libcfs] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa07c0a69&gt;] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa07ca89d&gt;] ptlrpc_main+0xafd/0x1780 [ptlrpc] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa07c9da0&gt;] ? ptlrpc_main+0x0/0x1780 [ptlrpc] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 an 20 13:42:18 nbp8-mds1 kernel: Pid: 17780, comm: mdt00_044 Jan 20 13:42:18 nbp8-mds1 kernel: Jan 20 13:42:18 nbp8-mds1 kernel: Call Trace: Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffff8155a9c2&gt;] schedule_timeout+0x192/0x2e0 Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffff81083300&gt;] ? process_timeout+0x0/0x10 Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa078af70&gt;] ? ldlm_expired_completion_wait+0x0/0x360 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa078f7a1&gt;] ldlm_completion_ast+0x4b1/0x920 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffff81061fe0&gt;] ? default_wake_function+0x0/0x20 Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa078ef00&gt;] ldlm_cli_enqueue_local+0x1f0/0x5e0 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa078f2f0&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0e72de0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0e7cc06&gt;] mdt_object_lock0+0x1b6/0xb30 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0e72de0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa078f2f0&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0e7d644&gt;] mdt_object_lock+0x14/0x20 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0e85b8e&gt;] mdt_getattr_name_lock+0x8fe/0x19d0 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa07df766&gt;] ? __req_capsule_get+0x166/0x710 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa07ba7b4&gt;] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0e86ef9&gt;] mdt_intent_getattr+0x299/0x480 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0e75c3e&gt;] mdt_intent_policy+0x3ae/0x770 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa076f2c5&gt;] ldlm_lock_enqueue+0x135/0x980 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0798ebb&gt;] ldlm_handle_enqueue0+0x51b/0x10c0 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0e76106&gt;] mdt_enqueue+0x46/0xe0 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0e7aada&gt;] mdt_handle_common+0x52a/0x1470 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0eb74a5&gt;] mds_regular_handle+0x15/0x20 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa07c80c5&gt;] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffff81061ff2&gt;] ? default_wake_function+0x12/0x20 Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa07ca89d&gt;] ptlrpc_main+0xafd/0x1780 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa07c9da0&gt;] ? ptlrpc_main+0x0/0x1780 [ptlrpc] Jan 20 13:42:19 nbp8-mds1 kernel: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 Jan 20 13:42:19 nbp8-mds1 kernel: Jan 20 13:42:19 nbp8-mds1 kernel: Pid: 16840, comm: mdt03_017 Jan 20 13:42:19 nbp8-mds1 kernel: Jan 20 13:42:19 nbp8-mds1 kernel: Call Trace: Jan 20 13:42:19 nbp8-mds1 kernel: [&lt;ffffffffa076a885&gt;] ? _ldlm_lock_debug+0x2d5/0x660 [ptlrpc] Jan 20 13:42:19 nbp8-mds1 kernel: [&lt;ffffffff8155a9c2&gt;] schedule_timeout+0x192/0x2e0 Jan 20 13:42:19 nbp8-mds1 kernel: [&lt;ffffffff81083300&gt;] ? process_timeout+0x0/0x10 Jan 20 13:42:19 nbp8-mds1 kernel: [&lt;ffffffffa078af70&gt;] ? ldlm_expired_completion_wait+0x0/0x360 [ptlrpc] Jan 20 13:42:19 nbp8-mds1 kernel: [&lt;ffffffffa078f7a1&gt;] ldlm_completion_ast+0x4b1/0x920 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffff81061fe0&gt;] ? default_wake_function+0x0/0x20 Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa078ef00&gt;] ldlm_cli_enqueue_local+0x1f0/0x5e0 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa078f2f0&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0e72de0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0e7cc06&gt;] mdt_object_lock0+0x1b6/0xb30 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0e72de0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa078f2f0&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0e7d644&gt;] mdt_object_lock+0x14/0x20 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0e85b8e&gt;] mdt_getattr_name_lock+0x8fe/0x19d0 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa07df766&gt;] ? __req_capsule_get+0x166/0x710 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa07ba7b4&gt;] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0e86ef9&gt;] mdt_intent_getattr+0x299/0x480 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0e75c3e&gt;] mdt_intent_policy+0x3ae/0x770 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa076f2c5&gt;] ldlm_lock_enqueue+0x135/0x980 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0798ebb&gt;] ldlm_handle_enqueue0+0x51b/0x10c0 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0e76106&gt;] mdt_enqueue+0x46/0xe0 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0e7aada&gt;] mdt_handle_common+0x52a/0x1470 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0eb74a5&gt;] mds_regular_handle+0x15/0x20 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa07c80c5&gt;] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa04f08d5&gt;] ? lc_watchdog_touch+0x65/0x170 [libcfs] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa07c0a69&gt;] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa07ca89d&gt;] ptlrpc_main+0xafd/0x1780 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa07c9da0&gt;] ? ptlrpc_main+0x0/0x1780 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Looks the same as <a href="https://jira.whamcloud.com/browse/LU-7232" title="racer deadlock" class="issue-link" data-issue-key="LU-7232"><del>LU-7232</del></a></p> <p>I can upload debug logs if needed.</p> lustre 2.5.3 LUDOC-323

LNet: Service thread Hung

Bug Major Closed Won't Fix Richard Henwood Mahmoud Hanafi Wed, 20 Jan 2016 22:13:19 +0000 Thu, 21 Jan 2016 20:25:12 +0000 Thu, 21 Jan 2016 20:25:12 +0000 0 1 <p>Why did this get filed under LUDOC?</p> <p>Sorry my mistake. Please close/cancel I'll reopen a new case.</p> Development Rank 1|hzxyof: Rank (Obsolete) 9223372036854775807 Severity