https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7692 Lustre <p>On MDT LNet service thread hung Dumped stack trace</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">an 20 13:42:10 nbp8-mds1 kernel: LNet: Service thread pid 17862 was inactive <span class="code-keyword">for</span> 424.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace <span class="code-keyword">for</span> debugging purposes: Jan 20 13:42:10 nbp8-mds1 kernel: Pid: 17862, comm: mdt00_057 Jan 20 13:42:14 nbp8-mds1 kernel: Jan 20 13:42:14 nbp8-mds1 kernel: Call Trace: Jan 20 13:42:14 nbp8-mds1 kernel: [&lt;ffffffff8155a9c2&gt;] schedule_timeout+0x192/0x2e0 Jan 20 13:42:14 nbp8-mds1 kernel: [&lt;ffffffff81083300&gt;] ? process_timeout+0x0/0x10 Jan 20 13:42:14 nbp8-mds1 kernel: [&lt;ffffffffa078af70&gt;] ? ldlm_expired_completion_wait+0x0/0x360 [ptlrpc] Jan 20 13:42:14 nbp8-mds1 kernel: [&lt;ffffffffa078f7a1&gt;] ldlm_completion_ast+0x4b1/0x920 [ptlrpc] Jan 20 13:42:14 nbp8-mds1 kernel: [&lt;ffffffff81061fe0&gt;] ? default_wake_function+0x0/0x20 Jan 20 13:42:14 nbp8-mds1 kernel: [&lt;ffffffffa078ef00&gt;] ldlm_cli_enqueue_local+0x1f0/0x5e0 [ptlrpc] Jan 20 13:42:14 nbp8-mds1 kernel: [&lt;ffffffffa078f2f0&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Jan 20 13:42:14 nbp8-mds1 kernel: [&lt;ffffffffa0e72de0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e7cde4&gt;] mdt_object_lock0+0x394/0xb30 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e72de0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa078f2f0&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e7d644&gt;] mdt_object_lock+0x14/0x20 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e7d801&gt;] mdt_object_find_lock+0x61/0x170 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0ea982c&gt;] mdt_reint_open+0x88c/0x21a0 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa04fb5d6&gt;] ? upcall_cache_get_entry+0x296/0x880 [libcfs] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0652ef0&gt;] ? lu_ucred+0x20/0x30 [obdclass] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e71935&gt;] ? mdt_ucred+0x15/0x20 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e8e51c&gt;] ? mdt_root_squash+0x2c/0x3f0 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa07df766&gt;] ? __req_capsule_get+0x166/0x710 [ptlrpc] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e92481&gt;] mdt_reint_rec+0x41/0xe0 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e77ed3&gt;] mdt_reint_internal+0x4c3/0x780 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e7845e&gt;] mdt_intent_reint+0x1ee/0x410 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e75c3e&gt;] mdt_intent_policy+0x3ae/0x770 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa076f2c5&gt;] ldlm_lock_enqueue+0x135/0x980 [ptlrpc] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0798ebb&gt;] ldlm_handle_enqueue0+0x51b/0x10c0 [ptlrpc] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e76106&gt;] mdt_enqueue+0x46/0xe0 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0e7aada&gt;] mdt_handle_common+0x52a/0x1470 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa0eb74a5&gt;] mds_regular_handle+0x15/0x20 [mdt] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa07c80c5&gt;] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa04f08d5&gt;] ? lc_watchdog_touch+0x65/0x170 [libcfs] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa07c0a69&gt;] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa07ca89d&gt;] ptlrpc_main+0xafd/0x1780 [ptlrpc] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffffa07c9da0&gt;] ? ptlrpc_main+0x0/0x1780 [ptlrpc] Jan 20 13:42:17 nbp8-mds1 kernel: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 an 20 13:42:18 nbp8-mds1 kernel: Pid: 17780, comm: mdt00_044 Jan 20 13:42:18 nbp8-mds1 kernel: Jan 20 13:42:18 nbp8-mds1 kernel: Call Trace: Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffff8155a9c2&gt;] schedule_timeout+0x192/0x2e0 Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffff81083300&gt;] ? process_timeout+0x0/0x10 Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa078af70&gt;] ? ldlm_expired_completion_wait+0x0/0x360 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa078f7a1&gt;] ldlm_completion_ast+0x4b1/0x920 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffff81061fe0&gt;] ? default_wake_function+0x0/0x20 Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa078ef00&gt;] ldlm_cli_enqueue_local+0x1f0/0x5e0 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa078f2f0&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0e72de0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0e7cc06&gt;] mdt_object_lock0+0x1b6/0xb30 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0e72de0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa078f2f0&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0e7d644&gt;] mdt_object_lock+0x14/0x20 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0e85b8e&gt;] mdt_getattr_name_lock+0x8fe/0x19d0 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa07df766&gt;] ? __req_capsule_get+0x166/0x710 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa07ba7b4&gt;] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0e86ef9&gt;] mdt_intent_getattr+0x299/0x480 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0e75c3e&gt;] mdt_intent_policy+0x3ae/0x770 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa076f2c5&gt;] ldlm_lock_enqueue+0x135/0x980 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0798ebb&gt;] ldlm_handle_enqueue0+0x51b/0x10c0 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0e76106&gt;] mdt_enqueue+0x46/0xe0 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0e7aada&gt;] mdt_handle_common+0x52a/0x1470 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa0eb74a5&gt;] mds_regular_handle+0x15/0x20 [mdt] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa07c80c5&gt;] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffff81061ff2&gt;] ? default_wake_function+0x12/0x20 Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa07ca89d&gt;] ptlrpc_main+0xafd/0x1780 [ptlrpc] Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 Jan 20 13:42:18 nbp8-mds1 kernel: [&lt;ffffffffa07c9da0&gt;] ? ptlrpc_main+0x0/0x1780 [ptlrpc] Jan 20 13:42:19 nbp8-mds1 kernel: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 Jan 20 13:42:19 nbp8-mds1 kernel: Jan 20 13:42:19 nbp8-mds1 kernel: Pid: 16840, comm: mdt03_017 Jan 20 13:42:19 nbp8-mds1 kernel: Jan 20 13:42:19 nbp8-mds1 kernel: Call Trace: Jan 20 13:42:19 nbp8-mds1 kernel: [&lt;ffffffffa076a885&gt;] ? _ldlm_lock_debug+0x2d5/0x660 [ptlrpc] Jan 20 13:42:19 nbp8-mds1 kernel: [&lt;ffffffff8155a9c2&gt;] schedule_timeout+0x192/0x2e0 Jan 20 13:42:19 nbp8-mds1 kernel: [&lt;ffffffff81083300&gt;] ? process_timeout+0x0/0x10 Jan 20 13:42:19 nbp8-mds1 kernel: [&lt;ffffffffa078af70&gt;] ? ldlm_expired_completion_wait+0x0/0x360 [ptlrpc] Jan 20 13:42:19 nbp8-mds1 kernel: [&lt;ffffffffa078f7a1&gt;] ldlm_completion_ast+0x4b1/0x920 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffff81061fe0&gt;] ? default_wake_function+0x0/0x20 Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa078ef00&gt;] ldlm_cli_enqueue_local+0x1f0/0x5e0 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa078f2f0&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0e72de0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0e7cc06&gt;] mdt_object_lock0+0x1b6/0xb30 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0e72de0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa078f2f0&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0e7d644&gt;] mdt_object_lock+0x14/0x20 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0e85b8e&gt;] mdt_getattr_name_lock+0x8fe/0x19d0 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa07df766&gt;] ? __req_capsule_get+0x166/0x710 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa07ba7b4&gt;] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0e86ef9&gt;] mdt_intent_getattr+0x299/0x480 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0e75c3e&gt;] mdt_intent_policy+0x3ae/0x770 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa076f2c5&gt;] ldlm_lock_enqueue+0x135/0x980 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0798ebb&gt;] ldlm_handle_enqueue0+0x51b/0x10c0 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0e76106&gt;] mdt_enqueue+0x46/0xe0 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0e7aada&gt;] mdt_handle_common+0x52a/0x1470 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa0eb74a5&gt;] mds_regular_handle+0x15/0x20 [mdt] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa07c80c5&gt;] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa04f08d5&gt;] ? lc_watchdog_touch+0x65/0x170 [libcfs] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa07c0a69&gt;] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa07ca89d&gt;] ptlrpc_main+0xafd/0x1780 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffffa07c9da0&gt;] ? ptlrpc_main+0x0/0x1780 [ptlrpc] Jan 20 13:42:20 nbp8-mds1 kernel: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Looks the same as <a href="https://jira.whamcloud.com/browse/LU-7232" title="racer deadlock" class="issue-link" data-issue-key="LU-7232"><del>LU-7232</del></a></p> <p>I can upload debug logs if needed.</p> LU-7692

LNet: Service thread Hung

Bug Major Resolved Cannot Reproduce Zhenyu Xu Mahmoud Hanafi Wed, 20 Jan 2016 22:17:10 +0000 Mon, 2 Oct 2017 19:37:09 +0000 Thu, 8 Sep 2016 18:06:39 +0000 Lustre 2.5.3 Lustre 2.7.0 0 8 <p>It seems another instance of <a href="https://jira.whamcloud.com/browse/LU-7640" title="stuck mdt thread required reboot of mds" class="issue-link" data-issue-key="LU-7640"><del>LU-7640</del></a> to me.</p> <p>So the problem at hand is that the lock a server queued is not being grnted for a long time - either becase a client is not releasing a conflicting lock or for some other reason.</p> <p>There's a whole bunch of different problems that could cause this and I think <a href="https://jira.whamcloud.com/browse/LU-7232" title="racer deadlock" class="issue-link" data-issue-key="LU-7232"><del>LU-7232</del></a> is not it due to that caused by a particular master patch that I bet you are not carrying.</p> <p>As for <a href="https://jira.whamcloud.com/browse/LU-7640" title="stuck mdt thread required reboot of mds" class="issue-link" data-issue-key="LU-7640"><del>LU-7640</del></a> - it might be a bug of a similar nature, but again hard to tell what's the underlying cause at this stage.</p> <p>Just as a wild guess - you do not happen to have any SELinux-enabled client nodes (even if permissive) do you?</p> <p>We are not running SELinux on any client. We had 3 incidents of this yesterday. Even with <a href="https://jira.whamcloud.com/browse/LU-7372" title="replay-dual test_26: test failed to respond and timed out" class="issue-link" data-issue-key="LU-7372"><del>LU-7372</del></a> applied.<br/> This is getting trigger by a particular user. I tried to find the client/clients triggering it, but didn't have any luck.</p> <p>Uploading debug dump to see if can help.</p> <p>is this a particular job of this user that can trigger it? how large is the scale of that job?</p> <p>Oleg</p> <p>NASA shared that there is one specific user who can trigger this issue with their job runs and they will supply details when they are able to</p> <p>Bobijam</p> <p>Could you please review the uploaded debug logs and see if it is possible to deduce anything about the situation?</p> <p>Thanks</p> <p>Peter</p> <p>I am checking with a suspected user now. His job is 484 node and 5832 Tasks. The nodes are part of a remote cluster connected via routers. </p> <p>Can a client eviction cause this? Here we see this client getting evicted. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Jan 21 13:16:19 nbp8-mds1 kernel: LustreError: 0:0:(ldlm_lockd.c:346:waiting_locks_callback()) ### lock callback timer expired after 251s: evicting client at 10.153.10.226@o2ib233 ns: mdt-nbp8-MDT0000_UUID lock: ffff883cfce50dc0/0x8ea5cf5f62ad392a lrc: 3/0,0 mode: PR/PR res: [0x3603b8f6b:0xb4b:0x0].0 bits 0x13 rrc: 584 type: IBT flags: 0x60200400000020 nid: 10.153.10.226@o2ib233 remote: 0x4103cc31cf8b204c expref: 131 pid: 16613 timeout: 4333267923 lvb_type: 0 Jan 21 13:17:34 nbp8-mds1 kernel: LNet: Service thread pid 16447 was inactive <span class="code-keyword">for</span> 300.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace <span class="code-keyword">for</span> debugging purposes: Jan 21 13:17:34 nbp8-mds1 kernel: Pid: 16447, comm: mdt01_085 Jan 21 13:17:38 nbp8-mds1 kernel: Jan 21 13:17:38 nbp8-mds1 kernel: Call Trace: Jan 21 13:17:38 nbp8-mds1 kernel: [&lt;ffffffffa076a885&gt;] ? _ldlm_lock_debug+0x2d5/0x660 [ptlrpc] Jan 21 13:17:38 nbp8-mds1 kernel: [&lt;ffffffff8155a9c2&gt;] schedule_timeout+0x192/0x2e0 Jan 21 13:17:38 nbp8-mds1 kernel: [&lt;ffffffff81083300&gt;] ? process_timeout+0x0/0x10 Jan 21 13:17:38 nbp8-mds1 kernel: [&lt;ffffffffa078af70&gt;] ? ldlm_expired_completion_wait+0x0/0x360 [ptlrpc] Jan 21 13:17:38 nbp8-mds1 kernel: [&lt;ffffffffa078f7a1&gt;] ldlm_completion_ast+0x4b1/0x920 [ptlrpc] Jan 21 13:17:38 nbp8-mds1 kernel: [&lt;ffffffff81061fe0&gt;] ? default_wake_function+0x0/0x20 Jan 21 13:17:38 nbp8-mds1 kernel: [&lt;ffffffffa078ef00&gt;] ldlm_cli_enqueue_local+0x1f0/0x5e0 [ptlrpc] Jan 21 13:17:38 nbp8-mds1 kernel: [&lt;ffffffffa078f2f0&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa0e73de0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa0e7dde4&gt;] mdt_object_lock0+0x394/0xb30 [mdt] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa0e73de0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa078f2f0&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa0e7e644&gt;] mdt_object_lock+0x14/0x20 [mdt] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa0e7e801&gt;] mdt_object_find_lock+0x61/0x170 [mdt] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa0eaa88c&gt;] mdt_reint_open+0x88c/0x21a0 [mdt] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa04fb5d6&gt;] ? upcall_cache_get_entry+0x296/0x880 [libcfs] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa0652ef0&gt;] ? lu_ucred+0x20/0x30 [obdclass] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa0e72935&gt;] ? mdt_ucred+0x15/0x20 [mdt] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa0e8f51c&gt;] ? mdt_root_squash+0x2c/0x3f0 [mdt] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa07df766&gt;] ? __req_capsule_get+0x166/0x710 [ptlrpc] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa0e93481&gt;] mdt_reint_rec+0x41/0xe0 [mdt] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa0e78ed3&gt;] mdt_reint_internal+0x4c3/0x780 [mdt] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa0e7945e&gt;] mdt_intent_reint+0x1ee/0x410 [mdt] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa0e76c3e&gt;] mdt_intent_policy+0x3ae/0x770 [mdt] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa076f2c5&gt;] ldlm_lock_enqueue+0x135/0x980 [ptlrpc] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa0798ebb&gt;] ldlm_handle_enqueue0+0x51b/0x10c0 [ptlrpc] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa0e77106&gt;] mdt_enqueue+0x46/0xe0 [mdt] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa0e7bada&gt;] mdt_handle_common+0x52a/0x1470 [mdt] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa0eb8505&gt;] mds_regular_handle+0x15/0x20 [mdt] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa07c80c5&gt;] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa04f08d5&gt;] ? lc_watchdog_touch+0x65/0x170 [libcfs] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa07c0a69&gt;] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa07ca89d&gt;] ptlrpc_main+0xafd/0x1780 [ptlrpc] Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 Jan 21 13:17:41 nbp8-mds1 kernel: [&lt;ffffffffa07c9da0&gt;] ? ptlrpc_main+0x0/0x1780 [ptlrpc] </pre> </div></div> <p>Here is the client console log</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[1453410882.304755] LNetError: 7223:0:(o2iblnd_cb.c:3018:kiblnd_check_txs_locked()) Timed out tx: active_txs, 3 seconds [1453410882.316755] LNetError: 7223:0:(o2iblnd_cb.c:3081:kiblnd_check_conns()) Timed out RDMA with 10.153.26.93@o2ib233 (13): c: 61, oc: 0, rc: 63 [1453410882.328756] LustreError: 7224:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc ffff8807b8798000 [1453410882.340756] Lustre: 7260:0:(client.c:1940:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1453410728/real 1453410728] req@ffff880c045b5000 x1523566274464988/t0(0) o4-&gt;nbp8-OST00eb-osc-ffff88060b4b8800@10.151.27.63@o2ib:6/4 lens 488/448 e 0 to 1 dl 1453411070 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [1453410882.340756] Lustre: 7260:0:(client.c:1940:ptlrpc_expire_one_request()) Skipped 82 previous similar messages [1453410882.340756] Lustre: nbp8-OST00eb-osc-ffff88060b4b8800: Connection to nbp8-OST00eb (at 10.151.27.63@o2ib) was lost; in progress operations using <span class="code-keyword">this</span> service will wait <span class="code-keyword">for</span> recovery to complete [1453410882.340756] Lustre: Skipped 133 previous similar messages [1453410882.404758] Lustre: nbp8-OST00eb-osc-ffff88060b4b8800: Connection restored to nbp8-OST00eb (at 10.151.27.63@o2ib) [1453410882.404758] Lustre: Skipped 56 previous similar messages [1453410919.469932] LustreError: 7232:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc ffff880bce378000 [1453411070.294708] Lustre: 24291:0:(client.c:1940:ptlrpc_expire_one_request()) @@@ Request sent has timed out <span class="code-keyword">for</span> slow reply: [sent 1453410728/real 1453410728] req@ffff88061692bc00 x1523566274464784/t0(0) o101-&gt;nbp8-MDT0000-mdc-ffff88060b4b8800@10.151.27.60@o2ib:12/10 lens 624/4944 e 0 to 1 dl 1453411070 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [1453411070.322709] Lustre: 24291:0:(client.c:1940:ptlrpc_expire_one_request()) Skipped 1 previous similar message [1453411070.334710] Lustre: nbp8-MDT0000-mdc-ffff88060b4b8800: Connection to nbp8-MDT0000 (at 10.151.27.60@o2ib) was lost; in progress operations using <span class="code-keyword">this</span> service will wait <span class="code-keyword">for</span> recovery to complete [1453411071.346742] Lustre: nbp9-OST00e3-osc-ffff880607dea400: Connection restored to nbp9-OST00e3 (at 10.151.26.9@o2ib) [1453411096.347534] Lustre: nbp8-OST00be-osc-ffff88060b4b8800: Connection to nbp8-OST00be (at 10.151.27.70@o2ib) was lost; in progress operations using <span class="code-keyword">this</span> service will wait <span class="code-keyword">for</span> recovery to complete [1453411096.363534] Lustre: Skipped 169 previous similar messages [1453411121.368326] Lustre: nbp2-OST001f-osc-ffff880607d01c00: Connection restored to nbp2-OST001f (at 10.151.26.108@o2ib) [1453411121.368326] Lustre: Skipped 178 previous similar messages [1453411146.369118] Lustre: 7247:0:(client.c:1940:ptlrpc_expire_one_request()) @@@ Request sent has timed out <span class="code-keyword">for</span> slow reply: [sent 1453410804/real 1453410804] req@ffff880a414bb400 x1523566274482484/t0(0) o400-&gt;nbp7-OST0014-osc-ffff880607e0e000@10.151.27.45@o2ib:28/4 lens 224/224 e 0 to 1 dl 1453411146 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [1453411146.369118] Lustre: nbp7-OST0023-osc-ffff880607e0e000: Connection to nbp7-OST0023 (at 10.151.27.51@o2ib) was lost; in progress operations using <span class="code-keyword">this</span> service will wait <span class="code-keyword">for</span> recovery to complete [1453411146.369118] Lustre: Skipped 34 previous similar messages </pre> </div></div> <p>Some info from the user. Each task reads a restart file does some processing and writes the results. All the 5832 restart files are in the same directory.</p> <p>The relationship between the message and the eviction is direct more or less.</p> <p>What happens here is:<br/> 1. client gets a lock, starts to do IO under the lock<br/> 2. Somebody else wants the lock, so asks the above client to finish the io and drop the lock<br/> 3. The IO encounters a timeout, client is evicted<br/> 4. Lock is cancelled<br/> 5. if some thread spent long time waiting for the lock - you'll get hte message.</p> <p>Now in your case the eviciton is on one server (OST), but the message is on another (MDT). They should not really influence each other as such.<br/> But if there's some message dropping going on, then if the blocking AST is lost in transint, you'll se exactly this:<br/> "lock callback timer expired after" on the server that then evicts the offending client and then the thread that was waiting for the lock is able to get it, do whatever processing it needed and then complain that processing took too long once it was done - all thanks to the long waiting to get the lock.</p> <p>This issues is triggered when our remote cluster loses connectivity to the servers. We are trying to find the root cause of the network errors. But the MDT threads shouldn't lockup.</p> <p>I think the patch <a href="http://review.whamcloud.com/#/c/17853/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/17853/</a> addresses the MDT thread lockup issue, where the thread won't come out of ldlm_completion_ast() waiting.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jan 21 13:17:38 nbp8-mds1 kernel: [&lt;ffffffffa076a885&gt;] ? _ldlm_lock_debug+0x2d5/0x660 [ptlrpc] Jan 21 13:17:38 nbp8-mds1 kernel: [&lt;ffffffff8155a9c2&gt;] schedule_timeout+0x192/0x2e0 Jan 21 13:17:38 nbp8-mds1 kernel: [&lt;ffffffff81083300&gt;] ? process_timeout+0x0/0x10 Jan 21 13:17:38 nbp8-mds1 kernel: [&lt;ffffffffa078af70&gt;] ? ldlm_expired_completion_wait+0x0/0x360 [ptlrpc] Jan 21 13:17:38 nbp8-mds1 kernel: [&lt;ffffffffa078f7a1&gt;] ldlm_completion_ast+0x4b1/0x920 [ptlrpc] ... </pre> </div></div> <p>ldlm_expired_completion_wait() should return -ETIMEDOUT instead of 0, otherwise the ldlm_completion_ast() won't finish and keeps waiting.</p> <p>I always wonder how I can tell which patchset of a review was cherry-picked into our tree. Note that we cherry-picked some patch not in b2_5_fe repo, so the commit id is different than the one that I cherry-picked from. I have been using date that I did cherry-pick and examining codes to judge. Is there a good way to tell?</p> <p>As to <a href="http://review.whamcloud.com/#/c/17853/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/17853/</a>, I picked patch set #2 of 17853 (see <a href="https://jira.whamcloud.com/browse/LU-7640" title="stuck mdt thread required reboot of mds" class="issue-link" data-issue-key="LU-7640"><del>LU-7640</del></a>) but MDS still hung. I noticed that the lastest patch set is #4. I will cherry-pick #4 when it passes your internal testing.</p> <p>I just posted a comment to <a href="https://jira.whamcloud.com/browse/LU-7372" title="replay-dual test_26: test failed to respond and timed out" class="issue-link" data-issue-key="LU-7372"><del>LU-7372</del></a>:</p> <p>"<br/> Zhenyu Xu ported an earlier version (patch set 3, IIRC) of #17853 for b2_5_fe at<br/> <a href="http://review.whamcloud.com/17976" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17976</a> (see comment at 12/Jan/16 9:26 PM)</p> <p>Could someone from Intel update 17976 to the latest patch set #4 for us? Thanks!<br/> "</p> <p>I checked the difference of patch set 3 and patch set 4 of <br/> <a href="http://review.whamcloud.com/#/c/17853/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/17853/</a><br/> Well, no difference. They are the same.</p> <p>Assuming <br/> <a href="http://review.whamcloud.com/17976" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17976</a> is a b2_5_fe port of 17853 patch set 3, and we already have that patch in our mds/mgs, then the patch did not address our problem.</p> <p>Hi Jay,</p> <p>can you please extract the log of all processes from crash dump and post it here?</p> <p>This doesn't look like a duplication of <a href="https://jira.whamcloud.com/browse/LU-7372" title="replay-dual test_26: test failed to respond and timed out" class="issue-link" data-issue-key="LU-7372"><del>LU-7372</del></a>.</p> <p>From the backtrace, the MDT thread is doing open; also it's conflicting with a PR lock with LOOK|UPDATE|LAYOUT, so the evicted client opened the file for execution. However, I don't know what type of lock the stuck MDT thread was requesting.</p> <p>Attaching the process trace and dmesg output. </p> <p>mds.processtrace<br/> mds.dmesg</p> <p>From the backtrace, ALL mdt threads were blocked at LDLM lock enqueue. This implied a deadlock case if the DLM lock that blocked those MDT threads was canceled by ELC so that it was piggy-backed in a RPC request, but that RPC request couldn't be served due to lack of MDT threads. This is really possible based on the behavior of that user;s application.</p> <p>Let's try to disable to ELC by writing this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl set_param ldlm.namespaces.*-mdc-*.early_lock_cancel=0 </pre> </div></div> <p>on all client nodes.</p> <p>This may have some negative impact on performance, definitely should not try it at peak hours.</p> <p>One thing I have noticed is this message from the mds:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;6&gt;Lustre: nbp8-MDT0000: Recovery over after 2:29, of 12307 clients 12307 recovered and 0 were evicted. </pre> </div></div> <p>12307 clients were recovered in 2:29 minutes, this is really impressive.</p> <p>Can you provide any additional info on a permanent solution. </p> <p>I need to understand the problem first and then propose a solution.</p> <p>wait, does it imply it actually helped by disabling ELC?</p> <p>We haven't tried 'disabling ELC' because the filesystem is in production. We know that this is triggered when clients are writing/read a large number of files and the clients get disconnected due to a network issue.</p> <p>Please close this case. No longer an issue.</p> <p>ok - thanks Mahmoud</p> <p>Just to make sure: Is this no longer an issue because it was fixed by patch 17853 for <a href="https://jira.whamcloud.com/browse/LU-7372" title="replay-dual test_26: test failed to respond and timed out" class="issue-link" data-issue-key="LU-7372"><del>LU-7372</del></a>, or simply because it was never seen again (maybe it was fixed on the application side) ?</p> <p>It was not fixed by <a href="https://jira.whamcloud.com/browse/LU-7372" title="replay-dual test_26: test failed to respond and timed out" class="issue-link" data-issue-key="LU-7372"><del>LU-7372</del></a>. We upgraded to 2.7 and was not seen again.</p> <p>Ok, thanks Mahmoud.</p> Related LU-7372 Development End date Fri, 29 Apr 2016 22:17:10 +0000 Rank 1|hzxyon: Rank (Obsolete) 9223372036854775807 Severity Start date Wed, 20 Jan 2016 22:17:10 +0000