https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1395 Lustre <p>We saw the following call traces on MDS and it hanged after it.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Apr 23 15:58:34 ALPL505 kernel: Call Trace: Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff88953a00&gt;] ldlm_expired_completion_wait+0x0/0x250 [ptlrpc] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff88955542&gt;] ldlm_completion_ast+0x4c2/0x880 [ptlrpc] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff8893a709&gt;] ldlm_lock_enqueue+0x9d9/0xb20 [ptlrpc] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff8008e421&gt;] default_wake_function+0x0/0xe Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff88935b6a&gt;] ldlm_lock_addref_internal_nolock+0x3a/0x90 [ptlrpc] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff889540bb&gt;] ldlm_cli_enqueue_local+0x46b/0x520 [ptlrpc] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff88caa157&gt;] enqueue_ordered_locks+0x387/0x4d0 [mds] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff889519a0&gt;] ldlm_blocking_ast+0x0/0x2a0 [ptlrpc] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff88955080&gt;] ldlm_completion_ast+0x0/0x880 [ptlrpc] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff88caa8e9&gt;] mds_get_parent_child_locked+0x649/0x960 [mds] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff88c9b652&gt;] mds_getattr_lock+0x632/0xc90 [mds] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff88c96dda&gt;] fixup_handle_for_resent_req+0x5a/0x2c0 [mds] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff88ca1d83&gt;] mds_intent_policy+0x623/0xc20 [mds] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff8893c270&gt;] ldlm_resource_putref_internal+0x230/0x460 [ptlrpc] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff88939eb6&gt;] ldlm_lock_enqueue+0x186/0xb20 [ptlrpc] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff889367fd&gt;] ldlm_lock_create+0x9bd/0x9f0 [ptlrpc] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff8895e870&gt;] ldlm_server_blocking_ast+0x0/0x83d [ptlrpc] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff8895bb39&gt;] ldlm_handle_enqueue+0xc09/0x1210 [ptlrpc] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff88ca0b30&gt;] mds_handle+0x40e0/0x4d10 [mds] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff800774ed&gt;] smp_send_reschedule+0x4e/0x53 Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff8008ddcd&gt;] enqueue_task+0x41/0x56 Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff8897fd55&gt;] lustre_msg_get_conn_cnt+0x35/0xf0 [ptlrpc] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff889896d9&gt;] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff88989e35&gt;] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff8008c85d&gt;] __wake_up_common+0x3e/0x68 Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff8898adc6&gt;] ptlrpc_main+0xf66/0x1120 [ptlrpc] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff8005dfb1&gt;] child_rip+0xa/0x11 Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff88989e60&gt;] ptlrpc_main+0x0/0x1120 [ptlrpc] Apr 23 15:58:34 ALPL505 kernel: [&lt;ffffffff8005dfa7&gt;] child_rip+0x0/0x11 </pre> </div></div> <p>this might be related to <a href="https://jira.whamcloud.com/browse/LU-59" title="call traces on MDS for ldlm_expired_completion_wait()" class="issue-link" data-issue-key="LU-59"><del>LU-59</del></a>, but please review on this.</p> LU-1395

MDS hangs after calltrace at ldlm_expired_completion_wait()

Bug Minor Resolved Cannot Reproduce Oleg Drokin Shuichi Ihara Thu, 10 May 2012 11:42:13 +0000 Mon, 29 May 2017 04:00:53 +0000 Mon, 29 May 2017 04:00:53 +0000 Lustre 1.8.6 0 6 <p>Oleg will look into this one</p> <p>Hi Peter, Oleg, <br/> could you plesae view on this quickly? we saw similar problems at a couple of sites.</p> <p>This trace is just a sign of client not responding to lock cancel request. It should be followed by a client being evicted.<br/> We need the client log to see what was happening there, I guess.</p> <p>This looks like a dupe of <a href="https://jira.whamcloud.com/browse/LU-500" title="MDS threads hang ldlm_expired_completion_wait+" class="issue-link" data-issue-key="LU-500"><del>LU-500</del></a> and <a href="https://jira.whamcloud.com/browse/LU-1269" title="speed up ASTs sending" class="issue-link" data-issue-key="LU-1269"><del>LU-1269</del></a>. I think because <a href="https://jira.whamcloud.com/browse/LU-1269" title="speed up ASTs sending" class="issue-link" data-issue-key="LU-1269"><del>LU-1269</del></a> is marked as an improvement instead of bug, it hasn't been getting the attention it should be. It appears as if there are several different ideas for fixing the issue. Can someone take a look at it? We have been hitting this bug regularly, most recently at IU.</p> <p>Close old ticket.</p> Development End date Fri, 19 Oct 2012 11:42:13 +0000 Rank 1|hzw1m7: Rank (Obsolete) 10343 Severity Start date Thu, 10 May 2012 11:42:13 +0000