https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1219 Lustre <p>We got the following call traces at the customer site, and one OST refuses to connect due to still busy with 1 active RPCs.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Mar 15 00:14:14 t2s007059 kernel: Pid: 13147, comm: ll_ost_io_229 Mar 15 00:14:14 t2s007059 kernel: Mar 15 00:14:14 t2s007059 kernel: Call Trace: Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff8002e024&gt;] __wake_up+0x38/0x4f Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff889fc7f3&gt;] jbd2_log_wait_commit+0xa3/0xf5 [jbd2] Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff800a2dff&gt;] autoremove_wake_function+0x0/0x2e Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff88b1390b&gt;] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs] Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff88b54144&gt;] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter] Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff886da3a6&gt;] lnet_ni_send+0x96/0xe0 [lnet] Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff88aeeede&gt;] ost_checksum_bulk+0x36e/0x5a0 [ost] Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff88af5d09&gt;] ost_brw_write+0x1c99/0x2480 [ost] Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff887ebac8&gt;] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc] Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff887b68b0&gt;] target_committed_to_req+0x40/0x120 [ptlrpc] Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff8008e7f9&gt;] default_wake_function+0x0/0xe Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff887f00a8&gt;] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff88af909e&gt;] ost_handle+0x2bae/0x55b0 [ost] Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff88741d00&gt;] class_handle2object+0xe0/0x170 [obdclass] Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff887aa19a&gt;] lock_res_and_lock+0xba/0xd0 [ptlrpc] Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff887af168&gt;] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff887ff6d9&gt;] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff887ffe35&gt;] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff8008cc1e&gt;] __wake_up_common+0x3e/0x68 Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff88800dc6&gt;] ptlrpc_main+0xf66/0x1120 [ptlrpc] Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff8005dfb1&gt;] child_rip+0xa/0x11 Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff887ffe60&gt;] ptlrpc_main+0x0/0x1120 [ptlrpc] Mar 15 00:14:14 t2s007059 kernel: [&lt;ffffffff8005dfa7&gt;] child_rip+0x0/0x11 Mar 15 00:14:14 t2s007059 kernel: </pre> </div></div> server : lustre-1.8.7, client : lustre-1.8.4.ddn2.2 LU-1219

The connection is refused due to still busy with 1 active RPCs

Bug Minor Resolved Incomplete Hongchao Zhang Shuichi Ihara Thu, 15 Mar 2012 01:15:52 +0000 Thu, 20 Dec 2012 14:29:34 +0000 Thu, 20 Dec 2012 14:29:34 +0000 Lustre 1.8.7 0 3 <p>Hongchao </p> <p>Could you please help with this one?</p> <p>Thanks</p> <p>Peter</p> <p>This is possibly a duplicate of <a href="https://jira.whamcloud.com/browse/LU-793" title="Reconnections should not be refused when there is a request in progress from this client." class="issue-link" data-issue-key="LU-793"><del>LU-793</del></a>, for which Oleg already has a patch.</p> <p>The underlying issue is the write stuck in jbd somehow, <a href="https://jira.whamcloud.com/browse/LU-793" title="Reconnections should not be refused when there is a request in progress from this client." class="issue-link" data-issue-key="LU-793"><del>LU-793</del></a> would not fix this, it just papers over some of the symptoms.</p> <p>the journal is stuck when committing the previous transaction, was the info of process in this node available? <br/> the stack trace of "kjournald2" should give some info about where it was stuck. thanks!</p> <p>Hongchao,</p> <p>Unfortunately, we only have /var/log/messages and /tmp/lustre-log.&lt;timestamp&gt;</p> <p>When if the problem happens again, we can collect all information you want.<br/> Please let us know, what should we run the commands before reboot servers. </p> <p>Hi, <br/> the customer is waiting for what they should get them when the same problem happens at the next time.<br/> please let me know what exactly we should do.</p> <p>Hi Shuichi</p> <p>Sorry for delayed response.<br/> please get the thread info of tasks by sysrq (echo "t" &gt;/proc/sysrq-trigger) of the node at where the OST resides,<br/> and it's better to get the current running address of process "kjournald2" ATM, Thanks!</p> <p>Close this old issue until there is more information available.</p> Development Rank 1|hzvh6v: Rank (Obsolete) 6434 Severity