https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1578 Lustre <p>At the our customer site, some of ll_ost_io_xx threads hang and OST's status get into "DOWN" with "lctl dl".<br/> We saw the following Call traces on these OSSs during the time.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jun 28 16:22:18 nos011i kernel: Pid: 32708, comm: ll_ost_io_188 Jun 28 16:22:18 nos011i kernel: Jun 28 16:22:18 nos011i kernel: Call Trace: Jun 28 16:22:18 nos011i kernel: [&lt;ffffffff888a914d&gt;] ldlm_cli_enqueue_local+0x4fd/0x520 [ptlrpc] Jun 28 16:22:18 nos011i kernel: [&lt;ffffffff800645e3&gt;] __down_write_nested+0x7a/0x92 Jun 28 16:22:18 nos011i kernel: [&lt;ffffffff88bd7b29&gt;] filter_destroy+0x969/0x1f90 [obdfilter] Jun 28 16:22:18 nos011i kernel: [&lt;ffffffff8876ecfd&gt;] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jun 28 16:22:18 nos011i kernel: [&lt;ffffffff888d2cd2&gt;] lustre_pack_reply_flags+0x8e2/0x950 [ptlrpc] Jun 28 16:22:18 nos011i kernel: [&lt;ffffffff888d2d69&gt;] lustre_pack_reply+0x29/0xb0 [ptlrpc] Jun 28 16:22:18 nos011i kernel: [&lt;ffffffff88b89070&gt;] ost_destroy+0x660/0x790 [ost] Jun 28 16:22:18 nos011i kernel: [&lt;ffffffff888ceef5&gt;] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Jun 28 16:22:18 nos011i kernel: [&lt;ffffffff88b92a46&gt;] ost_handle+0x1556/0x55b0 [ost] Jun 28 16:22:18 nos011i kernel: [&lt;ffffffff888de6d9&gt;] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jun 28 16:22:18 nos011i kernel: [&lt;ffffffff8008cc1e&gt;] __wake_up_common+0x3e/0x68 Jun 28 16:22:18 nos011i kernel: [&lt;ffffffff888dfdc6&gt;] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jun 28 16:22:18 nos011i kernel: [&lt;ffffffff8005dfb1&gt;] child_rip+0xa/0x11 Jun 28 16:22:18 nos011i kernel: [&lt;ffffffff888dee60&gt;] ptlrpc_main+0x0/0x1120 [ptlrpc] Jun 28 16:22:18 nos011i kernel: [&lt;ffffffff8005dfa7&gt;] child_rip+0x0/0x11 Jun 28 16:22:18 nos011i kernel: Jun 28 16:22:18 nos011i kernel: Lustre: Service thread pid 8870 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 28 16:22:18 nos011i kernel: LustreError: dumping log to /tmp/lustre-log.1340868138.32708 Jun 28 16:22:18 nos011i kernel: Pid: 8870, comm: ll_ost_io_37 Jun 28 16:22:20 nos011i kernel: Jun 28 16:22:20 nos011i kernel: Call Trace: Jun 28 16:22:21 nos011i kernel: [&lt;ffffffff800645e3&gt;] __down_write_nested+0x7a/0x92 Jun 28 16:22:21 nos011i kernel: [&lt;ffffffff88bd7b29&gt;] filter_destroy+0x969/0x1f90 [obdfilter] Jun 28 16:22:21 nos011i kernel: [&lt;ffffffff8876ecfd&gt;] libcfs_debug_vmsg2+0x70d/0x970 [libcfs] Jun 28 16:22:21 nos011i kernel: [&lt;ffffffff888d2cd2&gt;] lustre_pack_reply_flags+0x8e2/0x950 [ptlrpc] Jun 28 16:22:21 nos011i kernel: [&lt;ffffffff888d2d69&gt;] lustre_pack_reply+0x29/0xb0 [ptlrpc] Jun 28 16:22:21 nos011i kernel: [&lt;ffffffff88b89070&gt;] ost_destroy+0x660/0x790 [ost] Jun 28 16:22:21 nos011i kernel: [&lt;ffffffff888ceef5&gt;] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] Jun 28 16:22:21 nos011i kernel: [&lt;ffffffff88b92a46&gt;] ost_handle+0x1556/0x55b0 [ost] Jun 28 16:22:21 nos011i kernel: [&lt;ffffffff888de6d9&gt;] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] Jun 28 16:22:21 nos011i kernel: [&lt;ffffffff8008cc1e&gt;] __wake_up_common+0x3e/0x68 Jun 28 16:22:21 nos011i kernel: [&lt;ffffffff888dfdc6&gt;] ptlrpc_main+0xf66/0x1120 [ptlrpc] Jun 28 16:22:21 nos011i kernel: [&lt;ffffffff8005dfb1&gt;] child_rip+0xa/0x11 Jun 28 16:22:21 nos011i kernel: [&lt;ffffffff888dee60&gt;] ptlrpc_main+0x0/0x1120 [ptlrpc] Jun 28 16:22:21 nos011i kernel: [&lt;ffffffff8005dfa7&gt;] child_rip+0x0/0x11 Jun 28 16:22:21 nos011i kernel: </pre> </div></div> RHEL5.7 LU-1578

ll_ost_io_xx thread hangs and OST status get into down

Bug Minor Resolved Duplicate Niu Yawei Shuichi Ihara Thu, 28 Jun 2012 04:39:59 +0000 Fri, 31 Aug 2012 11:25:34 +0000 Fri, 31 Aug 2012 11:25:34 +0000 Lustre 1.8.7 0 4 <p>two OSS's log files are attached. we saw this situation on these two OSSs.</p> <p>Are you sure the disks were healthy during this time? <br/> Can you get a stack dump of all threads when you see the OST in 'down' state and attach to the bug? </p> <p>Could you try upgrading to 1.8.8-wc1? There have been a number of fixes since 1.8.7 that may address your problem.</p> <p>By the way, where did the 1.8.7 release you are running come from, Whamcloud directly or some other supplier? Can you upgrade to a Whamcloud release or is that path blocked for you?</p> <p>What is LU-XX affect this issue? really need detailed information if this ticket is duplicated or fixed in 1.8.8.<br/> And, yes, vanilla 1.8.7-wc1 is running.</p> <p>Currently suspect <a href="https://jira.whamcloud.com/browse/LU-952" title="Hung thread with HIGH OSS load" class="issue-link" data-issue-key="LU-952"><del>LU-952</del></a> as being related to your issue. Don't know for sure yet.</p> <p>we are monitoring all storage devices (not only health check, but latency, etc), but there are not much differences from normal. <br/> btw, I forgot an thing, we already applied patch for <a href="https://jira.whamcloud.com/browse/LU-952" title="Hung thread with HIGH OSS load" class="issue-link" data-issue-key="LU-952"><del>LU-952</del></a> to 1.8.7-wc1 at this site, <a href="https://jira.whamcloud.com/browse/LU-952" title="Hung thread with HIGH OSS load" class="issue-link" data-issue-key="LU-952"><del>LU-952</del></a> should be fixed in this site.</p> <p>Bob, what's progress on this?<br/> There was no storage side issues and patches for <a href="https://jira.whamcloud.com/browse/LU-952" title="Hung thread with HIGH OSS load" class="issue-link" data-issue-key="LU-952"><del>LU-952</del></a> were already applied on top of lustre-1.8.7-wc1.</p> <p>There hasn't been much progress. Attempts to reproduce haven't succeeded. We don't yet have a good theory about how this might happen.</p> <p>Have more instances happened? Were you able to gather the stack dump of all threads Cliff asked for 6/29?</p> <p>Today, we had same problem on few OSSs and some OSTs's status on the some clients due to refusing the connection by still busy with XX active RPCs. What's RPC waiting and where?<br/> I uploaded these clients and OSS's debug files and syslog on /uploads/<a href="https://jira.whamcloud.com/browse/LU-1578" title="ll_ost_io_xx thread hangs and OST status get into down" class="issue-link" data-issue-key="LU-1578"><del>LU-1578</del></a>.</p> <p>This happens very often here and critical. Please investigate deeply.</p> <p>Bob, </p> <p>Any progress on this? the cusotmer is waiting for responses. thanks!</p> <p>Hi, Ihara</p> <p>Is it possible to get the full stack trace of the OSS whe the "ll_ost_io_xx thread hangs" happen? that'll be easier for us to figure out if there is deadlock. Thanks.</p> <p>BTW: I found there is LBUG in the messages, it looks like <a href="https://jira.whamcloud.com/browse/LU-1563" title="lustre_quota.h:326:lqs_putref() LBUG" class="issue-link" data-issue-key="LU-1563"><del>LU-1563</del></a>, did you tried the fix from <a href="https://jira.whamcloud.com/browse/LU-1563" title="lustre_quota.h:326:lqs_putref() LBUG" class="issue-link" data-issue-key="LU-1563"><del>LU-1563</del></a>?</p> <p>Hi Niu,</p> <p>The patch for <a href="https://jira.whamcloud.com/browse/LU-1563" title="lustre_quota.h:326:lqs_putref() LBUG" class="issue-link" data-issue-key="LU-1563"><del>LU-1563</del></a> is not applied yet on this site. The next maintenance window on the next mask, we will apply these patches. So, we are still sometimes seeing this problem.</p> <p>Let me check we could get stack trace when ll_ost_io threads hangs.</p> <p>Niu, </p> <p>we got a similar situation on the another client, but it's ll_mdt_xx haning and MDT was down status. (got still busy with XX active RPCs) Should I file it as well? that situation, we could get stack traces when the problem happens.</p> <p>Yes, please get the full stack trace. I think it should be different problem, let's check the stack trace first. Thanks.</p> <p>filed nmd01.stack on uploads/<a href="https://jira.whamcloud.com/browse/LU-1578" title="ll_ost_io_xx thread hangs and OST status get into down" class="issue-link" data-issue-key="LU-1578"><del>LU-1578</del></a>.</p> <p>thank you!</p> <p>I think it's <a href="https://jira.whamcloud.com/browse/LU-1563" title="lustre_quota.h:326:lqs_putref() LBUG" class="issue-link" data-issue-key="LU-1563"><del>LU-1563</del></a>, a mdt thread hit LBUG in in quota_pending_commit(), which is holding ldlm lock and blocked other operations.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">ll_mdt_34 D ffff810021004420 0 7589 1 7590 7588 (L-TLB) ffff8105fc56f720 0000000000000046 0000000000000000 0000000000000286 ffff8105fc56f6e0 0000000000000009 ffff8105f72a07e0 ffffffff80316b60 00023eb511a49dac 000000000000109f ffff8105f72a09c8 000000008003bbca Call Trace: [&lt;ffffffff88767c26&gt;] :libcfs:lbug_with_loc+0xc6/0xd0 [&lt;ffffffff8876ffc0&gt;] :libcfs:tracefile_init+0x0/0x110 [&lt;ffffffff889c02ef&gt;] :lquota:quota_pending_commit+0x41f/0x5b0 [&lt;ffffffff88aff5d1&gt;] :jbd2:jbd2_journal_stop+0x209/0x215 [&lt;ffffffff88b40317&gt;] :ldiskfs:__ldiskfs_journal_stop+0x67/0x80 [&lt;ffffffff88c24d8b&gt;] :mds:mds_open+0x30eb/0x3758 [&lt;ffffffff888d1491&gt;] :ptlrpc:lustre_swab_buf+0x81/0x170 [&lt;ffffffff8002cc91&gt;] mntput_no_expire+0x19/0x89 [&lt;ffffffff8000d5a8&gt;] dput+0x2c/0x113 [&lt;ffffffff88bfb0d5&gt;] :mds:mds_reint_rec+0x365/0x550 [&lt;ffffffff88c259be&gt;] :mds:mds_update_unpack+0x1fe/0x280 [&lt;ffffffff88bededa&gt;] :mds:mds_reint+0x35a/0x420 [&lt;ffffffff88becdea&gt;] :mds:fixup_handle_for_resent_req+0x5a/0x2c0 [&lt;ffffffff88bf7c1c&gt;] :mds:mds_intent_policy+0x4ac/0xc20 [&lt;ffffffff88892270&gt;] :ptlrpc:ldlm_resource_putref_internal+0x230/0x460 [&lt;ffffffff8888feb6&gt;] :ptlrpc:ldlm_lock_enqueue+0x186/0xb20 [&lt;ffffffff8888c7fd&gt;] :ptlrpc:ldlm_lock_create+0x9bd/0x9f0 [&lt;ffffffff888b4870&gt;] :ptlrpc:ldlm_server_blocking_ast+0x0/0x83d [&lt;ffffffff888b1b39&gt;] :ptlrpc:ldlm_handle_enqueue+0xc09/0x1210 [&lt;ffffffff88bf6b40&gt;] :mds:mds_handle+0x40e0/0x4d10 [&lt;ffffffff8008e1a4&gt;] enqueue_task+0x41/0x56 [&lt;ffffffff8008e20f&gt;] __activate_task+0x56/0x6d [&lt;ffffffff888d5d55&gt;] :ptlrpc:lustre_msg_get_conn_cnt+0x35/0xf0 [&lt;ffffffff888df6d9&gt;] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00 </pre> </div></div> <p>OK, thanks. <br/> I will post stack trace when ll_ost_io_xx thread hangs on the next time.</p> <p>Niu, </p> <p>when we evicted on the client manually, it connected well well.<br/> But, we see same problem on the another node sometimes. If we evicted, is the holding lock released? or just connect with different thread and still keeping the lock? this is why client can connect? <br/> But, the root cause of LBUG can't fix, this is why we saw same issue on the another client. is this correct? </p> <p>Hello Niu,</p> <p>I just upload today's MDS's syslog on uploads/<a href="https://jira.whamcloud.com/browse/LU-1578" title="ll_ost_io_xx thread hangs and OST status get into down" class="issue-link" data-issue-key="LU-1578"><del>LU-1578</del></a>/messages.nmd01. We hit same problem today, but not only single client, but saw Down states on the many clients. As far as we see the log files, it seems to be same issue. <br/> Howerver, question is that as I asked before, what exctly happens if MDS does LBUG and ll_mdt_xx hangs?<br/> Please advise!</p> <p>yes, I checked log, it seems same problem (<a href="https://jira.whamcloud.com/browse/LU-1563" title="lustre_quota.h:326:lqs_putref() LBUG" class="issue-link" data-issue-key="LU-1563"><del>LU-1563</del></a>). If the MDS hit the LBUG, then MDS will not responsive and the whole cluster will not functional.</p> Development Rank 1|hzvgsn: Rank (Obsolete) 6370 Severity