https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1520 Lustre <p>An client (cluster1) failed connection to MDS and recovered, but failed connection again by some reasons.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jun 11 11:28:45 cluster1 kernel: Lustre: 30906:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1402727385081248 sent from lustre-MDT0000-mdc- ffff880c06249800 to NID 192.168.3.45@o2ib 995s ago has timed out (995s prior to deadline). Jun 11 11:28:45 cluster1 kernel: req@ffff880293aaf800 x1402727385081248/t0 o101-&gt;lustre-MDT0000_UUID@192.168.3.45@o2ib:12/10 lens 560/1616 e 3 to 1 dl 1339381725 ref 1 fl Rpc:/0/0 rc 0/0 </pre> </div></div> <p>few hours later, call traces showed up on another client (cluster3).</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jun 11 15:03:10 cluster3 kernel: Call Trace: Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffff814dbcd5&gt;] schedule_timeout+0x215/0x2e0 Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffffa086808d&gt;] ? lustre_msg_early_size+0x6d/0x70 [ptlrpc] Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffffa0996244&gt;] ? mdc_intent_open_pack+0x364/0x530 [mdc] Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffff8115a1ae&gt;] ? cache_alloc_refill+0x9e/0x240 Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffff814dcbf2&gt;] __down+0x72/0xb0 Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffff81093f61&gt;] down+0x41/0x50 Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffffa0997173&gt;] mdc_enqueue+0x283/0xa20 [mdc] Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffffa081fbef&gt;] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffffa081fbef&gt;] ? __ldlm_handle2lock+0x9f/0x3d0 [ptlrpc] Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffffa09987d2&gt;] mdc_intent_lock+0x102/0x440 [mdc] Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffffa0853e90&gt;] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc] Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffffa0a431a5&gt;] ? ll_lookup_it+0x405/0x870 [lustre] Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffffa0a40490&gt;] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffffa0a402ee&gt;] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffffa0a40490&gt;] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffffa083f770&gt;] ? ldlm_completion_ast+0x0/0x8a0 [ptlrpc] Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffffa0a402ee&gt;] ? ll_prepare_mdc_op_data+0xbe/0x120 [lustre] Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffffa0a430b5&gt;] ll_lookup_it+0x315/0x870 [lustre] Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffffa0a40490&gt;] ? ll_mdc_blocking_ast+0x0/0x5f0 [lustre] Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffffa06f97c1&gt;] ? cfs_alloc+0x91/0xf0 [libcfs] Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffffa0a43ac8&gt;] ll_lookup_nd+0x88/0x470 [lustre] Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffff8118ad4e&gt;] ? d_alloc+0x13e/0x1b0 Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffff81181c02&gt;] __lookup_hash+0x102/0x160 Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffff81181d3a&gt;] lookup_hash+0x3a/0x50 Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffff81182768&gt;] do_filp_open+0x2c8/0xd90 Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffff8118f1e2&gt;] ? alloc_fd+0x92/0x160 Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffff8116f989&gt;] do_sys_open+0x69/0x140 Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffff8116faa0&gt;] sys_open+0x20/0x30 Jun 11 15:03:10 cluster3 kernel: [&lt;ffffffff8100b172&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>I will upload the all log files soon.</p> LU-1520

client fails MDS connection and stack threads on another client

Bug Minor Resolved Won't Fix Hongchao Zhang Shuichi Ihara Thu, 14 Jun 2012 03:11:18 +0000 Fri, 29 Apr 2016 00:24:59 +0000 Fri, 29 Apr 2016 00:24:59 +0000 Lustre 1.8.7 Lustre 1.8.9 0 7 <p>all log files on /uploads/<a href="https://jira.whamcloud.com/browse/LU-1520" title="client fails MDS connection and stack threads on another client" class="issue-link" data-issue-key="LU-1520"><del>LU-1520</del></a></p> <p>Thanks!</p> <p>Hi Hongchao</p> <p>Could you please look into this issue?</p> <p>Thanks</p> <p>Peter</p> <p>the call traces seen in cluster3 is caused for waiting the rpc_lock in mdc_enqueue, which is the result of the bad performance<br/> of mds(mds01), it runs low of memory and could be more damaged by ldlm_pools_shrink(BZ24419 or <a href="https://jira.whamcloud.com/browse/LU-607" title="port bz24419 (ldlm namespace lock contention during oom)" class="issue-link" data-issue-key="LU-607"><del>LU-607</del></a>?).</p> <p>Hongchao,</p> <p>It seems <a href="https://jira.whamcloud.com/browse/LU-607" title="port bz24419 (ldlm namespace lock contention during oom)" class="issue-link" data-issue-key="LU-607"><del>LU-607</del></a> was landed in b1_8 once, but the patches were reverted by Johann.<br/> <a href="http://git.whamcloud.com/?p=fs/lustre-release.git;a=commit;h=447794d5ebb71dbd39d7378944c3c9eeb230f8d0" class="external-link" target="_blank" rel="nofollow noopener">http://git.whamcloud.com/?p=fs/lustre-release.git;a=commit;h=447794d5ebb71dbd39d7378944c3c9eeb230f8d0</a></p> <p>Any reason why this reverted?<br/> Also, the patches for BZ24419 was not landed in -wc branch. </p> <p>Which patches worth to try?</p> <p>Please advise.</p> <p>Ihara</p> <p>Just read <a href="https://jira.whamcloud.com/browse/LU-607" title="port bz24419 (ldlm namespace lock contention during oom)" class="issue-link" data-issue-key="LU-607"><del>LU-607</del></a> again and it seems the patch for <a href="https://jira.whamcloud.com/browse/LU-607" title="port bz24419 (ldlm namespace lock contention during oom)" class="issue-link" data-issue-key="LU-607"><del>LU-607</del></a> introduces the regressions and reverted.<br/> So, this problem is not fixed even yet.. <br/> we are seeing couple of times similar situation in month. Please investigate to avoid this problem.. </p> <p>Hi, Ihara</p> <p>sorry for delayed response! <br/> the patch ported from BZ24419 is reverted for it causes a LASSERT. the patch tracked at BZ24419 is to improve<br/> the performance of shrinking of LDLM, which could mitigate this issue. will port and test the newest patch in BZ24419<br/> to check whether it can fix the issue.</p> <p>Hi Hongchao,<br/> do you mind if you can port patch to b1_8, please?</p> <p>Hi Ihara<br/> Okay, will port it to b1_8</p> <p>the patch is tracked at <a href="http://review.whamcloud.com/#change,3270" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3270</a></p> <p>Thanks!<br/> does it need to review by someone before we try to test this patches?</p> <p>Hongchao, we will be applying backported patches at the customer site, but before apply them, I wonder if someone could review it.. Please.. </p> <p>Ihara</p> <p>oh, yes, sorry!! I missed your previous comment, I'll do it right now.</p> <p>Hello, I was wondering what the status was of this patch. It appears that there were some suggested changes, were those ever done? Can we get this landed?</p> <p>Hi, the updated patch is under creation &amp; test, and will upload it soon</p> <p>the updated patch has been pushed, <a href="http://review.whamcloud.com/#change,3270" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3270</a></p> <p>Hello, can we get an update? It looks like it has two +1 reviews, can it be landed? Thanks, Kit</p> <p>Kit</p> <p>The b1_8 version is ready to land but the master version is still being worked <a href="http://review.whamcloud.com/#change,3859" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3859</a>. We try to land to master first so as to avoid deltas between 1.8.x and 2.x arising.</p> <p>Peter</p> <p>Ah ok, thanks.</p> <p>Peter, <br/> Thanks for clarifying on this, but nobody listed on <a href="http://review.whamcloud.com/#change,3859" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3859</a> as reviewer.<br/> I really want this patches to review and land, land into b1_8 as well. otherwise we need to apply patches to top of b1_8 to solve current customer issue..</p> <p>It looks like the Maloo testing hit <a href="https://jira.whamcloud.com/browse/LU-479" title="Test failure on test suite sanity, subtest test_124a" class="issue-link" data-issue-key="LU-479"><del>LU-479</del></a>?</p> <p>there is a bug in the previous patch, which causes sanity subtest 124a failed.<br/> the updated patch has been pushed to Gerrit</p> <p>Hi Hongchao, <br/> Will the 1.8 version need to be updated too?</p> <p>no, the 1.8.x version has no such problem.</p> <p>any updates on the master port of this patch? We are carrying the 1.8.x version in our 1.8.9 build, but it would be nice to integrate it in the Intel version. The last update on the changeset was in October.</p> <p>Really old ticket. Peter we should close this as well.</p> <p>Probably we should defer to DDN on that?</p> <p>Hello Ihara,</p> <p>Do you want us to keep this open, or can we go ahead and mark it as resolved/won't fix?</p> <p>Thanks,<br/> ~ jfc.</p> <p>Hello Ihara,</p> <p>We have marked this as Resolved/Won't fix.</p> <p>Thanks,<br/> ~ jfc.</p> Related LU-6529 Development Rank 1|hzvncf: Rank (Obsolete) 7592 Severity