Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-793] Reconnections should not be refused when there is a request in progress from this client. https://jira.whamcloud.com/browse/LU-793 Lustre <p>While originally this was a useful workaround, it created a lot of other unintended problems.</p> <p>This code must be disabled and instead we just should disable handling several duplicate requests at the same time.</p> LU-793

Reconnections should not be refused when there is a request in progress from this client.

Bug Critical Resolved Fixed Mikhail Pershin Oleg Drokin JL Tue, 25 Oct 2011 11:41:16 +0000 Mon, 3 Jun 2019 17:52:49 +0000 Tue, 18 Feb 2014 22:03:19 +0000 Lustre 2.1.0 Lustre 2.2.0 Lustre 2.4.0 Lustre 1.8.6 Lustre 2.6.0 0 24 <p>Vitaly shared that a patch is under review at <a href="http://review.whamcloud.com/#change,1616" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1616</a></p> <p>Hi Oleg,</p> <p>since we are interested in this patch could you provide us with the next information:<br/> is <a href="http://review.whamcloud.com/#change,1616" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1616</a> a final version or you have a plan to add something there (if yes - what are your time estimates for such action)?</p> <p>thank you.</p> <p>Well, I cannot predict how well the inspections will pass.<br/> I think it's fine, though a bit of real testing is needed to ensure clients will not end up bombarding the servers with repeated resent requests too often (for the requests that were stuck on the server).</p> <p>Also see my comment for patchset 3 on Mar 13th.</p> <p>Hi Oleg.</p> <p>Is it possible to speed-up the review process somehow? Or we can do review in gerrit from our side since we are interested in this code?</p> <p>Thank you,<br/> Iurii Golovach</p> <p>In the Description for this bug, does "this was a useful workaround" refer to the <a href="http://review.whamcloud.com/#change,1616" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1616</a> patch?<br/> Is this Oleg's patch originally? If someone were to work on it, should they address Andreas' comments in the patch or try to follow the 2nd line of the Description?</p> <p>The workaround being referred to is the refusal of reconnection.</p> <p>The patch in change 1616 is originally mine, and I still work on it as I have time.<br/> But if you have someone with more time that can pick it up and finish it faster, that's cool too.</p> <p>The Andreas' comments are definitely valid and the request switching I think is needed, I was liking this idea from the tart, and now Andreas seems to like it too, so it needs to be done I guess.</p> <p><a href="http://review.whamcloud.com/#change,1616" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1616</a> needs rebasing.</p> <p>Long standing bug, duplicate of <a href="https://jira.whamcloud.com/browse/LU-7" title="Reconnect server->client connection" class="issue-link" data-issue-key="LU-7"><del>LU-7</del></a>, which in turn was a duplicate of years-old bugzilla bug.</p> <p>This scheme shows cases when we drop new packet and when we process it.</p> <p>Mike, <br/> Could you please have a look at this one?<br/> Thank you!</p> <p><a href="http://review.whamcloud.com/#change,4960" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4960</a></p> <p>I've just updated old patch to be compatible with new master code. It provides the same functionality as old one so far, without any attempts to use new reply buffers while answering on older request.</p> <p>I am still experiencing this issue on Hyperion, especially using mdtest file-per-process. In the last test, starting seeing these timeouts prior to the actual error.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Jan 14 22:35:08 hyperion-rst6 kernel: LNet: Service thread pid 11870 was inactive <span class="code-keyword">for</span> 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace <span class="code-keyword">for</span> debugging purposes: Jan 14 22:35:08 hyperion-rst6 kernel: Pid: 11870, comm: mdt03_001 Jan 14 22:35:08 hyperion-rst6 kernel: Jan 14 22:35:08 hyperion-rst6 kernel: Call Trace: Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffffa06cf072>] start_this_handle+0x282/0x500 [jbd2] Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffff81092170>] ? autoremove_wake_function+0x0/0x40 Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffffa06cf4f0>] jbd2_journal_start+0xd0/0x110 [jbd2] Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffffa072bab8>] ldiskfs_journal_start_sb+0x58/0x90 [ldiskfs] Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffffa072be7c>] ldiskfs_dquot_initialize+0x4c/0xc0 [ldiskfs] Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffffa0708590>] ? ldiskfs_delete_inode+0x0/0x250 [ldiskfs] Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffff811968a3>] generic_delete_inode+0x173/0x1d0 Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffff81196965>] generic_drop_inode+0x65/0x80 Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffff811957b2>] iput+0x62/0x70 Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffffa0e3fc24>] osd_object_delete+0x1d4/0x2f0 [osd_ldiskfs] Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffffa07e6059>] lu_object_free+0x89/0x1a0 [obdclass] Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffffa03a5351>] ? libcfs_debug_msg+0x41/0x50 [libcfs] Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffffa03aa5d2>] ? cfs_hash_bd_from_key+0x42/0xd0 [libcfs] Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffffa07e690d>] lu_object_put+0xad/0x320 [obdclass] Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffffa0f2c69d>] mdt_object_unlock_put+0x3d/0x110 [mdt] Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffffa0f4e0d7>] mdt_reint_unlink+0x637/0x850 [mdt] Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffffa0965fce>] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffffa0f48cf1>] mdt_reint_rec+0x41/0xe0 [mdt] Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffffa0f423a3>] mdt_reint_internal+0x4e3/0x7d0 [mdt] Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffffa0f426d4>] mdt_reint+0x44/0xe0 [mdt] Jan 14 22:35:08 hyperion-rst6 kernel: [<ffffffffa0f31a72>] mdt_handle_common+0x8e2/0x1680 [mdt] Jan 14 22:35:09 hyperion-rst6 kernel: [<ffffffffa0f67a25>] mds_regular_handle+0x15/0x20 [mdt] Jan 14 22:35:09 hyperion-rst6 kernel: [<ffffffffa09761ec>] ptlrpc_server_handle_request+0x41c/0xdf0 [ptlrpc] Jan 14 22:35:09 hyperion-rst6 kernel: [<ffffffffa039564e>] ? cfs_timer_arm+0xe/0x10 [libcfs] Jan 14 22:35:09 hyperion-rst6 kernel: [<ffffffffa096d669>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jan 14 22:35:09 hyperion-rst6 kernel: [<ffffffff81053463>] ? __wake_up+0x53/0x70 Jan 14 22:35:09 hyperion-rst6 kernel: [<ffffffffa0977776>] ptlrpc_main+0xbb6/0x1950 [ptlrpc] Jan 14 22:35:09 hyperion-rst6 kernel: [<ffffffffa0976bc0>] ? ptlrpc_main+0x0/0x1950 [ptlrpc] Jan 14 22:35:09 hyperion-rst6 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20 Jan 14 22:35:09 hyperion-rst6 kernel: [<ffffffffa0976bc0>] ? ptlrpc_main+0x0/0x1950 [ptlrpc] Jan 14 22:35:09 hyperion-rst6 kernel: [<ffffffffa0976bc0>] ? ptlrpc_main+0x0/0x1950 [ptlrpc] Jan 14 22:35:09 hyperion-rst6 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20 </pre> </div></div> <p>Here is a call trace from a client, while the MDS is reporting the RPC issue:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2013-01-15 09:45:57 INFO: task mdtest:6078 blocked <span class="code-keyword">for</span> more than 120 seconds. 2013-01-15 09:45:57 <span class="code-quote">"echo 0 > /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. 2013-01-15 09:45:57 mdtest D 0000000000000003 0 6078 6074 0x00000000 2013-01-15 09:45:57 ffff88017ede7c58 0000000000000086 0000000000000000 0000000000000086 2013-01-15 09:45:57 ffff88030aa62a20 ffff8801c5851960 ffffffff81aa5740 0000000000000286 2013-01-15 09:45:57 ffff8801bba69ab8 ffff88017ede7fd8 000000000000fb88 ffff8801bba69ab8 2013-01-15 09:45:57 Call Trace: 2013-01-15 09:45:57 [<ffffffff814ff6fe>] __mutex_lock_slowpath+0x13e/0x180 2013-01-15 09:45:57 [<ffffffff814ff59b>] mutex_lock+0x2b/0x50 2013-01-15 09:45:57 [<ffffffffa07eb80c>] mdc_reint+0x3c/0x3b0 [mdc] 2013-01-15 09:45:57 [<ffffffffa07ec880>] mdc_unlink+0x1b0/0x500 [mdc] 2013-01-15 09:45:57 [<ffffffffa0a5feb9>] lmv_unlink+0x199/0x7e0 [lmv] 2013-01-15 09:45:57 [<ffffffffa097dba6>] ll_unlink+0x176/0x670 [lustre] 2013-01-15 09:45:57 [<ffffffff8118923f>] vfs_unlink+0x9f/0xe0 2013-01-15 09:45:57 [<ffffffff81187f8a>] ? lookup_hash+0x3a/0x50 2013-01-15 09:45:57 [<ffffffff8118b773>] do_unlinkat+0x183/0x1c0 2013-01-15 09:45:57 [<ffffffff8119a960>] ? mntput_no_expire+0x30/0x110 2013-01-15 09:45:57 [<ffffffff8100c5b5>] ? math_state_restore+0x45/0x60 2013-01-15 09:45:57 [<ffffffff8118b7c6>] sys_unlink+0x16/0x20 2013-01-15 09:45:57 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Cliff, the refused-reconnection problem is the after-effect of whatever problem happened, and there are many ways it can be triggered. You should open a separate bug report for the initial problem of the server going unresponsive, or whatever happened.</p> <p>Hi gents, </p> <p>here is a patch version for this issue from our team:<br/> <a href="http://review.whamcloud.com/#change,5054" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5054</a> </p> <p>Since Andreas recommended to merge Mikhail patch and mine I updated <a href="http://review.whamcloud.com/#change,4960" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4960</a> with improvement request.</p> <p>Change #4960 failed autotest. Do you just need to resubmit?</p> <p>retriggered</p> <p>This patch doesn't work properly with bulk resends because bulks use new XID always, even for RESENT case. Therefore we cannot match original request that might be processed on server at the same time. It is not clear how to solve this in simple way in context of this patch, looks like patch should be reworked and will be more complex, e.g. we might need to change protocol and store original XID in bulk along with new one.<br/> Btw, the comment about bulk XID and replated code:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">int</span> ptlrpc_register_bulk(struct ptlrpc_request *req) { ... /* An XID is only used <span class="code-keyword">for</span> a single request from the client. * For retried bulk transfers, a <span class="code-keyword">new</span> XID will be allocated in * in ptlrpc_check_set() <span class="code-keyword">if</span> it needs to be resent, so it is not * using the same RDMA match bits after an error. */ ... } and void ptlrpc_resend_req(struct ptlrpc_request *req) { DEBUG_REQ(D_HA, req, <span class="code-quote">"going to resend"</span>); lustre_msg_set_handle(req->rq_reqmsg, &(struct lustre_handle){ 0 }); req->rq_status = -EAGAIN; spin_lock(&req->rq_lock); req->rq_resend = 1; req->rq_net_err = 0; req->rq_timedout = 0; <span class="code-keyword">if</span> (req->rq_bulk) { __u64 old_xid = req->rq_xid; <span class="code-comment">/* ensure previous bulk fails */</span> req->rq_xid = ptlrpc_next_xid(); CDEBUG(D_HA, <span class="code-quote">"resend bulk old x"</span>LPU64<span class="code-quote">" <span class="code-keyword">new</span> x"</span>LPU64<span class="code-quote">"\n"</span>, old_xid, req->rq_xid); } ptlrpc_client_wake_req(req); spin_unlock(&req->rq_lock); } </pre> </div></div> <p>patch was refreshed <a href="http://review.whamcloud.com/#/c/4960/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/4960/</a>. I doesn't handle bulk request for now, this will be solved in the following patch.</p> <p>This is still a support priority, but we need to finalize the fix before we consider it for inclusion in a release</p> <p>Patch is refreshed, now it handles all request including bulk. That requires protocol changes and works only with new clients, old clients will be handled as before - returning -EBUSY on connect request if there is another request in processing </p> <p>Patch was updated again and I hope it addresses all cases including bulk requests. It doesn't change protocol now. Cris, I expect this patch will be landed soon to the master, can you try it and see how it helps? </p> <p>Unfortunately, no, we won't be able to find out it it helps for some time. We are doing a major upgrade to Lustre 2.4 over the next 2-3 weeks on the SCF machines, but this patch missed the window for inclusion in that distribution. We will have to work it into the pipeline for the next upgrade.</p> <p>Can you explain a little more about what the patch will do? I see "Bulk requests are aborted upon reconnection by comparing connection count of request and export." in the patch comment. What happens when the bulk requests are aborted? Will the client transparently resend them?</p> <p>Also, what happens if there is more than one rpc outstanding? Is the client able to reconnect in that case?</p> <p>The behavior is almost the same as before for bulks. Currently all pending bulks are aborted if new reconnect arrived from client and reconnect is refused with -EBUSY until there will be no more active requests, this is how it is handled before patch. With this patch we accept reconnect even if there are active requests and all bulks from last connection are aborted. Basically it is the same behavior as before, but now the connection count is checked instead of specific flag.<br/> Client will resend aborted bulks, yes. Also the client is able to reconnect always, but resent bulk may stuck on original bulk until it is aborted.</p> <p>I think this patch introduced a timeout in conf-sanity (<a href="https://jira.whamcloud.com/browse/LU-4349" title="conf-sanity test_47: test failed to respond and timed out" class="issue-link" data-issue-key="LU-4349"><del>LU-4349</del></a>), so that needs to be addressed before this patch is introduced into the 2.4 release. </p> <p>Mike</p> <p>Could you please clarify what LLNL would need to port in order to use this fix on b2_4?</p> <p>Thanks</p> <p>Peter</p> <p>Peter, the <a href="https://jira.whamcloud.com/browse/LU-793" title="Reconnections should not be refused when there is a request in progress from this client." class="issue-link" data-issue-key="LU-793"><del>LU-793</del></a> and <a href="https://jira.whamcloud.com/browse/LU-4349" title="conf-sanity test_47: test failed to respond and timed out" class="issue-link" data-issue-key="LU-4349"><del>LU-4349</del></a> are needed.</p> <p>Backports to b2_4</p> <p><a href="http://review.whamcloud.com/#/c/9209/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9209/</a><br/> <a href="http://review.whamcloud.com/#/c/9210/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9210/</a><br/> <a href="http://review.whamcloud.com/#/c/9211/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9211/</a></p> <p>Hi </p> <p>We are seeing similar issues on Lustre 2.1.6 release, so is this patch compatible with 2.1.x release and if yes then can this be backported to branch b2_1. </p> <p>Thank You,<br/> Manish</p> <p>The patch</p> <p><a href="http://review.whamcloud.com/#/c/9211/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9211/</a></p> <p>was determined to not be needed for b2_4.</p> Duplicate LU-7 Related LU-2621 LU-4349 LU-4458 LU-7 LU-4359 LU-2429 LU-4480 Development Rank 1|hzve3b: Rank (Obsolete) 5914 Severity