https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4582 Lustre <p>Following are steps to reproduce the issue reliably:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">1. adjust obd_timeout from <span class="code-keyword">default</span> 100 to 300 lctl conf_param &lt;fsname&gt;.sys.timeout=300 2. mount and umount the client mount - lustre &lt;primary MGS ip&gt;:&lt;secondary MGS ip&gt;:/&lt;fsname&gt; /mnt/lustre 3. failover the MGS node to the secondary 4. mount the client again using the same command as in step 2 </pre> </div></div> <p>Then step 4 will fail with EIO.</p> LU-4582

After failing over Lustre MGS node to the secondary, client mount fails with -5

Bug Major Resolved Fixed Cliff White Cheng Shao patch Tue, 4 Feb 2014 20:12:19 +0000 Wed, 14 Mar 2018 13:53:33 +0000 Mon, 23 Jun 2014 15:59:08 +0000 Lustre 2.1.0 Lustre 2.2.0 Lustre 2.3.0 Lustre 2.4.0 0 11 <p>Here is the timeline of events according to the Lustre debug log. The beginning num is relative to the start of the mount op.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">+0 Client sent MGS_CONNECT req to primary MGS node with timeout set to (obd_timeout/20 + adaptive_timeout), which was 20 seconds in our test <span class="code-keyword">case</span>. +0 Client sent LDLM_ENQUEUE req to MGS node with rq_delay_limit set to 5 seconds. This is <span class="code-keyword">for</span> sptlrpc. The send will be delayed because the <span class="code-keyword">import</span> is still in connecting state. +5 The above req failed after the delayed sent expired. But <span class="code-keyword">this</span> is not fatal. +5 Client sent another LDLM_ENQUEUE req to MGS node with rq_delay_limit set to MGC_ENQUEUE_LIMIT, which is hard coded to 50 seconds. +20 MGS_CONNECT timed out. +55 The second LDLM_ENQUEUE req failed after the delayed sent expired. This will fail the whole client mount with error -5. </pre> </div></div> <p>The problem here is that, after the MGS_CONNECT failed to connect to the primary MGS, it didn't get a chance to connect to the secondary before the mount fails. We know that selecting a different MGS node is triggered by the pinger, which works at (obd_timeout/4) interval. Since we increased obd_timeout to 300, the interval became 75 seconds now. So the connection to the secondary will not happened prior to failure of the second LDLM_ENQUEUE req.</p> <p>The solution we proposed here is to redefine MGS_ENQUEUE_LIMIT as relative to obd_timeout, instead of a hard-coded value. By doing that, the second LDLM_ENQUEUE will wait long enough to go through after the connection to the secondary MGS node is established.</p> <p>Cheng, have you uploaded your patch to the whamcloud gerrit review site? If so, please post a link here. Thanks.</p> <p>Patch is up for review at <a href="http://review.whamcloud.com/#/c/9141/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9141/</a></p> <p>I wonder why is your patch against b2_5 and not master? Is master not affected?<br/> We generally prefer to land things to master first.</p> <p>Master is definitely affected as well. Will abandon this patch and submit a new one against master.</p> <p>New patch is at <a href="http://review.whamcloud.com/#/c/9217/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9217/</a>.</p> <p>unfortunately Cheng left Xyratex, but we still need to get this landed.</p> <p>Could someone review Cheng's patch?</p> <p>Reviewers have been assigned.</p> <p>It has been a while since there has been any activity on this bug. Who is reviewing Cheng's patch?</p> <p>Very sorry about the delay, will investigate</p> <p>Thanks, Cliff. <a href="http://review.whamcloud.com/#/c/9217/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9217/</a> has landed.</p> <p>Is it okay to close this isse?</p> <p>Yes.</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13718" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13718</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4582" title="After failing over Lustre MGS node to the secondary, client mount fails with -5" class="issue-link" data-issue-key="LU-4582"><del>LU-4582</del></a> mgc: replace hard-coded MGC_ENQUEUE_LIMIT value<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: 6c815c12a2d6dfabf55b43563b4f1062c123db99</p> Duplicate Related Development Rank 1|hzwe9z: Rank (Obsolete) 12524 Severity