https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14318 Lustre <p>Currently, the recovery time could be extended to several hours if the recovery<br/> between MDT is stuck or some other reasons, which can cause the cluster to be<br/> unavailable for long time if no one is noticed and use "lctl" to abort the recovery,<br/> then there is some option to be needed to add to limit the overall recovery time. </p> LU-14318

Add the option to limit the overall recovery time

Bug Minor Open Unresolved Hongchao Zhang Hongchao Zhang Mon, 11 Jan 2021 07:34:54 +0000 Tue, 14 Sep 2021 12:07:06 +0000 0 8 <p>Hongchao Zhang (hongchao@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41171" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41171</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14318" title="Add the option to limit the overall recovery time" class="issue-link" data-issue-key="LU-14318">LU-14318</a> ldlm: add recovery time limit<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 2<br/> Commit: 3f8c53ec33e518658d91d7ff3d07e13f5ad43327</p> <p>Currently, if some other MDT can't be connected during the MDT recovery, the recovery process can be extended to<br/> last several hours (maybe forever if not aborted by the lctl) in "check_for_recovery_ready"</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>static int check_for_recovery_ready(struct lu_target *lut) { struct obd_device *obd = lut-&gt;lut_obd; unsigned int clnts = atomic_read(&amp;obd-&gt;obd_connected_clients); CDEBUG(D_HA, "connected %d stale %d max_recoverable_clients %d abort %d expired %d\n", clnts, obd-&gt;obd_stale_clients, atomic_read(&amp;obd-&gt;obd_max_recoverable_clients), obd-&gt;obd_abort_recovery, obd-&gt;obd_recovery_expired); if (!obd-&gt;obd_abort_recovery &amp;&amp; !obd-&gt;obd_recovery_expired) { LASSERT(clnts &lt;= atomic_read(&amp;obd-&gt;obd_max_recoverable_clients)); if (clnts + obd-&gt;obd_stale_clients &lt; atomic_read(&amp;obd-&gt;obd_max_recoverable_clients)) return 0; } if (!obd-&gt;obd_abort_recov_mdt &amp;&amp; lut-&gt;lut_tdtd != NULL) { if (!lut-&gt;lut_tdtd-&gt;tdtd_replay_ready &amp;&amp; !obd-&gt;obd_abort_recovery &amp;&amp; !obd-&gt;obd_stopping) { /* * Let's extend recovery timer, in case the recovery * timer expired, and some clients got evicted */ extend_recovery_timer(obd, obd-&gt;obd_recovery_timeout, &lt;--- the recovery will be extended even if the timer expired true); CDEBUG(D_HA, "%s update recovery is not ready, extend recovery %d\n", obd-&gt;obd_name, obd-&gt;obd_recovery_timeout); return 0; } } return 1; } </pre> </div></div> <p>Hongchao, is this patch only needed for the case when the remote MDT is not available at all, or is there some other problem like <a href="https://jira.whamcloud.com/browse/LU-13608" title="MDT stuck in WAITING, abort_recov stuck too" class="issue-link" data-issue-key="LU-13608"><del>LU-13608</del></a> that is causing recovery to be stuck for a long time even when all of the MDTs are available?</p> <p>Rather than timing out recovery for a remote MDT completely, it would probably be better to keep the recovery <b>for that MDT</b> pending until the MDT is available again, and then do the remote recovery when the MDTs reconnect. That might only be a small (or no) difference in the normal case when all of the MDTs are available at mount, but I think this may give a very important improvement when some MDTs are unavailable.</p> <p>The big improvement would be if MDT0000 and other MDTs are available at restart time, it would complete recovery with all those MDTs quickly, and not block access to files/directories that are on available MDTs. It would allow most client access to work, and only remote/striped directories would be blocked and/or time out (allow CTRL-C for client processes). This would be better for users, if they are mostly using remote directories for subtrees of the filesystem, since only the subtrees on the missing MDTs would be inaccessible.</p> <p>Eventually, having mirrored entries for <tt>ROOT/</tt> on several/all MDTs could allow the filesystem to be accessible even if MDT0000 is unavailable, but that is definitely a separate project.</p> <p>HI,<br/> this patch is meant to solve the recovery hung caused by some unresponsive MDT during recovery, the patch in <a href="https://jira.whamcloud.com/browse/LU-13608" title="MDT stuck in WAITING, abort_recov stuck too" class="issue-link" data-issue-key="LU-13608"><del>LU-13608</del></a> is meant to fix<br/> the "lctl abort_recovery" hung when it was used to abort the recovery.</p> <p>Okay, I will create another patch to allow the recovery to continue if some of MDTs other than MDT0000 is unavailable during recovery.<br/> Thanks!</p> <p>Hongchao Zhang (hongchao@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41424" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41424</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14318" title="Add the option to limit the overall recovery time" class="issue-link" data-issue-key="LU-14318">LU-14318</a> ldlm: don't wait other MDT forever<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4c2b2b26bd29b1a8f522fbf705a3d145cb06eb9f</p> Related LU-13608 Development Rank 1|i01j1r: Rank (Obsolete) 9223372036854775807 Severity