https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1352 Lustre <p>A production MDS in recovery seemed to restart the recovery timer more than we would expect. Here is a paraphrased transcript of the console log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Apr 25 12:21:28 roc-mds1 kernel: Lustre: ls5-MDT0000: Starting recovery timer for 5:00 Apr 25 12:21:28 roc-mds1 kernel: Lustre: ls5-MDT0000: Starting recovery timer for 5:00 Apr 25 12:36:03 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports Apr 25 12:36:03 roc-mds1 kernel: Lustre: ls5-MDT0000: Denying connection for new client ..., waiting for 193 clients in recovery for 0:24 (repeats once) Apr 25 12:36:29 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports Apr 25 12:36:29 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports Apr 25 12:36:30 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports Apr 25 12:36:32 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports Apr 25 12:36:33 roc-mds1 kernel: Lustre: ls5-MDT0000: Starting recovery timer for 15:00 Apr 25 12:36:33 roc-mds1 kernel: Lustre: ls4-MDT0000: Denying connection for new client ..., waiting for 100 clients in recovery for 15:00 (repeats about ten times over ten minutes) Apr 25 12:51:33 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports Apr 25 12:51:34 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports Apr 25 12:51:35 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports Apr 25 12:51:35 roc-mds1 kernel: Lustre: ls5-MDT0000: Starting recovery timer for 15:00 Apr 25 12:55:20 roc-mds1 kernel: Lustre: ls5-MDT0000: Denying connection for new client ..., waiting for 100 clients in recovery for 11:15 Apr 25 13:05:20 roc-mds1 kernel: Lustre: ls5-MDT0000: Denying connection for new client ..., waiting for 100 clients in recovery for 1:15 Apr 25 13:06:35 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports Apr 25 13:06:35 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports Apr 25 13:06:36 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports Apr 25 13:06:37 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports Apr 25 13:06:39 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports Apr 25 13:06:39 roc-mds1 kernel: Lustre: ls5-MDT0000: Starting recovery timer for 15:00 Apr 25 13:15:20 roc-mds1 kernel: Lustre: ls5-MDT0000: Denying connection for new client ..., waiting for 100 clients in recovery for 6:19 Apr 25 13:21:39 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports Apr 25 13:21:40 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports Apr 25 13:21:41 roc-mds1 kernel: Lustre: ls5-MDT0000: Starting recovery timer for 15:00 </pre> </div></div> <p>This raises several questions.</p> <ul> <li>Is it unexpected to have multiple "Starting recovery timer" messages, or is this just normal extension of recovery timeout due to new client connections?</li> <li>Should it be possible to start the recovery timer twice in the same second, as at 12:21:28?</li> <li>How many times should "recovery is timed out, evict stale exports" appear? (Looking at target_recovery_overseer(), it seems it should be at most twice.)</li> <li>In check_and_start_recovery_timer(), is the cfs_timer_is_armed() check sufficient to avoid race conditions, or does it need to also check the obd_device recovery flags?</li> </ul> <p>Let's also think about how to improve the code comments, manual, and log messages to make the recovery process more transparent.</p> <a href="https://github.com/chaos/lustre/commits/2.1.1-4chaos">https://github.com/chaos/lustre/commits/2.1.1-4chaos</a> LU-1352

spurious recovery timer resets

Bug Major Resolved Fixed Hongchao Zhang Ned Bass llnl Fri, 27 Apr 2012 21:02:06 +0000 Sun, 3 Apr 2016 03:07:25 +0000 Sun, 3 Apr 2016 03:07:25 +0000 Lustre 2.1.1 Lustre 2.2.0 0 8 <p>Hongchao</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>in 2.1.1-4chaos, there could be a bug in reset_recovery_timer,<br/> ...<br/> left = cfs_time_sub(obd-&gt;obd_recovery_end, now); &lt;-- here, obd-&gt;obd_recovery_end is less than now</p> <p> if (extend &amp;&amp; (duration &gt; left)<br/> obd-&gt;obd_recovery_timeout += duration - left;<br/> else if (!extend &amp;&amp; (duration &gt; obd-&gt;obd_recovery_timeout))<br/> /* Track the client's largest expected replay time */<br/> obd-&gt;obd_recovery_timeout = duration;<br/> ...</p> <p>when the recovery timer expired, this function will be called by target_recovery_overseer(in our master branch, this has been<br/> changed to "extent_recovery_timer"), and obd-&gt;obd_recovery_end is less than "now" and the calculation result could be wrong, <br/> for the expired callback is only about 1 second interval (btw, cfs_time_t is "unsigned long" and cfs_duration_t is "long")<br/> ...<br/> Apr 25 12:36:29 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports<br/> Apr 25 12:36:29 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports<br/> Apr 25 12:36:30 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports..<br/> Apr 25 12:36:32 roc-mds1 kernel: Lustre: ls5-MDT0000: recovery is timed out, evict stale exports<br/> ...</p> <p>-Is it unexpected to have multiple "Starting recovery timer" messages, or is this just normal extension of recovery timeout due to new client connections?</p> <p>it's not a extension of recovery timer, and there is a chance for target_handle_connect/target_start_and_reset_recovery_timer<br/> to start a new recovery timer during these short timer armed by target_recovery_overseer/reset_recovery_timer</p> <p>-Should it be possible to start the recovery timer twice in the same second, as at 12:21:28?</p> <p>there is a race in check_and_start_recovery_timer (in our master branch, it has been fixed)<br/> static void check_and_start_recovery_timer(struct obd_device *obd)<br/> {<br/> cfs_spin_lock(&amp;obd-&gt;obd_recovery_task_lock);<br/> if (cfs_timer_is_armed(&amp;obd-&gt;obd_recovery_timer)) </p> { cfs_spin_unlock(&amp;obd-&gt;obd_recovery_task_lock); return; } <p> LCONSOLE_INFO("%s: Starting recovery timer for %d:%.02d\n",<br/> obd-&gt;obd_name, obd-&gt;obd_recovery_timeout / 60,<br/> obd-&gt;obd_recovery_timeout % 60);<br/> obd-&gt;obd_recovery_start = cfs_time_current_sec();<br/> cfs_spin_unlock(&amp;obd-&gt;obd_recovery_task_lock); <br/> &lt;-- race here, another call can do it again.<br/> reset_recovery_timer(obd, obd-&gt;obd_recovery_timeout, 0);<br/> }</p> <p>-How many times should "recovery is timed out, evict stale exports" appear? (Looking at target_recovery_overseer(), it seems it should be at most twice.)</p> <p>it will continue to appear if there are still healthy recovering clients.</p> <p>-In check_and_start_recovery_timer(), is the cfs_timer_is_armed() check sufficient to avoid race conditions, or does it need to also check the obd_device recovery flags?</p> <p>yes, it isn't sufficient</p> <p>status update:</p> <p>the related tickets are: <br/> ORNL-28 <a href="http://review.whamcloud.com/1292" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1292</a>, <a href="http://review.whamcloud.com/1293" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1293</a>, <a href="http://review.whamcloud.com/1620" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1620</a><br/> <a href="https://jira.whamcloud.com/browse/LU-889" title="rework extent_recovery_timer()" class="issue-link" data-issue-key="LU-889"><del>LU-889</del></a> <a href="http://review.whamcloud.com/1722" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1722</a></p> <p>the pending patch will try to backport these patches to b2_1</p> <p>status update:<br/> the patch is under test, and there are still some problems in it and hope to attach it before the weekend.</p> <p>the patch is tracked at <a href="http://review.whamcloud.com/#change,2719" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2719</a></p> <p>I think that when porting patches to another branch the patches need to remain separate. The commit messages should be mostly the same as well (especially the bug reference number and first line summary). It will be very difficult to make sense of what is in a branch using "git log --oneline" if we squash multiple commits into one on another branch.</p> <p>this patch is part of the patches landed on master, and it mainly bases on the implementation of IR(Imperative Recovery),<br/> and the patch <a href="http://review.whamcloud.com/1620" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1620</a> and <a href="http://review.whamcloud.com/1722" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1722</a> are fixing the problem in <br/> <a href="http://review.whamcloud.com/1292" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1292</a>, <a href="http://review.whamcloud.com/2674" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2674</a> is a separated patch, then how about creating two patches<br/> to backport it?</p> <p>Hi Hongchao,</p> <p>That approach sounds reasonable to me. Thanks</p> <p>status update:<br/> the first patch is tracked at <a href="http://review.whamcloud.com/#change,2719" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2719</a>, and the second patch is upon the first one<br/> and will attach it once the first is merged.</p> <p>If you have a second patch that is needed, please submit it to gerrit now as well. We can give it some testing now on our end.</p> <p>We also experienced the same issue at TGCC site, and what is not clear for me is if "http://review.whamcloud.com/#change,2719" patch is b2_1 only and just a backport of the new recovery timer algorithms as implemented in b2_3/master ??...</p> <p>Closing old 2.1 bug. Problem was fixed in 2.2 and fix was backported to 2.1.</p> Development Rank 1|hzv3a7: Rank (Obsolete) 4031 Severity