https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3844 Lustre <p>We frequently observe that the recovery process repeats itself after reaching the timer expiration. Often times it reaches the first timer expiration because a client has died, so it is going to go through the whole recovery period again the second time.</p> <p>In this example, recovery took 60s rather than the better case of 30. Does this fall under the case implied by the wording 'Will be in recovery for at least 30:00'. During the failure several OSS had to be rebooted.<br/> <span class="error">&#91;blakec@widow-mgmt3 ~&#93;</span>$ tail -10000 /data/log/apps/lustrekernel|grep oss12a2|grep -i recov|grep widow1-OST00b5:<br/> Aug 27 14:16:46 widow-oss12a2 kernel: [ 544.793348] Lustre: widow1-OST00b5: Now serving widow1-OST00b5 on /dev/mpath/widow-ddn12a-l48 with recovery enabled<br/> Aug 27 14:16:46 widow-oss12a2 kernel: [ 544.793363] Lustre: widow1-OST00b5: Will be in recovery for at least 30:00, or until 12345 clients reconnect<br/> Aug 27 14:17:13 widow-oss12a2 kernel: [ 572.156957] LustreError: 16171:0:(ldlm_lib.c:946:target_handle_connect()) widow1-OST00b5: denying connection for new client 172.30.221.31@o2ib3 (66a3b9fa-4a34-e8cb-c90e-1c98fa38f2e6): 12336 clients in recovery for 1772s<br/> Aug 27 14:19:01 widow-oss12a2 kernel: [ 679.468170] LustreError: 15390:0:(ldlm_lib.c:946:target_handle_connect()) widow1-OST00b5: denying connection for new client 172.30.221.25@o2ib3 (e65b92af-bef9-b8b2-5069-d8894f06fb19): 12294 clients in recovery for 1665s<br/> Aug 27 14:55:51 widow-oss12a2 kernel: [ 2885.008362] LustreError: 16132:0:(ldlm_lib.c:946:target_handle_connect()) widow1-OST00b5: denying connection for new client 172.30.221.18@o2ib3 (5ac80ce4-58aa-16fd-8c53-889df4ba3118): 12174 clients in recovery for 1254s<br/> Aug 27 14:59:35 widow-oss12a2 kernel: [ 3108.624936] Lustre: 15386:0:(ldlm_lib.c:1817:target_queue_last_replay_reply()) widow1-OST00b5: 12173 recoverable clients remain<br/> Aug 27 15:16:46 widow-oss12a2 kernel: [ 4137.296402] Lustre: widow1-OST00b5: Recovery period over after 60:00, of 12345 clients 12342 recovered and 2 were evicted.<br/> Aug 27 15:16:46 widow-oss12a2 kernel: [ 4137.296415] Lustre: widow1-OST00b5: sending delayed replies to recovered clients</p> <p>In an earlier case, only the MSD failed, and recovery finished in less that 30min because all clients reconnected. I'm sure there are cases where recovery just takes 30min, but mostly now when it goes to 30min, it will also go to 60 min</p> <p>obd_timeout is set to 600<br/> <span class="error">&#91;root@widow-oss10a1 ~&#93;</span># cat /proc/sys/lustre/timeout <br/> 600</p> <p>ldlm timeout is set to 200<br/> <span class="error">&#91;root@widow-oss10a1 ~&#93;</span># cat /proc/sys/lustre/ldlm_timeout <br/> 200</p> kernel 2.6.18-348.3.1.el5, rhel5.9, distribution-provided ofed, gni behind o2iblnd routers LU-3844

Double recovery period in 1.8.9 after OSS failure

Bug Minor Closed Fixed Hongchao Zhang Blake Caldwell Tue, 27 Aug 2013 19:39:41 +0000 Wed, 28 Aug 2013 21:46:00 +0000 Wed, 28 Aug 2013 21:45:59 +0000 Lustre 1.8.9 0 4 <p>Hongchao, </p> <p>Would you please comment on this one?</p> <p>Thanks,<br/> James</p> <p>Hi Blake,</p> <p>the extra recovery period is caused by VBR (version based recovery).</p> <p>in target_recovery_check_and_stop, after the first recovery period (3*obd_timeout = 1800s = 30m) is expired, obd_device-&gt;obd_version_recov will be set and<br/> the extra recovery period (30m) is started by calling "reset_recovery_timer".</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">int</span> target_recovery_check_and_stop(struct obd_device *obd) { <span class="code-object">int</span> abort_recovery = 0; <span class="code-keyword">if</span> (obd-&gt;obd_stopping || !obd-&gt;obd_recovering) <span class="code-keyword">return</span> 1; spin_lock_bh(&amp;obd-&gt;obd_processing_task_lock); abort_recovery = obd-&gt;obd_abort_recovery; obd-&gt;obd_abort_recovery = 0; spin_unlock_bh(&amp;obd-&gt;obd_processing_task_lock); <span class="code-keyword">if</span> (!abort_recovery) <span class="code-keyword">return</span> 0; <span class="code-comment">/** check <span class="code-keyword">if</span> fs version-capable */</span> <span class="code-keyword">if</span> (target_fs_version_capable(obd)) { class_handle_stale_exports(obd); } <span class="code-keyword">else</span> { CWARN(<span class="code-quote">"Versions are not supported by ldiskfs, VBR is OFF\n"</span>); class_disconnect_stale_exports(obd, exp_flags_from_obd(obd)); } <span class="code-comment">/* VBR: no clients are remained to replay, stop recovery */</span> spin_lock_bh(&amp;obd-&gt;obd_processing_task_lock); <span class="code-keyword">if</span> (obd-&gt;obd_recovering &amp;&amp; obd-&gt;obd_recoverable_clients == 0) { spin_unlock_bh(&amp;obd-&gt;obd_processing_task_lock); target_stop_recovery(obd, 0); <span class="code-keyword">return</span> 1; } <span class="code-comment">/* always check versions now */</span> obd-&gt;obd_version_recov = 1; cfs_waitq_signal(&amp;obd-&gt;obd_next_transno_waitq); spin_unlock_bh(&amp;obd-&gt;obd_processing_task_lock); <span class="code-comment">/* reset timer, recovery will proceed with versions now */</span> reset_recovery_timer(obd, OBD_RECOVERY_TIME_SOFT, 1); <span class="code-keyword">return</span> 0; } </pre> </div></div> <p>Thanks much for the explanation!</p> <p>Blake, </p> <p>Is there anything else we need to do under this ticket or should we close it?</p> <p>Thanks, <br/> James</p> <p>This can be closed.</p> Development Rank 1|hzvze7: Rank (Obsolete) 9949 Severity