https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2368 Lustre <p>MDS failover happend during OSTs recovery, and OST got two mds connections from different IP. First was processed by OST, second connection cause class_fail_export() at target_handle_connect(), and we got perpetual recovery.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oct 26 17:09:07 snx11001n008 kernel: [ 838.638847] Lustre: 90700:0:(ldlm_lib.c:2007:target_recovery_init()) RECOVERY: service snx11001-OST0012, 3 recoverable clients, last_transno 54526017 Oct 26 17:09:07 snx11001n008 kernel: [ 838.708354] Lustre: snx11001-OST0012: Now serving snx11001-OST0012/ on /dev/md4 with recovery enabled Oct 26 17:09:07 snx11001n008 kernel: [ 838.717732] Lustre: snx11001-OST0012: Will be in recovery for at least 15:00, or until 3 clients reconnect Oct 26 17:11:05 snx11001n008 kernel: [ 956.648093] LustreError: 88011:0:(ldlm_lib.c:927:target_handle_connect()) snx11001-OST0012: NID 10.10.101.3@o2ib1 (snx11001-MDT0000-mdtlov_UUID) reconnected with 1 conn_cnt; cookies not random? Oct 26 17:15:10 snx11001n008 kernel: [ 1201.718217] Lustre: 88009:0:(ldlm_lib.c:941:target_handle_connect()) snx11001-OST0012: connection from snx11001-MDT0000-mdtlov_UUID@10.10.101.3@o2ib1 recovering/t0 exp ffff88072cb90400 cur 1351289710 last 1351289346 Oct 26 17:18:40 snx11001n008 kernel: [ 1410.931800] Lustre: 88010:0:(ldlm_lib.c:854:target_handle_connect()) snx11001-OST0012: received MDS connection from NID 10.10.101.4@o2ib1, removing former export from NID 10.10.101.3@o2ib1 Oct 26 17:18:40 snx11001n008 kernel: [ 1410.948937] Lustre: 88010:0:(ldlm_lib.c:941:target_handle_connect()) snx11001-OST0012: connection from snx11001-MDT0000-mdtlov_UUID@10.10.101.4@o2ib1 recovering/t0 exp (null) cur 1351289920 last 0 Oct 26 17:18:40 snx11001n008 kernel: [ 1410.976334] LustreError: 88010:0:(ldlm_lib.c:974:target_handle_connect()) snx11001-OST0012: denying connection for new client 10.10.101.4@o2ib1 (snx11001-MDT0000-mdtlov_UUID): 0 clients in recovery for 381s </pre> </div></div> LU-2368

OSTs stuck in perpetual recovery

Bug Major Resolved Fixed Keith Mannthey Alexander Boyko patch Wed, 21 Nov 2012 01:23:17 +0000 Wed, 22 May 2013 20:54:04 +0000 Wed, 22 May 2013 20:54:04 +0000 Lustre 2.1.0 Lustre 2.1.6 0 5 <p><a href="http://review.whamcloud.com/4641" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4641</a></p> <p>Alexander, could you give more info about how does that causes perpetual recovery? Just logs showing that will be good. I wonder just isn't this the same as <a href="https://jira.whamcloud.com/browse/LU-2104" title="conf-sanity test 47 never completes, negative time to recovery" class="issue-link" data-issue-key="LU-2104"><del>LU-2104</del></a>?</p> <p>I can see updated recovery timer (recovery is timed out, evict stale exports), so this does not relate to <a href="https://jira.whamcloud.com/browse/LU-2104" title="conf-sanity test 47 never completes, negative time to recovery" class="issue-link" data-issue-key="LU-2104"><del>LU-2104</del></a>. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oct 26 17:18:40 snx11001n008 kernel: [ 1410.976334] LustreError: 88010:0:(ldlm_lib.c:974:target_handle_connect()) snx11001-OST0012: denying connection for new client 10.10.101.4@o2ib1 (snx11001-MDT0000-mdtlov_UUID): 0 clients in recovery for 381s Oct 26 17:21:31 snx11001n008 kernel: [ 1582.137247] LustreError: 88010:0:(ldlm_lib.c:974:target_handle_connect()) snx11001-OST0012: denying connection for new client 10.10.101.4@o2ib1 (snx11001-MDT0000-mdtlov_UUID): 0 clients in recovery for 209s Oct 26 17:24:01 snx11001n008 kernel: [ 1731.873364] Lustre: 88010:0:(ldlm_lib.c:941:target_handle_connect()) snx11001-OST0012: connection from snx11001-MDT0000-mdtlov_UUID@10.10.101.4@o2ib1 recovering/t0 exp (null) cur 1351290241 last 0 Oct 26 17:25:01 snx11001n008 kernel: [ 1791.526174] Lustre: snx11001-OST0012: disconnecting 1 stale clients Oct 26 17:25:16 snx11001n008 kernel: [ 1806.790647] LustreError: 88010:0:(ldlm_lib.c:974:target_handle_connect()) snx11001-OST0012: denying connection for new client 10.10.101.4@o2ib1 (snx11001-MDT0000-mdtlov_UUID): 0 clients in recovery for 54s Oct 26 17:26:11 snx11001n008 kernel: [ 1861.406988] Lustre: snx11001-OST0012: recovery is timed out, evict stale exports Oct 26 17:27:21 snx11001n008 kernel: [ 1931.296723] Lustre: snx11001-OST0012: recovery is timed out, evict stale exports Oct 26 17:28:31 snx11001n008 kernel: [ 2001.192849] Lustre: snx11001-OST0012: recovery is timed out, evict stale exports Oct 26 17:30:16 snx11001n008 kernel: [ 2106.354440] LustreError: 88010:0:(ldlm_lib.c:974:target_handle_connect()) snx11001-OST0012: denying connection for new client 10.10.101.4@o2ib1 (snx11001-MDT0000-mdtlov_UUID): 0 clients in recovery for 34s Oct 26 17:32:01 snx11001n008 kernel: [ 2210.862684] Lustre: snx11001-OST0012: recovery is timed out, evict stale exports Oct 26 17:39:01 snx11001n008 kernel: [ 2630.201143] Lustre: snx11001-OST0012: recovery is timed out, evict stale exports Oct 26 17:47:46 snx11001n008 kernel: [ 3154.655964] LustreError: 88010:0:(ldlm_lib.c:974:target_handle_connect()) snx11001-OST0012: denying connection for new client 10.10.101.4@o2ib1 (snx11001-MDT0000-mdtlov_UUID): 0 clients in recovery for 34s </pre> </div></div> <p>but recovery never ends still? Or it just lasts too long?</p> <p>Xyratex-bug-id: <a href="http://jira-nss.xy01.xyratex.com:8080/browse/MRP-738" class="external-link" target="_blank" rel="nofollow noopener">MRP-738</a></p> <p>I cannot access Xyratex site and check bug internals there</p> <p>&gt; but recovery never ends still? Or it just lasts too long?<br/> never ends<br/> above, you can see how timer was restarted during 30 mins </p> <p>Strictly speaking the situation with two MDS connection is not something special, old one is evicted through class_fail_export(), the second is not allowed until recovery is finished. So problem is why recovery cannot finish and the reason is that class_fail_export() call. During recovery all evicted/failed clients are counted in obd_stale_clients, inconsistent counter may cause recovery stuck. I think this patch should solve your problem:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /* <span class="code-keyword">if</span> called during recovery then should keep obd_stale_clients * consistent */ <span class="code-keyword">if</span> (exp-&gt;exp_obd-&gt;obd_recovering) exp-&gt;exp_obd-&gt;obd_stale_clients++; </pre> </div></div> <p>I'd prefer this solution because it fixes source of problem, your patch is correct too, but cover only case with that one particular call to class_fail_export(). I tried to simulate similar situation in master branch and patch above works, I'd appreciate if you will check does it help in your case? If this is not easy to do then I will agree with your patch for b2_1.</p> <p>The <a href="https://jira.whamcloud.com/browse/LU-2104" title="conf-sanity test 47 never completes, negative time to recovery" class="issue-link" data-issue-key="LU-2104"><del>LU-2104</del></a> is the same problem actually, but there is also miscalculation in recovery timer reset which stops timer at all so recovery exceed timeout without being woken up, but the reason is the same and master will be fixed with lu-2104 patch anyway.</p> <p>Thanks Mikhail, I will try it.</p> <p>I approve, the patch</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>/* if called during recovery then should keep obd_stale_clients * consistent */ if (exp-&gt;exp_obd-&gt;obd_recovering) exp-&gt;exp_obd-&gt;obd_stale_clients++; </pre> </div></div> <p>fix this issue. And is better than <a href="http://review.whamcloud.com/4641" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4641</a>.</p> <p>Mikhail, do you plan to land <a href="https://jira.whamcloud.com/browse/LU-2104" title="conf-sanity test 47 never completes, negative time to recovery" class="issue-link" data-issue-key="LU-2104"><del>LU-2104</del></a> patch to b2_1 branch, or small change?</p> <p>I had no such plan so far. I think that code you checked is enough for b2_1, I'd use just that fix as it is sufficient. Can you push it for b2_1 in context of this ticket?</p> <p>sure, I have changed my previos patch with this one.</p> <p>Both patches were landed for 2.1 . </p> Development Rank 1|hzvclz: Rank (Obsolete) 5631 Severity