https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9973 Lustre <p>Hi, </p> <p>When I re-mount mdts, MDT try to recovery all clients. But the mdts recovery status never completed ( several hours). The recovery process will not to be end even reach hard limit (900). I can only abort recovery process by "lctl --device MDTxxx abort_recovery".<br/> 1.Can I abort recovery process by "mount.lustre -o avort_recov xx" ? I can not execute command sucessful. (command hang) <br/> 2.Can I evict all stalled clients to end the recovery process?<br/> 3.Is there any side effect to abort recovery process by "lctl abort_recovery" ?</p> <p>Thanks.</p> <p>[ 570.019086] LNet: 14185:0:(o2iblnd_cb.c:3209:kiblnd_check_conns()) Timed out tx for 192.168.5.202@o2ib6: 7 seconds<br/> [ 622.166399] Lustre: 18740:0:(ldlm_lib.c:1784:extend_recovery_timer()) hpcfs-MDT0000: extended recovery timer reaching hard limit: 900, extend: 1<br/> [ 622.973445] Lustre: 26593:0:(ldlm_lib.c:1784:extend_recovery_timer()) hpcfs-MDT0001: extended recovery timer reaching hard limit: 900, extend: 1<br/> [ 622.973448] Lustre: 26593:0:(ldlm_lib.c:1784:extend_recovery_timer()) Skipped 2 previous similar messages<br/> [ 682.170260] Lustre: 18740:0:(ldlm_lib.c:1784:extend_recovery_timer()) hpcfs-MDT0000: extended recovery timer reaching hard limit: 900, extend: 1<br/> [ 682.170264] Lustre: 18740:0:(ldlm_lib.c:1784:extend_recovery_timer()) Skipped 2 previous similar messages<br/> [ 684.027392] LNet: 14185:0:(o2iblnd_cb.c:3209:kiblnd_check_conns()) Timed out tx for 192.168.7.202@o2ib8: 7 seconds<br/> [ 684.027397] LNet: 14185:0:(o2iblnd_cb.c:3209:kiblnd_check_conns()) Skipped 3 previous similar messages<br/> [ 693.713019] Lustre: 14236:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: <span class="error">&#91;sent 1505204590/real 0&#93;</span> req@ffff887c5585ad00 x1578320959384656/t0(0) o38-&gt;hpcfs-MDT0000-osp-MDT0001@192.168.8.202@o2ib9:24/4 lens 520/544 e 0 to 1 dl 1505204596 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1<br/> [ 693.713024] Lustre: 14236:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages<br/> [ 737.721925] LustreError: 11-0: hpcfs-OST0000-osc-MDT0001: operation ost_connect to node 192.168.17.212@o2ib18 failed: rc = -19</p> LU-9973

MDT recovery status never completed

Question/Request Minor Open Unresolved Lai Siyao sebg-crd-pm Tue, 12 Sep 2017 11:20:45 +0000 Tue, 19 Sep 2017 00:13:55 +0000 Lustre 2.10.0 0 3 <p>Lai</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>It looks like you may be having some sort of communication problem between MDT0000 and MDT0001? The MDS recovery can not complete with multiple MDTs until they are all available, and it looks like there timeouts during connect (o38 = <tt>MDS_CONNECT</tt>):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Request sent has timed out for sent delay: [sent 1505204590/real 0] req@ffff887c5585ad00 x1578320959384656/t0(0) o38-&gt;hpcfs-MDT0000-osp-MDT0001@192.168.8.202@o2ib9 </pre> </div></div> <p>What is also a bit strange to me is that you appear to have a large number of different IB networks? Just in the messages posted here, I see <tt>o2ib6</tt>, <tt>o2ib8</tt>, <tt>o2ib9</tt>, and <tt>o2ib18</tt> reported. Do you have an extremely large number of clients, several different filesystems, or are you configuring a separate LNet network for each host? That isn't <em>necessarily</em> a source of problems, but it is unusual and opens more chance of incorrect configuration causing problems.</p> <p>Thank you for your advise. I can get MDT recovery status completed now.<br/> Too many o2ibx made MDTs need spent much time to try create workable connections.<br/> And too many logs need spent more than 8 hours to update them. (The update log rate too slow, so I clear update log fininally)</p> Development Rank 1|hzzjzb: Rank (Obsolete) 9223372036854775807