https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13608 Lustre <p>when some network switch goes down, taking with it a number of compute nodes. <br/> a bunch of compute nodes were stuck, and also caused some oss to go down to hit<br/> LBUG (<a href="https://jira.whamcloud.com/browse/LU-12906" title="LBUG ASSERTION( rspt-&gt;rspt_cpt == cpt ) failed" class="issue-link" data-issue-key="LU-12906"><del>LU-12906</del></a>) and crash.</p> <p>the mds went into softlockups before crashing. when it got back, 3 out of 4 MDTs<br/> mounted and recovered, but one MDT went into WAITING and stayed there.<br/> lctl abort_recov had no effect on its status. so the mds was rebooted.</p> <p>when the MDT again went into WAITING state, a pre-emptive abort_recov was issued<br/> before it could time-out. but it did not help and the MDT continued to try to recover.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2020-05-08 18:52:15 [ 1640.880984] Pid: 15922, comm: mdt02_020 3.10.0-1062.4.1.el7_lustre.x86_64 #1 SMP Mon Oct 28 01:39:05 UTC 2019 2020-05-08 18:52:15 [ 1640.892505] Call Trace: 2020-05-08 18:52:15 [ 1640.895708] [&lt;ffffffffc169bdc0&gt;] ptlrpc_set_wait+0x480/0x790 [ptlrpc] 2020-05-08 18:52:15 [ 1640.903498] [&lt;ffffffffc169c153&gt;] ptlrpc_queue_wait+0x83/0x230 [ptlrpc] 2020-05-08 18:52:15 [ 1640.911383] [&lt;ffffffffc1a9eaf3&gt;] osp_remote_sync+0xd3/0x200 [osp] 2020-05-08 18:52:15 [ 1640.918760] [&lt;ffffffffc1a84c63&gt;] osp_attr_get+0x463/0x730 [osp] 2020-05-08 18:52:15 [ 1640.925917] [&lt;ffffffffc1a818cd&gt;] osp_object_init+0x16d/0x2d0 [osp] 2020-05-08 18:52:15 [ 1640.933361] [&lt;ffffffffc141c59b&gt;] lu_object_start.isra.35+0x8b/0x120 [obdclass] 2020-05-08 18:52:15 [ 1640.941977] [&lt;ffffffffc1420471&gt;] lu_object_find_at+0x1e1/0xa60 [obdclass] 2020-05-08 18:52:15 [ 1640.950100] [&lt;ffffffffc1420d06&gt;] lu_object_find+0x16/0x20 [obdclass] 2020-05-08 18:52:15 [ 1640.957737] [&lt;ffffffffc194a01b&gt;] mdt_object_find+0x4b/0x170 [mdt] 2020-05-08 18:52:15 [ 1640.965074] [&lt;ffffffffc194cc38&gt;] mdt_getattr_name_lock+0x848/0x1c30 [mdt] 2020-05-08 18:52:15 [ 1640.973194] [&lt;ffffffffc1954d25&gt;] mdt_intent_getattr+0x2b5/0x480 [mdt] 2020-05-08 18:52:15 [ 1640.980928] [&lt;ffffffffc1951bb5&gt;] mdt_intent_policy+0x435/0xd80 [mdt] 2020-05-08 18:52:15 [ 1640.988552] [&lt;ffffffffc1659d56&gt;] ldlm_lock_enqueue+0x356/0xa20 [ptlrpc] 2020-05-08 18:52:15 [ 1640.996483] [&lt;ffffffffc1682366&gt;] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] 2020-05-08 18:52:15 [ 1641.004811] [&lt;ffffffffc170ab02&gt;] tgt_enqueue+0x62/0x210 [ptlrpc] 2020-05-08 18:52:15 [ 1641.012076] [&lt;ffffffffc17112ea&gt;] tgt_request_handle+0xaea/0x1580 [ptlrpc] 2020-05-08 18:52:15 [ 1641.020213] [&lt;ffffffffc16b629b&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] 2020-05-08 18:52:15 [ 1641.029217] [&lt;ffffffffc16b9bfc&gt;] ptlrpc_main+0xb2c/0x1460 [ptlrpc] 2020-05-08 18:52:15 [ 1641.056176] Pid: 14912, comm: mdt03_006 3.10.0-1062.4.1.el7_lustre.x86_64 #1 SMP Mon Oct 28 01:39:05 UTC 2019 2020-05-08 18:52:15 [ 1641.067667] Call Trace: 2020-05-08 18:52:15 [ 1641.070833] [&lt;ffffffffc1672b96&gt;] ldlm_completion_ast+0x4e6/0x860 [ptlrpc] 2020-05-08 18:52:15 [ 1641.078951] [&lt;ffffffffc167492f&gt;] ldlm_cli_enqueue_fini+0x96f/0xdf0 [ptlrpc] 2020-05-08 18:52:15 [ 1641.087272] [&lt;ffffffffc167751e&gt;] ldlm_cli_enqueue+0x40e/0x920 [ptlrpc] 2020-05-08 18:52:15 [ 1641.095108] [&lt;ffffffffc1a997f2&gt;] osp_md_object_lock+0x162/0x2d0 [osp] 2020-05-08 18:52:15 [ 1641.102832] [&lt;ffffffffc10cb193&gt;] lod_object_lock+0xf3/0x7b0 [lod] 2020-05-08 18:52:15 [ 1641.110179] [&lt;ffffffffc1a2eeee&gt;] mdd_object_lock+0x3e/0xe0 [mdd] 2020-05-08 18:52:15 [ 1641.117429] [&lt;ffffffffc194a341&gt;] mdt_remote_object_lock_try+0x1e1/0x750 [mdt] 2020-05-08 18:52:15 [ 1641.125926] [&lt;ffffffffc194a8da&gt;] mdt_remote_object_lock+0x2a/0x30 [mdt] 2020-05-08 18:52:15 [ 1641.133847] [&lt;ffffffffc195f2ae&gt;] mdt_rename_lock+0xbe/0x4b0 [mdt] 2020-05-08 18:52:15 [ 1641.141189] [&lt;ffffffffc1961605&gt;] mdt_reint_rename+0x2c5/0x2b90 [mdt] 2020-05-08 18:52:15 [ 1641.148819] [&lt;ffffffffc196a693&gt;] mdt_reint_rec+0x83/0x210 [mdt] 2020-05-08 18:52:15 [ 1641.155965] [&lt;ffffffffc19471b3&gt;] mdt_reint_internal+0x6e3/0xaf0 [mdt] 2020-05-08 18:52:15 [ 1641.163689] [&lt;ffffffffc1952567&gt;] mdt_reint+0x67/0x140 [mdt] 2020-05-08 18:52:15 [ 1641.170469] [&lt;ffffffffc17112ea&gt;] tgt_request_handle+0xaea/0x1580 [ptlrpc] 2020-05-08 18:52:15 [ 1641.178601] [&lt;ffffffffc16b629b&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] 2020-05-08 18:52:15 [ 1641.187639] [&lt;ffffffffc16b9bfc&gt;] ptlrpc_main+0xb2c/0x1460 [ptlrpc] </pre> </div></div> Servers: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Centos 7.7 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Kernel 3.10.0-1062.4.1.el7_lustre.x86_64 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Lustre 2.12.3 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OFED-internal-4.7-1.0.0 LU-13608

MDT stuck in WAITING, abort_recov stuck too

Bug Minor Resolved Fixed Hongchao Zhang Hongchao Zhang Thu, 28 May 2020 12:00:13 +0000 Tue, 10 Jan 2023 19:17:45 +0000 Fri, 23 Oct 2020 04:33:17 +0000 Lustre 2.12.3 Lustre 2.14.0 Lustre 2.12.6 Lustre 2.12.7 0 8 <p>what version did you use? any logs?</p> <p>Hongchao Zhang (hongchao@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38746" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38746</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13608" title="MDT stuck in WAITING, abort_recov stuck too" class="issue-link" data-issue-key="LU-13608"><del>LU-13608</del></a> tgt: abort recovery while reading update llog<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8c4b7bf0bf5321a937d39ee1f5e7e907b2c20903</p> <p>I've seen a similar MDS hang, and I want to add my thoughts<br/> For my occurrence, MDT02 recovery have not finished because of lod_sub_recovery_thread(), it tried to read update log files which was not exist.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 113444 TASK: ffff8ed602b4a140 CPU: 0 COMMAND: "lod0002_rec0000" #0 [ffff8ed5197635c8] __schedule at ffffffff9416d0bc #1 [ffff8ed519763658] schedule at ffffffff9416d6e9 #2 [ffff8ed519763668] schedule_timeout at ffffffff9416b01e #3 [ffff8ed519763710] ptlrpc_set_wait at ffffffffc10d42f8 [ptlrpc] #4 [ffff8ed5197637b8] ptlrpc_queue_wait at ffffffffc10d4653 [ptlrpc] #5 [ffff8ed5197637d8] osp_remote_sync at ffffffffc168a783 [osp] #6 [ffff8ed519763828] osp_attr_get at ffffffffc16704d7 [osp] #7 [ffff8ed5197638a8] osp_object_init at ffffffffc166d00d [osp] #8 [ffff8ed5197638e0] lu_object_start at ffffffffc0dff50b [obdclass] #9 [ffff8ed519763938] lu_object_find_at at ffffffffc0e033f1 [obdclass] #10 [ffff8ed519763a00] dt_locate_at at ffffffffc0e049bd [obdclass] #11 [ffff8ed519763a20] llog_osd_open at ffffffffc0dc517f [obdclass] #12 [ffff8ed519763a98] llog_open at ffffffffc0db1f8a [obdclass] #13 [ffff8ed519763ae0] llog_cat_id2handle at ffffffffc0dbad45 [obdclass] #14 [ffff8ed519763b50] llog_cat_process_common at ffffffffc0dbb1b9 [obdclass] #15 [ffff8ed519763ba8] llog_cat_process_cb at ffffffffc0dbc191 [obdclass] #16 [ffff8ed519763bf8] llog_process_thread at ffffffffc0db586f [obdclass] #17 [ffff8ed519763d08] llog_process_or_fork at ffffffffc0db6b3c [obdclass] #18 [ffff8ed519763d70] llog_cat_process_or_fork at ffffffffc0db89e9 [obdclass] #19 [ffff8ed519763de8] llog_cat_process at ffffffffc0db8b9e [obdclass] #20 [ffff8ed519763e08] lod_sub_recovery_thread at ffffffffc142b526 [lod] #21 [ffff8ed519763ea8] kthread at ffffffff93ac2016 #22 [ffff8ed519763f50] ret_from_fork_nospec_begin at ffffffff9417abdd </pre> </div></div> <p>MDT02 sent out request to MDT0, at MDT0 osd_fid_lookup happened, and triggered OI scrub, because of file did not exist. MDT0 replied with EINPROGRESS, and MDT02 resent request. And again osd_fid_lookup trigger scrub etc.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00080000:02000400:11.0:1593417562.046666:0:224060:0:(osd_handler.c:1221:osd_fid_lookup()) fs1-MDT0000: trigger OI scrub by RPC for the [0x200032c9a:0x1:0x0] with flags 0x4a, rc = 0 00100000:10000000:17.0:1593417562.411181:0:2919:0:(osd_scrub.c:536:osd_scrub_post()) fs1-MDT0000: OI scrub post with result = 1 00100000:10000000:17.0:1593417562.411207:0:2919:0:(scrub.c:239:scrub_file_store()) fs1-MDT0000: store scrub file: rc = 0 00100000:10000000:17.0:1593417562.411208:0:2919:0:(osd_scrub.c:1322:osd_scrub_main()) fs1-MDT0000: OI scrub: stop, pos = 3041940257: rc = 1 00100000:10000000:17.0:1593417570.677557:0:2950:0:(osd_scrub.c:459:osd_scrub_prep()) fs1-MDT0000: OI scrub prep, flags = 0x4e 00100000:10000000:17.0:1593417570.677574:0:2950:0:(scrub.c:132:scrub_file_reset()) fs1-MDT0000: reset OI scrub file, old flags = 0x0, add flags = 0x0 00100000:10000000:17.0:1593417570.677600:0:2950:0:(scrub.c:239:scrub_file_store()) fs1-MDT0000: store scrub file: rc = 0 00100000:10000000:17.0:1593417570.677603:0:2950:0:(osd_scrub.c:1307:osd_scrub_main()) fs1-MDT0000: OI scrub start, flags = 0x4e, pos = 12 00080000:02000400:7.0:1593417570.677625:0:225112:0:(osd_handler.c:1221:osd_fid_lookup()) fs1-MDT0000: trigger OI scrub by RPC for the [0x200032c9b:0x2:0x0] with flags 0x4a, rc = 0 00100000:10000000:17.0:1593417571.031439:0:2950:0:(osd_scrub.c:536:osd_scrub_post()) fs1-MDT0000: OI scrub post with result = 1 00100000:10000000:17.0:1593417571.031464:0:2950:0:(scrub.c:239:scrub_file_store()) fs1-MDT0000: store scrub file: rc = 0 00100000:10000000:17.0:1593417571.031465:0:2950:0:(osd_scrub.c:1322:osd_scrub_main()) fs1-MDT0000: OI scrub: stop, pos = 3041940257: rc = 1 00100000:10000000:17.0:1593417573.046732:0:2951:0:(osd_scrub.c:459:osd_scrub_prep()) fs1-MDT0000: OI scrub prep, flags = 0x4e 00100000:10000000:17.0:1593417573.046735:0:2951:0:(scrub.c:132:scrub_file_reset()) fs1-MDT0000: reset OI scrub file, old flags = 0x0, add flags = 0x0 00100000:10000000:17.0:1593417573.046744:0:2951:0:(scrub.c:239:scrub_file_store()) fs1-MDT0000: store scrub file: rc = 0 00100000:10000000:17.0:1593417573.046747:0:2951:0:(osd_scrub.c:1307:osd_scrub_main()) fs1-MDT0000: OI scrub start, flags = 0x4e, pos = 12 00080000:02000400:8.0:1593417573.046823:0:224060:0:(osd_handler.c:1221:osd_fid_lookup()) fs1-MDT0000: trigger OI scrub by RPC for the [0x200032c9a:0x1:0x0] with flags 0x4a, rc = 0 00100000:10000000:17.0:1593417573.403695:0:2951:0:(osd_scrub.c:536:osd_scrub_post()) fs1-MDT0000: OI scrub post with result = 1 00100000:10000000:17.0:1593417573.403720:0:2951:0:(scrub.c:239:scrub_file_store()) fs1-MDT0000: store scrub file: rc = 0 00100000:10000000:17.0:1593417573.403721:0:2951:0:(osd_scrub.c:1322:osd_scrub_main()) fs1-MDT0000: OI scrub: stop, pos = 3041940257: rc = 1 00100000:10000000:17.0:1593417581.677557:0:3081:0:(osd_scrub.c:459:osd_scrub_prep()) fs1-MDT0000: OI scrub prep, flags = 0x4e 00100000:10000000:17.0:1593417581.677574:0:3081:0:(scrub.c:132:scrub_file_reset()) fs1-MDT0000: reset OI scrub file, old flags = 0x0, add flags = 0x0 00100000:10000000:17.0:1593417581.677600:0:3081:0:(scrub.c:239:scrub_file_store()) fs1-MDT0000: store scrub file: rc = 0 00100000:10000000:17.0:1593417581.677603:0:3081:0:(osd_scrub.c:1307:osd_scrub_main()) fs1-MDT0000: OI scrub start, flags = 0x4e, pos = 12 00080000:02000400:7.0:1593417581.677628:0:225112:0:(osd_handler.c:1221:osd_fid_lookup()) fs1-MDT0000: trigger OI scrub by RPC for the [0x200032c9b:0x2:0x0] with flags 0x4a, rc = 0 00100000:10000000:17.0:1593417582.027254:0:3081:0:(osd_scrub.c:536:osd_scrub_post()) fs1-MDT0000: OI scrub post with result = 1 00100000:10000000:17.0:1593417582.027278:0:3081:0:(scrub.c:239:scrub_file_store()) fs1-MDT0000: store scrub file: rc = 0 00100000:10000000:17.0:1593417582.027279:0:3081:0:(osd_scrub.c:1322:osd_scrub_main()) fs1-MDT0000: OI scrub: stop, pos = 3041940257: rc = 1 00100000:10000000:17.0:1593417584.046608:0:3083:0:(osd_scrub.c:459:osd_scrub_prep()) fs1-MDT0000: OI scrub prep, flags = 0x4e 00100000:10000000:17.0:1593417584.046611:0:3083:0:(scrub.c:132:scrub_file_reset()) fs1-MDT0000: reset OI scrub file, old flags = 0x0, add flags = 0x0 00100000:10000000:17.0:1593417584.046622:0:3083:0:(scrub.c:239:scrub_file_store()) fs1-MDT0000: store scrub file: rc = 0 00100000:10000000:17.0:1593417584.046624:0:3083:0:(osd_scrub.c:1307:osd_scrub_main()) fs1-MDT0000: OI scrub start, flags = 0x4e, pos = 12 00080000:02000400:11.0:1593417584.046686:0:224060:0:(osd_handler.c:1221:osd_fid_lookup()) fs1-MDT0000: trigger OI scrub by RPC for the [0x200032c9a:0x1:0x0] with flags 0x4a, rc = 0 </pre> </div></div> <p>For me it looks strange that OI scrub does not return an error after full scan and repeated FID.</p> <p> <a href="https://jira.whamcloud.com/browse/LU-13608" title="MDT stuck in WAITING, abort_recov stuck too" class="issue-link" data-issue-key="LU-13608"><del>LU-13608</del></a> tgt: abort recovery while reading update llog would help recovery process at MDT, but I'm not sure about lod_sub_recovery_thread().  It looped at pltrpc layer with repeating a request for EINPROGRESS.</p> <p>Here is the test to reproduce update log deletion. The fix doesn't help. I checked abort_recovery, etc.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/lustre/tests/sanity.sh b/lustre/tests/sanity.sh index 414d95e..b2f02f0 100755 --- a/lustre/tests/sanity.sh +++ b/lustre/tests/sanity.sh @@ -24177,6 +24177,29 @@ test_902() { } run_test 902 "test short write doesn't hang lustre" +test_903() { + [ $MDSCOUNT -lt 2 ] &amp;&amp; skip "needs &gt;= 2 MDTs" + + local timeout + + timeout=$(do_facet mds2 "$LCTL get_param -n mdt.$FSNAME-MDT0001.recovery_time_hard") + for idx in $(seq $MDSCOUNT); do + stop mds${idx} + done + + do_facet mds1 "mkdir -p /tmp/test_903 &amp;&amp; mount -t ldiskfs -o loop $(mdsdevname 1) /tmp/test_903 &amp;&amp; + rm -f /tmp/test_903/update_log_dir/* &amp;&amp; umount /tmp/test_903 &amp;&amp; rm -rf /tmp/test_903" + + for idx in $(seq $MDSCOUNT); do + start mds${idx} $(mdsdevname $idx) $MDS_MOUNT_OPTS || + error "mount mds$idx failed" + done + + wait_recovery_complete mds2 $timeout + +} +run_test 903 "don't hang MDS recovery when failed to get update log" + </pre> </div></div> <p>Maybe rq_no_retry_einprogress = 1 ? @Alex Zhuravlev  </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl get_param mdt.*.recovery_status mdt.lustre-MDT0000.recovery_status= status: COMPLETE recovery_start: 1593460707 recovery_duration: 246400 completed_clients: 1/2 replayed_requests: 0 last_transno: 4294967310 VBR: DISABLED IR: DISABLED mdt.lustre-MDT0001.recovery_status= status: WAITING non-ready MDTs: 0000 recovery_start: 1593707106 time_waited: 1429 root 16048 0.0 0.0 11232 708 pts/2 D+ 12:30 0:00 lctl --device 14 abort_recovery sudo cat /proc/16048/stack [&lt;ffffffffc0e9474d&gt;] target_stop_recovery_thread.part.20+0x3d/0xd0 [ptlrpc] [&lt;ffffffffc0e947f8&gt;] target_stop_recovery_thread+0x18/0x20 [ptlrpc] [&lt;ffffffffc11bcf68&gt;] mdt_iocontrol+0x558/0xb00 [mdt] [&lt;ffffffffc0a17eeb&gt;] class_handle_ioctl+0x16bb/0x1cc0 [obdclass] [&lt;ffffffffc0a18565&gt;] obd_class_ioctl+0x75/0x170 [obdclass] [&lt;ffffffff9d256490&gt;] do_vfs_ioctl+0x3a0/0x5a0 [&lt;ffffffff9d256731&gt;] SyS_ioctl+0xa1/0xc0 [&lt;ffffffff9d776ddb&gt;] system_call_fastpath+0x22/0x27 [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff </pre> </div></div> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38746/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38746/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13608" title="MDT stuck in WAITING, abort_recov stuck too" class="issue-link" data-issue-key="LU-13608"><del>LU-13608</del></a> tgt: abort recovery while reading update llog<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0496cdf20451f07befebd1cb8a770544ec0f57df</p> <p>Li Dongyang (dongyangli@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/39284" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39284</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13608" title="MDT stuck in WAITING, abort_recov stuck too" class="issue-link" data-issue-key="LU-13608"><del>LU-13608</del></a> tgt: abort recovery while reading update llog<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: be2b297c3edd91d7caab97408bf7f539612a47e6</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=hongchao.zhang" class="user-hover" rel="hongchao.zhang">hongchao.zhang</a> is the issue reported by <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=aboyko" class="user-hover" rel="aboyko">aboyko</a> a different issue?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=pjones" class="user-hover" rel="pjones">pjones</a><br/> Hi, this could be another case of this issue, I will create a new patch to fix it.</p> <p>HongChao Zhang, could you please reproduce the original issue and show that abort recovery fix it?  It's better to have a regression test for it or some reproducer.</p> <p>Alexander Boyko (alexander.boyko@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/39538" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39538</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13608" title="MDT stuck in WAITING, abort_recov stuck too" class="issue-link" data-issue-key="LU-13608"><del>LU-13608</del></a> out: don't return einprogress error<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 48cdb839e17fa99a2cbfe442b4ad86c6a6746e2b</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39538/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39538/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13608" title="MDT stuck in WAITING, abort_recov stuck too" class="issue-link" data-issue-key="LU-13608"><del>LU-13608</del></a> out: don't return einprogress error<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 865aa3f692bccdd9cf7ff6cafeee350e06bb8d76</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39284/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39284/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13608" title="MDT stuck in WAITING, abort_recov stuck too" class="issue-link" data-issue-key="LU-13608"><del>LU-13608</del></a> tgt: abort recovery while reading update llog<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 4142f05811b019df453ce52f6b690ec81fa5897f</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41183" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41183</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13608" title="MDT stuck in WAITING, abort_recov stuck too" class="issue-link" data-issue-key="LU-13608"><del>LU-13608</del></a> out: don't return einprogress error<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 2c6b286c1e596b850cabe0b185c1552b0133496d</p> <p>Alexander Boyko (alexander.boyko@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/39539" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39539</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13608" title="MDT stuck in WAITING, abort_recov stuck too" class="issue-link" data-issue-key="LU-13608"><del>LU-13608</del></a> tests: check MDS recovery hang<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2329127abf2ace5333a10352b77202c55f8da0aa</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/41183/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41183/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13608" title="MDT stuck in WAITING, abort_recov stuck too" class="issue-link" data-issue-key="LU-13608"><del>LU-13608</del></a> out: don't return einprogress error<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 7817acc39ee1d6859c2737f75619748dc8e37f95</p> Related LU-14318 LU-14119 Development Rank 1|i011gf: Rank (Obsolete) 9223372036854775807 Severity