https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8815 Lustre <p>After previous failover, MDT0003 is failed back to lola-11:<br/> Nov 8 22:10:14 lola-10 kernel: Lustre: server umount soaked-MDT0003 complete</p> <p>MDS failover fault induced, lola-11 fails MDT0003 to lola-10:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">016-11-08 22:24:25,959:fsmgmt.fsmgmt:INFO Failing over soaked-MDT0003 ... 2016-11-08 22:24:25,959:fsmgmt.fsmgmt:INFO Mounting soaked-MDT0003 on lola-10 ... </pre> </div></div> <p>Recovery starts on MDS: Nov 8 22:25:29 lola-10 kernel: Lustre: soaked-MDT0003: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450</p> <p>Task block on lola-10:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">ov 8 22:29:17 lola-10 kernel: INFO: task mdt_out01_007:6305 blocked <span class="code-keyword">for</span> more than 120 seconds. Nov 8 22:29:17 lola-10 kernel: Tainted: P -- ------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Nov 8 22:29:17 lola-10 kernel: <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. Nov 8 22:29:17 lola-10 kernel: mdt_out01_007 D 0000000000000018 0 6305 2 0x00000080 Nov 8 22:29:17 lola-10 kernel: ffff8808104b7b40 0000000000000046 0000000000000000 ffff8808104b7d00 Nov 8 22:29:17 lola-10 kernel: ffff8804337ea800 ffff880824300118 000014206fe1f7c1 ffff880824300118 Nov 8 22:29:17 lola-10 kernel: ffff8808104b7b40 00000001014d3a26 ffff880834add068 ffff8808104b7fd8 Nov 8 22:29:17 lola-10 kernel: Call Trace: Nov 8 22:29:17 lola-10 kernel: [&lt;ffffffffa07bca3d&gt;] lu_object_find_at+0x3d/0xe0 [obdclass] Nov 8 22:29:17 lola-10 kernel: [&lt;ffffffffa09dd442&gt;] ? __req_capsule_get+0x162/0x6e0 [ptlrpc] Nov 8 22:29:17 lola-10 kernel: [&lt;ffffffff81067650&gt;] ? default_wake_function+0x0/0x20 Nov 8 22:29:17 lola-10 kernel: [&lt;ffffffffa09b2af0&gt;] ? lustre_swab_object_update_reply+0x0/0xc0 [ptlrpc] Nov 8 22:29:17 lola-10 kernel: [&lt;ffffffffa07bd7cc&gt;] dt_locate_at+0x1c/0xa0 [obdclass] Nov 8 22:29:17 lola-10 kernel: [&lt;ffffffffa0a24147&gt;] out_handle+0x1067/0x18d0 [ptlrpc] Nov 8 22:29:17 lola-10 kernel: [&lt;ffffffff8105e9b6&gt;] ? enqueue_task+0x66/0x80 Nov 8 22:29:17 lola-10 kernel: [&lt;ffffffff8105ab8d&gt;] ? check_preempt_curr+0x6d/0x90 Nov 8 22:29:17 lola-10 kernel: [&lt;ffffffffa067ac8a&gt;] ? lc_watchdog_touch+0x7a/0x190 [libcfs] Nov 8 22:29:17 lola-10 kernel: [&lt;ffffffff8153afce&gt;] ? mutex_lock+0x1e/0x50 Nov 8 22:29:17 lola-10 kernel: [&lt;ffffffffa0a142ca&gt;] ? req_can_reconstruct+0x6a/0x120 [ptlrpc] Nov 8 22:29:17 lola-10 kernel: [&lt;ffffffffa0a1b4bc&gt;] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 8 22:29:17 lola-10 kernel: [&lt;ffffffffa09c7bc1&gt;] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 8 22:29:17 lola-10 kernel: [&lt;ffffffff81539b0e&gt;] ? thread_return+0x4e/0x7d0 Nov 8 22:29:17 lola-10 kernel: [&lt;ffffffffa09c6e90&gt;] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 8 22:29:17 lola-10 kernel: [&lt;ffffffff810a138e&gt;] kthread+0x9e/0xc0 Nov 8 22:29:17 lola-10 kernel: [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 Nov 8 22:29:17 lola-10 kernel: [&lt;ffffffff810a12f0&gt;] ? kthread+0x0/0xc0 Nov 8 22:29:18 lola-10 kernel: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Second blocked task:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Nov 8 22:29:32 lola-10 kernel: Call Trace: Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffffa07bca3d&gt;] lu_object_find_at+0x3d/0xe0 [obdclass] Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffffa09dd442&gt;] ? __req_capsule_get+0x162/0x6e0 [ptlrpc] Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffff81067650&gt;] ? default_wake_function+0x0/0x20 Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffffa09b2af0&gt;] ? lustre_swab_object_update_reply+0x0/0xc0 [ptlrpc] Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffffa07bd7cc&gt;] dt_locate_at+0x1c/0xa0 [obdclass] Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffffa0a24147&gt;] out_handle+0x1067/0x18d0 [ptlrpc] Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffff8105e9b6&gt;] ? enqueue_task+0x66/0x80 Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffff8105ab8d&gt;] ? check_preempt_curr+0x6d/0x90 Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffffa067ac8a&gt;] ? lc_watchdog_touch+0x7a/0x190 [libcfs] Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffff8153afce&gt;] ? mutex_lock+0x1e/0x50 Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffffa0a142ca&gt;] ? req_can_reconstruct+0x6a/0x120 [ptlrpc] Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffffa0a1b4bc&gt;] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffffa09c7bc1&gt;] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffff81539b0e&gt;] ? thread_return+0x4e/0x7d0 Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffffa09c6e90&gt;] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffff810a138e&gt;] kthread+0x9e/0xc0 Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffff810a12f0&gt;] ? kthread+0x0/0xc0 Nov 8 22:29:32 lola-10 kernel: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Recovery timer competes, but recovery does not:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">mdt.soaked-MDT0003.recovery_status= status: RECOVERING recovery_start: 1478672734 time_remaining: 0 connected_clients: 21/21 req_replay_clients: 4 lock_repay_clients: 5 completed_clients: 16 evicted_clients: 0 replayed_requests: 7 queued_requests: 4 next_transno: 77309522798 </pre> </div></div> <p>System finally notices: </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Nov 9 08:22:35 lola-10 kernel: Lustre: soaked-MDT0003: Recovery already passed deadline 588:42, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. </pre> </div></div> <p>Attached lustre-log dumps from lola-10</p> Soak test cluster, tip of lustre-master lustre: 2.8.59_79_gb8811a0 LU-8815

MDS fails to complete recovery

Bug Critical Closed Duplicate nasf Cliff White soak Wed, 9 Nov 2016 17:54:42 +0000 Fri, 30 Jun 2017 01:51:09 +0000 Fri, 30 Jun 2017 01:51:09 +0000 Lustre 2.9.0 0 9 <p>Prior to this issue, soak test had 4 successful MDS failovers, 8 successful MDS restarts.</p> <p>The recovery stuck here</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>tgt_recover_3 S 0000000000000008 0 9320 2 0x00000080 ffff8803539037f0 0000000000000046 0000000000000000 ffff8803c5a25520 ffff88041d3fa040 ffff8803c5a25520 0000345d232c77d9 ffff8807ba8ae1c0 ffff8807fec20000 00000001036a5b96 ffff8803c5a25ad8 ffff880353903fd8 Call Trace: [&lt;ffffffff8153a9b2&gt;] schedule_timeout+0x192/0x2e0 [&lt;ffffffff81089fa0&gt;] ? process_timeout+0x0/0x10 [&lt;ffffffffa09abbb1&gt;] ptlrpc_set_wait+0x321/0x960 [ptlrpc] [&lt;ffffffffa09a0ce0&gt;] ? ptlrpc_interrupted_set+0x0/0x120 [ptlrpc] [&lt;ffffffff81067650&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa09b80a5&gt;] ? lustre_msg_set_jobid+0xf5/0x130 [ptlrpc] [&lt;ffffffffa09ac271&gt;] ptlrpc_queue_wait+0x81/0x220 [ptlrpc] [&lt;ffffffffa12b6cc2&gt;] osp_remote_sync+0xf2/0x1e0 [osp] [&lt;ffffffffa129b1c7&gt;] osp_attr_get+0x447/0x710 [osp] [&lt;ffffffff811791ea&gt;] ? kmem_cache_alloc+0x18a/0x190 [&lt;ffffffffa129bd25&gt;] osp_object_init+0x1f5/0x360 [osp] [&lt;ffffffffa07bb508&gt;] lu_object_alloc+0xd8/0x320 [obdclass] [&lt;ffffffffa07bc8f1&gt;] lu_object_find_try+0x151/0x260 [obdclass] [&lt;ffffffffa07bcab1&gt;] lu_object_find_at+0xb1/0xe0 [obdclass] [&lt;ffffffffa07bbac5&gt;] ? lu_object_put+0x135/0x3b0 [obdclass] [&lt;ffffffffa07bd7cc&gt;] dt_locate_at+0x1c/0xa0 [obdclass] [&lt;ffffffffa0a3457a&gt;] update_recovery_exec+0xfa/0x1ce0 [ptlrpc] [&lt;ffffffffa0a37da1&gt;] distribute_txn_replay_handle+0x271/0xcf0 [ptlrpc] [&lt;ffffffffa0979402&gt;] target_recovery_thread+0xa12/0x1dd0 [ptlrpc] [&lt;ffffffffa09789f0&gt;] ? target_recovery_thread+0x0/0x1dd0 [ptlrpc] [&lt;ffffffff810a138e&gt;] kthread+0x9e/0xc0 [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 [&lt;ffffffff810a12f0&gt;] ? kthread+0x0/0xc0 [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>And on the remote MDT, most thread stuck at</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mdt_out01_012 D 0000000000000009 0 19980 2 0x00000080 ffff8803facd3b40 0000000000000046 0000000000000000 ffff8803facd3d00 ffff8804337ea800 ffff880824e00118 0000213d70f62b10 ffff880824e00118 ffff8803facd3b40 00000001022959fb ffff8803f16c1ad8 ffff8803facd3fd8 Call Trace: [&lt;ffffffffa07bca3d&gt;] lu_object_find_at+0x3d/0xe0 [obdclass] [&lt;ffffffffa09dd442&gt;] ? __req_capsule_get+0x162/0x6e0 [ptlrpc] [&lt;ffffffff81067650&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa09b2af0&gt;] ? lustre_swab_object_update_reply+0x0/0xc0 [ptlrpc] [&lt;ffffffffa07bd7cc&gt;] dt_locate_at+0x1c/0xa0 [obdclass] [&lt;ffffffffa0a24147&gt;] out_handle+0x1067/0x18d0 [ptlrpc] [&lt;ffffffff8105e9b6&gt;] ? enqueue_task+0x66/0x80 [&lt;ffffffff8105ab8d&gt;] ? check_preempt_curr+0x6d/0x90 [&lt;ffffffffa067ac8a&gt;] ? lc_watchdog_touch+0x7a/0x190 [libcfs] [&lt;ffffffff8153afce&gt;] ? mutex_lock+0x1e/0x50 [&lt;ffffffffa0a142ca&gt;] ? req_can_reconstruct+0x6a/0x120 [ptlrpc] [&lt;ffffffffa0a1b4bc&gt;] tgt_request_handle+0x8ec/0x1440 [ptlrpc] [&lt;ffffffffa09c7bc1&gt;] ptlrpc_main+0xd31/0x1800 [ptlrpc] [&lt;ffffffff81539b0e&gt;] ? thread_return+0x4e/0x7d0 [&lt;ffffffffa09c6e90&gt;] ? ptlrpc_main+0x0/0x1800 [ptlrpc] [&lt;ffffffff810a138e&gt;] kthread+0x9e/0xc0 [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 [&lt;ffffffff810a12f0&gt;] ? kthread+0x0/0xc0 [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Just glance the trace a bit, no obvious evidence why it stuck here. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>The abort_recovery command never completes:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LustreError: 72214:0:(mdt_handler.c:5942:mdt_iocontrol()) soaked-MDT0003: Aborting recovery <span class="code-keyword">for</span> device LustreError: 72214:0:(ldlm_lib.c:2596:target_stop_recovery_thread()) soaked-MDT0003: Aborting recovery LustreError: 6078:0:(ldlm_lib.c:2816:target_queue_recovery_request()) @@@ dropping resent queued req req@ffff88 03f78f03c0 x1550360654345360/t0(77309522798) o36-&gt;3d670dc0-f7e5-756a-f555-c09f70a29db2@192.168.1.117@o2ib100:-1/ -1 lens 768/0 e 0 to 0 dl 1478715043 ref 2 fl Interpret:/6/ffffffff rc 0/-1 LustreError: 6078:0:(ldlm_lib.c:2816:target_queue_recovery_request()) Skipped 169 previous similar messages </pre> </div></div> <p>Hit this again testing 2.9.0 GA </p> <p>Any more information here? console or stack trace. Thanks.</p> <p>Currently, system cannot recover. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Jan 20 08:59:41 lola-8 kernel: Pid: 6236, comm: OI_scrub Tainted: P --L------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ Jan 20 08:59:41 lola-8 kernel: RIP: 0010:[&lt;ffffffffa10d6e98&gt;] [&lt;ffffffffa10d6e98&gt;] osd_inode_iteration+0x498/0xd80 [osd_ldiskfs] Jan 20 08:59:41 lola-8 kernel: RSP: 0018:ffff8804173d7d10 EFLAGS: 00000202 Jan 20 08:59:41 lola-8 kernel: RAX: 0000000000000004 RBX: ffff8804173d7e00 RCX: ffff8804173d7dc0 Jan 20 08:59:41 lola-8 kernel: RDX: ffff8804173d7da0 RSI: ffff8808296fc000 RDI: ffff88082d394000 Jan 20 08:59:41 lola-8 kernel: RBP: ffffffff8100bc0e R08: 0000000000000001 R09: 0000000000000004 Jan 20 08:59:41 lola-8 kernel: R10: ffff880417371b80 R11: ffff880417371b90 R12: ffff8804173d7e00 Jan 20 08:59:41 lola-8 kernel: R13: ffff8808296fd000 R14: ffff8808296fd0e0 R15: 0000000000000001 Jan 20 08:59:41 lola-8 kernel: FS: 0000000000000000(0000) GS:ffff880038680000(0000) knlGS:0000000000000000 Jan 20 08:59:41 lola-8 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Jan 20 08:59:41 lola-8 kernel: CR2: 00000039036acd90 CR3: 0000000001a8d000 CR4: 00000000000407e0 Jan 20 08:59:41 lola-8 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 20 08:59:41 lola-8 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 20 08:59:41 lola-8 kernel: <span class="code-object">Process</span> OI_scrub (pid: 6236, threadinfo ffff8804173d4000, task ffff880435f34040) Jan 20 08:59:41 lola-8 kernel: Stack: Jan 20 08:59:41 lola-8 kernel: 0000000000015a00 ffff8808331d4c00 ffff8808000005a6 ffff8804173d7e10 Jan 20 08:59:41 lola-8 kernel: &lt;d&gt; ffff8804173d7dd0 ffffffff81539b0e ffff880435f34040 ffff880435f34040 Jan 20 08:59:41 lola-8 kernel: &lt;d&gt; ffffffffa10d1e20 ffffffffa10d3be0 ffff88082d394000 ffff8808345e0000 Jan 20 08:59:41 lola-8 kernel: Call Trace: Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffff81539b0e&gt;] ? thread_return+0x4e/0x7d0 Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffffa10d1e20&gt;] ? osd_scrub_exec+0x0/0x1dc0 [osd_ldiskfs] Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffffa10d3be0&gt;] ? osd_scrub_next+0x0/0x4b0 [osd_ldiskfs] Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffffa10d8905&gt;] ? osd_scrub_main+0x885/0xec0 [osd_ldiskfs] Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffff81067650&gt;] ? default_wake_function+0x0/0x20 Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffffa10d8080&gt;] ? osd_scrub_main+0x0/0xec0 [osd_ldiskfs] Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffff810a138e&gt;] ? kthread+0x9e/0xc0 Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffff8100c28a&gt;] ? child_rip+0xa/0x20 Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffff810a12f0&gt;] ? kthread+0x0/0xc0 Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 Jan 20 08:59:41 lola-8 kernel: Code: 00 02 05 00 00 48 c7 05 a3 59 02 00 00 00 00 00 c7 05 91 59 02 00 01 00 00 00 e8 d4 ad 7b ff e9 d2 fb ff ff 0f 1f 80 00 00 00 00 &lt;83&gt; f8 05 0f 84 67 03 00 00 83 f8 06 0f 84 a6 04 00 00 41 89 c1 Jan 20 08:59:41 lola-8 kernel: Call Trace: Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffff81539b0e&gt;] ? thread_return+0x4e/0x7d0 Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffffa10d1e20&gt;] ? osd_scrub_exec+0x0/0x1dc0 [osd_ldiskfs] Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffffa10d3be0&gt;] ? osd_scrub_next+0x0/0x4b0 [osd_ldiskfs] Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffffa10d8905&gt;] ? osd_scrub_main+0x885/0xec0 [osd_ldiskfs] Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffff81067650&gt;] ? default_wake_function+0x0/0x20 Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffffa10d8080&gt;] ? osd_scrub_main+0x0/0xec0 [osd_ldiskfs] Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffff810a138e&gt;] ? kthread+0x9e/0xc0 Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffff8100c28a&gt;] ? child_rip+0xa/0x20 Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffff810a12f0&gt;] ? kthread+0x0/0xc0 Jan 20 08:59:41 lola-8 kernel: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>According to the bug description, it seems that one MDT was trying to recover/replay some cross-MDTs RPC, that triggered OUT_ATTR_GET RPC. Unfortunately, such OUT RPC was blocked on remote MDT inside lu_object_find_at(). Usually, the lu_object_find_at() will be blocked if the target object is in-RAM but marked as dying. If someone hold the reference on the dying object, then others will be blocked until all the reference have been released. So we need to find out what the target object is, and who hold the reference on such dying object.</p> <p>Do we have any clew about the first question: what the target object is?<br/> Di, is it possible an update_log file on remote MDT?</p> <p>As for the stack trace for OI scrub in above comment, that seems not related. Because OI scrub operates the inode directly, NOT reference the up layer object.</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/25260" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25260</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8815" title="MDS fails to complete recovery" class="issue-link" data-issue-key="LU-8815"><del>LU-8815</del></a> obdclass: debug information for lu_object_find_at<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e8c1940e2e7b0afa0b54d139c70359425bf92d40</p> <p>Cliff, would you please to verify the debug patch <a href="https://review.whamcloud.com/#/c/25260/?" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/25260/?</a> That can tell us which FID caused the blocking. Thanks!</p> <p>+1 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/b54d8c9c-eee8-11e6-9756-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b54d8c9c-eee8-11e6-9756-5254006e85c2</a></p> <p>Cliff, do you have any change to reproduce the issue with the patch <a href="https://review.whamcloud.com/#/c/25260/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/25260/</a> applied? Thanks!</p> <p>We are moving soak hardware this week, will add that to the list.</p> <p>Any update with the patch applied? Thanks!</p> <p>We have been busy with the move, but have not seen any recovery issues. I should be able to try the patch this week, is there a build?</p> <p><a href="https://review.whamcloud.com/#/c/25260/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/25260/</a><br/> set 4 build status:<br/> Overall Build Status: SUCCESS<br/> Builder CentOS 6.7 x86_64 (BUILD) succeeded (build successful) - <a href="http://build.lustre.org/builders/CentOS%206.7%20x86_64%20%28BUILD%29/builds/7691" class="external-link" target="_blank" rel="nofollow noopener">http://build.lustre.org/builders/CentOS%206.7%20x86_64%20%28BUILD%29/builds/7691</a><br/> Builder CentOS 6.8 x86_64 (BUILD) succeeded (build successful) - <a href="http://build.lustre.org/builders/CentOS%206.8%20x86_64%20%28BUILD%29/builds/3827" class="external-link" target="_blank" rel="nofollow noopener">http://build.lustre.org/builders/CentOS%206.8%20x86_64%20%28BUILD%29/builds/3827</a><br/> Builder CentOS 7.2 x86_64 (BUILD) succeeded (build successful) - <a href="http://build.lustre.org/builders/CentOS%207.2%20x86_64%20%28BUILD%29/builds/7677" class="external-link" target="_blank" rel="nofollow noopener">http://build.lustre.org/builders/CentOS%207.2%20x86_64%20%28BUILD%29/builds/7677</a><br/> Builder Ubuntu 14.04 x86_64 (BUILD) succeeded (build successful) - <a href="http://build.lustre.org/builders/Ubuntu%2014.04%20x86_64%20%28BUILD%29/builds/7637" class="external-link" target="_blank" rel="nofollow noopener">http://build.lustre.org/builders/Ubuntu%2014.04%20x86_64%20%28BUILD%29/builds/7637</a></p> <p>We have also moved to RHEL 7 with new soak, can you re-trigger your build? Jenkins no longer has the bits. I get RPMS from Jenkins. </p> <p>New build:<br/> <a href="https://build.hpdd.intel.com/job/lustre-reviews/46754/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-reviews/46754/</a></p> <p>It may be that this is fixed with patch <a href="https://review.whamcloud.com/26965" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26965</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-9049" title="DNE MDT Never completes recovery" class="issue-link" data-issue-key="LU-9049"><del>LU-9049</del></a> obdclass: unhash object when destroying object</tt>".</p> <p>We have not seen a recovery failure since that time. However, we've had other hard bugs that reduced runtime in some cases. </p> <p>Ok I will remove the fix version for the time being and we can continue to monitor</p> <p>Another instance of <a href="https://jira.whamcloud.com/browse/LU-9049" title="DNE MDT Never completes recovery" class="issue-link" data-issue-key="LU-9049"><del>LU-9049</del></a>.</p> Related LU-9049 Development Rank 1|hzyv1j: Rank (Obsolete) 9223372036854775807 Severity