https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5128 Lustre <p>MDS failovered and once MDS's recovery finished, many OSS crahsed due to following ASSERTION.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2014-05-30 17:39:07 Lustre: Skipped 3 previous similar messages 2014-05-30 17:39:07 LustreError: 18967:0:(ldlm_lib.c:1851:target_next_replay_req()) ASSERTION( atomic_read(&amp;obd-&gt;obd_req_replay_clients) == 0 ) failed: 2014-05-30 17:39:07 LustreError: 18967:0:(ldlm_lib.c:1851:target_next_replay_req()) LBUG 2014-05-30 17:39:07 Pid: 18967, comm: tgt_recov 2014-05-30 17:39:07 2014-05-30 17:39:07 Call Trace: 2014-05-30 17:39:07 [&lt;ffffffffa0353895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 2014-05-30 17:39:07 [&lt;ffffffffa0353e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 2014-05-30 17:39:07 [&lt;ffffffffa066f48c&gt;] target_recovery_thread+0x14ac/0x1970 [ptlrpc] 2014-05-30 17:39:07 [&lt;ffffffffa066dfe0&gt;] ? target_recovery_thread+0x0/0x1970 [ptlrpc] 2014-05-30 17:39:07 [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 2014-05-30 17:39:07 [&lt;ffffffffa066dfe0&gt;] ? target_recovery_thread+0x0/0x1970 [ptlrpc] 2014-05-30 17:39:07 [&lt;ffffffffa066dfe0&gt;] ? target_recovery_thread+0x0/0x1970 [ptlrpc] 2014-05-30 17:39:07 [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 2014-05-30 17:39:07 2014-05-30 17:39:07 Kernel panic - not syncing: LBUG 2014-05-30 17:39:07 Pid: 18967, comm: tgt_recov Not tainted 2.6.32-358.18.1.el6_lustre.x86_64 #1 2014-05-30 17:39:07 Call Trace: 2014-05-30 17:39:07 [&lt;ffffffff8150de58&gt;] ? panic+0xa7/0x16f 2014-05-30 17:39:07 [&lt;ffffffffa0353eeb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] 2014-05-30 17:39:07 [&lt;ffffffffa066f48c&gt;] ? target_recovery_thread+0x14ac/0x1970 [ptlrpc] 2014-05-30 17:39:07 [&lt;ffffffffa066dfe0&gt;] ? target_recovery_thread+0x0/0x1970 [ptlrpc] 2014-05-30 17:39:07 [&lt;ffffffff8100c0ca&gt;] ? child_rip+0xa/0x20 2014-05-30 17:39:07 [&lt;ffffffffa066dfe0&gt;] ? target_recovery_thread+0x0/0x1970 [ptlrpc] 2014-05-30 17:39:07 [&lt;ffffffffa066dfe0&gt;] ? target_recovery_thread+0x0/0x1970 [ptlrpc] 2014-05-30 17:39:07 [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p><a href="https://jira.whamcloud.com/browse/LU-1522" title="ASSERTION(cfs_atomic_read(&amp;obd-&gt;obd_req_replay_clients) == 0) failed" class="issue-link" data-issue-key="LU-1522"><del>LU-1522</del></a> and <a href="https://jira.whamcloud.com/browse/LU-2397" title="Assertion triggered in check_for_next_transno" class="issue-link" data-issue-key="LU-2397"><del>LU-2397</del></a> reported similar problem, but these patches have been merged in b2_4, already.</p> Lustre-2.4.3 LU-5128

ASSERTION( atomic_read(&obd->obd_req_replay_clients) == 0 ) failed

Bug Minor Resolved Fixed Hongchao Zhang Shuichi Ihara duu mn4 Sun, 1 Jun 2014 02:37:21 +0000 Mon, 18 Aug 2014 15:21:29 +0000 Wed, 6 Aug 2014 14:58:58 +0000 Lustre 2.4.3 Lustre 2.7.0 Lustre 2.5.3 0 5 <p>Hongchao</p> <p>Could you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Hi,</p> <p>Could you please attach the whole logs of this issue, thanks!<br/> btw, did the OSS also failovered along with MDS?</p> <p>Thanks</p> <p>there could be a race between "target_process_req_flags" and "class_export_recovery_cleanup", and if the replay request contains the flag<br/> "MSG_REQ_REPLAY_DONE", the "exp-&gt;exp_req_replay_needed" will be cleared and "obd-&gt;obd_req_replay_clients" will be decreased with<br/> protection "exp_lock" in "target_process_req_flags". but "class_export_recovery_cleanup" checks the "exp_req_replay_needed" without the lock<br/> "exp_lock", then it could decrement "obd_req_replay_clients" once more and causes this issue.</p> <p>the patch against b2_4 is tracked at <a href="http://review.whamcloud.com/#/c/10628/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10628/</a></p> <p>this only happens on b2_4 branch or same problem maybe occur even on b2_5?</p> <p>the issue tracked at <a href="http://review.whamcloud.com/#/c/10628/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10628/</a> also exists on b2_5</p> <p>the patch against master is tracked at <a href="http://review.whamcloud.com/#/c/10849/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10849/</a></p> <p>Here is the patch for b2_5: <a href="http://review.whamcloud.com/#/c/11102/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11102/</a></p> <p>Landed for 2.7</p> Development Rank 1|hzwni7: Rank (Obsolete) 14153 Severity