https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13614 Lustre <p>This issue was created by maloo for jianyu &lt;yujian@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/7ecf29f1-b3ea-45ad-8eaa-e759af2b2c8c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/7ecf29f1-b3ea-45ad-8eaa-e759af2b2c8c</a></p> <p>test_117 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>trevis-35vm9 crashed during replay-single test_117 Lustre: 19264:0:(ldlm_lib.c:1893:extend_recovery_timer()) Skipped 1 previous similar message LustreError: 19264:0:(ldlm_lib.c:2601:replay_request_or_update()) ASSERTION( atomic_read(&amp;obd-&gt;obd_req_replay_clients) == 0 ) failed: LustreError: 19264:0:(ldlm_lib.c:2601:replay_request_or_update()) LBUG Pid: 19264, comm: tgt_recover_0 3.10.0-1062.18.1.el7_lustre.x86_64 #1 SMP Wed May 27 23:19:17 UTC 2020 Call Trace: [&lt;ffffffffc09901ac&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [&lt;ffffffffc099025c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [&lt;ffffffffc128bd47&gt;] replay_request_or_update.isra.24+0x867/0x8d0 [ptlrpc] [&lt;ffffffffc128c4e5&gt;] target_recovery_thread+0x735/0x11a0 [ptlrpc] [&lt;ffffffffbb0c6321&gt;] kthread+0xd1/0xe0 [&lt;ffffffffbb78ed37&gt;] ret_from_fork_nospec_end+0x0/0x39 [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff Kernel panic - not syncing: LBUG CPU: 0 PID: 19264 Comm: tgt_recover_0 Kdump: loaded Tainted: P OE ------------ 3.10.0-1062.18.1.el7_lustre.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: [&lt;ffffffffbb77b416&gt;] dump_stack+0x19/0x1b [&lt;ffffffffbb774a0b&gt;] panic+0xe8/0x21f [&lt;ffffffffc09902ab&gt;] lbug_with_loc+0x9b/0xa0 [libcfs] [&lt;ffffffffc128bd47&gt;] replay_request_or_update.isra.24+0x867/0x8d0 [ptlrpc] [&lt;ffffffffc128c4e5&gt;] target_recovery_thread+0x735/0x11a0 [ptlrpc] [&lt;ffffffffc128bdb0&gt;] ? replay_request_or_update.isra.24+0x8d0/0x8d0 [ptlrpc] [&lt;ffffffffbb0c6321&gt;] kthread+0xd1/0xe0 [&lt;ffffffffbb0c6250&gt;] ? insert_kthread_work+0x40/0x40 [&lt;ffffffffbb78ed37&gt;] ret_from_fork_nospec_begin+0x21/0x21 [&lt;ffffffffbb0c6250&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> <p>&lt;&lt;Please provide additional information about the failure here&gt;&gt;</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> replay-single test_117 - trevis-35vm9 crashed during replay-single test_117</p> LU-13614

replay-single test_117: LBUG: ASSERTION( atomic_read(&obd->obd_req_replay_clients) == 0 ) failed

Bug Major Resolved Fixed Vladimir Saveliev Maloo Sat, 30 May 2020 06:32:43 +0000 Sun, 20 Aug 2023 14:57:02 +0000 Mon, 12 Oct 2020 12:48:08 +0000 Lustre 2.14.0 Lustre 2.14.0 0 6 <p>Vladimir Saveliev (c17830@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/39532" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39532</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13614" title="replay-single test_117: LBUG: ASSERTION( atomic_read(&amp;obd-&gt;obd_req_replay_clients) == 0 ) failed" class="issue-link" data-issue-key="LU-13614"><del>LU-13614</del></a> ldlm: revert <a href="https://jira.whamcloud.com/browse/LU-11762" title="replay-single test 0d fails with &#39;post-failover df failed&#39;" class="issue-link" data-issue-key="LU-11762"><del>LU-11762</del></a><br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ecd083651da5648936acb73170756a8bd3cd57cf</p> <blockquote> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 19264:0:(ldlm_lib.c:2601:replay_request_or_update()) ASSERTION( atomic_read(&amp;obd-&gt;obd_req_replay_clients) == 0 ) failed: </pre> </div></div></blockquote> <p><a href="https://review.whamcloud.com/#/c/35627/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/35627/</a> could be responsible for this assertion.</p> <p>That patch cares about target_recovery_overseer() so that it did not call</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> wait_event_timeout(check_for_next_transno) </pre> </div></div> <p>with inactive recovery timer.</p> <p>But that seems to be impossible. If recovery timeout == hard recovery timeout and if the timer is inactive then with goto repeat target_recovery_overseer() goes to:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (obd-&gt;obd_recovery_start != 0 &amp;&amp; ktime_get_seconds() &gt;= (obd-&gt;obd_recovery_start + obd-&gt;obd_recovery_time_hard)) { ... </pre> </div></div> <p>where abort recovery flag will be set.</p> <p>So, I propose to reverse the "<a href="https://jira.whamcloud.com/browse/LU-11762" title="replay-single test 0d fails with &#39;post-failover df failed&#39;" class="issue-link" data-issue-key="LU-11762"><del>LU-11762</del></a> ldlm: ensure the recovery timer is armed".</p> <p>If I miss something please point where I am wrong.<br/>  </p> <p> </p> <p> </p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39532/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39532/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13614" title="replay-single test_117: LBUG: ASSERTION( atomic_read(&amp;obd-&gt;obd_req_replay_clients) == 0 ) failed" class="issue-link" data-issue-key="LU-13614"><del>LU-13614</del></a> ldlm: revert <a href="https://jira.whamcloud.com/browse/LU-11762" title="replay-single test 0d fails with &#39;post-failover df failed&#39;" class="issue-link" data-issue-key="LU-11762"><del>LU-11762</del></a><br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 2d24238a80be9ca924369d142148d4f6f1891102</p> <p>Landed for 2.14</p> Duplicate Related LU-11762 LU-13339 Development Rank 1|i011nz: Rank (Obsolete) 9223372036854775807 Severity