https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5651 Lustre <p>Client doesn't restore import state correctly<br/> on reconnect during replay. It resends lock replay<br/> when final ping was queued by server.<br/> Server fails with "target_queue_recovery_request())<br/> ASSERTION( req-&gt;rq_export-&gt;exp_lock_replay_needed ) failed"</p> <p>Solution is to add imp_replay_state to store last replay state. <br/> During reconnect imp_state is restored from imp_replay_state.</p> LU-5651

ASSERTION( req->rq_export->exp_lock_replay_needed ) failed

Bug Minor Resolved Fixed Niu Yawei Andriy Skulysh patch Tue, 23 Sep 2014 11:16:29 +0000 Fri, 29 May 2015 12:32:28 +0000 Thu, 15 Jan 2015 07:44:03 +0000 Lustre 2.7.0 Lustre 2.5.4 0 12 <p>patch: <a href="http://review.whamcloud.com/12015" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12015</a></p> <p>This looks dup of <a href="https://jira.whamcloud.com/browse/LU-5287" title="(ldlm_lib.c:2253:target_queue_recovery_request()) ASSERTION( req-&gt;rq_export-&gt;exp_lock_replay_needed ) failed" class="issue-link" data-issue-key="LU-5287"><del>LU-5287</del></a> and I posted a fix at: <a href="http://review.whamcloud.com/#/c/11871/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11871/</a></p> <blockquote> <p>Client doesn't restore import state correctly<br/> on reconnect during replay. It resends lock replay<br/> when final ping was queued by server.<br/> Server fails with "target_queue_recovery_request())<br/> ASSERTION( req-&gt;rq_export-&gt;exp_lock_replay_needed ) failed"</p></blockquote> <p>If the final ping has been queued on server, the recovery on this export should be finished (exp_in_recovery == 0) when the resent lock replay reach server. I don't see why that could trigger the assertion, could you explain it in detail?</p> <p>The recovery ends only on receiving and processing final pings from all clients. I can happen that server accepted final ping from client1 but waits for requests from client2 and client1 reconnects. This situation was simulated in test by adding timeout before processing final pings. </p> <blockquote> <p>The recovery ends only on receiving and processing final pings from all clients.</p></blockquote> <p>I was saying recovery on this export is done. (exp_in_recovery == 0).</p> <p>exp_in_recovery is zeroed during recovery final stage 3, client1 reconnects during lock replay stage 2.</p> <p>step by step explanation: <br/> 1) server goes to recovery stage 2 (lock replay) with at least 2 clients.<br/> 2) client1 and client 2 send lock relay,<br/> 3) client1 sends final ping<br/> 4) server queues final ping from client1 and set exp_lock_replay_needed to 0<br/> 5) client2 still in lock replay stage (waits for lock replies form server)<br/> 6) client1 reconnects<br/> 7) client1 replays locks from the beginning <br/> 8) assertion fails </p> <blockquote> <p>step by step explanation: <br/> 1) server goes to recovery stage 2 (lock replay) with at least 2 clients.<br/> 2) client1 and client 2 send lock relay,<br/> 3) client1 sends final ping<br/> 4) server queues final ping from client1 and set exp_lock_replay_needed to 0<br/> 5) client2 still in lock replay stage (waits for lock replies form server)<br/> 6) client1 reconnects</p></blockquote> <p>Is the final ping from client1 still in queue? If it's still in queue, client1 can't reconnect because there is inflgiht RPC; If the final ping has been processed, the exp_in_recovery should have been cleared.</p> <p>no. request reaches timeout on reply, client reconnects </p> <blockquote> <p>no. request reaches timeout on reply, client reconnects</p></blockquote> <p>Could you explain how can client1 reconnect when the final ping is in queue (server would reject the reconnect because the export has inflight RPC)? Is there defect in reconnect path?</p> <p>I don't understand why reconnect should be rejected. The only check for inflight RPC on a export is </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>no_export: OBD_FAIL_TIMEOUT(OBD_FAIL_TGT_DELAY_CONNECT, 2 * obd_timeout); } else if (req-&gt;rq_export == NULL &amp;&amp; atomic_read(&amp;export-&gt;exp_rpc_count) &gt; 0) { LCONSOLE_WARN("%s: Client %s (at %s) refused connection, " "still busy with %d references\n", target-&gt;obd_name, cluuid.uuid, libcfs_nid2str(req-&gt;rq_peer.nid), atomic_read(&amp;export-&gt;exp_refcount)); GOTO(out, rc = -EBUSY); </pre> </div></div> <p>but reconnect request has valid export handle.</p> <p>Ah, my mistake, the code has been changed by <a href="https://jira.whamcloud.com/browse/LU-793" title="Reconnections should not be refused when there is a request in progress from this client." class="issue-link" data-issue-key="LU-793"><del>LU-793</del></a>, so client can reconnect now. I'll review the patch soon, thanks for your explanation.</p> <p>ORNL hit this issue today in production after upgrading to Lustre 2.5.2 on both server and client.</p> <p>Patch for b2_5 at <a href="http://review.whamcloud.com/#/c/12163/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12163/</a></p> <p>The master patch has landed. Is there more work needed to complete this ticket or should it be closed?</p> <p>I believe the replay-single test needs to be updated to test for lustre versions so interop testing passes.</p> <p>James Simmons (uja.ornl@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/12942" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12942</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5651" title="ASSERTION( req-&gt;rq_export-&gt;exp_lock_replay_needed ) failed" class="issue-link" data-issue-key="LU-5651"><del>LU-5651</del></a> test: run replay-single test 93 only when supported.<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7bc6d9b8c2e09027444b31c40dd320e244c2412d</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12942/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12942/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5651" title="ASSERTION( req-&gt;rq_export-&gt;exp_lock_replay_needed ) failed" class="issue-link" data-issue-key="LU-5651"><del>LU-5651</del></a> test: run replay-single test 93 only when supported.<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: afde9f17260650d0cb80d53613fb5afda0a39384</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12163/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12163/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5651" title="ASSERTION( req-&gt;rq_export-&gt;exp_lock_replay_needed ) failed" class="issue-link" data-issue-key="LU-5651"><del>LU-5651</del></a>: ptlrpc: fix import state during replay<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: <br/> Commit: 5748379d28846c672f793ba1f8e143e63531dd05</p> <p>Patches landed to master and b2_5 branches.</p> Duplicate LU-5719 Related LU-5287 Development Rank 1|hzwwwf: Rank (Obsolete) 15837 Severity