https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6928 Lustre <p>During 24 hours DNE failover test, one of client fails because of Version mismatch during replay.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: 7977:0:(client.c:2828:ptlrpc_replay_interpret()) @@@ Version mismatch during replay req@ffff8806698209c0 x1508081941826144/t21475879788(21475879788) o36-&gt;lustre-MDT0001-mdc-ffff880821cfdc00@192.168.2.126@o2ib:12/10 lens 608/424 e 1 to 0 dl 1438262329 ref 2 fl Interpret:R/4/0 rc -75/-75 Lustre: 7977:0:(import.c:1301:completed_replay_interpret()) lustre-MDT0001-mdc-ffff880821cfdc00: version recovery fails, reconnecting LustreError: 167-0: lustre-MDT0001-mdc-ffff880821cfdc00: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. LustreError: 9213:0:(vvp_io.c:1475:vvp_io_init()) lustre: refresh file layout [0x240002341:0x1d74:0x0] error -5. LustreError: 29913:0:(lmv_obd.c:1332:lmv_fid_alloc()) Can't alloc new fid, rc -19 Lustre: lustre-MDT0001-mdc-ffff880821cfdc00: Connection restored to lustre-MDT0001 (at 192.168.2.126@o2ib) Lustre: DEBUG MARKER: ==== Checking the clients loads BEFORE failover -- failure NOT OK ELAPSED=43433 DURATION=86400 PERIOD=1800 Lustre: DEBUG MARKER: Client load failed on node c01, rc=1 Lustre: DEBUG MARKER: Duration: 86400 Lustre: Unmounted lustre-client </pre> </div></div> LU-6928

Version mismatch during DNE replay

Bug Major Resolved Duplicate Niu Yawei Di Wang Thu, 30 Jul 2015 15:03:17 +0000 Thu, 27 Aug 2015 16:41:25 +0000 Thu, 27 Aug 2015 16:41:25 +0000 Lustre 2.8.0 Lustre 2.8.0 0 5 <p>Di - I'm seeing a very similar error in replay-single test 48 in review-dne-part-2. Do you think this is the same issue? The logs are at: <a href="https://testing.hpdd.intel.com/test_sets/8281843c-365f-11e5-830b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8281843c-365f-11e5-830b-5254006e85c2</a></p> <p>The client console shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>18:10:31:Lustre: DEBUG MARKER: == replay-single test 48: MDS-&gt;OSC failure during precreate cleanup (2824) == 18:08:48 (1438193328) 18:10:31:Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname) 18:10:31:Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi 18:10:31:Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 18:10:31:LustreError: 7733:0:(client.c:2816:ptlrpc_replay_interpret()) request replay timed out, restarting recovery 18:10:31:Lustre: 7733:0:(client.c:2828:ptlrpc_replay_interpret()) @@@ Version mismatch during replay 18:10:31: req@ffff8800379e8c80 x1508053244555028/t197568495717(197568495717) o36-&gt;lustre-MDT0000-mdc-ffff88007b65b000@10.1.4.62@tcp:12/10 lens 640/416 e 2 to 0 dl 1438193365 ref 1 fl Interpret:R/6/0 rc -75/-75 18:10:31:Lustre: 7733:0:(import.c:1306:completed_replay_interpret()) lustre-MDT0000-mdc-ffff88007b65b000: version recovery fails, reconnecting 18:10:31:LustreError: 167-0: lustre-MDT0000-mdc-ffff88007b65b000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. 18:10:31:LustreError: Skipped 1 previous similar message 18:10:31:LustreError: 13949:0:(lmv_obd.c:1473:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff88007b65b000), error -5 18:10:31:LustreError: 13949:0:(llite_lib.c:1707:ll_statfs_internal()) md_statfs fails: rc = -5 18:10:31:Lustre: DEBUG MARKER: /usr/sbin/lctl mark replay-single test_48: @@@@@@ FAIL: client_up failed 18:10:31:Lustre: DEBUG MARKER: replay-single test_48: @@@@@@ FAIL: client_up failed </pre> </div></div> <p>Yes, looks like same issue, I believe it is a bug caused by multiple slot patch. I will cook a patch.</p> <p>Hmm, I think this is what happened</p> <p>During replay</p> <p>1. client send replay request to MDS01, and MDS01 did not handle it in time. <br/> 2. So client did not get reply, so it expires and reconnects the import. And during sending this CONNECT request, it will first find out the min XID first, <br/> but it only tries to find such XID in sending and replay list, which is wrong, because the replay request (just timeout) are not in these list (see ptlrpc_check_set()) . So usually, it will get the xid of this connect request, which is obviously larger than the xid of replay request.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /* find the lowest unreplied XID */ list_for_each(tmp, &amp;imp-&gt;imp_delayed_list) { struct ptlrpc_request *r; r = list_entry(tmp, struct ptlrpc_request, rq_list); if (r-&gt;rq_xid &lt; min_xid) min_xid = r-&gt;rq_xid; } list_for_each(tmp, &amp;imp-&gt;imp_sending_list) { struct ptlrpc_request *r; r = list_entry(tmp, struct ptlrpc_request, rq_list); if (r-&gt;rq_xid &lt; min_xid) min_xid = r-&gt;rq_xid; } </pre> </div></div> <p>Delete req from sending list after req failure.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /* Request already may be not on sending or delaying list. This * may happen in the case of marking it erroneous for the case * ptlrpc_import_delay_req(req, status) find it impossible to * allow sending this rpc and returns *status != 0. */ if (!list_empty(&amp;req-&gt;rq_list)) { list_del_init(&amp;req-&gt;rq_list); atomic_dec(&amp;imp-&gt;imp_inflight); } spin_unlock(&amp;imp-&gt;imp_lock); </pre> </div></div> <p>3. And MDT handle that replay request (mentioned in 1) at this time, which will add the lrd of this req into the reply_list.<br/> 4. Then connect request arrives, and it has bigger reply XID, which then delete the lrd created in step 3<br/> 5. After connect, client will resend the replay request, but server can not identify it as a already received request, because lrd has been deleted in 3. So the replay request has been executed twice, then it causes this Version mismatch issue. </p> <p>so the fix might be in step 2, i.e. we need consider the expired request when finding this min XID, or we just do not pack this min xid for CONNECT request. </p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15812" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15812</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6928" title="Version mismatch during DNE replay" class="issue-link" data-issue-key="LU-6928"><del>LU-6928</del></a> ptlrpc: Do not pack min XID for connect req<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: acb892354ac119dfe80edf60f5f2a2923cdcc980</p> <p>Niu: could you please confirm if your patch can fix this problem? Thanks</p> <p>Yes, the patch for <a href="https://jira.whamcloud.com/browse/LU-5951" title="sanity test_39k: mtime is lost on close" class="issue-link" data-issue-key="LU-5951"><del>LU-5951</del></a> (<a href="http://review.whamcloud.com/#/c/15473/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15473/</a>) will fix this problem.</p> <p>Duplicate with <a href="https://jira.whamcloud.com/browse/LU-5951" title="sanity test_39k: mtime is lost on close" class="issue-link" data-issue-key="LU-5951"><del>LU-5951</del></a></p> Related LU-6831 Development Rank 1|hzxjcv: Rank (Obsolete) 9223372036854775807 Severity