Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-6841] replay-single test_30: multiop 20786 failed https://jira.whamcloud.com/browse/LU-6841 Lustre <p>This issue was created by maloo for John Hammond &lt;john.hammond@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/3f0a5c78-27ba-11e5-b37a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3f0a5c78-27ba-11e5-b37a-5254006e85c2</a>.</p> <p>The sub-test test_30 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>multiop 20786 failed </pre> </div></div> <p>Please provide additional information about the failure here.</p> <p>Info required for matching: replay-single 30</p> LU-6841 replay-single test_30: multiop 20786 failed Bug Critical Closed Duplicate Hongchao Zhang Maloo Mon, 13 Jul 2015 14:23:35 +0000 Thu, 28 Jan 2016 08:18:44 +0000 Thu, 28 Jan 2016 08:18:44 +0000 0 11 <p>We've hit this several times in review-dne-part-2:<br/> 2015-07-02 22:11:37 - <a href="https://testing.hpdd.intel.com/test_sets/2a34f08c-214e-11e5-b398-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2a34f08c-214e-11e5-b398-5254006e85c2</a><br/> 2015-07-06 01:11:19 - <a href="https://testing.hpdd.intel.com/test_sets/5dfd340e-23c6-11e5-a5be-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5dfd340e-23c6-11e5-a5be-5254006e85c2</a><br/> 2015-07-07 14:35:41 - <a href="https://testing.hpdd.intel.com/test_sets/964e83c2-2505-11e5-9d0f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/964e83c2-2505-11e5-9d0f-5254006e85c2</a><br/> 2015-07-07 16:17:47 - <a href="https://testing.hpdd.intel.com/test_sets/0dc78356-251d-11e5-bf7b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0dc78356-251d-11e5-bf7b-5254006e85c2</a><br/> 2015-07-08 13:58:58 - <a href="https://testing.hpdd.intel.com/test_sets/bd146d1c-25c5-11e5-a6b1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/bd146d1c-25c5-11e5-a6b1-5254006e85c2</a><br/> 2015-07-11 01:17:04 - <a href="https://testing.hpdd.intel.com/test_sets/3f0a5c78-27ba-11e5-b37a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3f0a5c78-27ba-11e5-b37a-5254006e85c2</a><br/> 2015-07-23 22:17:07 - <a href="https://testing.hpdd.intel.com/test_sets/c3e8da08-31d5-11e5-84cf-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c3e8da08-31d5-11e5-84cf-5254006e85c2</a><br/> 2015-07-23 14:54:47 - <a href="https://testing.hpdd.intel.com/test_sets/a8bae426-3192-11e5-a788-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a8bae426-3192-11e5-a788-5254006e85c2</a><br/> 2015-07-23 10:36:11 - <a href="https://testing.hpdd.intel.com/test_sets/0da58c80-3178-11e5-9558-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0da58c80-3178-11e5-9558-5254006e85c2</a><br/> 2015-07-23 08:49:47 - <a href="https://testing.hpdd.intel.com/test_sets/54d88a1a-312f-11e5-9558-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/54d88a1a-312f-11e5-9558-5254006e85c2</a><br/> 2015-07-20 11:51:53 - <a href="https://testing.hpdd.intel.com/test_sets/9bb5ee50-2f22-11e5-bb5a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9bb5ee50-2f22-11e5-bb5a-5254006e85c2</a><br/> 2015-07-18 05:32:09 - <a href="https://testing.hpdd.intel.com/test_sets/4abb96d6-2d5a-11e5-a112-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4abb96d6-2d5a-11e5-a112-5254006e85c2</a><br/> 2015-07-17 13:17:20 - <a href="https://testing.hpdd.intel.com/test_sets/56c54190-2db2-11e5-8ad8-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/56c54190-2db2-11e5-8ad8-5254006e85c2</a><br/> 2015-07-13 19:14:15 - <a href="https://testing.hpdd.intel.com/test_sets/c30d68dc-2a01-11e5-b04d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c30d68dc-2a01-11e5-b04d-5254006e85c2</a></p> <p>Di - Would you please review these errors and see if they are related to recent DNE landings?<br/> Thank you.</p> <p>Hmm, this is replay-single 30, and no DNE operation in this test. So it is unlikely related with DNE changes.</p> <p>Hongchao</p> <p>Could you please look into this issue?</p> <p>Thanks</p> <p>Peter</p> <p>status update: still analyzing the logs, and still need more time to find the cause of it.</p> <p>this could be related to <a href="https://jira.whamcloud.com/browse/LU-5319" title="Support multiple slots per client in last_rcvd file" class="issue-link" data-issue-key="LU-5319"><del>LU-5319</del></a> (multiple slots per client in last_rcvd), for it was encountered firstly in the test of the patch of it. <a href="https://testing.hpdd.intel.com/sub_tests/153ee6d4-ffba-11e4-ac6b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/153ee6d4-ffba-11e4-ac6b-5254006e85c2</a> (at 2015-05-20)</p> <p>then this issue disappeared and occurred again just after the patch (<a href="http://review.whamcloud.com/#/c/14860/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14860/</a>) was landed at July 1th.</p> <p>the reason of the bug is the committed replay request (open request) is timed out during replaying</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00100000:0.0:1437707068.959718:0:6229:0:(client.c:2591:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff88007c3bb800: committing for last_committed 163208757256 gen 3 00000100:00100000:0.0:1437707068.959727:0:6229:0:(client.c:2622:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff88007a0c7000 x1507534108223584/t163208757253(163208757253) o101-&gt;lustre-MDT0000-mdc-ffff88007c3bb800@10.1.5.29@tcp:12/10 lens 816/600 e 0 to 0 dl 1437707075 ref 1 fl Complete:RP/4/0 rc 0/0 00000100:00100000:0.0:1437707068.959731:0:6229:0:(client.c:2622:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff88007a0c5e00 x1507534108223592/t163208757254(163208757254) o101-&gt;lustre-MDT0000-mdc-ffff88007c3bb800@10.1.5.29@tcp:12/10 lens 816/600 e 0 to 0 dl 1437707075 ref 1 fl Complete:RP/4/0 rc 0/0 ... 00000100:00080000:0.0:1437707094.279383:0:5882:0:(import.c:1188:ptlrpc_connect_interpret()) ffff88007c3bf000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:0.0:1437707094.279386:0:5882:0:(import.c:1448:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1437707094.279389:0:5882:0:(client.c:2591:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff88007c3bb800: committing for last_committed 163208757256 gen 3 00000100:00100000:0.0:1437707094.279394:0:5882:0:(client.c:2617:ptlrpc_free_committed()) @@@ stopping search req@ffff88007a0c4000 x1507534108223680/t163208757257(163208757257) o36-&gt;lustre-MDT0000-mdc-ffff88007c3bb800@10.1.5.29@tcp:12/10 lens 640/416 e 0 to 0 dl 1437707076 ref 1 fl Complete:R/4/0 rc 0/0 00000100:00080000:0.0:1437707094.279401:0:5882:0:(recover.c:93:ptlrpc_replay_next()) import ffff88007c3bf000 from lustre-MDT0000_UUID committed 163208757256 last 0 00000100:00080000:0.0:1437707094.279408:0:5882:0:(client.c:2928:ptlrpc_replay_req()) @@@ REPLAY req@ffff88007a0c7000 x1507534108223584/t163208757253(163208757253) o101-&gt;lustre-MDT0000-mdc-ffff88007c3bb800@10.1.5.29@tcp:12/10 lens 816/600 e 0 to 0 dl 1437707075 ref 1 fl New:RP/4/0 rc 0/0 ... 00000100:00000400:0.0:1437707100.279594:0:5882:0:(client.c:2020:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1437707094/real 1437707094] req@ffff88007a0c7000 x1507534108223584/t163208757253(163208757253) o101-&gt;lustre-MDT0000-mdc-ffff88007c3bb800@10.1.5.29@tcp:12/10 lens 816/600 e 1 to 1 dl 1437707096 ref 2 fl Rpc:XP/4/ffffffff rc 0/-1 00000100:00100000:0.0:1437707100.279621:0:5882:0:(client.c:2049:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY (now=REPLAY) req@ffff88007a0c7000 x1507534108223584/t163208757253(163208757253) o101-&gt;lustre-MDT0000-mdc-ffff88007c3bb800@10.1.5.29@tcp:12/10 lens 816/600 e 1 to 1 dl 1437707096 ref 2 fl Rpc:XP/4/ffffffff rc 0/-1 00000100:00020000:0.0:1437707100.279637:0:5882:0:(client.c:2816:ptlrpc_replay_interpret()) request replay timed out, restarting recovery 00000100:00080000:0.0:1437707100.282045:0:5882:0:(import.c:673:ptlrpc_connect_import()) ffff88007c3bf000 lustre-MDT0000_UUID: changing import state from REPLAY to CONNECTING 00000100:00080000:0.0:1437707100.282053:0:5882:0:(import.c:518:import_select_connection()) lustre-MDT0000-mdc-ffff88007c3bb800: connect to NID 10.1.5.29@tcp last attempt 4306605556 00000100:00080000:0.0:1437707100.282062:0:5882:0:(import.c:596:import_select_connection()) lustre-MDT0000-mdc-ffff88007c3bb800: import ffff88007c3bf000 using connection 10.1.5.29@tcp/10.1.5.29@tcp 00000100:00100000:0.0:1437707100.282089:0:5882:0:(import.c:760:ptlrpc_connect_import()) @@@ (re)connect request (timeout 5) req@ffff88007a0c4f00 x0/t0(0) o38-&gt;lustre-MDT0000-mdc-ffff88007c3bb800@10.1.5.29@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00100000:0.0:1437707100.282101:0:5882:0:(client.c:1951:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_rcv:89a94e57-b8e5-bb46-dc76-c5424df90dcb:5882:1507534108223584:10.1.5.29@tcp:101 00000100:00100000:0.0:1437707100.282113:0:5882:0:(client.c:1574:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_rcv:89a94e57-b8e5-bb46-dc76-c5424df90dcb:5882:1507534108223988:10.1.5.29@tcp:38 00000100:00080000:0.0:1437707100.283658:0:5882:0:(import.c:1004:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff88007c3bb800: connect to target with instance 39 00000100:00080000:0.0:1437707100.283671:0:5882:0:(import.c:1163:ptlrpc_connect_interpret()) reconnected to lustre-MDT0000_UUID@10.1.5.29@tcp after partition 00000100:00080000:0.0:1437707100.283673:0:5882:0:(import.c:1173:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff88007c3bb800: reconnected to lustre-MDT0000_UUID during replay 00000100:00080000:0.0:1437707100.283676:0:5882:0:(import.c:1179:ptlrpc_connect_interpret()) ffff88007c3bf000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:0.0:1437707100.283679:0:5882:0:(import.c:1448:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1437707100.283682:0:5882:0:(client.c:2591:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff88007c3bb800: committing for last_committed 163208757256 gen 3 00000100:00100000:0.0:1437707100.283686:0:5882:0:(client.c:2617:ptlrpc_free_committed()) @@@ stopping search req@ffff88007a0c4000 x1507534108223680/t163208757257(163208757257) o36-&gt;lustre-MDT0000-mdc-ffff88007c3bb800@10.1.5.29@tcp:12/10 lens 640/416 e 0 to 0 dl 1437707076 ref 1 fl Complete:R/4/0 rc 0/0 00000100:00080000:0.0:1437707100.283693:0:5882:0:(recover.c:93:ptlrpc_replay_next()) import ffff88007c3bf000 from lustre-MDT0000_UUID committed 163208757256 last 0 00000100:00080000:0.0:1437707100.283699:0:5882:0:(client.c:2928:ptlrpc_replay_req()) @@@ REPLAY req@ffff88007a0c5e00 x1507534108223592/t163208757254(163208757254) o101-&gt;lustre-MDT0000-mdc- .. 00000100:00000400:0.0:1437707106.283630:0:5882:0:(client.c:2020:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1437707100/real 1437707100] req@ffff88007a0c5e00 x1507534108223592/t163208757254(163208757254) o101-&gt;lustre-MDT0000-mdc-ffff88007c3bb800@10.1.5.29@tcp:12/10 lens 816/600 e 1 to 1 dl 1437707102 ref 2 fl Rpc:XP/6/ffffffff rc 0/-1 00000100:00100000:0.0:1437707106.283659:0:5882:0:(client.c:2049:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY (now=REPLAY) req@ffff88007a0c5e00 x1507534108223592/t163208757254(163208757254) o101-&gt;lustre-MDT0000-mdc-ffff88007c3bb800@10.1.5.29@tcp:12/10 lens 816/600 e 1 to 1 dl 1437707102 ref 2 fl Rpc:XP/6/ffffffff rc 0/-1 00000100:00020000:0.0:1437707106.283676:0:5882:0:(client.c:2816:ptlrpc_replay_interpret()) request replay timed out, restarting recovery 00000100:00080000:0.0:1437707106.285949:0:5882:0:(import.c:673:ptlrpc_connect_import()) ffff88007c3bf000 lustre-MDT0000_UUID: changing import state from REPLAY to CONNECTING 00000100:00080000:0.0:1437707106.285956:0:5882:0:(import.c:518:import_select_connection()) lustre-MDT0000-mdc-ffff88007c3bb800: connect to NID 10.1.5.29@tcp last attempt 4306611560 00000100:00080000:0.0:1437707106.285965:0:5882:0:(import.c:596:import_select_connection()) lustre-MDT0000-mdc-ffff88007c3bb800: import ffff88007c3bf000 using connection 10.1.5.29@tcp/10.1.5.29@tcp 00000100:00100000:0.0:1437707106.285996:0:5882:0:(import.c:760:ptlrpc_connect_import()) @@@ (re)connect request (timeout 5) req@ffff88007a0c5800 x0/t0(0) o38-&gt;lustre-MDT0000-mdc-ffff88007c3bb800@10.1.5.29@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:N/0/ffffffff rc 0/-1 00000100:00100000:0.0:1437707106.286008:0:5882:0:(client.c:1951:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_rcv:89a94e57-b8e5-bb46-dc76-c5424df90dcb:5882:1507534108223592:10.1.5.29@tcp:101 00000100:00100000:0.0:1437707106.286020:0:5882:0:(client.c:1574:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_rcv:89a94e57-b8e5-bb46-dc76-c5424df90dcb:5882:1507534108224040:10.1.5.29@tcp:38 00000100:00080000:0.0:1437707106.287815:0:5882:0:(import.c:1004:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff88007c3bb800: connect to target with instance 39 00000100:00080000:0.0:1437707106.287828:0:5882:0:(import.c:1163:ptlrpc_connect_interpret()) reconnected to lustre-MDT0000_UUID@10.1.5.29@tcp after partition 00000100:00080000:0.0:1437707106.287831:0:5882:0:(import.c:1173:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff88007c3bb800: reconnected to lustre-MDT0000_UUID during replay 00000100:00080000:0.0:1437707106.287833:0:5882:0:(import.c:1179:ptlrpc_connect_interpret()) ffff88007c3bf000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:0.0:1437707106.287836:0:5882:0:(import.c:1448:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1437707106.287839:0:5882:0:(client.c:2591:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff88007c3bb800: committing for last_committed 163208757256 gen 3 00000100:00100000:0.0:1437707106.287843:0:5882:0:(client.c:2617:ptlrpc_free_committed()) @@@ stopping search req@ffff88007a0c4000 x1507534108223680/t163208757257(163208757257) o36-&gt;lustre-MDT0000-mdc-ffff88007c3bb800@10.1.5.29@tcp:12/10 lens 640/416 e 0 to 0 dl 1437707076 ref 1 fl Complete:R/4/0 rc 0/0 00000100:00080000:0.0:1437707106.287852:0:5882:0:(recover.c:93:ptlrpc_replay_next()) import ffff88007c3bf000 from lustre-MDT0000_UUID committed 163208757256 last 0 00000100:00080000:0.0:1437707106.287857:0:5882:0:(client.c:2928:ptlrpc_replay_req()) @@@ REPLAY req@ffff88007a0c5e00 x1507534108223592/t163208757254(163208757254) o101-&gt;lustre-MDT0000-mdc-ffff88007c3bb800@10.1.5.29@tcp:12/10 lens 816/600 e 1 to 1 dl 1437707102 ref 1 fl New:EXP/6/ffffffff rc 0/-1 </pre> </div></div> <p>Hi Yujian,<br/> In <a href="https://testing.hpdd.intel.com/sub_tests/61999b00-32d7-11e5-a4fd-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/61999b00-32d7-11e5-a4fd-5254006e85c2</a> (July 24),<br/> the debug logs at MDS side during recovery is dumped to "/tmp/lustre-log.1437707274.9697",<br/> and how I can get that log? Thanks!!</p> <p>Hi Hongchao,<br/> I think the logs were missing because the nodes were re-provisioned for testing other patches.</p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15814" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15814</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6841" title="replay-single test_30: multiop 20786 failed" class="issue-link" data-issue-key="LU-6841"><del>LU-6841</del></a> target: debug patch to collect more logs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d8fa2f729b6dd8fa5c9ae604308f39306ddd6aa9</p> <p>Hi Gregoire, have you seen any problems similar to this in your testing?</p> <p>I don't remember having seen this test fail during my local testing.</p> <p>However, I have to admit that the kernel I use does not include the <tt>dev_read_only</tt> patch, which means the MDT device cannot be made readonly by <tt>replay_barrier</tt>. This means the test environment is not exactly the same.</p> <p>This issue has not been seen on any branch since 7/24 and may have been resolved by related landings. We are going to reduce the severity and leave it open.</p> <p>Let's close the ticket as cannot report and reopen if we do see it again</p> <p>Just encountered on master but test replay-single/test_31<br/> <a href="https://testing.hpdd.intel.com/test_sets/7708f4da-7e14-11e5-9c23-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7708f4da-7e14-11e5-9c23-5254006e85c2</a></p> <p>Status update:<br/> as per the logs, this issue is caused by the resent replay request during recovery, but MDT doesn't find the corresponding reply<br/> of the resent request, and It could be related to the Multiple Slots in last_rcvd. Still analysis the related code lines to find where<br/> the problem is and will update the status once there is any progress.</p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17303" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17303</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6841" title="replay-single test_30: multiop 20786 failed" class="issue-link" data-issue-key="LU-6841"><del>LU-6841</del></a> target: check the last reply for resent<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 21583ba45fefda43841c36ab3609e529dbadda1d</p> <p>This issue could be marked as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-5951" title="sanity test_39k: mtime is lost on close" class="issue-link" data-issue-key="LU-5951"><del>LU-5951</del></a></p> <p>closed as duplicate of <a href="https://jira.whamcloud.com/browse/LU-5951" title="sanity test_39k: mtime is lost on close" class="issue-link" data-issue-key="LU-5951"><del>LU-5951</del></a></p> Related LU-5319 LU-3127 Development Rank 1|hzxi1r: Rank (Obsolete) 9223372036854775807 Severity