https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6200 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/be3ebe76-a817-11e4-93dd-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/be3ebe76-a817-11e4-93dd-5254006e85c2</a>.</p> <p>The sub-test test_failover_ost failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test_failover_ost returned 1 </pre> </div></div> <p>client 3 shows</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>tar: etc/sysconfig/quota_nld: Cannot write: No such file or directory tar: etc/sysconfig/quota_nld: Cannot utime: No such file or directory tar: etc/sysconfig/sandbox: Cannot write: No such file or directory tar: etc/sysconfig/nfs: Cannot write: No such file or directory tar: Exiting with failure status due to previous errors </pre> </div></div> client and server: lustre-master build # 2835 RHEL6 LU-6200

Failover recovery-mds-scale test_failover_ost: test_failover_ost returned 1

Bug Major Resolved Duplicate Hongchao Zhang Maloo p4hc Tue, 3 Feb 2015 01:40:38 +0000 Tue, 26 Mar 2019 07:57:01 +0000 Tue, 26 Mar 2019 07:57:01 +0000 Lustre 2.7.0 Lustre 2.8.0 Lustre 2.10.0 Lustre 2.11.0 Lustre 2.10.4 0 13 <p>Hi Hongchao,</p> <p>Is this similar to <a href="https://jira.whamcloud.com/browse/LU-4621" title="recovery-mds-scale: test_failover_ost" class="issue-link" data-issue-key="LU-4621"><del>LU-4621</del></a>?</p> <p>Hongchao, what is the severity of this bug? Is it something that will break normal failover for users or is it only affecting testing?</p> <p>Is this a new regression from a recently landed patch?</p> <p>Hi Andreas, <br/> this issue should not affecting testing only.<br/> there is no corresponding object when replaying the "OST_SETATTR"(op=2) request, and return -2(ENOENT).</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>11:14:39:LustreError: 2377:0:(client.c:2809:ptlrpc_replay_interpret()) @@@ status -2, old was 0 req@ffff8800448fd980 x1491531070413056/t30064787885(30064787885) o2-&gt;lustre-OST0000-osc-ffff880037b57c00@10.2.4.145@tcp:28/4 lens 440/400 e 0 to 0 dl 1422443475 ref 2 fl Interpret:R/4/0 rc -2/-2 11:14:39:LustreError: 2377:0:(client.c:2809:ptlrpc_replay_interpret()) Skipped 12 previous similar messages 11:14:39:Lustre: lustre-OST0000-osc-ffff880037b57c00: Connection restored to lustre-OST0000 (at 10.2.4.145@tcp) 11:14:39:Lustre: Skipped 2 previous similar messages 11:14:39:Lustre: 2377:0:(client.c:1942:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1422443458/real 1422443458] req@ffff88004303c680 x1491531070428380/t0(0) o8-&gt;lustre-OST0001-osc-ffff880037b57c00@10.2.4.141@tcp:28/4 lens 400/544 e 0 to 1 dl 1422443483 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 11:14:39:Lustre: 2377:0:(client.c:1942:ptlrpc_expire_one_request()) Skipped 26 previous similar messages 11:14:40:LustreError: 2377:0:(client.c:2809:ptlrpc_replay_interpret()) @@@ status -2, old was 0 req@ffff88004293c680 x1491531070413096/t30064787867(30064787867) o2-&gt;lustre-OST0001-osc-ffff880037b57c00@10.2.4.145@tcp:28/4 lens 440/400 e 0 to 0 dl 1422443544 ref 2 fl Interpret:R/4/0 rc -2/-2 11:14:40:LustreError: 2377:0:(client.c:2809:ptlrpc_replay_interpret()) Skipped 7 previous similar messages 11:14:40:Lustre: lustre-OST0001-osc-ffff880037b57c00: Connection restored to lustre-OST0001 (at 10.2.4.145@tcp) 11:14:40:Lustre: 2377:0:(client.c:1942:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1422443593/real 1422443593] req@ffff880043240080 x1491531070431228/t0(0) o8-&gt;lustre-OST0005-osc-ffff880037b57c00@10.2.4.141@tcp:28/4 lens 400/544 e 0 to 1 dl 1422443619 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 11:14:40:Lustre: 2377:0:(client.c:1942:ptlrpc_expire_one_request()) Skipped 46 previous similar messages 11:14:40:LustreError: 2377:0:(client.c:2809:ptlrpc_replay_interpret()) @@@ status -2, old was 0 req@ffff880042d84c80 x1491531070413176/t30064787863(30064787863) o2-&gt;lustre-OST0003-osc-ffff880037b57c00@10.2.4.145@tcp:28/4 lens 440/400 e 0 to 0 dl 1422443660 ref 2 fl Interpret:R/4/0 rc -2/-2 11:14:40:LustreError: 2377:0:(client.c:2809:ptlrpc_replay_interpret()) Skipped 17 previous similar messages 11:14:40:Lustre: lustre-OST0003-osc-ffff880037b57c00: Connection restored to lustre-OST0003 (at 10.2.4.145@tcp) 11:14:40:Lustre: Skipped 1 previous similar message 11:14:40:LustreError: 2377:0:(client.c:2809:ptlrpc_replay_interpret()) @@@ status -2, old was 0 req@ffff880042dbec80 x1491531070412976/t30064787924(30064787924) o2-&gt;lustre-OST0005-osc-ffff880037b57c00@10.2.4.145@tcp:28/4 lens 440/400 e 0 to 0 dl 1422443725 ref 2 fl Interpret:R/4/0 rc -2/-2 11:31:06:LustreError: 2377:0:(client.c:2809:ptlrpc_replay_interpret()) Skipped 16 previous similar messages 11:31:06:Lustre: lustre-OST0005-osc-ffff880037b57c00: Connection restored to lustre-OST0005 (at 10.2.4.145@tcp) </pre> </div></div> <p>how about recreating those missing objects in "ofd_setattr_hdl" just like "ofd_preprw_write"? </p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13668" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13668</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6200" title="Failover recovery-mds-scale test_failover_ost: test_failover_ost returned 1" class="issue-link" data-issue-key="LU-6200"><del>LU-6200</del></a> ofd: recreate objects for setattr<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c22faf7b7a1721ca3da24f31f2164b18d0cfb666</p> <p>the patch for this ticket has been merged with the patch for <a href="https://jira.whamcloud.com/browse/LU-5526" title="recovery-mds-scale test failover_mds: dd: No space left on device" class="issue-link" data-issue-key="LU-5526"><del>LU-5526</del></a>, and the "No such file or directory" seems to have been<br/> fixed by the patch, and the issue of <a href="https://jira.whamcloud.com/browse/LU-5526" title="recovery-mds-scale test failover_mds: dd: No space left on device" class="issue-link" data-issue-key="LU-5526"><del>LU-5526</del></a>(No space left on device) still occurs, and its cause is known but the best way to fix it<br/> is still under way. </p> <p><a href="https://testing.hpdd.intel.com/test_sets/09d2e80a-b798-11e4-9d63-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/09d2e80a-b798-11e4-9d63-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/78c3db02-b5d1-11e4-a70c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/78c3db02-b5d1-11e4-a70c-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/815c59b4-b3a7-11e4-add6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/815c59b4-b3a7-11e4-add6-5254006e85c2</a></p> <p>master, build# 3264, 2.7.64 tag<br/> Hard Failover: EL6.7 Server/Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/7b412132-9edd-11e5-87a9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7b412132-9edd-11e5-87a9-5254006e85c2</a></p> <p>Hit this issue for every hard failover configs(6), if the patch for <a href="https://jira.whamcloud.com/browse/LU-5526" title="recovery-mds-scale test failover_mds: dd: No space left on device" class="issue-link" data-issue-key="LU-5526"><del>LU-5526</del></a> can not be landed soon, could we have a separated patch for this particular problem? </p> <p>Per the discussion in triage call, with separated patch the test would still hit <a href="https://jira.whamcloud.com/browse/LU-5526" title="recovery-mds-scale test failover_mds: dd: No space left on device" class="issue-link" data-issue-key="LU-5526"><del>LU-5526</del></a> every time, so no improvement in this case with separated patch</p> <p>Another instance found for hardfailover: EL6.7 Server/Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/3e92c154-bc93-11e5-8f65-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3e92c154-bc93-11e5-8f65-5254006e85c2</a></p> <p>Another instance found for hardfailover: EL6.7 Server/SLES11 SP3 Clients<br/> <a href="https://testing.hpdd.intel.com/test_sets/762762d0-ba4c-11e5-9a07-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/762762d0-ba4c-11e5-9a07-5254006e85c2</a></p> <p>Another instance found for hardfailover : EL6.7 Server/Client, tag 2.7.66, master build 3314<br/> <a href="https://testing.hpdd.intel.com/test_sessions/7c5e8006-cb2d-11e5-b3e8-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/7c5e8006-cb2d-11e5-b3e8-5254006e85c2</a></p> <p>Another instance found for hardfailover : EL6.7 Server/Client - ZFS, tag 2.7.66, master build 3314<br/> <a href="https://testing.hpdd.intel.com/test_sessions/766ea3ec-cb55-11e5-b49e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/766ea3ec-cb55-11e5-b49e-5254006e85c2</a></p> <p>Another instance found for hardfailover : EL7 Server/Client, tag 2.7.66, master build 3314<br/> <a href="https://testing.hpdd.intel.com/test_sessions/8d13249a-ca8f-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/8d13249a-ca8f-11e5-9609-5254006e85c2</a></p> <p>Another instance found for hardfailover : EL7 Server/SLES11 SP3 Client, tag 2.7.66, master build 3316<br/> <a href="https://testing.hpdd.intel.com/test_sessions/2fbf67e4-cd4c-11e5-b1fa-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/2fbf67e4-cd4c-11e5-b1fa-5254006e85c2</a></p> <p>Another instance found for hardfailover : EL7 Server/Client - ZFS, tag 2.7.66, master build 3314<br/> <a href="https://testing.hpdd.intel.com/test_sessions/f0dd9616-ca6e-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/f0dd9616-ca6e-11e5-9609-5254006e85c2</a></p> <p>Another instance found on b2_8 for failover testing , build# 6.<br/> <a href="https://testing.hpdd.intel.com/test_sessions/0aed3028-da39-11e5-a8a6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/0aed3028-da39-11e5-a8a6-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/eaf85780-d65e-11e5-afe8-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/eaf85780-d65e-11e5-afe8-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/54ec62da-d99d-11e5-9ebe-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/54ec62da-d99d-11e5-9ebe-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/eb9f29ec-d8da-11e5-83e2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/eb9f29ec-d8da-11e5-83e2-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/2f0aa9f6-d5a5-11e5-9cc2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/2f0aa9f6-d5a5-11e5-9cc2-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/c5a8e44c-d9c7-11e5-85dd-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/c5a8e44c-d9c7-11e5-85dd-5254006e85c2</a></p> <p>the patch <a href="https://review.whamcloud.com/#/c/13668/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/13668/</a> has been updated.</p> <p>Hongchao,</p> <p>i don't have an access to gerrit now, but you patch <br/> <a href="https://git.hpdd.intel.com/?p=fs/lustre-release.git;a=commitdiff;h=daa98c46817c98d6fbf70dafa9fbdde678f8b9ba;hp=32d1a1c5d610d054ad4609c1cf332172e8310805" class="external-link" target="_blank" rel="nofollow noopener">https://git.hpdd.intel.com/?p=fs/lustre-release.git;a=commitdiff;h=daa98c46817c98d6fbf70dafa9fbdde678f8b9ba;hp=32d1a1c5d610d054ad4609c1cf332172e8310805</a><br/> is bad.</p> <p>Looks, You can't use a </p> <p>+ /* Do sync create if the seq is about to used up */<br/> + if (fid_seq_is_idif(seq) || fid_seq_is_mdt0(seq)) {<br/> + if (unlikely(oid &gt;= IDIF_MAX_OID - 1))<br/> + sync = 1;</p> <p>because ost id in this case need to account lower 16 bits from seq, please look to the ost id macros.</p> <p>the patch has been updated as per the review feedback.</p> <p>the patch <a href="https://review.whamcloud.com/#/c/13668/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/13668/</a> has been updated.</p> <p>2.11.0:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/e6578085-2eed-486d-8601-e5214bac4bb0" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/e6578085-2eed-486d-8601-e5214bac4bb0</a></p> <p>the patch <a href="https://review.whamcloud.com/#/c/13668/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/13668/</a> has been updated</p> <p>Hi,</p> <p>Description of the problem looks similar with I've already fixed in <a href="https://review.whamcloud.com/#/c/33836/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/33836/</a> .<br/> Please look carefully and If I am right this could be resolved as a dup of <a href="https://jira.whamcloud.com/browse/LU-11765" title="during failover test run, mdtest job fails, numerous stat failures &#39;No such file or directory&#39;" class="issue-link" data-issue-key="LU-11765"><del>LU-11765</del></a>.</p> <p>Hi Sergey,</p> <p>Thanks!<br/> This should be the same as <a href="https://jira.whamcloud.com/browse/LU-11765" title="during failover test run, mdtest job fails, numerous stat failures &#39;No such file or directory&#39;" class="issue-link" data-issue-key="LU-11765"><del>LU-11765</del></a>, only the fixing ways are different, the patch in this ticket recreates the object<br/> if it doesn't exist and the patch in <a href="https://jira.whamcloud.com/browse/LU-11765" title="during failover test run, mdtest job fails, numerous stat failures &#39;No such file or directory&#39;" class="issue-link" data-issue-key="LU-11765"><del>LU-11765</del></a> returns -EAGAIN to notify the caller to retry.</p> <p>Resolved as duplicate of <a href="https://jira.whamcloud.com/browse/LU-11765" title="during failover test run, mdtest job fails, numerous stat failures &#39;No such file or directory&#39;" class="issue-link" data-issue-key="LU-11765"><del>LU-11765</del></a></p> Duplicate LU-463 Related LU-5483 LU-5526 Development Rank 1|hzx5gv: Rank (Obsolete) 17329 Severity