https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3939 Lustre <p>This issue was created by maloo for John Hammond &lt;john.hammond@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/b8769b2e-1b6b-11e3-a00a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/b8769b2e-1b6b-11e3-a00a-52540035b04c</a>.</p> <p>The sub-test test_40 failed with the following error:</p> <blockquote> <p>requests did not complete</p></blockquote> <p>Info required for matching: sanity-hsm 40</p> LU-3939

Test failure on test suite sanity-hsm, subtest test_40

Technical task LU-3647 Major Resolved Fixed Bruno Faccini Maloo HSM Thu, 12 Sep 2013 14:43:56 +0000 Thu, 19 Mar 2015 15:43:18 +0000 Fri, 16 Jan 2015 19:42:39 +0000 Lustre 2.5.0 Lustre 2.7.0 0 13 <p>John, I had a look to the very verbose output for this test, and it seems that the list of archive requests is draining/reducing but very slowly and thus easily reach the 100s limit for the test !!</p> <p>Indeed. I can see that from <a href="https://maloo.whamcloud.com/test_logs/27a32fc6-1b6c-11e3-a00a-52540035b04c/show_text" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_logs/27a32fc6-1b6c-11e3-a00a-52540035b04c/show_text</a> by the change of FID.</p> <p>Should we fix this by getting rid of `--bandwidth 1' from copy tool, at least for the test where lots of file have to archived?</p> <p>Another one <a href="https://maloo.whamcloud.com/test_sets/41ff0516-1b8a-11e3-bede-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/41ff0516-1b8a-11e3-bede-52540035b04c</a> <br/> Also later tests are failing in sanity-hsm</p> <p>Are the 52-57 and 90 errors related or do we need a new LU? </p> <p>No, I strongly suspect failures in following tests are likely to be due to the archive requests back-log still being processed.</p> <p>Can test 40 clean up in it's error state such what when it fails it damage is mostly contained?</p> <p>We can cancel the requests but from my understanding, it's not necessary to do that because there is nothing wrong, it's simply too slow to access the NFS share.</p> <p>Pushed <a href="http://review.whamcloud.com/7703" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7703</a> to force sanity-hsm/test_40 to use a local/tmp hsm-root/HSM_ARCHIVE anyway.</p> <p>Humm seems that actually auto-tests/maloo imperatively bypass sanity-hsm/test_40 (via $SANITY_HSM_EXCEPT env var usage ?) due to these current/concerned failures. But then, and even if my local testing was successful, this prevents my patch exposure to auto-tests ...</p> <p>Bruno I have a suggestion here. It may seem like a farce but I think it will work.</p> <ol> <li>Submit a patch to add 40 and 251 to ALWAYS_EXCEPT in sanity-hsm.sh.</li> <li>Wait for it to land.</li> <li>Wait for one week after that.</li> <li>Ask Mike to revert the changes in TEI-570.</li> <li>Wait for that to go in to effect.</li> <li>Update <a href="http://review.whamcloud.com/7703" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7703</a> to remove 40 from ALWAYS_EXCEPT and resubmit.</li> </ol> <p>If it is a fortestonly type of patch you could also just make it test_1000 (or whatever makes sense). </p> <p><a href="http://review.whamcloud.com/7772" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7772</a> just submitted to insert sanity-hsm/test_40 in ALWAYS_EXCEPT list.</p> <p>When landed, it will allow us to ask the Tools team to revert their changes in TEI-570, and then to remove it on-demand from exception list ...</p> <p>On the other hand, I am waiting from current auto-tests results of <a href="http://review.whamcloud.com/7703" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7703</a>, to address comments and any other issues before to push a new+definitive patch-set.</p> <p>Submitted patch-set #4 of <a href="http://review.whamcloud.com/7703" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7703</a>, including a hack to avoid TEI-570 changes effect and thus allow my changes to sanity-hsm/test_40 to be exposed in auto-tests environment.</p> <p>On the other hand I am trying to get test_40 be included in sanity-hsm sub-tests exception list, by the mean of <a href="http://review.whamcloud.com/7772" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7772</a> or better within Jinshan's change <a href="http://review.whamcloud.com/7374" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7374</a> for <a href="https://jira.whamcloud.com/browse/LU-3815" title="Fix sanity-hsm test 34: &quot;Remove file during restore&quot;" class="issue-link" data-issue-key="LU-3815"><del>LU-3815</del></a>, in order to be able to ask Tools-team for TEI-570 changes to be reverted.</p> <p>Jinshan just added test_40 to sanity-hsm exclusion-list in his change <a href="http://review.whamcloud.com/7374" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7374</a> for <a href="https://jira.whamcloud.com/browse/LU-3815" title="Fix sanity-hsm test 34: &quot;Remove file during restore&quot;" class="issue-link" data-issue-key="LU-3815"><del>LU-3815</del></a>, so I will abandon <a href="http://review.whamcloud.com/7772" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7772</a>.</p> <p>Need to wait now for Jinshan's patch to land (+ a few days to wait everybody time to rebase !!) in order to ask Tools-team to revert their changes in TEI-570.</p> <p>Since change #7374 for <a href="https://jira.whamcloud.com/browse/LU-3815" title="Fix sanity-hsm test 34: &quot;Remove file during restore&quot;" class="issue-link" data-issue-key="LU-3815"><del>LU-3815</del></a>, excluding test_40 sub-test in sanity-hsm, has land since about 10 days now, it sounds reasonable to ask Tools-team to revert their changes in TEI-570. So I re-open TEI-570.</p> <p>Submitted definitive version/patch-set #6 of <a href="http://review.whamcloud.com/7703" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7703</a>, which additionally re-enable test_40.</p> <p>Humm, even with my change line to re-enable test_40 in patch-set #6, auto-tests report still show it as "skipping excluded test 40" … Could it be that TEI-570 setting was not reverted by TEI-1041 as expected ??</p> <p>Now that TEI-1041 has definitely reverted ALL auto-tests settings causing sanity-hsm/test_40 sub-test to be disabled, I have re-triggered build (can not only re-trigger auto-tests since build has been removed during interval) of Change #7703/patch-set #6.</p> <p>BTW, new auto-test session Change #7703/patch-set #6, now show successful run of test_40 with a local/tmp hsm-root/HSM_ARCHIVE.</p> <p>Hi Bruno, I have a suggestion here. Can we determine which tests actually <b>require</b> that the HSM archive be on shared storage? I was thinking that perhaps <b>only</b> those tests should use shared storage and the other (normal) tests should use local /tmp on the copytool client. Based on a sanity-hsm run on 4 of my own VMs it seems that most tests are fine with a local HSM archive.</p> <p>While validating patches for Lustre b2_5 branch, this failure occurred frequently:<br/> <a href="https://maloo.whamcloud.com/test_sets/631324a2-76bc-11e3-9ce8-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/631324a2-76bc-11e3-9ce8-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/a365b8f0-7737-11e3-9ce8-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a365b8f0-7737-11e3-9ce8-52540035b04c</a></p> <p>Back-port patch <a href="http://review.whamcloud.com/7703/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7703/</a> to Lustre b2_5 branch: <a href="http://review.whamcloud.com/8771" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8771</a></p> <p>seen in master:<br/> <a href="https://maloo.whamcloud.com/test_sessions/bcd51864-9459-11e3-b8a9-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/bcd51864-9459-11e3-b8a9-52540035b04c</a></p> <p>Bob this failure, even if it looks of the same kind of problem, is against test_90 not test_40.<br/> BTW, it is a good point for John's remark that others tests that do not require a shared storage should use the same fix I applied for test_40.</p> <p>Fixed for 2.5.1 and 2.6. Similar fixes for other tests can be tracked under a separate ticket.</p> <p>This issue is still occurring on master:</p> <p>review-zfs:<br/> <a href="https://maloo.whamcloud.com/test_sets/0aa1078c-e2f3-11e3-8561-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0aa1078c-e2f3-11e3-8561-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/47ce19b2-dff4-11e3-9854-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/47ce19b2-dff4-11e3-9854-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/75f2680a-c5f7-11e3-a760-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/75f2680a-c5f7-11e3-a760-52540035b04c</a></p> <p>full (ldiskfs):<br/> <a href="https://maloo.whamcloud.com/test_sets/6637dbbc-aa80-11e3-bd80-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6637dbbc-aa80-11e3-bd80-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/be4e5838-a5e1-11e3-aac5-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/be4e5838-a5e1-11e3-aac5-52540035b04c</a></p> <p>The logs of these auto-tests failures clearly show that hsm_root is in an NFS mounted fie-system</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity-hsm test 40: Parallel archive requests == 20:25:46 (1394594746) CMD: shadow-7vm6 pkill -CONT -x lhsmtool_posix Purging archive on shadow-7vm6 CMD: shadow-7vm6 rm -rf /home/autotest/.autotest/shared_dir/2014-03-11/095905-70358207241380/arc1/* Starting copytool agt1 on shadow-7vm6 CMD: shadow-7vm6 mkdir -p /home/autotest/.autotest/shared_dir/2014-03-11/095905-70358207241380/arc1 CMD: shadow-7vm6 lhsmtool_posix --daemon --hsm-root /home/autotest/.autotest/shared_dir/2014-03-11/095905-70358207241380/arc1 --bandwidth 1 /mnt/lustre &lt; /dev/null &gt; /logdir/test_logs/2014-03-11/lustre-master-el6-x86_64-vs-lustre-b2_5-el6-x86_64--full--2_9_1__1937__-70358207241380-095905/sanity-hsm.test_40.copytool_log.shadow-7vm6.log 2&gt;&amp;1 ........... </pre> </div></div> <p>and again a too slow draining of the 100 archive requests for test_40 ...</p> <p>So could it be that something has changed in auto-tests/Maloo tools env. vars set, causing #7703 patch to need more to be done to force hsm_root in a local file-system ?<br/> How can I check current auto-tools master sessions environment/configuration ?</p> <p>Maloo reports show no new occurrence for this particular failure of sanity-hsm/test_40 since 2014-07-19 04:36:22 UTC. This last failure shows slow but constant evolution of the 400 archive requests, even with local/tmp filesystem usage.</p> <p>So I think we can assume that an other issue (last 4 failures between 2014-05-24 01:41:13 UTC and 2014-07-19 04:36:22 UTC have occured during review-zfs sessions, so any ZFS related slowness?) was causing slow archives even on a non-NFS/local archive area and it has been fixed.</p> <p>And does anybody agree we can we close this issue as fixed ?</p> <p>I've just noticed that the patch for b2_5 does not enable test 40; sanity-hsm.sh still has:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># bug number for skipped test: 3815 3939 ALWAYS_EXCEPT="$SANITY_HSM_EXCEPT 34 35 36 40" </pre> </div></div> <p>Is there any reason why test 40 is not enabled for b2_5?</p> <blockquote><p>Is there any reason why test 40 is not enabled for b2_5?</p></blockquote> <p>On master branch, test 40 was enabled by patch <a href="http://review.whamcloud.com/7703" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7703</a>. While back-porting the patch to Lustre b2_5 branch in <a href="http://review.whamcloud.com/8771" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8771</a> (patch set 1), test 40 was not disabled at that time. After that, patch <a href="http://review.whamcloud.com/7374" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7374</a> was cherry-picked to Lustre b2_5 branch to make test 40 disabled. So, this is caused by the order of patch landing.</p> <p>Here is the patch to enable test 40 on Lustre b2_5 branch: <a href="http://review.whamcloud.com/12754" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12754</a></p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/12754" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12754</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3939" title="Test failure on test suite sanity-hsm, subtest test_40" class="issue-link" data-issue-key="LU-3939"><del>LU-3939</del></a> tests: enable sanity-hsm test 40<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: f82e4e9449211c5be5c60006fe9c7b7c442cd58f</p> <p>It sounds like everything has landed to master and it is just landings to maintenance branches for interop testing still in flight</p> Related LU-5474 LU-4126 Development Rank 1|hzw22f: Rank (Obsolete) 10416