https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4643 Lustre <p>While validating patch <a href="http://review.whamcloud.com/9290" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9290</a> on Lustre b2_5 branch, sanity-hsm test 60 failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Updated after 9s: wanted '5242880' got '5242880' sanity-hsm test_60: @@@@@@ FAIL: Expected progress update within 10 seconds </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/b8823022-987c-11e3-98a2-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b8823022-987c-11e3-98a2-52540035b04c</a></p> <p>More instances occurred while validating patches on Lustre b2_5 branch:<br/> <a href="http://review.whamcloud.com/9260" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9260</a><br/> <a href="http://review.whamcloud.com/9103" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9103</a><br/> <a href="http://review.whamcloud.com/9071" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9071</a></p> <p>The same failure also occurred while validating patches on master branch:<br/> <a href="http://review.whamcloud.com/9243" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9243</a><br/> <a href="http://review.whamcloud.com/9221" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9221</a></p> <p>The test was introduced by the patches for <a href="https://jira.whamcloud.com/browse/LU-4512" title="POSIX copytool option &quot;--report&quot; is non-obvious and functionality is broken" class="issue-link" data-issue-key="LU-4512"><del>LU-4512</del></a>.</p> LU-4643

sanity-hsm test_60: FAIL: Expected progress update within 10 seconds

Bug Critical Resolved Fixed Bruno Faccini Jian Yu Tue, 18 Feb 2014 14:31:17 +0000 Tue, 29 Sep 2015 12:49:02 +0000 Tue, 29 Sep 2015 12:48:18 +0000 Lustre 2.6.0 Lustre 2.5.1 Lustre 2.6.0 Lustre 2.5.1 0 11 <p>Hi Michael,</p> <p>This failure is a regression introduced by the patch for <a href="https://jira.whamcloud.com/browse/LU-4512" title="POSIX copytool option &quot;--report&quot; is non-obvious and functionality is broken" class="issue-link" data-issue-key="LU-4512"><del>LU-4512</del></a>. Could you please take a look at this failure? Thanks.</p> <p>Another similar failure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Update not seen after 100s: wanted '5242880' got '' sanity-hsm test_60: @@@@@@ FAIL: request on 0x400000401:0x1c1:0x0 has not made progress 5242880 on mds1 </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/8e7ec8f2-97b0-11e3-a9f6-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/8e7ec8f2-97b0-11e3-a9f6-52540035b04c</a></p> <p>For "Expected progress update within 10 seconds" failure, I checked the outputs of all of the failed tests and found that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Updated after 9s: wanted '5242880' got '5242880' Updated after 9s: wanted '5242880' got '5242880' Updated after 12s: wanted '5242880' got '5242880' Updated after 12s: wanted '5242880' got '5242880' Updated after 11s: wanted '5242880' got '5242880' Updated after 11s: wanted '5242880' got '5242880' Updated after 11s: wanted '5242880' got '5242880' Updated after 13s: wanted '5242880' got '5242880' </pre> </div></div> <p>The current values of "interval" and "progress_timeout" in sanity-hsm test_60() are as follows:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> local interval=5 local progress_timeout=$((interval * 2)) </pre> </div></div> <p>I just uploaded patches to increase the value of "progress_timeout".<br/> For master branch: <a href="http://review.whamcloud.com/9304" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9304</a><br/> For b2_5 branch: <a href="http://review.whamcloud.com/9305" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9305</a></p> <p>For "Update not seen after 100s: wanted '5242880' got ''" failure, I checked the outputs of all of the failed tests and found that the updating of "done=" values were all as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Changed after 0s: from '' to '0' Changed after 4s: from '0' to '4194304' Changed after 9s: from '4194304' to '13631488' Changed after 13s: from '13631488' to '28311552' Changed after 17s: from '28311552' to '48234496' Changed after 22s: from '48234496' to '73400320' Changed after 26s: from '73400320' to '103809024' Changed after 30s: from '103809024' to '139460608' Changed after 34s: from '139460608' to '' </pre> </div></div> <p>In sanity-hsm test_60(), the expected value of "data_moved" is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> wait_request_progress $fid ARCHIVE 5242880 </pre> </div></div> <p>I'm confused here about how "5242880" was calculated because the size of archived file was 39000000 bytes:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> dd <span class="code-keyword">if</span>=/dev/urandom of=$file2 count=39 bs=1000000 conv=fsync || error <span class="code-quote">"cannot create $file2"</span> </pre> </div></div> <p>To allow for further debugging of sanity-hsm/test_60 issues, I pushed the 2 patches <a href="http://review.whamcloud.com/9313" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9313</a> and <a href="http://review.whamcloud.com/9314" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9314</a>, respectively for master and b2_5, that will disable it from inside sanity-hsm suite itself. Then it could be re-enabled as a further patch that will fix the issues.</p> <p>Possible issues to be addressed then are 1) a mismatch in the way the CT reports progress and how the CDT interpret/report it in the log, 2) a too short grace_period request causing request to disappear from the log, 3) a wrong 5242880 size to be checked, …??</p> <p>Hi, sorry I'm just now replying. Was on vacation last week.</p> <p>So, I think the problem with this test is that it's too fragile. The intent of the test is to ensure that changing the copytool reporting interval via command-line arguments works. The bug that <a href="https://jira.whamcloud.com/browse/LU-4512" title="POSIX copytool option &quot;--report&quot; is non-obvious and functionality is broken" class="issue-link" data-issue-key="LU-4512"><del>LU-4512</del></a> fixed was that the --report argument wasn't parsed correctly and I believe what happened was that the interval was interpreted as milliseconds rather than seconds. In any case, it wasn't working, and it works now.</p> <p>I believe that this is an intermittent failure due to load variations on the test VMs. The tests in this suite artificially limit bandwidth to 1MB/sec (see line 164 in copytool_setup()) which is fine when the cluster is actually capable of 1MB/sec. When it's not, test_60 can fail because the expected value isn't seen.</p> <p>In hindsight, I see now that assuming the test could always count on 1MB/sec was a mistake. Also, the test is sort of testing the wrong thing. We don't really care what the value is, just that there was a progress report within the expected window. I will work on a patch to make this test more robust.</p> <p>proposed improvement:<br/> master: <a href="http://review.whamcloud.com/#/c/9376/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9376/</a><br/> b2_5: <a href="http://review.whamcloud.com/#/c/9378/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9378/</a></p> <p>Michael, not sure if you have seen that patches #9313/#9314, respectively for master and b2_5, should land soon to disable sanity-hsm/test_60 until problem is fixed.<br/> This means that when it will have landed you will need to re-enable it inside your patches #9376/#9378.</p> <p>Given that my patches have +1s for code review and are working through testing, I wonder why we would land the patches to disable the test? Wouldn't it be better to just land the fixed version of the test and not bother with disabling it?</p> <p>The patches to disable the failing test is frequent way we use now to reduce the time/frequency tests sessions are impacted with the issue and then allow for enough time and no pressure during the test debugging. It is easier then to re-enable the test with and additional line/change (to modify ALWAYS_EXCEPT setting) in the final patch/fix.</p> <p>I understand, but as a hypothetical example, if the patch to disable the test and the patch to fix the test are both ready at the same time for landing, it doesn't make sense to me to disable the test and then re-enable it with two separate patches. Am I missing something?</p> <p><a href="http://review.whamcloud.com/#/c/9376/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9376/</a> has 2 +1 code-reviews, a +1 from jenkins, and a +1 from maloo. Just waiting for the gatekeeper to cherry-pick it and land it.</p> <p>I have no problem at all with that Michael, if you are sure/confident about imminent landing of #9376/#9378 just abandon #9313/#9314 in this case.</p> <p>Landed for 2.5.1 and 2.6</p> <p>Encountered similar issue for interop testing for 2.7.60 tag<br/> Client : master branch/ build 3194, RHEL 7<br/> Server: b2_7 branch</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>5242880 bytes copied in 3 seconds. sanity-hsm test_60: @@@@@@ FAIL: Expected progress update after at least 5 seconds </pre> </div></div> <p>Saurabh</p> <p>Given how old this issue is it would be much better to open a new ticket even if the issue does appear similar to this older one</p> <p>Peter</p> Development Rank 1|hzwfaf: Rank (Obsolete) 12700 Severity