https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13449 Lustre <p>This issue was created by maloo for jianyu &lt;yujian@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/f1c30a21-d730-4d65-a379-d78f2e5fa72a" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/f1c30a21-d730-4d65-a379-d78f2e5fa72a</a></p> <p>test_140b failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>recovery took too long 4362 20 </pre> </div></div> <p>&lt;&lt;Please provide additional information about the failure here&gt;&gt;</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> recovery-small test_140b - recovery took too long 4362 20</p> LU-13449

recovery-small test_140b: FAIL: recovery took too long 4362 20

Bug Minor Resolved Fixed Andreas Dilger Maloo Mon, 13 Apr 2020 19:13:10 +0000 Sat, 23 Oct 2021 00:15:49 +0000 Wed, 17 Feb 2021 22:38:02 +0000 Lustre 2.14.0 Lustre 2.12.6 Lustre 2.14.0 0 8 <p>The failure occurred consistently on master branch:<br/> <a href="https://testing.whamcloud.com/test_sets/f1c30a21-d730-4d65-a379-d78f2e5fa72a" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/f1c30a21-d730-4d65-a379-d78f2e5fa72a</a><br/> <a href="https://testing.whamcloud.com/test_sets/f28a48fa-4f64-47d4-a58b-85b2a76e6037" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/f28a48fa-4f64-47d4-a58b-85b2a76e6037</a><br/> <a href="https://testing.whamcloud.com/test_sets/3c3c41f6-1b79-4ccb-8ee8-9a39e079c7c7" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/3c3c41f6-1b79-4ccb-8ee8-9a39e079c7c7</a></p> <p>It looks like a bug in environment or bash. $SECONDS return a something wrong.</p> <p>Looking at one of the test results, the entire test took 58s to finish (as is also confirmed by the subtest banners):<br/> <a href="https://testing.whamcloud.com/test_sets/f1c30a21-d730-4d65-a379-d78f2e5fa72a" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/f1c30a21-d730-4d65-a379-d78f2e5fa72a</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== recovery-small test 140b: local mount is excluded from recovery ==== 10:41:23 (1586713283) : : trevis-64vm1: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec trevis-64vm2: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec recovery-small test_140b: @@@@@@ FAIL: recovery took too long 4362 20 FAIL 140b (58s) == recovery-small test 141: do not lose locks on MGS restart ==== 10:42:21 (1586713341) </pre> </div></div> <p>There does appear to be a typo in the error message (note '<tt>bsfore</tt>'), but that does not affect the actual check:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> @@ -2973,8 +2973,8 @@ test_140b() { local before=$SECONDS fail mds1 local after=$SECONDS (( $after-$before &lt; $TIMEOUT*2 )) || - error <span class="code-quote">"recovery took too <span class="code-object">long</span>"</span> $((after-bsfore)) $TIMEOUT + error <span class="code-quote">"recovery took too <span class="code-object">long</span> $after-$before &gt; $TIMEOUT"</span> } run_test 140b <span class="code-quote">"local mount is excluded from recovery"</span> </pre> </div></div> <p>Maybe the timeout needs to be scaled by the MDT count? Some further investigation is needed to see where the time is being spent, and if this indicates a defect in the "<tt>localrecov</tt>" code, or is unrelated to the test failure.</p> <p>Andreas,</p> <p>console messages say import stay in active in 0s, if you look a console time stamps this test was executed much less 5300s reported in fail.<br/> i think $before is zero or something similar, so my suggestion stop a use $SECONDS and replace with date -%s instead.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=shadow" class="user-hover" rel="shadow">shadow</a> the error message itself is incorrect. The "<tt>bsfore</tt>" variable is unset (note '<tt>s</tt>' instead of '<tt>e</tt>'), since it should be "<tt>before</tt>", so "<tt>$((after-bsfore))</tt>" is just printed as "<tt>$after</tt>" (which is the "<tt>4362</tt>" printed in the error message). I don't think there is anything wrong with <tt>$SECONDS</tt>, just that the test took 58s to complete, but <tt>$TIMEOUT</tt> is only 20s, or even 40s that is being checked. </p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38366" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38366</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13449" title="recovery-small test_140b: FAIL: recovery took too long 4362 20 " class="issue-link" data-issue-key="LU-13449"><del>LU-13449</del></a> tgt: fix recovery timer comparisons<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6f041a32a487c7f31c80525175a27e2990cbbf05</p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sessions/edb20dfa-e703-4bb7-a756-2d3d49bd74a7" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/edb20dfa-e703-4bb7-a756-2d3d49bd74a7</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sessions/4c20c06a-f6a6-48e7-8cdf-34cc5e8524a4" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/4c20c06a-f6a6-48e7-8cdf-34cc5e8524a4</a></p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38366/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38366/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13449" title="recovery-small test_140b: FAIL: recovery took too long 4362 20 " class="issue-link" data-issue-key="LU-13449"><del>LU-13449</del></a> tgt: fix recovery timer comparisons<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 814bb394843434883a94fe6432cd8c656035a3e1</p> <p>+1 on master. <br/> <a href="https://testing.whamcloud.com/test_sets/3cc6a545-5e58-426d-adf2-3f083d542c2f" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/3cc6a545-5e58-426d-adf2-3f083d542c2f</a></p> <p>This test has failed 21/188 runs on all branches in the past week. It looks like most of those failures are on older branches that have the old error messages, but there are 4/121 failures on master in the past week with the fixed error messages. Most of those are:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>recovery took too long 42 &gt; 40 </pre> </div></div> <p>which means that recovery <b>did</b> finish, just a few seconds later than expected. Should that be considered an error that needs to be investigated? Fixing the test to "pass" in that case is easy, but may be removing the entire value of the test itself?</p> <p>There were a few other failures like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>recovery-small.sh: line 2977: ((: 08: value too great for base (error token is "08") recovery-small test_140b: @@@@@@ FAIL: recovery took too long 08 &gt; 40 recovery-small.sh: line 2977: ((: 09: value too great for base (error token is "09") recovery-small test_140b: @@@@@@ FAIL: recovery took too long 09 &gt; 40 </pre> </div></div> <p>which looks like bash was unhappy with the "octal" format of "08" and "09", since the error message is printed in MM:SS format "<tt>Recovery over after 0:08, of 2 clients 2 recovered</tt>".</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> local recovery=$(do_facet mds1 dmesg | awk -F: <span class="code-quote">'/Recovery over after/ { print $4 }'</span> | cut -d, -f1 | tail -1) (( $recovery &lt; $TIMEOUT*2 )) || error <span class="code-quote">"recovery took too <span class="code-object">long</span> $recovery &gt; $((TIMEOUT * 2))"</span> </pre> </div></div> <p>Also, it seems that this check is ignoring the minutes part of the timeout, which is not ideal. It would be better to "unparse" the MM:SS format and handle it properly as seconds. Something like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # Lustre: tfs-MDT0000: Recovery over after 0:03, of 2 clients 2 rec... local recovery=$(do_facet mds1 dmesg | awk <span class="code-quote">'/Recovery over after/ { print $6 }'</span> | tail -1 | awk -F: <span class="code-quote">'{ print $1 * 60 + $2 }'</span>) (( recovery &lt; TIMEOUT * 2 )) || error <span class="code-quote">"recovery took too <span class="code-object">long</span> $recovery &gt; $((TIMEOUT * 2))"</span> </pre> </div></div> <p>appears to work, but it would be good to decide whether a "+5" should be added into the check or not?</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39909" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39909</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13449" title="recovery-small test_140b: FAIL: recovery took too long 4362 20 " class="issue-link" data-issue-key="LU-13449"><del>LU-13449</del></a> tests: fix recovery-small test_140b check<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1f932570fc9aec1e0f35f4d3f4c93a1af2dd1e37</p> <p>Hmm, looking at the MDS console logs for one of the "<tt>42 &gt; 40</tt>" failures I see:<br/> <a href="https://testing.whamcloud.com/sub_tests/72a6a27d-09f4-44fa-b6e8-435416f8c536" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/72a6a27d-09f4-44fa-b6e8-435416f8c536</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 216.334657] Lustre: lustre-MDT0000: Recovery over after 1:42, of 3 clients 2 recovered and 1 was evicted. </pre> </div></div> <p>so it is clear that this is <b>way</b> over the expected recovery time and probably needs some larger fix.</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/41000" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41000</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13449" title="recovery-small test_140b: FAIL: recovery took too long 4362 20 " class="issue-link" data-issue-key="LU-13449"><del>LU-13449</del></a> tgt: fix recovery timer comparisons<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: ce247bcbe51e597fda2cc0c552ba55251467aa1b</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39909/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39909/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13449" title="recovery-small test_140b: FAIL: recovery took too long 4362 20 " class="issue-link" data-issue-key="LU-13449"><del>LU-13449</del></a> tests: fix recovery-small test_140b check<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 463a70eb645df7519efeab49e558cbe6f4ecafa5</p> Development Rank 1|i00xun: Rank (Obsolete) 9223372036854775807 Severity