https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16632 Lustre <p>This issue was created by maloo for S Buisson &lt;sbuisson@ddn.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/b7260618-4120-4ad7-ad92-b6aae2bb0332" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/b7260618-4120-4ad7-ad92-b6aae2bb0332</a></p> <p>test_56xh failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>'lfs migrate -W' too slow in VM (80 &gt; 2 * 25 2) </pre> </div></div> <p>Test session details:<br/> clients: <a href="https://build.whamcloud.com/job/lustre-reviews/92966" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/92966</a> - 4.18.0-348.7.1.el8_5.x86_64<br/> servers: <a href="https://build.whamcloud.com/job/lustre-reviews/92966" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/92966</a> - 4.18.0-348.23.1.el8_lustre.x86_64</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity test_56xh - 'lfs migrate -W' too slow in VM (80 &gt; 2 * 25 2)</p> LU-16632

sanity test_56xh: 'lfs migrate -W' too slow

Bug Minor Resolved Fixed Jian Yu Maloo Fri, 10 Mar 2023 19:14:40 +0000 Tue, 14 Nov 2023 22:37:19 +0000 Wed, 29 Mar 2023 03:30:22 +0000 Lustre 2.16.0 0 4 <p>The "lfs migrate -W" option and corresponding test were added in <a href="https://jira.whamcloud.com/browse/LU-13482" title="add verbose stats to lfs_migrate and &quot;lfs migrate&quot;" class="issue-link" data-issue-key="LU-13482"><del>LU-13482</del></a>. </p> <p>We can't reliably measure performance inside the VMs, so we likely need to increase the margin of error allowed for this test. It currently checks:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> (( elapsed &lt;= size_mb * 150 / 100 )) || error <span class="code-quote">"<span class="code-quote">'lfs migrate -W'</span> too slow in VM ($elapsed &gt; 2 * $size_mb 2)"</span> </pre> </div></div> <p>And the error message says it is 2x the time, but the check is only 1.5x the time. The check and message should be fixed to use "<tt>&#42; 350 / 100</tt>", along with removing the stray "2" at the end of the error (test took 80s vs 25s without throttle, which is 3.2x longer).</p> <p>I wonder if this is exacerbated by the addition of many sleeps, which gives the VM more chances to reschedule this thread under contention, or possibly inaccuracies in the nanosecond clock that are hurting the rate calculations?</p> <p>I wonder if the algorithm should be designed to "catch up" in those cases (possibly exceeding the bandwidth cap if it was previously starved, or at least not going to sleep in those cases), or is the intent that it should not exceed the bandwidth limit even if previous intervals were slower than necessary (notwithstanding the fact that it is very bursty with its writes)?</p> <p>Rescheduling seems more likely to me than clock inaccuracies. Looking at a handful of past runs shows that the test very reliably hits 25s for the migrate job. Not much variability.</p> <p> </p> <p>I think it's preferable that the bandwidth cap is respected as much as possible, even if the migrate falls behind a bit. I think it makes more sense from a user perspective. I agree that the algorithm could be made less bursty in its writes.</p> <p>"Timothy Day &lt;timday@amazon.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50262" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50262</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16632" title="sanity test_56xh: &#39;lfs migrate -W&#39; too slow" class="issue-link" data-issue-key="LU-16632"><del>LU-16632</del></a> tests: more margin of error for 56xh<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2227fdcd8901262f09ccb669aed8fb3b4addc1a5</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50262/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50262/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16632" title="sanity test_56xh: &#39;lfs migrate -W&#39; too slow" class="issue-link" data-issue-key="LU-16632"><del>LU-16632</del></a> tests: more margin of error for sanity/56xh<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 36cbba150bce9e2890c8b462ec2ce4af2d6353a5</p> <p>Landed for 2.16</p> Related LU-13482 Development Rank 1|i03g33: Rank (Obsolete) 9223372036854775807 Severity