https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9069 Lustre <p>sanity: FAIL: test_255a Speedup with willread is less than XX%, got YY%</p> <p>Test is expecting willreads to be faster than cache reads, but willreads are slower:</p> <p>From test_log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Iter 1/10: cache speedup: 228% Iter 1/10: ladvise speedup: 24% Iter 2/10: cache speedup: 147% Iter 2/10: ladvise speedup: 20% Iter 3/10: cache speedup: 170% Iter 3/10: ladvise speedup: 10% </pre> </div></div> <p>Related to <a href="https://jira.whamcloud.com/browse/LU-8565" title="sanity test 255a fails with ‘Speedup with willread is less than X%, got Y%’" class="issue-link" data-issue-key="LU-8565"><del>LU-8565</del></a>, but that issue was attributed to VM slowness. This issue is also seen on physical machines.</p> <p>Info required for matching: sanity 255a</p> onyx-64 thru 67<br/> Full Group test, RHEL7.3/ZFS<br/> master branch, b3499 LU-9069

ZFS sanity: FAIL: test_255a Speedup with willread is less than XX%, got YY%

Bug Minor Open Unresolved WC Triage James Casper always_except zfs Tue, 31 Jan 2017 23:26:52 +0000 Fri, 1 Dec 2017 19:53:33 +0000 Lustre 2.10.0 0 8 <p>It turns out that all of these real-world test failures are only failing with ZFS backends. That implies some problem with ladvise not prefetching the pages properly for ZFS, since the cached reads run much faster but ladvise reads sometimes do not run faster with ZFS.</p> <p>Looking at the output more closely, it appears that the cases where ZFS is passing are only because the cached speedup is below the threshold:<br/> <a href="https://testing.hpdd.intel.com/sub_tests/64172ba6-f9df-11e6-8b38-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/64172ba6-f9df-11e6-8b38-5254006e85c2</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Iter 1/10: reading without willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 1/10: Reading again without willread hint Iter 1/10: reading with willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 1/10: cache speedup: 6.79256/6.53006=4% Iter 1/10: ladvise speedup: 6.59651/6.53006=1% Iter 2/10: reading without willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 2/10: Reading again without willread hint Iter 2/10: reading with willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 2/10: cache speedup: 7.24601/6.77954=6% Iter 2/10: ladvise speedup: 6.651/6.77954=-1% Iter 3/10: reading without willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 3/10: Reading again without willread hint Iter 3/10: reading with willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 3/10: cache speedup: 6.82836/6.4925=5% Iter 3/10: ladvise speedup: 6.39501/6.4925=-1% Iter 4/10: reading without willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 4/10: Reading again without willread hint Iter 4/10: reading with willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 4/10: cache speedup: 6.80656/6.68438=1% Iter 4/10: ladvise speedup: 6.52108/6.68438=-2% Iter 5/10: reading without willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 5/10: Reading again without willread hint Iter 5/10: reading with willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 5/10: cache speedup: 7.23196/6.44395=12% Iter 5/10: ladvise speedup: 6.67935/6.44395=3% Iter 6/10: reading without willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 6/10: Reading again without willread hint Iter 6/10: reading with willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 6/10: cache speedup: 6.90188/6.73113=2% Iter 6/10: ladvise speedup: 6.61056/6.73113=-1% Iter 7/10: reading without willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 7/10: Reading again without willread hint Iter 7/10: reading with willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 7/10: cache speedup: 6.93889/6.67545=3% Iter 7/10: ladvise speedup: 6.4859/6.67545=-2% Iter 8/10: reading without willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 8/10: Reading again without willread hint Iter 8/10: reading with willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 8/10: cache speedup: 7.03663/6.86692=2% Iter 8/10: ladvise speedup: 6.60757/6.86692=-3% Iter 9/10: reading without willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 9/10: Reading again without willread hint Iter 9/10: reading with willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 9/10: cache speedup: 7.18227/6.64533=8% Iter 9/10: ladvise speedup: 6.56245/6.64533=-1% Iter 10/10: reading without willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 10/10: Reading again without willread hint Iter 10/10: reading with willread hint CMD: trevis-34vm4 echo 3 &gt; /proc/sys/vm/drop_caches Iter 10/10: cache speedup: 6.96441/6.69985=3% Iter 10/10: ladvise speedup: 6.46659/6.69985=-3% Speedup with cache is less than 20% (4%), skipping check of speedup with willread: -1% </pre> </div></div> <p>So it looks like the real problem is that "echo 3 &gt; /proc/sys/vm/drop_caches" doesn't actually flush the cache for ZFS servers, so the "uncached speed" test is invalid.</p> <p>What is needed is to implement the "dontneed" ladvice for ZFS so that the test can use it to flush the server ARC cache for this file and run a valid test.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/24907" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/24907</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9069" title="ZFS sanity: FAIL: test_255a Speedup with willread is less than XX%, got YY%" class="issue-link" data-issue-key="LU-9069">LU-9069</a> tests: improve output of sanity test_255a<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 5<br/> Commit: 99c7ca0110affd39c33afda073d09e6c1335ac81</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/24907/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/24907/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9069" title="ZFS sanity: FAIL: test_255a Speedup with willread is less than XX%, got YY%" class="issue-link" data-issue-key="LU-9069">LU-9069</a> tests: improve output of sanity test_255a<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: bdd470ff9765cb81d94b888ee1067e1c420303bc</p> Related LU-6254 LU-10314 LU-8565 Development Rank 1|hzz25z: Rank (Obsolete) 9223372036854775807 Severity