https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16420 Lustre <p>This issue was created by maloo for jianyu &lt;yujian@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/19ce9196-c2db-40c4-9140-ebea855a919f" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/19ce9196-c2db-40c4-9140-ebea855a919f</a></p> <p>test_51d failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>OST0 has 265 objects, 86 are index 0 OST1 has 322 objects, 145 are index 0 OST2 has 320 objects, 85 are index 0 OST3 has 322 objects, 86 are index 0 OST4 has 265 objects, 87 are index 0 OST5 has 323 objects, 143 are index 0 OST6 has 324 objects, 88 are index 0 OST7 has 323 objects, 88 are index 0 'stripecount=3: OST 1 has more objects vs. OST 0 (341 &gt; 197 x 5/4)' </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity test_51d - 'stripecount=3: OST 1 has more objects vs. OST 0 (341 &gt; 197 x 5/4)'</p> LU-16420

sanity test_51d: FAIL: stripecount=3: OST 1 has more objects vs. OST 0

Bug Major Resolved Fixed Andreas Dilger Maloo Wed, 21 Dec 2022 07:30:22 +0000 Fri, 16 Jun 2023 16:25:52 +0000 Wed, 14 Jun 2023 22:01:59 +0000 Lustre 2.16.0 Lustre 2.16.0 0 4 <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/628ac757-7fa7-4dcb-9940-091fea92816f" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/628ac757-7fa7-4dcb-9940-091fea92816f</a></p> <p>There is some chance that patch <a href="https://review.whamcloud.com/50996" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/50996</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-16830" title="mdtest SEL jobs aborted with ENOSPC during automated FOFB testing" class="issue-link" data-issue-key="LU-16830"><del>LU-16830</del></a> lod: improve rr allocation</tt>" will avoid this issue. It landed yesterday, so need to wait a bit to see if it makes a difference. </p> <p>It seems possible that patch <a href="https://review.whamcloud.com/50532" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/50532</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-13748" title="&#39;lfs setstripe -C -1&#39; stripes too widely" class="issue-link" data-issue-key="LU-13748"><del>LU-13748</del></a> mdt: remove LASSERT in mdt_dump_lmm()</tt>" has caused this test to start failing much more often than before. It was failing only occasionally on master before 2023-05-22, with the exception of the above patch was failing test_51d regularly before it landed on 2023-05-19. </p> <p>The code change of that patch is almost impossible to be the cause (it just removes an <tt>LASSERT</tt>), but the added <tt>test_27Ch()</tt> that creates two fully overstriped files appears to somehow affect <tt>test_51d()</tt> in some obscure way. I don't really know why this is the case, since it is run about 20 minutes earlier. The patch stopped failing its testing once the overstriped file and directory were removed at the end of the test, though I also don't know why that was the case (it could have also been coincidence that all of the test runs passed in that case).</p> <p>Since I don't think the code change itself is causing the test failures, I'm instead going to try skipping the new test and/or move it later in the test series to see if this avoids the problem. It would also be good to understand exactly why this test is causing the problem. </p> <p>Some unsupported theories include:</p> <ul class="alternate" type="square"> <li>lingering exhaustion of objects on some OST after overstriping consumed all objects (but for 20 minutes?)</li> <li>create thread stuck or with bad state (but that wouldn't just have an imbalance?)</li> <li>MDS delete thread is still deleting OST objects from overstriped files, somehow affecting create performance?</li> <li>creating two overstriped files (on two different MDTs) somehow got the OST starting index "in sync" with other OSTs and imbalances the starting OST for the files just enough to trigger the failure?</li> <li>MDS is somehow stuck in QOS allocation mode, even after overstriped file has been removed?</li> </ul> <p>Note that a passing run is having almost perfect distribution:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>OST0 has 302 objects, 101 are index 0 OST1 has 302 objects, 101 are index 0 OST2 has 302 objects, 100 are index 0 OST3 has 302 objects, 100 are index 0 OST4 has 302 objects, 100 are index 0 OST5 has 302 objects, 100 are index 0 OST6 has 302 objects, 100 are index 0 </pre> </div></div> <p>unlike a failing run, which (AFAICS) alway has two OSTs that have fewer objects and the following OSTs have more first objects, which is what causes this test to fail:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>OST0 has 265 objects, 86 are index 0 OST1 has 322 objects, 145 are index 0 OST2 has 320 objects, 85 are index 0 OST3 has 322 objects, 86 are index 0 OST4 has 265 objects, 87 are index 0 OST5 has 323 objects, 143 are index 0 OST6 has 324 objects, 88 are index 0 OST7 has 323 objects, 88 are index 0 </pre> </div></div> <p>I also see that the passing results are with 7 OSTs (review-ldiskfs-* 0/176 failed) and the failing ones are 8 OSTs (review-dne-part-1, 65/269 failed runs in past week). That lends some credence to "phase shifted MDT creates" causing the problem, since creates between MDTs are not coordinated for performance reasons. </p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51268" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51268</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16420" title="sanity test_51d: FAIL: stripecount=3: OST 1 has more objects vs. OST 0" class="issue-link" data-issue-key="LU-16420"><del>LU-16420</del></a> tests: move overstriping test to sanity-pfl<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b05e0394254ca95c0e516acec1a12c945fea2f42</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51268/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51268/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16420" title="sanity test_51d: FAIL: stripecount=3: OST 1 has more objects vs. OST 0" class="issue-link" data-issue-key="LU-16420"><del>LU-16420</del></a> tests: move overstriping test to sanity-pfl<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0a21aa0623b87c643d4b56b9bd45fdcc1e566d4b</p> <p>Landed for 2.16</p> Duplicate LU-16864 LU-16902 Related LU-13748 LU-16830 Development Rank 1|i038gn: Rank (Obsolete) 9223372036854775807 Severity