https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14659 Lustre <p>This issue was created by maloo for Vitaly Fertman &lt;vitaly_fertman@xyratex.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/9b91c699-3875-4fd0-a21b-f78bb5335464" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/9b91c699-3875-4fd0-a21b-f78bb5335464</a></p> <p>test_413a failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>subdirs shouldn't be evenly distributed </pre> </div></div> <p>it seems <a href="https://jira.whamcloud.com/browse/LU-12495" title="sanity test_413b: subdirs shouldn&#39;t be evenly distributed" class="issue-link" data-issue-key="LU-12495"><del>LU-12495</del></a> appeared again</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity test_413a - subdirs shouldn't be evenly distributed</p> LU-14659

sanity test_413a: subdirs shouldn't be evenly distributed

Bug Minor Resolved Fixed Lai Siyao Maloo Fri, 30 Apr 2021 17:38:01 +0000 Sat, 21 Oct 2023 17:41:13 +0000 Sun, 17 Oct 2021 23:09:30 +0000 Lustre 2.15.0 0 6 <p>Vitaly,<br/> the test failure you linked here is on an early version of my patch <a href="https://review.whamcloud.com/#/c/43445/3" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/43445/3</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-13439" title="DNE3: MDT QOS tuning to avoid full MDTs completely" class="issue-link" data-issue-key="LU-13439"><del>LU-13439</del></a> lmv: qos stay on current MDT if less full</tt>" that is modifying the DNE MDT selection algorithm and had a bug in it. However, that bug was fixed and the more recent versions of that patch have been passing test_413<span class="error">&#91;abc&#93;</span>, so it isn't clear why you filed this issue?</p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/857c693d-64af-48d6-a029-b641395d9b1a" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/857c693d-64af-48d6-a029-b641395d9b1a</a></p> <p>This should be fixed by patch <a href="https://review.whamcloud.com/43997" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43997</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-14762" title="qos subdirectory creation stay on parent MDT if less full than average doesn&#39;t work as expected" class="issue-link" data-issue-key="LU-14762"><del>LU-14762</del></a> lmv: compare space to mkdir on parent MDT</tt>"</p> <p>Lai, it looks like this is still being hit on master. Looking at a recent failure, I think there are two issues:<br/> <a href="https://testing.whamcloud.com/test_sets/3d05b60b-1939-4440-859d-55b5a2bec858" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/3d05b60b-1939-4440-859d-55b5a2bec858</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Check for uneven MDTs: weight diff=0% must be &gt; 100% ...Fill MDT0 with 100 files: loop 0 weight diff=0% must be &gt; 100% ...Fill MDT0 with 100 files: loop 1 weight diff=0% must be &gt; 100% ...Fill MDT0 with 100 files: loop 2 MDT filesfree available: 834976 834538 835384 835497 MDT blocks available: 416644 1045656 1044312 1042392 weight diff=150% </pre> </div></div> <p>It looks like statfs is being cached inside the loop, so the loop is running more times than needed. This is probably also causing <a href="https://jira.whamcloud.com/browse/LU-14898" title="sanity test_413a: (max - min) * 100 / min: division by 0 (error token is &quot;min&quot;)" class="issue-link" data-issue-key="LU-14898"><del>LU-14898</del></a> (divide by zero error) because the loop doesn't stop until the MDT has no free space left. Also, since inodes are weighted higher than space, maybe we need to re-add some amount of regular file creation into the loop (maybe creating only 64KB DoM files instead of 1MB) so the MDTs are more imbalanced by inodes also?</p> <p>The second issue is that the "imbalance check" appears to be using the minimum imbalance instead of the maximum imbalance?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>179 directories created on MDT0 188 directories created on MDT1 222 directories created on MDT2 211 directories created on MDT3 sanity test_413a: @@@@@@ FAIL: subdirs shouldn't be evenly distributed: 188 - 179 &lt; 20 </pre> </div></div> <p>The 189 directories on MDT1 is only higher by 9, but the other MDTs are higger by 43 and 32. Since the MDT selection is random, maybe this should compare the average difference &gt; 20, and that no single MDT is less than 5 higher?</p> <p>Mmm, this can be improved. And the reason the "imbalance check" looks to be minimum imbalance is the most empty MDT is checked before IO, due to the cached statfs result and randomness in MDT choosing, it may not be the MDT most directories are created. It should be changed to check after creation.</p> <p>"Lai Siyao &lt;lai.siyao@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/44649" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44649</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14659" title="sanity test_413a: subdirs shouldn&#39;t be evenly distributed" class="issue-link" data-issue-key="LU-14659"><del>LU-14659</del></a> test: improve generate_uneven_mdts() in sanity.sh<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 074354d0a3282a72957eef178338d1e8cd14af7b</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/44649/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44649/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14659" title="sanity test_413a: subdirs shouldn&#39;t be evenly distributed" class="issue-link" data-issue-key="LU-14659"><del>LU-14659</del></a> test: improve generate_uneven_mdts() in sanity.sh<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d45be79a069f527657c1ce91630183031ea42b27</p> <p>Landed for 2.15</p> Duplicate LU-14762 Related LU-12831 LU-14824 LU-14898 Development Rank 1|i01tiv: Rank (Obsolete) 9223372036854775807 Severity