https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7584 Lustre <p>This issue was created by maloo for Saurabh Tandan &lt;saurabh.tandan@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/3eac1b80-a26d-11e5-bdef-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3eac1b80-a26d-11e5-bdef-5254006e85c2</a>.</p> <p>The sub-test test_129 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: shadow-53vm3 test -e /sys/fs/ldiskfs/dm-3/max_dir_size CMD: shadow-53vm3 echo 0 &gt;/sys/fs/ldiskfs/dm-3/max_dir_size return code 28 received as expected current dir size 24576, previous limit 24576 </pre> </div></div> <p>Appears to be similar to <a href="https://jira.whamcloud.com/browse/LU-2479" title="sanity.sh test_129: max dir size limit work incorrectly." class="issue-link" data-issue-key="LU-2479"><del>LU-2479</del></a></p> EL7.1 Server/EL7.1 Client - DNE<br/> Master, build# 3270 LU-7584

sanity test_129: current dir size 24576, previous limit 24576

Bug Critical Resolved Fixed Jian Yu Maloo Fri, 18 Dec 2015 23:47:18 +0000 Tue, 16 May 2017 12:50:14 +0000 Fri, 5 Feb 2016 15:28:39 +0000 Lustre 2.8.0 Lustre 2.8.0 Lustre 2.10.0 0 10 <p>it looks similar as <a href="https://jira.whamcloud.com/browse/LU-4654" title="sanity test_129: current dir size 12288, previous limit 12288" class="issue-link" data-issue-key="LU-4654"><del>LU-4654</del></a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/57908c32-b69d-11e5-88ef-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/57908c32-b69d-11e5-88ef-5254006e85c2</a></p> <p><a href="https://testing.hpdd.intel.com/test_sets/173b63f6-b575-11e5-bf32-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/173b63f6-b575-11e5-bf32-5254006e85c2</a></p> <p>another instance, not sure if it is the same case<br/> server: lustre-master build#2976 RHEL7<br/> client 2.7.1</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 129: test directory size limit ========================== 12:59:04 (1452113944) striped dir -i1 -c2 /mnt/lustre/d129.sanity open(O_RDWR|O_CREAT): No space left on device return code 28 received as expected sanity test_129: @@@@@@ FAIL: current dir size 8192, previous limit 24576 </pre> </div></div> <p>More failures on master:<br/> 2016-01-12 00:53:37 - <a href="https://testing.hpdd.intel.com/test_sets/a33af3f4-b8ee-11e5-825c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a33af3f4-b8ee-11e5-825c-5254006e85c2</a><br/> 2016-01-12 04:15:31 - <a href="https://testing.hpdd.intel.com/test_sets/8b6097d2-b913-11e5-80e0-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8b6097d2-b913-11e5-80e0-5254006e85c2</a></p> <p>more on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/2ae7caee-bf0b-11e5-b113-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2ae7caee-bf0b-11e5-b113-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/81e6a2f2-bf33-11e5-9bdc-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/81e6a2f2-bf33-11e5-9bdc-5254006e85c2</a></p> <p>more instance on master <br/> client and server: lustre-master build#3305 RHEL7.1 DNE<br/> <a href="https://testing.hpdd.intel.com/test_sets/0ef23214-bc15-11e5-8ede-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0ef23214-bc15-11e5-8ede-5254006e85c2</a></p> <p>another instance on master, sles12sp1 client/server:<br/> <a href="https://testing.hpdd.intel.com/test_sets/a415d5c2-c08d-11e5-a8e5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a415d5c2-c08d-11e5-a8e5-5254006e85c2</a></p> <p>I'm starting to think this problem may be a kernel version issue. many (all?) of the instances reported here are el7 or sles12.</p> <p>not seen in most recent test runs of el7.2 on master. continues to be seen a lot in recent test runs of sles12sp1 on master.</p> <p>Yes I concur. I have been testing the upstream client and seeing this same error.</p> <p>another on master, with el7.2:<br/> <a href="https://testing.hpdd.intel.com/test_sets/1ada50fe-c3e3-11e5-8866-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1ada50fe-c3e3-11e5-8866-5254006e85c2</a></p> <p>I think this bug is becoming a serious blocker for el7 and sles12.</p> <p>again on master with el7.2:<br/> <a href="https://testing.hpdd.intel.com/test_sets/bc6f6f14-c47c-11e5-a651-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/bc6f6f14-c47c-11e5-a651-5254006e85c2</a></p> <p>seem to be hitting this 100% with el7.2 on master</p> <p>As per <a href="http://review.whamcloud.com/15548" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15548</a>, ext4-give-warning-with-dir-htree-growing.patch is also needed in the new ldiskfs-3.10-rhel7.2.series and ldiskfs-3.12-sles12.series. I'm creating the patch.</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/18169" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18169</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7584" title="sanity test_129: current dir size 24576, previous limit 24576" class="issue-link" data-issue-key="LU-7584"><del>LU-7584</del></a> ldiskfs: add dir htree growing warning patch<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7f7a1eb405f17c974fe0d2747c0b871629aaf25c</p> <p>The above patch just added warning messages while directory size growing but didn't resolve the issue in this ticket. All of the failure instances occurred under DNE configuration.</p> <p>Di, do you think the following comparison in the current sanity test_129() is correct under DNE configuration?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> I=$(stat -c%s <span class="code-quote">"$DIR/$tdir"</span>) <span class="code-keyword">if</span> [ $(lustre_version_code $SINGLEMDS) -lt \ $(version_code 2.4.51) ] then [[ $I -eq $MAX ]] &amp;&amp; <span class="code-keyword">return</span> 0 <span class="code-keyword">else</span> [[ $I -gt $MAX ]] &amp;&amp; <span class="code-keyword">return</span> 0 fi error_exit <span class="code-quote">"current dir size $I, previous limit $MAX"</span> </pre> </div></div> <p><a href="http://review.whamcloud.com/#/c/17874" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/17874</a> already adds the missing patch to the el7.2 patch series, but the failure still happens anyway.</p> <p>Hmm, I saw it only create 1 more file then check if it pass the limit, which seems not right to me. Because the previous ENOSPC failure might happen on different stripes, i.e. if the new file is created in different stripe, then this check will fail. </p> <p>IMHO, probably you can just replace test_mkdir with mkdir and only do single MDS check (of course also removed those STRIPE_COUNT thing), since this test is for checking ldiskfs parameters, this probably makes sense.</p> <p>Or</p> <p>you need create the new file on the specific stripe by lfs mkdir, instead of using multiop, but then you also need find out which stripe is "FULL" here.</p> <p>Thank you, Di. Let me create a patch.</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/18192" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18192</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7584" title="sanity test_129: current dir size 24576, previous limit 24576" class="issue-link" data-issue-key="LU-7584"><del>LU-7584</del></a> tests: create file on single MDS in sanity test 129<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a3518d6fd8e2af08f43c41bb413edee19ee56f64</p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/be6a1f10-c555-11e5-b0fc-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/be6a1f10-c555-11e5-b0fc-5254006e85c2</a></p> <p>Another instance for FULL - EL7.1 Server/EL7.1 Client - DNE, master, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/6b7e1dae-cac5-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6b7e1dae-cac5-11e5-9609-5254006e85c2</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/18192/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18192/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7584" title="sanity test_129: current dir size 24576, previous limit 24576" class="issue-link" data-issue-key="LU-7584"><del>LU-7584</del></a> tests: create file on single MDS in sanity test 129<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1299e2aff9af57fc8a79a6fa09c1676a61cbfa4b</p> <p>Patch has landed for 2.8</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/26874" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26874</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7584" title="sanity test_129: current dir size 24576, previous limit 24576" class="issue-link" data-issue-key="LU-7584"><del>LU-7584</del></a> tests: clean up sanity test_129<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 18a15a480d3e1138ac73a07db0a3418912ee6687</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/26874/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26874/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7584" title="sanity test_129: current dir size 24576, previous limit 24576" class="issue-link" data-issue-key="LU-7584"><del>LU-7584</del></a> tests: clean up sanity test_129<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5611dbe56b95e79df0723e2668f977492abd68ed</p> Related Development Rank 1|hzxwdj: Rank (Obsolete) 9223372036854775807 Severity