https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13187 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;adilger@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run:<br/> <a href="https://testing.whamcloud.com/test_sets/2c10b9da-44b8-11ea-bffa-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/2c10b9da-44b8-11ea-bffa-52540065bddc</a></p> <p>test_129 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>current dir size 4096, previous limit 20480 </pre> </div></div> <p>It looks like this started on 2020-01-28 when a number of patches landed.</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity test_129 - current dir size 4096, previous limit 20480<br/> sanity test_129 - dirsize 4096 &lt; 32768 after 93 files</p> LU-13187

sanity test_129: current dir size 4096, previous limit 20480

Bug Major Resolved Fixed Dongyang Li Maloo Sat, 1 Feb 2020 07:22:24 +0000 Wed, 23 Sep 2020 14:58:33 +0000 Thu, 10 Sep 2020 20:12:01 +0000 Lustre 2.12.5 Lustre 2.14.0 Lustre 2.12.6 0 16 <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37683" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37683</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13187" title="sanity test_129: current dir size 4096, previous limit 20480" class="issue-link" data-issue-key="LU-13187"><del>LU-13187</del></a> tests: add ONLY_REPEAT parameter to repeat subtests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 23dd0d69dc325e206a653d1379661e27d8320fd9</p> <p>+42 failures in the past 4 weeks (2020-04-04 - 2020-04-30)</p> <p>Since this failure is only relevant for the ldiskfs code, it seems likely that the <b>only</b> ldiskfs patch that landed on 2020-01-28 is the culprit, namely patch <a href="https://review.whamcloud.com/37116" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37116</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-12977" title="fix i_mutex for ldiskfs_truncate() in osd_execute_truncate()" class="issue-link" data-issue-key="LU-12977"><del>LU-12977</del></a> ldiskfs: properly take inode_lock() for truncates</tt>". Unfortunately, I can't seem to get the test to fail consistently even when run in a loop, so it isn't possible to know whether reverting this patch would fix the test failures.</p> <p>+1 on Master: <a href="https://testing.whamcloud.com/sub_tests/d7a74936-6a65-4d13-8de0-1749e351fb84" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/d7a74936-6a65-4d13-8de0-1749e351fb84</a></p> <p>Dongyang, can you please look into this. It is being hit repeatedly with e2fsprogs testing, but also for regular review testing. </p> <p>I'm having a hard time to reproduce this and the patch <a href="https://review.whamcloud.com/37116" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37116</a> is just about truncate, should have nothing to do with the size of the dir.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38550" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38550</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13187" title="sanity test_129: current dir size 4096, previous limit 20480" class="issue-link" data-issue-key="LU-13187"><del>LU-13187</del></a> tests: improve sanity test_129 checking<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e8a90531a24453778e7a39e2fbdf7f8ff138b547</p> <p>+1 on master <a href="https://testing.whamcloud.com/sub_tests/963e28b3-1f52-4221-ad58-5c7b5871893b" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/963e28b3-1f52-4221-ad58-5c7b5871893b</a></p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38550/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38550/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13187" title="sanity test_129: current dir size 4096, previous limit 20480" class="issue-link" data-issue-key="LU-13187"><del>LU-13187</del></a> tests: improve sanity test_129 checking<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: eb80e37f023b85ef0e610ab65cee1d8ee07235fc</p> <p>So the patch has landed, but the test is still failing (although a bit more verbosely than before):<br/> <a href="https://testing.whamcloud.com/test_sets/3611de75-e067-402d-adce-920f82e2e2cc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/3611de75-e067-402d-adce-920f82e2e2cc</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: trevis-13vm8 echo 32768 &gt;/sys/fs/ldiskfs/dm-5/max_dir_size CMD: trevis-13vm8 echo 24576 &gt;/sys/fs/ldiskfs/dm-5/warning_dir_size mcreate: cannot create `/mnt/lustre/d129.sanity/file_base_93' with mode 0100644: No space left on device rc=28 returned as expected after 93 files total: 5 open/close in 0.01 seconds: 634.44 ops/second dirsize 4096 &lt; 32768 after 93 files </pre> </div></div> <p>Looking at the debug logs it is fairly clear that the MDS is returning <tt>-ENOSPC=-28</tt> after creating only 93 files, but it isn't printing any errors related to reaching the directory limits.</p> <p>So it seems that this problem may be a defect in the RHEL7.8 ldiskfs code, or possibly in the upstream ext4 code for that kernel? Could someone with a RHEL7.8 or 8.0 kernel run a manual test to see if this <tt>sys/fs/ldiskfs/XXX/max_dir_size</tt> setting is working at all?</p> <p>I've done that on RHEL7.8 some time ago and it's working:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@centos7 ~]# echo 20480 &gt; /sys/fs/ldiskfs/vdb/max_dir_size [root@centos7 ~]# echo 20480 &gt; /sys/fs/ldiskfs/vdb/warning_dir_size [root@centos7 ~]# sh ./test_129.sh open(O_RDWR|O_CREAT): No space left on device 28 655 [root@centos7 ~]# stat /mnt/testdir File: ‘/mnt/testdir’ Size: 20480 Blocks: 40 IO Block: 4096 directory Device: fd10h/64784d Inode: 1048577 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2020-05-07 21:35:00.688000000 +1000 Modify: 2020-05-07 21:45:55.832000000 +1000 Change: 2020-05-07 21:45:55.832000000 +1000 Birth: -</pre> </div></div> <p>the test_129.sh just creates files under testdir using multiop and prints errcode (28) and number of files created(655)</p> <p>I haven't done it on RHEL8 but I think ldiskfs side is fine.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dongyang" class="user-hover" rel="dongyang">dongyang</a> can you please submit a patch to test_129 that sets debug=-1 on the client and MDS, and maybe adds CDEBUG() to osd-ldiskfs so we can see where the -28 is coming from. I was looking at the debug logs from the current failure and they didn't show enough details</p> <p>Li Dongyang (dongyangli@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/38700" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38700</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13187" title="sanity test_129: current dir size 4096, previous limit 20480" class="issue-link" data-issue-key="LU-13187"><del>LU-13187</del></a> tests: get more debug info from test_129<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: db1fab110e4a0e57afbbc0300e14f4fbbd2d6f94</p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/52fe5475-a2f3-4bd6-a2b7-c1dc99b20590" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/52fe5475-a2f3-4bd6-a2b7-c1dc99b20590</a></p> <p>+1 on Lustre b2_12 branch: <a href="https://testing.whamcloud.com/test_sets/5e20954f-2daf-47ff-9b6c-eacbf7a41dfc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5e20954f-2daf-47ff-9b6c-eacbf7a41dfc</a></p> <p>more on master: <br/> <a href="https://testing.whamcloud.com/test_sets/353838f4-221f-4336-accc-ccaea50e17e3" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/353838f4-221f-4336-accc-ccaea50e17e3</a><br/> <a href="https://testing.whamcloud.com/test_sets/629cec52-dd19-40c0-b0f2-0c22435f81df" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/629cec52-dd19-40c0-b0f2-0c22435f81df</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/1e95c770-b87b-4de6-9a58-08d40241c712" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/1e95c770-b87b-4de6-9a58-08d40241c712</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/62ce778c-aac4-4504-a1dc-ecd559e78533" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/62ce778c-aac4-4504-a1dc-ecd559e78533</a></p> <p>Note Neil also ran into this problem on SUSE15 and pushed a fix here:</p> <p><a href="https://review.whamcloud.com/#/c/39571/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/39571/</a></p> <p>The same problem could be for RHEL platforms.</p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/18024581-0159-4d24-84ee-9ae6554ced77" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/18024581-0159-4d24-84ee-9ae6554ced77</a></p> <p>+3 on master:<br/> <a href="https://testing.whamcloud.com/test_sessions/7b47cafb-4b4e-4cc3-ae57-971c31e4ce84" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/7b47cafb-4b4e-4cc3-ae57-971c31e4ce84</a><br/> <a href="https://testing.whamcloud.com/test_sessions/70e01f6b-f61c-4d82-a3c6-fa141eb170fe" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/70e01f6b-f61c-4d82-a3c6-fa141eb170fe</a><br/> <a href="https://testing.whamcloud.com/test_sessions/27615c0d-2da3-42c0-8bb9-230da1f3acb2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/27615c0d-2da3-42c0-8bb9-230da1f3acb2</a></p> <p> </p> <p>There were 55 failures of this subtest in the last week, which is about a 10% failure rate, but since sanity is running multiple times per patch, it is affecting landing more than this. </p> <p>James Simmons (jsimmons@infradead.org) uploaded a new patch: <a href="https://review.whamcloud.com/39773" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39773</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13187" title="sanity test_129: current dir size 4096, previous limit 20480" class="issue-link" data-issue-key="LU-13187"><del>LU-13187</del></a> ldiskfs: Fix max_dir_size_kb for RHEL7<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 43fe1051ee6cab1c9f8b85863ec91aec2c06b251</p> <p>RHE8 and Ubuntu overlap for the ldiskfs patches so they will need to be updated at the same time. I can update Ubuntu but I don't have a RHEL8 system to fix it up on.</p> <p>I think this problem is caused by some metadata directory trying to grow.</p> <p>I added some tracing and found that the call to osd_ldiskfs_append() in iam_new_node() was failing with ENOSPC.</p> <p>Maybe the best fix would be to add a test to ldiskfs_append() to check if it is a special lustre metadata directory, and if so to bypass the dir limit.</p> <p>Is there an easy way to detect lustre metadata directories?</p> <p>Great, I just could not reproduce the problem.</p> <p>if it's failing with ENOSPC from osd_ldiskfs_append(), I think we can add a new param to ldiskfs/ext4_append() to bypass the limit check for the oi related code path, like iam_new_node(), iam_lfix_create() and iam_lvar_create().</p> <p>the normal dir is using a different code path, osd_ldiskfs_add_entry()-&gt;__ldiskfs/ext4_add_entry()</p> <p> </p> <p>Neil, thanks for tracking this down.</p> <p>Dongyang, I think it would be better avoid changing the API for <tt>ext4_append()</tt>, as that would need even more changes to the core ext4 code.</p> <p>I think there are two options that are relatively simple:</p> <ul class="alternate" type="square"> <li>split <tt>ext4_append()</tt> into an outer function of the same name that checks the directory size limit, and a second internal function (e.g. <tt>ext4_append_nolimit()</tt> or similar) that can be called directly from the <tt>iam_&#42;</tt> functions</li> <li>set a new <tt>EXT4_STATE_IAM</tt> flag on the IAM objects when they are opened, and check that inside <tt>ext4_append()</tt> when checking the size limit</li> </ul> <p>Probably the second option is less intrusive, as it is likely that patch could avoid conflicts with <tt>ext4-pdirop.patch</tt> and <tt>ext4-misc.patch</tt>, and hopefully would not need to be different for every kernel.</p> <p>Li Dongyang (dongyangli@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/39823" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39823</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13187" title="sanity test_129: current dir size 4096, previous limit 20480" class="issue-link" data-issue-key="LU-13187"><del>LU-13187</del></a> osd-ldiskfs: don't enforce max dir size limit on IAM objects<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 07c97b04411a396572d7124f3217a7d561a96d2b</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39823/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39823/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13187" title="sanity test_129: current dir size 4096, previous limit 20480" class="issue-link" data-issue-key="LU-13187"><del>LU-13187</del></a> osd-ldiskfs: don't enforce max dir size limit on IAM objects<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 03e6db505be90d35ccacb3af7e15277784e5d448</p> <p>Landed for 2.14</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39882" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39882</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13187" title="sanity test_129: current dir size 4096, previous limit 20480" class="issue-link" data-issue-key="LU-13187"><del>LU-13187</del></a> osd-ldiskfs: don't enforce max dir size limit on IAM objects<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: d8c40507e87798e37d05988d190ccef78b528c42</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39882/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39882/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13187" title="sanity test_129: current dir size 4096, previous limit 20480" class="issue-link" data-issue-key="LU-13187"><del>LU-13187</del></a> osd-ldiskfs: don't enforce max dir size limit on IAM objects<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: a73f4e566debadfc156b6d8c48237a2e34ac75ba</p> Related LU-11310 LU-13916 Development Rank 1|i00t0n: Rank (Obsolete) 9223372036854775807 Severity