https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5473 Lustre <p>This issue was created by maloo for Nathaniel Clark &lt;nathaniel.l.clark@intel.com&gt;</p> <p>This issue relates to the following test suite run:<br/> <a href="https://testing.hpdd.intel.com/test_sets/09a62af8-1feb-11e4-8610-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/09a62af8-1feb-11e4-8610-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/b21a51b0-0aff-11e4-8dbb-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b21a51b0-0aff-11e4-8dbb-5254006e85c2</a></p> <p>The sub-test test_51b failed with the following error:</p> <blockquote> <p>test_51b failed with 1</p></blockquote> <p>Info required for matching: sanity 51b</p> LU-5473

Test failure sanity test_51b: test_51b failed: fnum: No space left on device

Bug Major Resolved Fixed Nathaniel Clark Maloo HB zfs Mon, 11 Aug 2014 20:07:07 +0000 Wed, 21 Sep 2016 05:26:58 +0000 Wed, 21 Sep 2016 05:26:57 +0000 Lustre 2.7.0 Lustre 2.5.3 Lustre 2.9.0 0 13 <p>I think this is some sort of test env issue?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mkdir(/mnt/lustre/d51.sanity/d61732) error: No space left on device total: 61732 creates in 390.32 seconds: 158.16 creates/second /usr/lib64/lustre/tests/sanity.sh: line 4004: /mnt/lustre/d51.sanity/fnum: No space left on device </pre> </div></div> <p>The same failure occurred while verifying patch <a href="http://review.whamcloud.com/11435" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11435</a> on Lustre b2_5 branch with FSTYPE=zfs over IB network:<br/> <a href="https://testing.hpdd.intel.com/test_sets/ae3f2c6a-261f-11e4-9fe5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ae3f2c6a-261f-11e4-9fe5-5254006e85c2</a></p> <p>The configuration was:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>MDSCOUNT=1 MDSSIZE=2097152 OSTCOUNT=2 OSTSIZE=8388608 </pre> </div></div> <p>+1 at <a href="https://testing.hpdd.intel.com/test_sets/890a458c-2675-11e4-a34b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/890a458c-2675-11e4-a34b-5254006e85c2</a>.</p> <p>By searching on Maloo, I found that all of the failures occurred under the following configuration:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>MDSCOUNT=1 MDSSIZE=2097152 OSTCOUNT=2 OSTSIZE=8388608 NETTYPE=o2ib FSTYPE=zfs </pre> </div></div> <p>The same test passed on Lustre 2.5.2 with FSTYPE=zfs and NETTYPE=tcp.</p> <p>Here is a for-test-only patch to perform the test against Lustre 2.5.2 with FSTYPE=zfs and NETTYPE=o2ib: <a href="http://review.whamcloud.com/11510" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11510</a></p> <p>Test results showed that the same failure occurred on Lustre 2.5.2 with FSTYPE=zfs and NETTYPE=o2ib.</p> <p>Dmesg on MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 15047:0:(osd_handler.c:211:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC. Metadata overhead is underestimated or grant_ratio is too low. </pre> </div></div> <p>Just increased the MDSSIZE to run the test again.</p> <p>After increasing MDSSIZE from 2GB to 4GB, the tests passed with FSTYPE=zfs and NETTYPE=o2ib:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/c5442360-332c-11e4-a323-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/c5442360-332c-11e4-a323-5254006e85c2</a></p> <p>I just changed the size to 3GB to see if the tests pass. And will create a TEI ticket.</p> <p>After increasing the MDSSIZE to 3GB, the tests also passed:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/200dd82c-3648-11e4-81c9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/200dd82c-3648-11e4-81c9-5254006e85c2</a></p> <p>I created TEI-2623.</p> <p>One more instance on Lustre b2_5 branch:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/7643a3c8-35c3-11e4-8a7f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/7643a3c8-35c3-11e4-8a7f-5254006e85c2</a></p> <p>+1 <a href="https://testing.hpdd.intel.com/test_sets/9269b584-3d6a-11e4-af25-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9269b584-3d6a-11e4-af25-5254006e85c2</a></p> <p>Another failure instance:<br/> <a href="https://testing.hpdd.intel.com/test_sets/a333f7ac-419e-11e4-8023-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a333f7ac-419e-11e4-8023-5254006e85c2</a></p> <p>The test_51b() code itself checks to see if the filesystem is reporting at least $NUMTEST free inodes on the MDT where the test directory is located, and at least 4kB of free space for each file. For creating <tt>NUMTEST=70000</tt> empty directories this should consume about 272MB of space on a ZFS MDT, so it is surprising that we can't fit this on a 2GB MDT.</p> <p>I pushed <a href="http://review.whamcloud.com/12185" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12185</a> to add some debugging to this test to see why it is failing, and to collect more information about ZFS space usage per inode before and after the test on the MDT even if it is not failing.</p> <p>Yu Jian wrote in <a href="http://review.whamcloud.com/12185:" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12185:</a></p> <blockquote> <p>&gt; Tests received by maloo, run on CentOS release 6.5/x86_64: (<a href="https://maloo.whamcloud.com/test_sessions/723d3cb6-5bee-11e4-a35f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/723d3cb6-5bee-11e4-a35f-5254006e85c2</a>). Ran 3 tests. 1 tests failed: sanity.<br/> The failure was reproduced over IB network.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> mkdir(/mnt/lustre/d51.sanity/d62645) error: No space left on device UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2029824 2029824 0 100% /mnt/lustre[MDT:0] UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 63039 63039 0 100% /mnt/lustre[MDT:0] </pre> </div></div> <p>Both space and inodes were consumed 100%.<br/> Before running createmany, the space and inodes were:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2031488 6784 2022656 0% /mnt/lustre[MDT:0] UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 228934 191 228743 0% /mnt/lustre[MDT:0] </pre> </div></div> <p>It was strange that the total inodes number was reduced from 228934 to 63039 .</p> <p>&gt; Tests received by maloo, run on CentOS release 6.5/x86_64: (<a href="https://maloo.whamcloud.com/test_sessions/7622fce2-5c81-11e4-b08a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/7622fce2-5c81-11e4-b08a-5254006e85c2</a>). Ran 3 tests. No failures.<br/> With TCP network (other test parameters were the same), the same test passed.<br/> Before running createmany, the space and inodes were:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2031872 4096 2025728 0% /mnt/lustre[MDT:0] UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 300941 191 300750 0% /mnt/lustre[MDT:0] </pre> </div></div></blockquote> <p>The failing test run shows about 32KB of space used per inode on the MDT both before and after the test is run. This is more than I would have expected, which is about 4KB per ZFS inode. It is expected that the free space and free inodes would run out at the same time on a ZFS filesystem, since the ZFS inodes are not preallocated as they are on ldiskfs. The total number of inodes and the number of free inodes is an estimate that is based on the space used and number of inodes used (average bytes per inode) and the number of free blocks. </p> <p>It would be interesting to see what the average space used per inode is for other ZFS filesystems. </p> <p>Isaac</p> <p>What do you suggest here?</p> <p>Peter</p> <p>Is there any way that we can figure out why the MDS is consuming 32KB per inode? I now recall a patch landing:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 47c0b97421b21dab686b05d6bf829ebcaf62d5db Author: Isaac Huang &lt;he.huang@intel.com&gt; Date: Tue Jul 22 16:42:03 2014 -0600 LU-5391 osd-zfs: ZAP object block sizes too small Currently osd-zfs ZAP objects use 4K for both leaf and indirect blocks. This patch increases: - leaf block to 16K, which equals ZFS fzap_default_block_shift - indirect block to 16K, the default used by ZPL directories Signed-off-by: Isaac Huang &lt;he.huang@intel.com&gt; Change-Id: I5b476414d27822a14afb25e1307991fbd2e3a59e Reviewed-on: http://review.whamcloud.com/11182 </pre> </div></div> <p>which might account for some of this. If the ZAP leaf and indirect blocks are being updated randomly due to many file creations, it may be that the indirect blocks are taking up a lot of space in the metadnode and saved snapshots?</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12185/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12185/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5473" title="Test failure sanity test_51b: test_51b failed: fnum: No space left on device" class="issue-link" data-issue-key="LU-5473"><del>LU-5473</del></a> tests: print space usage in sanity test_51b<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6e45c6d3ae4c46a0312bbb95b7e9ff09761f037d</p> <p>Patch landed to Master.</p> <p>Only a debug patch was landed, and the test was skipped. The core problem is not fixed.</p> <p>Nathaniel</p> <p>Isaac thinks that this failure is due to a flaw in the test script but does not have the bandwidth to dig into to it atm. Are you able to investigate?</p> <p>Thanks</p> <p>Peter</p> <p>The test is skipped (SLOW) on review-zfs, but it is still being run during full test runs, and is consistently passing on ZFS. The average space usage before the test run is about 11KB/inode on the MDT, and it reports plenty of free inodes before the test is passing, but the MDT filesystem is 3GB in size so it would have enough space for 95k inodes even at 32KB/inode that we were seeing before. I think 11KB/inode is reasonable for the number of actual files created in the filesystem at this point, since this includes all of the other filesystem overhead. It would be more useful to know the average per-inode space usage once all 70k files were created to get a better average and/or the differential usage for just those inodes.</p> <p><a href="https://testing.hpdd.intel.com/sub_tests/7b8bcf2a-4c80-11e5-b77f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/7b8bcf2a-4c80-11e5-b77f-5254006e85c2</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 51b: exceed 64k subdirectory nlink limit == 05:28:37 (1440480517) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 3063424 12416 3048960 0% /mnt/lustre[MDT:0] lustre-OST0000_UUID 2031360 5632 2023680 0% /mnt/lustre[OST:0] lustre-OST0001_UUID 2031360 4608 2024704 0% /mnt/lustre[OST:1] lustre-OST0002_UUID 2031104 3968 2025088 0% /mnt/lustre[OST:2] lustre-OST0003_UUID 2031360 4224 2025088 0% /mnt/lustre[OST:3] lustre-OST0004_UUID 2031360 5760 2023552 0% /mnt/lustre[OST:4] lustre-OST0005_UUID 2031104 5888 2023168 0% /mnt/lustre[OST:5] lustre-OST0006_UUID 2031360 6912 2022400 0% /mnt/lustre[OST:6] filesystem summary: 14219008 36992 14167680 0% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 162442 1363 161079 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 76956 423 76533 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 73555 323 73232 0% /mnt/lustre[OST:1] lustre-OST0002_UUID 74679 320 74359 0% /mnt/lustre[OST:2] lustre-OST0003_UUID 74436 324 74112 0% /mnt/lustre[OST:3] lustre-OST0004_UUID 71288 323 70965 0% /mnt/lustre[OST:4] lustre-OST0005_UUID 70901 321 70580 0% /mnt/lustre[OST:5] lustre-OST0006_UUID 69146 322 68824 0% /mnt/lustre[OST:6] filesystem summary: 162442 1363 161079 1% /mnt/lustre </pre> </div></div> <p>The osd_statfs-&gt;osd_objs_count_estimate() information that is being computed for ZFS is using about 21KB/inode for its free inodes estimate, which is conservative but reasonable given how few inodes are actually in use at this point.</p> <p>It wouldn't be terrible to print another <tt>lfs df</tt> and <tt>lfs df -i</tt> after the test, regardless of pass/fail result, to see what the average space usage is on ZFS, and then if it is still reasonable (i.e. going down from 11KB/inode) this bug could be closed.</p> <p>This test hasn't failed since 2015-06-15:<br/> <a href="https://testing.hpdd.intel.com/test_sets/b3e189ee-13ca-11e5-b4b0-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b3e189ee-13ca-11e5-b4b0-5254006e85c2</a></p> <p>Same MDS error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>15:41:02:LustreError: 19303:0:(osd_handler.c:209:osd_trans_start()) lustre-MDT0000: failed to start transaction due to ENOSPC. Metadata overhead is underestimated or grant_ratio is too low. </pre> </div></div> <p>Nathaniel Clark (nathaniel.l.clark@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/21821" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21821</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5473" title="Test failure sanity test_51b: test_51b failed: fnum: No space left on device" class="issue-link" data-issue-key="LU-5473"><del>LU-5473</del></a> tests: Add debug to sanity/51b<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d0f82be70ef01af4ccea83d870ddb4d9178690ac</p> <p>11KB/inode does seem to be a good estimate for ZFS based custom runs from <a href="http://review.whamcloud.com/21821" class="external-link" target="_blank" rel="nofollow noopener">21821</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/21821/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21821/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5473" title="Test failure sanity test_51b: test_51b failed: fnum: No space left on device" class="issue-link" data-issue-key="LU-5473"><del>LU-5473</del></a> tests: sanity/51b Account for ZFS inode size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 15dd813536ad06a119dfb2358f00281eed22a98b</p> <p>Landed for 2.9</p> Related Development Rank 1|hzwte7: Rank (Obsolete) 15250 Severity