https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2253 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/c15b1d7e-22eb-11e2-afb4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c15b1d7e-22eb-11e2-afb4-52540035b04c</a>.</p> <p>The sub-test test_2 failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>test log shows no space left on device error and user create failure</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mknod(/mnt/lustre/d0.sanity-quota/d2/f.sanity-quota.2-01048326) error: No space left on device total: 1048326 creates in 1873.45 seconds: 559.57 creates/second Disk quotas for user quota_usr (uid 60000): Filesystem kbytes quota limit grace files quota limit grace /mnt/lustre 0 0 0 - 1048326 0 1048576 - lustre-MDT0000_UUID 0 - 0 - 1048326 - 1048576 - lustre-OST0000_UUID 0 - 0 - - - - - lustre-OST0001_UUID 0 - 0 - - - - - lustre-OST0002_UUID 0 - 0 - - - - - lustre-OST0003_UUID 0 - 0 - - - - - lustre-OST0004_UUID 0 - 0 - - - - - lustre-OST0005_UUID 0 - 0 - - - - - lustre-OST0006_UUID 0 - 0 - - - - - Files for user (quota_usr): File: `/mnt/lustre/d0.sanity-quota/d2/f.sanity-quota.2-0977084' Size: 0 Blocks: 0 IO Block: 2097152 regular empty file Device: 2c54f966h/743766374d Inode: 144115205373225167 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2012-10-30 04:15:05.000000000 -0700 Modify: 2012-10-30 04:15:05.000000000 -0700 Change: 2012-10-30 04:15:05.000000000 -0700 File: `/mnt/lustre/d0.sanity-quota/d2/f.sanity-quota.2-0164550' Size: 0 Blocks: 0 IO Block: 2097152 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272535769 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2012-10-30 03:49:44.000000000 -0700 Modify: 2012-10-30 03:49:44.000000000 -0700 Change: 2012-10-30 03:49:44.000000000 -0700 File: `/mnt/lustre/d0.sanity-quota/d2/f.sanity-quota.2-0461849' Size: 0 Blocks: 0 IO Block: 2097152 regular empty file Device: 2c54f966h/743766374d Inode: 144115205306125356 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2012-10-30 03:56:08.000000000 -0700 Modify: 2012-10-30 03:56:08.000000000 -0700 Change: 2012-10-30 03:56:08.000000000 -0700 File: `/mnt/lustre/d0.sanity-quota/d2/f.sanity-quota.2-0602708' Size: 0 Blocks: 0 IO Block: 2097152 regular empty file Device: 2c54f966h/743766374d Inode: 144115205322912359 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2012-10-30 04:00:27.000000000 -0700 Modify: 2012-10-30 04:00:27.000000000 -0700 Change: 2012-10-30 04:00:27.000000000 -0700 sanity-quota test_2: @@@@@@ FAIL: user create failure, but expect success Trace dump: </pre> </div></div> Test was run with SLOW=yes LU-2253

Test failure on sanity-quota test_2: error: No space left on device

Bug Blocker Resolved Fixed Niu Yawei Maloo mn8 yuc2 Wed, 31 Oct 2012 00:37:04 +0000 Thu, 19 Dec 2013 12:20:59 +0000 Tue, 3 Sep 2013 06:41:20 +0000 Lustre 2.4.0 Lustre 1.8.9 Lustre 2.4.2 Lustre 2.5.0 0 7 <p>Hit same issue on SLEL11 SP2 client<br/> <a href="https://maloo.whamcloud.com/test_sets/30b98780-2541-11e2-9e7c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/30b98780-2541-11e2-9e7c-52540035b04c</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mknod(/mnt/lustre/d0.sanity-quota/d2/f.sanity-quota.2-01047886) error: No space left on device total: 1047886 creates in 1872.26 seconds: 559.69 creates/second </pre> </div></div> <p>Niu,<br/> Can you have a look at this one? Do you already have a patch for this?</p> <p>Yes, I think it has been fixed by 984f4ce51fd38caaf0bd2b706a130f7f17c51638 (which has been landed). Now we check the free inodes in test_2, and skip the test if free inodes isn't enough.</p> <p>patch landed. <a href="http://review.whamcloud.com/4275" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4275</a> (see <a href="https://jira.whamcloud.com/browse/LU-2153" title="Quota reintegration holes " class="issue-link" data-issue-key="LU-2153"><del>LU-2153</del></a>)</p> <p>This failure is not seen in tag-2.3.56 testing</p> <p>Lustre client build: <a href="http://build.whamcloud.com/job/lustre-b1_8/258/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b1_8/258/</a> (1.8.9-wc1)<br/> Lustre server build: <a href="http://build.whamcloud.com/job/lustre-b2_4/31/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/31/</a></p> <p>sanity-quota test is blocked by the issue in this ticket:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mknod(/mnt/lustre/d0.sanity-quota/d2/f2-01048289) error: No space left on device total: 1048289 creates in 1784.48 seconds: 587.45 creates/second </pre> </div></div> <p><a href="https://maloo.whamcloud.com/test_sets/96c0392a-0489-11e3-90ba-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/96c0392a-0489-11e3-90ba-52540035b04c</a></p> <p>More instances:<br/> <a href="https://maloo.whamcloud.com/test_sets/a59bbd40-ec68-11e2-8011-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a59bbd40-ec68-11e2-8011-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/ae15b2c6-0236-11e3-ba9e-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ae15b2c6-0236-11e3-ba9e-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/6bb6a738-c941-11e2-8cff-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6bb6a738-c941-11e2-8cff-52540035b04c</a></p> <p>The free inode checking in s-q of b1_8 is wrong:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> local FREE_INODES=$(lfs_df -i | grep <span class="code-quote">"summary"</span> | awk <span class="code-quote">'{print $5}'</span>) </pre> </div></div> <p>It should be 'print $4' but not 'print $5', we must forgot to change it from 5 to 4 when replace 'lfs df' with 'lfs_df'.</p> <p>patch for b1_8 to fix the typo in test_2 of s-q: <a href="http://review.whamcloud.com/7341" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7341</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/28/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/28/</a><br/> MDSCOUNT=4</p> <p>The same failure also occurred under DNE configuration:<br/> <a href="https://maloo.whamcloud.com/test_sets/b33e632c-027e-11e3-b384-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b33e632c-027e-11e3-b384-52540035b04c</a></p> <p>In DNE, we ususally want to get the free inodes on MDT0 but not the whole filesystem: <a href="http://review.whamcloud.com/7375" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7375</a></p> <p>patches landed on master &amp; b1_8.</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/69/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/69/</a> (2.4.2 RC1)<br/> MDSCOUNT=4</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/a1a2c1d0-6877-11e3-a9a3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a1a2c1d0-6877-11e3-a9a3-52540035b04c</a></p> Development Rank 1|hzvbd3: Rank (Obsolete) 5382 Severity