https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3633 Lustre <p>sanity.sh test_101d 'dd failed' for not enough space.</p> <p><a href="https://maloo.whamcloud.com/sub_tests/b1487964-f4b3-11e2-b8a2-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/b1487964-f4b3-11e2-b8a2-52540035b04c</a></p> <p>console log from client:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">08:52:27:Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled =================== 08:52:25 (1374681145) 08:52:38:LustreError: 24293:0:(vvp_io.c:1094:vvp_io_commit_write()) Write page 46696 of inode ffff880062769b78 failed -28 08:52:38:LustreError: 24293:0:(vvp_io.c:1094:vvp_io_commit_write()) Write page 46696 of inode ffff880062769b78 failed -28 08:52:39:Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity test_101d: @@@@@@ FAIL: dd failed 08:52:39:Lustre: DEBUG MARKER: sanity test_101d: @@@@@@ FAIL: dd failed 08:52:39:Lustre: DEBUG MARKER: /usr/sbin/lctl dk &gt; /logdir/test_logs/2013-07-24/lustre-reviews-el6-x86_64--review--1_2_1__16787__-69987943176880-074027/sanity.test_101d.debug_log.$(hostname -s).1374681155.log; 08:52:39: dmesg &gt; /logdir/test_logs/2013-07-24/lustre-reviews-el6-x86_64--rev 08:52:50:Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity test 101e: check read-ahead <span class="code-keyword">for</span> small read\(1k\) <span class="code-keyword">for</span> small files\(500k\) == 08:52:40 \(1374681160\) </pre> </div></div> LU-3633

sanity.sh test_101d failed for 'dd failed'

Bug Blocker Resolved Fixed James Nunez Niu Yawei Thu, 25 Jul 2013 04:02:35 +0000 Fri, 13 Sep 2013 03:55:57 +0000 Wed, 7 Aug 2013 20:06:26 +0000 Lustre 2.5.0 Lustre 2.4.1 Lustre 2.5.0 0 5 <p>Just for clarity -28 is:</p> <p>#define ENOSPC 28 /* No space left on device */</p> <p>From the OST console:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>08:52:26:Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled =================== 08:52:25 (1374681145) 08:52:38:LustreError: 16787:0:(ofd_grant.c:255:ofd_grant_space_left()) lustre-OST0001: cli 795f3143-5aac-1e12-ab67-46f017cf8245/ffff88007bd67000 left 44511232 &lt; tot_grant 59851008 unstable 5242880 pending 5242880 08:52:38:LustreError: 842:0:(ofd_grant.c:255:ofd_grant_space_left()) lustre-OST0001: cli 795f3143-5aac-1e12-ab67-46f017cf8245/ffff88007bd67000 left 41365504 &lt; tot_grant 56705280 unstable 5242880 pending 3145728 08:52:39:LustreError: 16787:0:(ofd_grant.c:255:ofd_grant_space_left()) lustre-OST0001: cli 795f3143-5aac-1e12-ab67-46f017cf8245/ffff88007bd67000 left 20393984 &lt; tot_grant 37220608 unstable 4194304 pending 4194304 08:52:39:LustreError: 16787:0:(ofd_grant.c:255:ofd_grant_space_left()) Skipped 8 previous similar messages 08:52:39:LustreError: 840:0:(ofd_grant.c:255:ofd_grant_space_left()) lustre-OST0001: cli 795f3143-5aac-1e12-ab67-46f017cf8245/ffff88007bd67000 left 0 &lt; tot_grant 16392448 unstable 4194304 pending 4194304 08:52:39:LustreError: 840:0:(ofd_grant.c:255:ofd_grant_space_left()) Skipped 13 previous similar messages 08:52:39:Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity test_101d: @@@@@@ FAIL: dd failed </pre> </div></div> <p>There is a good set of messages in the debug log for the ost as well (just filter for grant messages for ost1). Andreas mentioned on skype this seems like a grant accounting issue. </p> <p>I didn't get a chance to fully parse that log yet but it looks to have very good info in it. </p> <p>Seems like the test fs is running out of space, I'm not sure what consumed so much space, but I think the test script is simply wrong, the test 101d checks if the whole fs avail space is greater than 'size' first, then it tries to write 'size' to a file, however, the file isn't necessary full stripe file, so it's quite possible that the write failed for out of space.</p> <p>Nui,<br/> The grant messages don't seem to line up for me:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> left 44511232 &lt; tot_grant 59851008 unstable 5242880 pending 5242880 left 41365504 &lt; tot_grant 56705280 unstable 5242880 pending 3145728 left 20393984 &lt; tot_grant 37220608 unstable 4194304 pending 4194304 left 0 &lt; tot_grant 16392448 unstable 4194304 pending 4194304 </pre> </div></div> <p>I don't know this code well but it I would sort of assume that left + tot_grant == Total resource in the pool.</p> <p>We see the total grant drastically shrink and the "left" go to 0 ie nothing close to a static resource pool. Where did all the resource go? Maybe I am just looking at these error messages wrong. </p> <p>Proposed patch at</p> <p><a href="http://review.whamcloud.com/#/c/7179" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7179</a></p> <p>This patch only fixes the issue that Niu brought up about test 101d. It does not address any issues that may exist with grants.</p> <p>James your patch looks like a good test change to me. I am likely just reading the error messages wrong. </p> <blockquote> <p>I don't know this code well but it I would sort of assume that left + tot_grant == Total resource in the pool.</p></blockquote> <p>I think left + tot_grant == Total unused space.</p> <blockquote> <p>We see the total grant drastically shrink and the "left" go to 0 ie nothing close to a static resource pool. Where did all the resource go? Maybe I am just looking at these error messages wrong.</p></blockquote> <p>I think it's possible, the space allocated for file isn't included in the 'left + tot_granted'.</p> <p>I see I am just reading the error messages wrong then. Thanks for the information. </p> <p>Both <a href="https://jira.whamcloud.com/browse/LU-3640" title="Test failure on test suite sanity, test_116a, when OST is full" class="issue-link" data-issue-key="LU-3640"><del>LU-3640</del></a> and this ticket are a result of a single OST filling.</p> <p>This ticket is stopping the landing of the last 2 HSM tickets, and thereby blocking the feature freeze. Escalating to a blocker.</p> <p>Landed for 2.5</p> <p>Hit this issue when running interop test between 2.4.0 client and 2.5 server:</p> <p><a href="https://maloo.whamcloud.com/test_sets/bfe9f440-09ec-11e3-ad93-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/bfe9f440-09ec-11e3-ad93-52540035b04c</a></p> Related LU-3640 LU-3720 Development Rank 1|hzvw4f: Rank (Obsolete) 9355 Severity