https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4482 Lustre <p>Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ========= 13:06:48 (1389690408)<br/> LustreError: 14188:0:(ofd_grant.c:255:ofd_grant_space_left()) lustre-OST0010: cli 17a94b5b-8940-b62d-589f-082653aa3e82/ffff88003789e6c8 left 0 &lt; tot_grant 34105472 unstable 0 pending 0<br/> LustreError: 18856:0:(ofd_grant.c:255:ofd_grant_space_left()) lustre-OST0010: cli 17a94b5b-8940-b62d-589f-082653aa3e82/ffff88003789e6c8 left 0 &lt; tot_grant 27814016 unstable 8388608 pending 8388608<br/> LustreError: 18856:0:(ofd_grant.c:255:ofd_grant_space_left()) Skipped 26 previous similar messages<br/> LustreError: 14178:0:(ofd_grant.c:255:ofd_grant_space_left()) lustre-OST0010: cli 17a94b5b-8940-b62d-589f-082653aa3e82/ffff88003789e6c8 left 0 &lt; tot_grant 36505728 unstable 0 pending 0<br/> LustreError: 14178:0:(ofd_grant.c:255:ofd_grant_space_left()) Skipped 33 previous similar messages<br/> LustreError: 25596:0:(ofd_grant.c:255:ofd_grant_space_left()) lustre-OST0010: cli 17a94b5b-8940-b62d-589f-082653aa3e82/ffff88003789e6c8 left 3014656 &lt; tot_grant 37619840 unstable 0 pending 0<br/> LustreError: 25596:0:(ofd_grant.c:255:ofd_grant_space_left()) Skipped 57 previous similar messages<br/> LustreError: 14192:0:(ofd_grant.c:255:ofd_grant_space_left()) lustre-OST0010: cli 17a94b5b-8940-b62d-589f-082653aa3e82/ffff88003789e6c8 left 3063808 &lt; tot_grant 43010176 unstable 0 pending 0<br/> LustreError: 14192:0:(ofd_grant.c:255:ofd_grant_space_left()) Skipped 65 previous similar messages</p> LU-4482

OST grants bugs

Bug Blocker Resolved Fixed Niu Yawei Alexey Lyashkov MB Tue, 14 Jan 2014 09:08:10 +0000 Tue, 25 Feb 2014 01:27:21 +0000 Tue, 25 Feb 2014 01:27:21 +0000 Lustre 2.6.0 0 6 <p>I just checked my logs and I frequently see this in case of OST being full. It's probably been thee for a while since I see this all the way back to when my logs started.</p> <p>We need to get to the root of this as this potentially can lead to unexpected data loss on the client side.</p> <p>Also it seems to started at around May 25 2013 in my test logs as I now see</p> <p>Niu</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Seems it's introduced by "<a href="https://jira.whamcloud.com/browse/LU-1030" title="new IO engine" class="issue-link" data-issue-key="LU-1030"><del>LU-1030</del></a> osc: new IO engine implementation", it looks to me that sync write doesn't consume grant anymore from 9fe4b52ad2ffadf125d9b5c78bb2ff9a01725707, I think we'd add it back.</p> <p>Xiong, could you take a look at this? I think it's an unintentional change, right?</p> <p>It's changed that way on purpose because I think it doesn't need to consume grant if the application can see the errors with a sync write.</p> <p>If it can cause grant issue, then grant algorithm has BUGs because pages without FROM_GRANT flag shouldn't consume reserved space.</p> <blockquote> <p>If it can cause grant issue, then grant algorithm has BUGs because pages without FROM_GRANT flag shouldn't consume reserved space.</p></blockquote> <p>Any kind of write (include sync write or direct io) should consume grant if the client has available grant (and the FROM_GRANT flag should be set on these pages), otherwise, OST could run of of space with client still holding lots of grant.</p> <p>obviously the issue here is not for ENOSPC. the reserved space is less than granted bytes. Did I miss something?</p> <p>If sync write doesn't consume grant, the grant hold by client will not decreased on sync write, however, free space on OST will be decreased, at the end, OST space will be used up by sync writes, however, client still hold grants, and further cached data will be lost.</p> <p>The error message shows that available space is less than total granted bytes (which means client has grant, but OST hasn't enough space for grant) it's because sync write doesn't consume grant but consumes space.</p> <p>Well, there are two problems: </p> <ul class="alternate" type="square"> <li>client should consume grant for sync write (as I mentioned above);</li> <li>osd_statfs() shouldn't cache statfs data because grant mechanism relies on dt_statfs() returning fresh data.</li> </ul> <p><a href="http://review.whamcloud.com/8911" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8911</a></p> <p>patch landed on master for 2.6</p> Related LU-4664 Development Rank 1|hzwcrz: Rank (Obsolete) 12273 Severity