https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7510 Lustre <p>We have some production apps and rsync processes failing writes with ENOSPC errors on the ZFS backed FS only. It is currently at ~79%. There are no server side errors, -28 errors as above appeare in the client logs.</p> <p>I see that <a href="https://jira.whamcloud.com/browse/LU-3522" title="sanity-benchmark test_iozone: &quot;no space left on device&quot; on ZFS" class="issue-link" data-issue-key="LU-3522"><del>LU-3522</del></a> and <a href="https://jira.whamcloud.com/browse/LU-2049" title="add support for OBD_CONNECT_GRANT_PARAM" class="issue-link" data-issue-key="LU-2049"><del>LU-2049</del></a> may have a bearing on this issue, is there a 2.5 backport or equivalent fix available? </p> Servers and clients: 2.5.4-11chaos-11chaos--PRISTINE-2.6.32-573.7.1.1chaos.ch5.4.x86_64<br/> ZFS back end LU-7510

(vvp_io.c:1088:vvp_io_commit_write()) Write page 962977 of inode ffff880fbea44b78 failed -28

Bug Major Resolved Done Nathaniel Clark Ruth Klundt llnl Tue, 1 Dec 2015 21:13:15 +0000 Wed, 8 Jun 2016 22:23:43 +0000 Wed, 8 Jun 2016 22:23:43 +0000 Lustre 2.5.3 0 7 <p>Ruth, could you please post the output of "<tt>lfs df</tt>" and "<tt>lfs df -i</tt>" on your filesystem(s). On the OSS nodes, could you please collect "<tt>lctl get_param obdfilter.*.tot_granted</tt>" to see if this is the actual cause of the ENOSPC errors. Also, how many clients are connected to the filesystem?</p> <p>One potential workaround is to release some of the grant from the clients using "<tt>lctl set_param osc.&#42;.cur_grant_bytes=2M</tt>" and then check <tt>lctl get_param obdfilter.&#42;.tot_granted</tt>" on the OSS nodes again to see if the total grant space has been reduced.</p> <p>Have you modified the client's maximum RPC size (via <tt>lctl set_param osc.&#42;.max_pages_per_rpc=4M</tt>, e.g. to have 4MB RPC size), or the ZFS maximum blocksize (via <tt>zfs set recordsize=1048576 lustre/lustre-OST0000</tt> or similar)? That will aggravate this problem until the patch from <a href="https://jira.whamcloud.com/browse/LU-3522" title="sanity-benchmark test_iozone: &quot;no space left on device&quot; on ZFS" class="issue-link" data-issue-key="LU-3522"><del>LU-3522</del></a> is landed.</p> <p>The max_pages_per_rpc value is default of 256, and the zfs recordsize is 128K. We have 3 OSTs on each OSS rather than just one. We have ~6500 clients mounting the file system.</p> <p>We requested that some of the heavy users clean up, so the FS is at 75% now. Also moved a couple of affected users to the other (ldiskfs) file system. </p> <p>No messages so far today. I will go ahead and release some grant if you think it's still necessary or beneficial. </p> <p>I was guessing a combination of the bug + heavy user activity + high fs usage may have triggered this. Our FS usage tends to run high around here.</p> <p>Ruth, the <tt>cur_grant_bytes</tt> command to release grants is something that you can try as a workaround if the <tt>-ENOSPC</tt> errors are being hit again. It doesn't hurt to run this now, or occasionally, though it may cause a very brief hiccup in IO performance as the grant is released. The main reason this isn't useful to do (much) in advance of the problem is that this command asks clients try to return their grant to the server, but if the server isn't low on space it will just return the grant back to the client.</p> <p>The real fix for this problem is indeed the <a href="https://jira.whamcloud.com/browse/LU-2049" title="add support for OBD_CONNECT_GRANT_PARAM" class="issue-link" data-issue-key="LU-2049"><del>LU-2049</del></a> patch. The reason you see this problem when LLNL does not is that you have many more clients connected directly to the filesystem (6500 vs 768) and their OSTs are 72TB vs 30TB so they wouldn't hit this until they reach 99% full.</p> <p>We are working to get the <a href="https://jira.whamcloud.com/browse/LU-2049" title="add support for OBD_CONNECT_GRANT_PARAM" class="issue-link" data-issue-key="LU-2049"><del>LU-2049</del></a> patch landed to resolve this issue permanently. </p> <p>thanks, much appreciated. I'll keep an eye on those messages, they have not resumed since the usage went down. </p> <p>We are resolving this as a duplicate.</p> <p>Ruth &#8211; if the problem recurs and you need more help please either ask us to reopen this ticket, or open a new one, as you prefer.</p> <p>Thanks,<br/> ~ jfc.</p> <p>John,<br/> FYI...We've been hitting this at LLNL the last week or so. I'll note it on <a href="https://jira.whamcloud.com/browse/LU-2049" title="add support for OBD_CONNECT_GRANT_PARAM" class="issue-link" data-issue-key="LU-2049"><del>LU-2049</del></a> as well.</p> <p>We are running the workaround to release grant on the clients from the epilog (ie after each job), just to proactively keep that under control. We have not seen enospc errs in logs again, and usage has hit the 80% mark several times since. </p> <p>Ruth,<br/> Thanks for letting us know you went down the grant release route. I had noticed that 32 of our 80 OSTs were ~90% full (the others ~ 65%), so I deactivated those 32 fuller OSTs and that seems to have resolved the problem for now.</p> <p>Yesterday we had a server go down with this LBUG:</p> <p>LustreError: 8117:0:(ofd_grant.c:352:ofd_grant_incoming()) fscratch-OST001d: cli 8c1795e2-8806-4e65-5865-4e42489eac9b/ffff8807dee68400 dirty 33554432 pend 0 grant -54657024<br/> LustreError: 8117:0:(ofd_grant.c:354:ofd_grant_incoming()) LBUG</p> <p>The client side grant release doesn't seem to be taking effect on that node. The values of tot_granted on the server side were increased to ~5x10^12 on that node only, all the other nodes had values ~2x10^12.</p> <p>This node has gone down several times in the last few weeks, this is the first time we got any log messages before it died. It seems we should also deactivate those OSTs, and increase the priority of upgrading the servers - although it appears a 2.5 upgrade path is not yet available (looking at <a href="https://jira.whamcloud.com/browse/LU-2049" title="add support for OBD_CONNECT_GRANT_PARAM" class="issue-link" data-issue-key="LU-2049"><del>LU-2049</del></a>). </p> <p>But I'm puzzled at why the grant release doesn't work for just that node. All the others have not been re-booted during this time period since we started the workaround.</p> <p>It turns out that the grant release does work, even on the problem node, once the grant on the problem server reaches ~4.9T. It decreased to ~4.0T over the course of a day before the lbug on 2 different targets. The other servers respond to grant release at levels of as low as 1.4T. The usage levels are similar, all osts between 75-80%. The only difference I can find is that the other zpools last item in history is activation of compression back in march. So this one server was rebooted after that compression activation, all the rest were not. Wondering if the size computation is affected by whether compression is on or off? All zpools are reporting 1.03-1.05 ratios.</p> <p>FYI: I don't have an exact version for 2.5.4-11chaos (12chaos and 4chaos are tagged in our system, so I have a good idea).</p> <p>Do you have any logging leading up to the LBUG, by any chance?</p> <p>There is nothing prior to the lbug. Here are the traces. </p> <p>The ofd code at least does not differ between 11chaos and 12chaos versions afaics.</p> <p>after deactivating the osts on that node, the rate of increase is slower, but it still is much larger than all the others and not decreasing so far at about ~3.7T.</p> <p>Each of the osts have shown a couple of decreases, in the 3.8-3.9 T range. </p> <p>Nearly all OSS nodes on this file system became inaccessible yesterday, 3 of them showed the LBUG at ofd_grant.c:352:ofd_grant_incoming with negative grant values. I disabled the automated grant release workaround in case it is related to this occurence. The OSTs are 77-79% full at the moment. After that another OSS went down with the same LBUG. </p> <p>This coincides with the addition of a new cluster, but we haven't done any I/O from it so far, just mounting. Any advice/thoughts? </p> <p>And a specific question, is the LBUG likely addressed by changes upstream or should this be a separate ticket? </p> <p>The LBUG in question hasn't been changed, though the grant code has been reworked (a la <a href="https://jira.whamcloud.com/browse/LU-2049" title="add support for OBD_CONNECT_GRANT_PARAM" class="issue-link" data-issue-key="LU-2049"><del>LU-2049</del></a>) upstream. The negative grant resulting in LBUG should be separate bug, though it's probably 2.5 only.</p> <p>The file system usage has been reduced to ~70%, and we haven't seen -28 issues or LBUGs since then. </p> <p>You can close this one, we'll consider the fix for -28 issues to be upgrade to 2.8 lustre on the servers at some point in the future.</p> <p>If the LBUG re-occurs I'll open a new ticket. </p> <p>Thanks,<br/> Ruth</p> <p>Thanks Ruth.</p> <p>~ jfc.</p> Related LU-8007 LU-2049 Development End date Thu, 26 May 2016 21:13:15 +0000 Rank 1|hzxuon: Rank (Obsolete) 9223372036854775807 Severity Start date Tue, 1 Dec 2015 21:13:15 +0000