https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12687 Lustre <p>On direct I/O a client gets substantial amount of grants without consuming them so that direct I/O writes face with ENOSPC long before disk space is over.<br/> The below example shows how dd oflag=direct fails to write 400kb to ~70mb ost.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@sl75master tests]# OSTSIZE=100000 sh llmount.sh ... Updated after 6s: wanted 'procname_uid' got 'procname_uid' disable quota as required [root@sl75master tests]# lfs df -h UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 122.4M 1.9M 109.5M 2% /mnt/lustre[MDT:0] lustre-OST0000_UUID 69.4M 1.2M 61.4M 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 69.4M 1.2M 61.4M 2% /mnt/lustre[OST:1] filesystem_summary: 138.9M 2.5M 122.7M 2% /mnt/lustre [root@sl75master tests]# dd if=/dev/zero of=/mnt/lustre/file bs=4k count=100 oflag=direct dd: error writing ‘/mnt/lustre/file’: No space left on device 54+0 records in 53+0 records out 217088 bytes (217 kB) copied, 0.138233 s, 1.6 MB/s [root@sl75master tests]# </pre> </div></div> LU-12687

Fast ENOSPC on direct I/O

Bug Minor Resolved Fixed Vladimir Saveliev Vladimir Saveliev Fri, 23 Aug 2019 10:45:44 +0000 Thu, 7 Apr 2022 15:57:27 +0000 Thu, 24 Sep 2020 13:45:09 +0000 Lustre 2.14.0 Lustre 2.12.6 0 10 <p>Vladimir Saveliev (c17830@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/35896" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35896</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12687" title="Fast ENOSPC on direct I/O" class="issue-link" data-issue-key="LU-12687"><del>LU-12687</del></a> osc: consume grants for direct I/O<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9492bed7688f492c756901c9c357dbacbace4485</p> <p>Vladimir,</p> <p>This is a dupe of <a href="https://jira.whamcloud.com/browse/LU-4664" title="sync write should consume grant on client" class="issue-link" data-issue-key="LU-4664"><del>LU-4664</del></a></p> <p>We have a patch in work there - In fact, I think it's ready except for fixing the test.</p> <p>I think having direct i/o consume grants <b>correctly</b> rather than not using grant at all is our preferred solution, and that's what <a href="https://jira.whamcloud.com/browse/LU-4664" title="sync write should consume grant on client" class="issue-link" data-issue-key="LU-4664"><del>LU-4664</del></a> implements.  Is there a specific reason why direct i/o should not request grant, rather than fixing the consume side as in <a href="https://jira.whamcloud.com/browse/LU-4664" title="sync write should consume grant on client" class="issue-link" data-issue-key="LU-4664"><del>LU-4664</del></a>?</p> <p>Sorry, patch is:<br/> <a href="https://review.whamcloud.com/#/c/9454/26" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/9454/</a></p> <p>I agree with Patrick here that the <a href="https://jira.whamcloud.com/browse/LU-4664" title="sync write should consume grant on client" class="issue-link" data-issue-key="LU-4664"><del>LU-4664</del></a> patch is the right way to go. If we don't consume grants on <tt>O_DIRECT</tt> writes, then it is possible for those writes to run out of space when there is still grant available on the client. </p> <p>Conversely, if <tt>O_DIRECT</tt> consumes all of the grant, but doesn't request more (as with the patch here) then if there are cached writes after this it will not have any grant and will start with small (<tt>PAGE_SIZE</tt>) synchronous writes until it has some grant.</p> <p>It would be great if that patch could be refreshed and landed.</p> <p>I'm hitting this almost every run of sanity.sh in 248b</p> <p>Alex, if you are able to reproduce this easily, could you please verify patch <a href="https://review.whamcloud.com/35896" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35896</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-12687" title="Fast ENOSPC on direct I/O" class="issue-link" data-issue-key="LU-12687"><del>LU-12687</del></a> osc: consume grants for direct I/O</tt>" fixes it for you? That is about to land.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35896/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35896/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12687" title="Fast ENOSPC on direct I/O" class="issue-link" data-issue-key="LU-12687"><del>LU-12687</del></a> osc: consume grants for direct I/O<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 05f326a7988a7a0d6954d1b0d318315526209ae6</p> <p>Landed for 2.14</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39386" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39386</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12687" title="Fast ENOSPC on direct I/O" class="issue-link" data-issue-key="LU-12687"><del>LU-12687</del></a> osc: consume grants for direct I/O<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: c89c8423cf730df97ddc19a8981f978c79fabdfa</p> <p><del>Olaf Faaland-LLNL (faaland1@llnl.gov) uploaded a new patch:</del> <a href="https://review.whamcloud.com/39517" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39517</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-12687" title="Fast ENOSPC on direct I/O" class="issue-link" data-issue-key="LU-12687"><del>LU-12687</del></a> canary: comment-only change</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: bcec79eb40dda0c26bffb9ec950a142034866df6</del></p> <p>Regarding landed patch <a href="https://review.whamcloud.com/35896/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35896/</a> - I've noted that added code does pretty the same as <tt>osc_enter_cache_try()</tt> but skips the following checks:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (cli-&gt;cl_dirty_pages &lt; cli-&gt;cl_dirty_max_pages) { <span class="code-keyword">if</span> (atomic_long_add_return(1, &amp;obd_dirty_pages) &lt;= obd_max_dirty_pages) { ... </pre> </div></div> <p>for <tt>cl_dirty_max_pages</tt> limit. Instead of that we just consuming write grants always. Was that intentional or the same limits check should be added for DIO case too?</p> <p>Vladimir, Andreas, I reopened this due to question I have about already landed patch, please check my previous comment.</p> <p>Hi Mike, I don't <em>think</em> that the max_dirty_pages limit applies in the case of <tt>O_DIRECT</tt> writes, because the client is not caching those pages locally. It is immediately sending the pages for write to the OST, so it doesn't actually <em>need</em> the grant or to check the dirty limit. The grant usage is only to prevent clients from holding grant that cannot be used by the <tt>O_DIRECT</tt> writes. In most cases, the client will get grant back immediately in the reply, unless the filesystem is low on free space.</p> <p>yes, I understand that, meanwhile we don't just consume grants but also increase <tt>cl_dirty_pages</tt> and <tt>obd_dirty_pages</tt> counters during that and do that without regards to limits, so technically these values can become greater than their cap and prevent cache entering in <tt>osc_enter_cache_try()</tt> for a non-DIO writers. So actually the question is - should we increase <tt>cl_dirty_pages</tt> and <tt>obd_dirty_pages</tt> at all? Or better keep them untouched with DIO writes. Probably that is not big deal because as you said - it returns back quite quickly</p> <blockquote> <p>I've noted that added code does pretty the same as osc_enter_cache_try() but skips the following checks:</p></blockquote> <p>osc_enter_cache_try() checks the dirty page counter in order to throttle dirty page generation. In case of DIO we want to submit I/O as soon as possible. So, I guess that is why check for dirty page counter is missing in the patch.</p> <blockquote> <p>So actually the question is - should we increase cl_dirty_pages and obd_dirty_pages at all? Or better keep them untouched with DIO writes. Probably that is not big deal because as you said - it returns back quite quickly</p></blockquote> <p>cl_dirty_pages and obd_dirty_pages are increased for DIO case in order to minimize changes: osc_free_grant() remains unchanged with that.</p> <blockquote> <p>technically these values can become greater than their cap and prevent cache entering in osc_enter_cache_try() for a non-DIO writers.</p></blockquote> <p>I see no problem with that as it decreases write rpc congestion on network and on server.</p> <p>OK, if you see no problem with that then ticket can be closed </p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39386/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39386/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12687" title="Fast ENOSPC on direct I/O" class="issue-link" data-issue-key="LU-12687"><del>LU-12687</del></a> osc: consume grants for direct I/O<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: ab95f0c361a1e0e65ac51fe15ada4cb1e7eeaa1e</p> <p>Seems to be landed</p> Related LU-4664 LU-12757 LU-13766 LU-13847 LU-4198 Development Rank 1|i00lnr: Rank (Obsolete) 9223372036854775807 Severity