https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7015 Lustre <p>Our system space utilization on one of our systems is high, and as we work to prune some of this data, we're exploring some other space tunings.</p> <p>One of our admins noted the "cur_grant_bytes" osc parameter. When we looked at a few clients, we saw that this variable often exceeds the max_dirty_mb, sometimes by an order of magnitude. We usually use 64MB of dirty cache per osc per client. Is there an upper limit to this cur_grants_bytes parameter? What are the side effects of setting this value to some lower value (or 0)? Can we reduce this client grant while there is active I/O, and can we do this for all osc connections simultaneously (for a collective of millions of osc connections) for a system? Is this documented well anywhere? </p> <p>Additionally, we are looking into tuning the reserved_blocks_percent parameter. The Lustre manual states that 5% is the minimum, but is that a sane value for all OST sizes?</p> <p>Thanks,<br/> &#8211;<br/> Jesse</p> RHEL-6.6, lustre-2.5.4 LU-7015

Grant space and reserved blocks percent parameters

Question/Request Minor Resolved Fixed Oleg Drokin Jesse Hanley Mon, 17 Aug 2015 20:30:06 +0000 Wed, 14 Mar 2018 12:28:21 +0000 Thu, 10 Sep 2015 18:20:55 +0000 Lustre 2.5.4 Lustre 2.8.0 0 13 <p>To include some numbers:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># ls exports | wc -l 20190 # echo "$(cat tot_granted) / $(ls exports | wc -l)" | bc 116961044 # echo "100 * $(cat tot_granted) / 1024 / $(cat kbytestotal)" | bc 15 </pre> </div></div> <p>Our 20,000 clients average 116MB of grants per OST, resulting in 15% of the OST reserved for grants. That means when any OST hits 85% full, users start getting ENOSPC. I picked a random client and grant sizes range from 2MB to 343MB per OSC.</p> <p>cur_grant_bytes is how much grant was received from this OST by a client. It has no direct relation to max_dirty_mb other than max_dirty_mb cannot be higher than this.</p> <p>Technically the calculation for the grant request is</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> long max_in_flight = (cli-&gt;cl_max_pages_per_rpc &lt;&lt; PAGE_CACHE_SHIFT) * (cli-&gt;cl_max_rpcs_in_flight + 1); oa-&gt;o_undirty = max(cli-&gt;cl_dirty_max_pages &lt;&lt; PAGE_CACHE_SHIFT, max_in_flight); </pre> </div></div> <p>This is how much every client RPC requests.<br/> It's a max of either your num RPCs in flight or max_dirty_mb</p> <p>Theoretically we should not exceed this value (want = o_undirty):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /* Grant some fraction of the client's requested grant space so that * they are not always waiting for write credits (not all of it to * avoid overgranting in face of multiple RPCs in flight). This * essentially will be able to control the OSC_MAX_RIF for a client. * * If we do have a large disparity between what the client thinks it * has and what we think it has, don't grant very much and let the * client consume its grant first. Either it just has lots of RPCs * in flight, or it was evicted and its grants will soon be used up. */ if (curgrant &gt;= want || curgrant &gt;= fed-&gt;fed_grant + grant_chunk) RETURN(0); ... grant = min(want, left); /* round grant upt to the next block size */ grant = (grant + (1 &lt;&lt; ofd-&gt;ofd_blockbits) - 1) &amp; ~((1ULL &lt;&lt; ofd-&gt;ofd_blockbits) - 1); /* Limit to ofd_grant_chunk() if not reconnect/recovery */ if ((grant &gt; grant_chunk) &amp;&amp; conservative) grant = grant_chunk; ... ofd-&gt;ofd_tot_granted += grant; fed-&gt;fed_grant += grant; </pre> </div></div> <p>So I imagine biggest case could be that if a client sends a bunch of requests while the grant is nearly at the max, then every one of those RPCs would return 2M of grant each,<br/> which I guess theoretically should only allow to get a client to receive at most 2x of the max_dirty_mb or max_requests_in_flight megabytes (though if you are in recovery then every request could bring as much grant).<br/> Overall it seems there's seem to be a bit of a logic flaw in server-side granting logic where after initial checks want should be recalculated as want -= grant or something like that.</p> <p>While you can write a low value into the proc file, it would only have an effect of releasing the extra grant above the value you write there immediately, but the value does not stick and the grant would keep accumulating according to the calculations above.</p> <p>As for the reserved_blocks_percent, do you mean the ext4 reservation? I think recent e2fspogs reduced that to smaller value for large filesystem sizes by default already.</p> <p>Thanks Oleg for the detail!</p> <p>These servers were originally formatted with Lustre 2.4. When I checked, it looks like the OSTs are at 5% reserved:</p> <p>Block count: 3755999232<br/> Reserved block count: 187799961</p> <p>187799961 / 3755999232 * 100 ~= 5%</p> <p>With that being the case, can/should we lower this to a smaller reserved block count?</p> <p>Also, do I need to submit a new case about the server logic?</p> <p>Thanks!<br/> &#8211;<br/> Jesse</p> <p>I guess until we get usage down or a patch for this, we will need to periodically shrink grants on clients to avoid ENOSPC.</p> <p>The source of the question about reserved space was to better understand when a user might get ENOSPC. Would it be when a client has exhausted its grant and (kbytesfree - (tot_granted/1024)) &lt;= 0 or does it use (kbytesavail - (tot_granted/1024)) &lt;= 0 ?</p> <p>The Lustre Operations manual has a pretty strong warning about lowering the reserved space:</p> <blockquote><p>Reducing the space reservation can cause severe performance degradation as the OST file system becomes more than 95% full, due to difficulty in locating large areas of contiguous free space. This performance degradation may persist even if the space usage drops below 95% again. It is recommended NOT to reduce the reserved disk space below 5%.</p></blockquote> <p>But if that will give us a little headroom, we know grants will help keep us from getting <b>too</b> close to completely empty.</p> <p>It looks like ofd_grant_space_left() uses ofd-&gt;ofd_osfs.os_bavail, so it appears to take the reserved space into account.</p> <p>I just ran a quick test on our TDS system. I took a newly mounted client and created 50 files striped across OST 0. I backgrounded 50 <em>dd</em> processes against those files and gathered logs with +cache enabled on the client and server.</p> <p>The first thing I noticed it that the server very quickly increased the grant to the client, maybe even before the client had a chance to realize it.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00002000:00000020:4.0:1439910325.382841:0:36645:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 0 granting: 8388608 00002000:00000020:4.0:1439910325.383027:0:31053:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 0 granting: 8388608 00002000:00000020:4.0:1439910325.383615:0:36646:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 0 granting: 8388608 00002000:00000020:4.0:1439910325.383775:0:36647:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 0 granting: 8388608 00002000:00000020:4.0:1439910325.384272:0:36648:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 0 granting: 8388608 00002000:00000020:4.0:1439910325.385007:0:36649:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 0 granting: 8388608 00002000:00000020:4.0:1439910325.385154:0:36650:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 0 granting: 8388608 00002000:00000020:6.0:1439910325.416668:0:36648:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 335872 granting: 8388608 00002000:00000020:5.0:1439910325.417207:0:36649:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 0 granting: 8388608 00002000:00000020:6.0:1439910325.417262:0:36645:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 0 granting: 8388608 00002000:00000020:6.0:1439910325.433766:0:31053:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 29917184 granting: 8388608 00002000:00000020:4.0:1439910325.433773:0:36646:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 8187904 granting: 8388608 00002000:00000020:5.0:1439910325.433789:0:31052:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 22822912 granting: 8388608 00002000:00000020:6.0:1439910325.434528:0:31054:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 25923584 granting: 8388608 00002000:00000020:4.0:1439910325.434534:0:36647:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 25845760 granting: 8388608 00002000:00000020:5.0:1439910325.591676:0:36650:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 32403456 granting: 8388608 00002000:00000020:4.0:1439910325.591852:0:36652:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 32382976 granting: 8388608 00002000:00000020:5.0:1439910325.591860:0:36647:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 32608256 granting: 8388608 00002000:00000020:6.0:1439910325.593790:0:31054:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 30371840 granting: 8388608 00002000:00000020:5.0:1439910325.595378:0:36651:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 29700096 granting: 8388608 00002000:00000020:4.0:1439910325.595384:0:31052:0:(ofd_grant.c:662:ofd_grant()) atlastds-OST0000: cli ce2fb5d0-e502-410d-675d-3b8d0dd26305/ffff880806fe3c00 wants: 33554432 current grant 29696000 granting: 8388608 </pre> </div></div> <p>The server granted it 56MB before the client even reported having a grant.</p> <p>I haven't read all of the grant-related code, so take this analysis with a grain of salt.</p> <p>Is the <em>want</em> parameter supposed to be an absolute or relative value?</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>lustre/ofd/ofd_grant.c:ofd_grant()</b></div><div class="codeContent panelContent"> <pre class="code-java"> /* Grant some fraction of the client's requested grant space so that * they are not always waiting <span class="code-keyword">for</span> write credits (not all of it to * avoid overgranting in face of multiple RPCs in flight). This * essentially will be able to control the OSC_MAX_RIF <span class="code-keyword">for</span> a client. * * If we <span class="code-keyword">do</span> have a large disparity between what the client thinks it * has and what we think it has, don't grant very much and let the * client consume its grant first. Either it just has lots of RPCs * in flight, or it was evicted and its grants will soon be used up. */ <span class="code-keyword">if</span> (curgrant &gt;= want || curgrant &gt;= fed-&gt;fed_grant + grant_chunk) RETURN(0); </pre> </div></div> <p>This looks like <em>want</em> is being used as an absolute value. Assuming <em>want</em> should be absolute, do we also need a check to ensure that <em>fed-&gt;fed_grant</em> isn't much larger than <em>want</em>?</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>lustre/ofd/ofd_grant.c:ofd_grant()</b></div><div class="codeContent panelContent"> <pre class="code-java">grant = min(want, left); ... <span class="code-comment">/* Limit to ofd_grant_chunk() <span class="code-keyword">if</span> not reconnect/recovery */</span> <span class="code-keyword">if</span> ((grant &gt; grant_chunk) &amp;&amp; conservative) grant = grant_chunk; ... ofd-&gt;ofd_tot_granted += grant; fed-&gt;fed_grant += grant; </pre> </div></div> <p>This looks like <em>want</em> is a relative value.</p> <p>So the clients repeatedly says "I want 32MB" and the server takes that request, lowers it to <em>grant_chunk</em> (8MB), and grants it 8MB repeatedly until the client claims it has at least 32MB.</p> <p>According to Andreas in <a href="https://jira.whamcloud.com/browse/LU-3859" title="grant shrinker floods OST and produce a large load" class="issue-link" data-issue-key="LU-3859"><del>LU-3859</del></a>, OBD_CONNECT_GRANT_SHRINK isn't set, so this is never cleaned up automatically. Is there a reason this is disabled?</p> <p>The upper limit for wanted grant is typically at least 2x max_dirty_mb, so by having max_dirty_mb=64M doubles the potential amount of grant per client. With 22,000 clients that will result in a large amount of grant, as you have seen. Presumably the change to max_dirty_mb=64M was done to improve single-client write performance and/or increase writeback caching on the client without blocking the IO?</p> <p>Unfortunately, without the grant shrinker active, writing to cur_grant_bytes will not permanently affect the amount of grant held by the client unless the filesystem is nearly out of space. Otherwise, the client will try to surrender the grant but the server will reply that there is still grant available and return the same amount back. Only when the available space begins to get constrained will the OST not return the grant, but this can be done on clients as an emergency measure when available space is running short, something like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>pdsh -a &lt;clients&gt; lctl set_param osc.*.cur_grant_bytes=2M </pre> </div></div> <p>(or even 1MB if necessary) and then clients which do not need any grant will not get any more.</p> <p>The grant shrinking code had problems when it was first introduced (before 2.0 was released) and has never been fixed since then.</p> <p>Oleg Drokin (oleg.drokin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16216" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16216</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7015" title="Grant space and reserved blocks percent parameters" class="issue-link" data-issue-key="LU-7015"><del>LU-7015</del></a> ofd: Fix wanted grant calculation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 22c2ad105d9d420058f653f03030ce2e4a3f017b</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16216/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16216/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7015" title="Grant space and reserved blocks percent parameters" class="issue-link" data-issue-key="LU-7015"><del>LU-7015</del></a> ofd: Fix wanted grant calculation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 091988499717c729f8870b331ab3774b249d5818</p> <p>Landed for 2.8.0</p> Duplicate Related LU-3859 Development Rank 1|hzxknr: Rank (Obsolete) 9223372036854775807