https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8895 Lustre <p>When there are several requests inflight a server may grant a client with more grants than it requests.</p> <p>In real environment it has been reported that clients were granted ~800mb while max_dirty_mb set to 256.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ cat compute_clients_cur_grant_bytes.txt | grep OST019c | awk -F= 'BEGIN \{ grants=0 } \{ if (grants &lt; $2) grants = $2 } END \{ print grants }' 805306368 </pre> </div></div> <p>In total clients were granted ~960gb</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ cat compute_clients_cur_grant_bytes.txt | grep OST019c | awk -F= '{ grants += $2 } END { print grants }' 971171270656 </pre> </div></div> <p>That resulted in ENOSPC condition when osts had about 1tb of free space.</p> LU-8895

server grants clients with more grants that the clients ask

Bug Minor Resolved Fixed Vladimir Saveliev Vladimir Saveliev Fri, 2 Dec 2016 14:23:25 +0000 Mon, 21 Jan 2019 15:31:13 +0000 Thu, 4 Jan 2018 13:27:02 +0000 Lustre 2.11.0 Lustre 2.10.7 0 6 <p>Vladimir Saveliev (vladimir.saveliev@seagate.com) uploaded a new patch: <a href="https://review.whamcloud.com/24096" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/24096</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8895" title="server grants clients with more grants that the clients ask" class="issue-link" data-issue-key="LU-8895"><del>LU-8895</del></a> ofd: do not grant more than asked<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ad1a4cfae54b235c853951ed2ca50588a6fa5b40</p> <p>I don't think it's a problem that 'granted' &gt; 'max_dirty_mb'. In my opinion, 'max_dirty_mb' is used to limit the dirty pages on client, and 'granted' is used to reserve space on server, so that to make sure later dirty flush won't failed for ENOSPC, it'll be fine as long as 'granted' &gt;= 'dirty'. I don't see why that can result in ENOSPC error.</p> <p>When a client is actively consuming grant, it's reasonable to grant more space for this client (to avoid the client running out of grant), probably we need to throttle the granting speed to make sure not granting too much? But that looks not a very serious problem to me, because we do have shrink mechanism to reclaim unused grant.</p> <blockquote> <p>I don't think it's a problem that 'granted' &gt; 'max_dirty_mb'.</p></blockquote> <p>ok, however the below code from ofd_grant_alloc() tries to not grant more than <b>want</b>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (curgrant &gt;= want || curgrant &gt;= fed-&gt;fed_grant + chunk) RETURN(0); </pre> </div></div> <p>Note that <b>want</b> which clients send their rpcs with is equal to max_dirty_mb. See osc_announce_cached():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> nrpages = cli-&gt;cl_max_pages_per_rpc; nrpages *= cli-&gt;cl_max_rpcs_in_flight + 1; nrpages = max(nrpages, cli-&gt;cl_dirty_max_pages); oa-&gt;o_undirty = nrpages &lt;&lt; PAGE_SHIFT; </pre> </div></div> <p> </p> <blockquote> <p>But that looks not a very serious problem to me, because we do have shrink mechanism to reclaim unused grant.</p></blockquote> <p>Yes, as grant shrink is off currently (afaics), this quick patch limiting grants might allow to delay occurrence of virtual enospc condition. If the grant shrink is going to be turned on soon, then this patch is not needed.</p> <p>I was not aware of that grant shrink is disabled by default, but no matter if there is a shrinking mechanism, I do agree that grant shouldn't be far too larger than dirty_max of client. I just don't think it's necessary to restrain it strictly less than dirty_max, exceeding dirty_max little bit looks not a problem to me.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/24096/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/24096/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8895" title="server grants clients with more grants that the clients ask" class="issue-link" data-issue-key="LU-8895"><del>LU-8895</del></a> target: limit grant allocation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 82e494a36e9ea4f51ec163ab15beb9fdda7fa8d6</p> <p>Landed for 2.11</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/32907" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32907</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8895" title="server grants clients with more grants that the clients ask" class="issue-link" data-issue-key="LU-8895"><del>LU-8895</del></a> target: limit grant allocation<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: f078f27f0de19c898e5eda2a45b6c33732a4e4ab</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32907/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32907/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8895" title="server grants clients with more grants that the clients ask" class="issue-link" data-issue-key="LU-8895"><del>LU-8895</del></a> target: limit grant allocation<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: e800ee9409c4fae46e0d52f8d84a432a7f3ff428</p> Duplicate Related LU-8708 Development Rank 1|hzyx93: Rank (Obsolete) 9223372036854775807 Severity