https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2513 Lustre <p>static int osc_shrink_grant(struct client_obd *cli)<br/> {<br/> long target = (cli-&gt;cl_max_rpcs_in_flight + 1) *<br/> cli-&gt;cl_max_pages_per_rpc;</p> <p> client_obd_list_lock(&amp;cli-&gt;cl_loi_list_lock);<br/> if (cli-&gt;cl_avail_grant &lt;= target)<br/> target = cli-&gt;cl_max_pages_per_rpc;<br/> client_obd_list_unlock(&amp;cli-&gt;cl_loi_list_lock);</p> <p> return osc_shrink_grant_to_target(cli, target);<br/> }</p> <p>that function looks have a typo - cl_avail_grant in bytes, but target will be in pages so &lt;&lt; CFS_PAGE_SHIFT looks lost.</p> LU-2513

typo in shrink osc grants

Bug Major Resolved Fixed John Hammond Alexey Lyashkov LB Thu, 20 Dec 2012 12:07:28 +0000 Mon, 22 Apr 2013 01:30:29 +0000 Mon, 22 Apr 2013 01:30:29 +0000 Lustre 2.1.0 Lustre 2.2.0 Lustre 2.3.0 Lustre 2.4.0 Lustre 2.4.0 0 3 <p>looks also in osc_shrink_grant_to_target</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> client_obd_list_lock(&amp;cli-&gt;cl_loi_list_lock); /* Don't shrink <span class="code-keyword">if</span> we are already above or below the desired limit * We don't want to shrink below a single RPC, as that will negatively * impact block allocation and <span class="code-object">long</span>-term performance. */ <span class="code-keyword">if</span> (target &lt; cli-&gt;cl_max_pages_per_rpc) target = cli-&gt;cl_max_pages_per_rpc; <span class="code-keyword">if</span> (target &gt;= cli-&gt;cl_avail_grant) { client_obd_list_unlock(&amp;cli-&gt;cl_loi_list_lock); RETURN(0); } client_obd_list_unlock(&amp;cli-&gt;cl_loi_list_lock); OBD_ALLOC_PTR(body); <span class="code-keyword">if</span> (!body) RETURN(-ENOMEM); osc_announce_cached(cli, &amp;body-&gt;oa, 0); client_obd_list_lock(&amp;cli-&gt;cl_loi_list_lock); body-&gt;oa.o_grant = cli-&gt;cl_avail_grant - target; cli-&gt;cl_avail_grant = target; </pre> </div></div> <p>but..</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">osc_cache.c: <span class="code-keyword">if</span> (cli-&gt;cl_avail_grant &gt;= bytes) { osc_cache.c: cli-&gt;cl_avail_grant -= bytes; </pre> </div></div> <p>so we really in bytes..</p> <p>one more bug in same function</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">int</span> osc_shrink_grant_to_target(struct client_obd *cli, <span class="code-object">long</span> target) { <span class="code-object">int</span> rc = 0; struct ost_body *body; ENTRY; client_obd_list_lock(&amp;cli-&gt;cl_loi_list_lock); /* Don't shrink <span class="code-keyword">if</span> we are already above or below the desired limit * We don't want to shrink below a single RPC, as that will negatively * impact block allocation and <span class="code-object">long</span>-term performance. */ <span class="code-keyword">if</span> (target &lt; cli-&gt;cl_max_pages_per_rpc) target = cli-&gt;cl_max_pages_per_rpc; <span class="code-keyword">if</span> (target &gt;= cli-&gt;cl_avail_grant) { client_obd_list_unlock(&amp;cli-&gt;cl_loi_list_lock); RETURN(0); } client_obd_list_unlock(&amp;cli-&gt;cl_loi_list_lock); OBD_ALLOC_PTR(body); <span class="code-keyword">if</span> (!body) RETURN(-ENOMEM); osc_announce_cached(cli, &amp;body-&gt;oa, 0); client_obd_list_lock(&amp;cli-&gt;cl_loi_list_lock); body-&gt;oa.o_grant = cli-&gt;cl_avail_grant - target; </pre> </div></div> <p>so we have a release a lock and may sleep for any time and may have negative result of cl_avail_grant - target after it.</p> <p>second question - is this operation 'atomic' over client + ost.</p> <p>few more bugs with grants.<br/> 1) wrong rounding to block size (fixed in 2.4).<br/> 2) <br/> [ 933.237133] LustreError: 29642:0:(ofd_grant.c:250:ofd_grant_space_left()) lustre-OST0000: cli lustre-OST0000_UUID/ffff8800a72809a0 left 62275584 &lt; tot_grant 65467236 unstable 0 pending 0<br/> Dec 28 07:45:39 rhel6-64 kernel: [ 933.237133] LustreError: 29642:0:(ofd_grant.c:250:ofd_grant_space_left()) lustre-OST0000: cli lustre-OST0000_UUID/ffff8800a72809a0 left 62275584 &lt; tot_grant 65467236 unstable 0 pending 0</p> <p>with simple test</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[root@rhel6-64 tests]# OSTCOUNT=1 DEBUG_SIZE=400 SUBSYSTEM=<span class="code-quote">"ost osc filter"</span> PTLDEBUG=-1 sh llmount.sh <span class="code-keyword">for</span> i in {1..25000}; <span class="code-keyword">do</span> dd <span class="code-keyword">if</span>=/dev/zero of=lustre/$i bs=4096 count=2 <span class="code-keyword">if</span> [ $? -ne 0 ]; then <span class="code-keyword">break</span>; fi; done lctl dk &gt; /tmp/grant.log </pre> </div></div> <p>3) send up to max rpc in flight grant requests in parallel /reproduced with 2.1 but exist in 2.4/</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">82297.783585:0:5381:0:(filter_io.c:249:filter_grant()) lustre-OST0000: cli 846bbe16-0fc2-273c-ff5f-ff1e24116f3f/ffff8800908e1438 wants: 33554432 current grant 33603584 granting: 0 82297.783601:0:5373:0:(filter_io.c:249:filter_grant()) lustre-OST0000: cli 846bbe16-0fc2-273c-ff5f-ff1e24116f3f/ffff8800908e1438 wants: 33554432 current grant 33603584 granting: 0 82297.802358:0:5373:0:(filter_io.c:249:filter_grant()) lustre-OST0000: cli 846bbe16-0fc2-273c-ff5f-ff1e24116f3f/ffff8800908e1438 wants: 33554432 current grant 33546240 granting: 1048576 82297.802404:0:5381:0:(filter_io.c:249:filter_grant()) lustre-OST0000: cli 846bbe16-0fc2-273c-ff5f-ff1e24116f3f/ffff8800908e1438 wants: 33554432 current grant 33546240 granting: 1048576 82297.802757:0:5373:0:(filter_io.c:249:filter_grant()) lustre-OST0000: cli 846bbe16-0fc2-273c-ff5f-ff1e24116f3f/ffff8800908e1438 wants: 33554432 current grant 33546240 granting: 1048576 82297.829384:0:5381:0:(filter_io.c:249:filter_grant()) lustre-OST0000: cli 846bbe16-0fc2-273c-ff5f-ff1e24116f3f/ffff8800908e1438 wants: 33554432 current grant 41869312 granting: 0 </pre> </div></div> <p>with that example - client got a 2 extra grant updates.</p> <p>but we really don't need a have grants more then dirty cache size, because we have stick in waiting dirty space.<br/> limiting a grant request don't need to be to max rpc in flight, because that is more related to dirty cache when network transfer.</p> <p>Xyratex-Bug-Id: MRP-809.</p> <p>Peter,</p> <p>I have fix for it issue, and have plan to submit in short time.<br/> Fix touched client side only.</p> <p>2.1 need additional chunk </p> <p>diff --git a/lustre/obdfilter/filter_io.c b/lustre/obdfilter/filter_io.c<br/> index bb2b8d1..5096950 100644<br/> &#8212; a/lustre/obdfilter/filter_io.c<br/> +++ b/lustre/obdfilter/filter_io.c<br/> @@ -603,7 +603,7 @@ static int filter_grant_check(struct obd_export *exp, struct obdo *oa,<br/> ungranted, fed-&gt;fed_grant, fed-&gt;fed_dirty);</p> <p> /* Rough calc in case we don't refresh cached statfs data */</p> <ul class="alternate" type="square"> <li>using = (used + ungranted + 1 ) &gt;&gt;<br/> + using = (used + ungranted + blocksize - 1) &gt;&gt;<br/> exp-&gt;exp_obd-&gt;u.obt.obt_sb-&gt;s_blocksize_bits;<br/> if (exp-&gt;exp_obd-&gt;obd_osfs.os_bavail &gt; using)<br/> exp-&gt;exp_obd-&gt;obd_osfs.os_bavail -= using;</li> </ul> <p>remote: New Changes:<br/> remote: <a href="http://review.whamcloud.com/5017" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5017</a></p> <p>if 2.1 fix need - i may submit it also.</p> <p>John, can you please submit a patch that only fixes the bytes-vs-pages issue independent of the "number of rpcs in flight" change that is blocking the current patch from landing.</p> <p>Please see <a href="http://review.whamcloud.com/5495" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5495</a>.</p> <p>Patch landed.</p> <p>Andreas, my last version have dropped an limiting a number of rpc from a patch, and i don't understand a situation why it's don't inspected and new patch created?<br/> It's hard to inspect a existent patch instead of create own?</p> <p>I really don't understand...</p> <p>Shadow, you submitted the original patch on Jan 11 and I inspected it on Jan 17, then Jinshan inspected it again on feb 11. I wanted to make sure that at least the basic fix was landed, so a month after not hearing from you (Feb 20) I asked John to submit the basic fix without changing the whole grant protocol. In cases like this it is easier to separate the basic fix (incompatible units for grant variables) from a much more major change that is controversial. You didn't refresh the original the patch until March 20. </p> <p>I agree that John should have resubmitted his patch using the same Change-Id as your original patch, since this makes it a lot easier to reinspect and compare the two patches. That would also have made it more clear to you that there was a new version of your fix . Sorry for the confusion. </p> Development Rank 1|hzve33: Rank (Obsolete) 5913 Severity