https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11798 Lustre <p>The Lustre 2.10.5 / zfs 0.7.11 OST is not allocating grant to the clients.</p> <p>I set +cache debug flag on both client and server.</p> <p>On the client, under /proc/fs/lustre/osc/lustre3-OST....:<br/> cur_dirty_bytes == 0<br/> cur_grant_bytes starts out as 2097152</p> <p>I then issue<br/> dd if=/dev/urandom of=/p/lustre3/foo/toss-4377/ost0/test.10M bs=5K count=2000</p> <p>cur_grant_bytes goes down, and never up, as writes to the OST occur. Eventually it reaches 0 and then stays == 0.</p> <p>There is no connection issue - no log messages indicating connections lost/reconnects, and osc/lustre3-OSTXXXX.*/state has "current_state: FULL" for all OSTs.</p> client: <br/> lustre-2.8.2_5.chaos-1.ch6.x86_64 <br/> <br/> servers: <br/> zfs-0.7.11-4llnl.ch6.x86_64 <br/> lustre-2.10.5_2.chaos-3.ch6.x86_64 LU-11798

cur_grant goes to 0 and never increases with 2.8 client and 2.10 server

Bug Major Resolved Fixed Alex Zhuravlev Olaf Faaland llnl Mon, 17 Dec 2018 21:43:27 +0000 Sat, 13 Apr 2019 16:50:22 +0000 Mon, 1 Apr 2019 12:31:45 +0000 Lustre 2.8.0 Lustre 2.10.5 Lustre 2.13.0 Lustre 2.12.1 0 11 <p>debug logs records on the OST show the following:</p> <p>tgt_grant_check()) lustre3-OST0000: cli 74c91b6f-783d-4891-fd8c-4d39dd24af9b/ffff88e174d62000 granted: 0 ungranted: 1703936 grant: 0 dirty: 0</p> <p>and no other CDEBUG message from tgt_grant_check(). These messages always show granted == 0 and ungranted &gt; 0, grant == 0, dirty == 0. The logs contain no messages from tgt_grant_alloc() or tgt_grant_shrink(), which I believe implies OBD_MD_FLGRANT is not set.</p> <p>The connection flags reported by the client are:<br/> flags=0x20404af0e3440478<br/> write_grant server_lock version request_portal truncate_lock max_byte_per_rpc early_lock_cancel adaptive_timeouts lru_resize alt_checksum_algorithm fid_is_enabled version_recovery full20 layout_lock 64bithash object_max_bytes jobstats einprogress lvb_type lfsck bulk_mbits</p> <p>so neither OBD_CONNECT_GRANT_SHRINK nor OBD_CONNECT_GRANT_PARAM are set.</p> <p>I'm unable to reproduce this on our test clusters so far, unfortunately. Still working on it.</p> <p>This could be why you were able to trigger <a href="https://jira.whamcloud.com/browse/LU-11663" title="corrupt data after page-unaligned write with zfs backend lustre 2.10" class="issue-link" data-issue-key="LU-11663"><del>LU-11663</del></a> fairly easily. When the client has no grant it will immediately send synchronous writes to the OSS because it doesn't know if it is safe to cache a write if there is no space on the OST.</p> <p>I agree. I forgot to add the link, thanks for adding that.</p> <p>@paf here is the ticket for grant going to 0 with 2.10.5 server.</p> <p>I've uploaded debug logs from OST 0 and from a client.</p> <p>The client version is 2.8.2_6.chaos plus two patches which add some CDEBUGs to the client, see branch 2.8.2-llnl-lu-11798-troubleshoot in the lustre-release-fe-llnl project. The extra CDEBUGs are not likely helpful, but I'm making it available to you so the line numbers you see in the client debug logs match up with the code. </p> <p> <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/31663/31663_toss-4377.catalyst107.dec13.tgz" title="toss-4377.catalyst107.dec13.tgz attached to LU-11798">toss-4377.catalyst107.dec13.tgz^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/31664/31664_toss-4377.ost0.dec13.tar.gz" title="toss-4377.ost0.dec13.tar.gz attached to LU-11798">toss-4377.ost0.dec13.tar.gz^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> </p> <p>Hi Alex</p> <p>Could you please investigate?</p> <p>Thanks</p> <p>Peter</p> <p>did you try to reproduce with 2.10 client? I'm building 2.8 locally, but just in case you tried already..</p> <p> </p> <p>Alex, I wasn't able to test with 2.10 client on the catalyst cluster. </p> <p>thanks..  just tried to reproduce with 2.10 - seem to work well so far.</p> <p> </p> <p>There's some bug which I cannot find right now where the ZFS block size confuses the grant code with older clients...  And the server... gets confused about how much grant to send to them or something?  It is resolved by the various grant fixes to enable OBD_GRANT_PARAM and the fixes atop it.  I cannot recall a single specific patch.</p> <p>I talked about this at LUG this year with some of the DDN Lustre guys, because IU (Cray client, DDN server) was having trouble with it.  Perhaps Li Xi remembers better...?  (I believe that's who I talked to, but I'm not 100% sure.)</p> <p>Found some notes.  Olaf - What is your record size on ZFS?  Reducing it from 1 MiB to 128K made the problem disappear for us, so that would be a good indication if it's the same issue.</p> <p>OK, finally found what I was looking for.</p> <p>Here's the details of a very similar problem we saw with a 2.5 client/2.10 server interop with a 2.5 client that did not have the OBD_GRANT_PARAM patches.  We did not track down the precise fix that resolved the issue - We just had them use Cray's 2.7, which has the OBD_GRANT_PARAM and other related fixes integrated and worked fine.</p> <p>Note that the DDN 2.5 client was OK - I never confirmed, but I suspect DDN had the OBD_GRANT_PARAM fixes integrated in to that client.<br/> "</p> <p>With 1M record sizes, the server is not giving the client any grant:<br/> 00002000:00000001:27.0:1522265343.971722:0:109178:0:(tgt_grant.c:909:tgt_grant_alloc()) Process leaving (rc=0 : 0 : 0)</p> <p>With 128K record sizes, it is:<br/> 00002000:00000001:16.0:1522265535.951623:0:34090:0:(tgt_grant.c:951:tgt_grant_alloc()) Process leaving (rc=33554432 : 33554432 : 2000000)</p> <p>The result of all this is that the client can only send 1 RPC at a time. This works better at with 4 MiB RPCs than 1 MiB RPCs, for obvious reasons, but sending only 1 RPC at a time is the real problem.</p> <p>There are a few different possible fixes on the client side, but again, it will be interesting to know what DDN has here.</p> <p>The check which is failing in tgt_grant_alloc <b>appears</b> to be:<br/> /* Grant some fraction of the client's requested grant space so that</p> <ul> <li>they are not always waiting for write credits (not all of it to</li> <li>avoid overgranting in face of multiple RPCs in flight). This</li> <li>essentially will be able to control the OSC_MAX_RIF for a client.<br/> *</li> <li>If we do have a large disparity between what the client thinks it</li> <li>has and what we think it has, don't grant very much and let the</li> <li>client consume its grant first. Either it just has lots of RPCs</li> <li>in flight, or it was evicted and its grants will soon be used up. */<br/> if (curgrant &gt;= want || curgrant &gt;= ted-&gt;ted_grant + chunk)<br/> RETURN(0);</li> </ul> <p>"</p> <p>(As noted at the top, I didn't actually root cause the problem - We just noticed Cray's 2.7 <span class="error">&#91;which likely has other fixes than those in your 2.8&#93;</span> worked and moved on.)</p> <p>Thank you Patrick. In this case tgt_grant_alloc is never called (nor tgt_grant_shrink), based on the debug log contents. So I believe OBD_MD_FLGRANT is not set. For some reason that's not the case on another 2.8 client / 2.10 server instance I have for comparison.</p> <p><font color="#172b4d"><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ofaaland" class="user-hover" rel="ofaaland">ofaaland</a> please check I'm using correct version to reproduce:</font></p> <p>commit 5c48e8c507bed4d4c35b10a3c61ae17e57ec1372 (<b>tag: 2.8.2_6.chaos</b>, <b>origin/2.8.2-llnl</b>)</p> <p>Author: Jian Yu &lt;yujian@whamcloud.com&gt;</p> <p>Date:   Sun Oct 28 11:28:23 2018 -0700</p> <p> </p> <p>I just tried with 2.8.2-chaos branch (client) and master (server) - failed to reproduce. and same with 2.10.5 on the server. ZFS record size changed nothing. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@cl4 tests]# ../utils/lctl get_param osc.*.*grant* osc.lustre-OST0000-osc-ffff8801d7ebe800.cur_grant_bytes=790528 osc.lustre-OST0000-osc-ffff8801d7ebe800.cur_lost_grant_bytes=0 osc.lustre-OST0000-osc-ffff8801d7ebe800.grant_shrink_interval=1200 [root@cl4 tests]# dd <span class="code-keyword">if</span>=/dev/urandom of=/mnt/lustre/test.10M bs=5K count=2000 &gt;&amp;/dev/<span class="code-keyword">null</span> [root@cl4 tests]# ../utils/lctl get_param osc.*.*grant* osc.lustre-OST0000-osc-ffff8801d7ebe800.cur_grant_bytes=12288 osc.lustre-OST0000-osc-ffff8801d7ebe800.cur_lost_grant_bytes=0 osc.lustre-OST0000-osc-ffff8801d7ebe800.grant_shrink_interval=1200 [root@cl4 tests]# sync [root@cl4 tests]# ../utils/lctl get_param osc.*.*grant* osc.lustre-OST0000-osc-ffff8801d7ebe800.cur_grant_bytes=761856 osc.lustre-OST0000-osc-ffff8801d7ebe800.cur_lost_grant_bytes=0 osc.lustre-OST0000-osc-ffff8801d7ebe800.grant_shrink_interval=1200 </pre> </div></div> <p>AFAICS, the client was asking for grant every RPC:</p> <p>00000008:00000020:1.0:1544742792.404120:0:65915:0:(osc_request.c:611:osc_announce_cached()) dirty: 0 undirty: 33554432 dropped 0 grant: 0</p> <p>undirty (which is how much the client asks for) is 32MB</p> <p>for old pre-GRANT_PARAM clients we've got the following:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (!exp_grant_param_supp(exp)) { oa-&gt;o_undirty = tgt_grant_inflate(tgd, oa-&gt;o_undirty); <span class="code-keyword">static</span> inline u64 tgt_grant_inflate(struct tg_grants_data *tgd, u64 val) { <span class="code-keyword">if</span> (tgd-&gt;tgd_blockbits &gt; COMPAT_BSIZE_SHIFT) /* Client does not support such large block size, grant * is thus inflated. We already significantly overestimate * overhead, no need to add the extent tax in <span class="code-keyword">this</span> <span class="code-keyword">case</span> */ <span class="code-keyword">return</span> val &lt;&lt; (tgd-&gt;tgd_blockbits - COMPAT_BSIZE_SHIFT); #define COMPAT_BSIZE_SHIFT 12</pre> </div></div> <p> and finally:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> #define o_undirty o_mode __u32 o_mode;</pre> </div></div> <p>my understanding that for blocksize=512K (1^19) and larger tgt_grant_inflate() generates a number overflowing o_undirty and making it 0</p> <p>my previous try to reproduce that failed because blocksize for grants is learnt at mount, it doesn't reflect subsequent changes made with zfs set recordsize=.<br/> if the recordsize is set before mount, then it's trivial to reproduce.</p> <p>That's right, recordsize was different on the test system than it was on the production system, and setting recordsize=1M, then stopping and starting the target, triggers the issue on the test system.</p> <p>Thank you Alex, thank you Patrick!</p> <p>just attached a quick "fix" which essentially limits client's "wish" on the server side. <br/> this is far from a good solution, I'm afraid, but at least not every 4K IO is sync.<br/> with this approach the larger ZFS block size, the more sync IO.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ofaaland" class="user-hover" rel="ofaaland">ofaaland</a> is this kind of stopgap fix sufficient to allow you to move forward? I know that you are planning to complete your migration to 2.10.x in the new year.</p> <p>Peter, I think this will work. </p> <p>We have a couple of file systems with ~7000 clients, and so this would mean unused grant would "occupy" about 7TB per OST. The OSTs each have &gt;400TB free right now, so that's OK.</p> <p>This fix is not ideal, as Alex says, but the existing code is badly broken. Is there a reason not to put this patch through the usual system and get it landed to the 2.10 branch?</p> <p>I think the best solution is to upgrade the clients <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <blockquote><p>I think the best solution is to upgrade the clients </p></blockquote> <p>Alex, I have no argument with that! But a significant number of sites still use lustre versions &lt; 2.10 according to the last survey results I saw:<br/> <a href="https://www.eofs.eu/_media/events/lad18/02_peter_jones_community_release_update_lad_2018_final.pdf" class="external-link" target="_blank" rel="nofollow noopener">https://www.eofs.eu/_media/events/lad18/02_peter_jones_community_release_update_lad_2018_final.pdf</a></p> <p>So this needs to be fixed, or at least made less bad, in master. And since you're about to stop maintaining 2.10, it seems a shame to leave it this broken.</p> <blockquote><p><font color="#172b4d">So this needs to be fixed, or at least made less bad, in master. And since you're about to stop maintaining 2.10, it seems a shame to leave it this broken.</font></p></blockquote> <p>Happy New Year!   What say you?  Or is there a patch in gerrit I overlooked? Thanks.</p> <p>Olaf Faaland-LLNL (faaland1@llnl.gov) uploaded a new patch: <a href="https://review.whamcloud.com/33948" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33948</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11798" title="cur_grant goes to 0 and never increases with 2.8 client and 2.10 server" class="issue-link" data-issue-key="LU-11798"><del>LU-11798</del></a> grant: prevent overflow of o_undirty<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0b13ea79c3477ed835f40c12a03e38d92e7b77d4</p> <p>I've pushed p0 as a patch against Master to gerrit, which I will cherry-pick from for our local build.</p> <p>Olaf</p> <p>Thanks for pushing the patch to master. Alex is out for the holidays but I chatted briefly with him earlier and he confirmed that we could land this fix to master (and then to b2_10 and b2_12 afterwards)</p> <p>Peter</p> <p>Great, thank you both.</p> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33957" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33957</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11798" title="cur_grant goes to 0 and never increases with 2.8 client and 2.10 server" class="issue-link" data-issue-key="LU-11798"><del>LU-11798</del></a> grant: prevent overflow of o_undirty<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: a8d73df2469a4966f93d61fc4668f581c1dd7422</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33948/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33948/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11798" title="cur_grant goes to 0 and never increases with 2.8 client and 2.10 server" class="issue-link" data-issue-key="LU-11798"><del>LU-11798</del></a> grant: prevent overflow of o_undirty<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d6f5219162116778e49865d83724276b17b9ce3a</p> <p>Landed for 2.13</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34561" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34561</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11798" title="cur_grant goes to 0 and never increases with 2.8 client and 2.10 server" class="issue-link" data-issue-key="LU-11798"><del>LU-11798</del></a> grant: prevent overflow of o_undirty<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 9786a2a7bf1d27539ea5aaa27200a5f651d9b430</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34561/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34561/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11798" title="cur_grant goes to 0 and never increases with 2.8 client and 2.10 server" class="issue-link" data-issue-key="LU-11798"><del>LU-11798</del></a> grant: prevent overflow of o_undirty<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: e13cc954a776f6dec06f01299a55a55018ae6235</p> Related LU-11663 LU-11844 Development Rank 1|i0088f: Rank (Obsolete) 9223372036854775807 Severity