https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9704 Lustre <p>Hi,</p> <p>After about 2 months of production, our new Lustre 2.9-based system (Oak) started to show the following Lustre errors in the logs on our OSS servers, just a few days ago:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Jun 22 10:45:06 oak-io1-s1 kernel: LustreError: 26807:0:(ofd_grant.c:641:ofd_grant_check()) oak-OST0014: cli 1b1c6319-5ec1-dbcc-f6e7-f85575c95c4c claims 131072 GRANT, real grant 0 </pre> </div></div> <p>Attaching an lquota subsystem debug trace on one of the OSS and a splunk graph showing that these messages are actually new. I first thought that this could be related to <a href="https://jira.whamcloud.com/browse/LU-9671" title="Group quota not enforced on clients with mapped GID" class="issue-link" data-issue-key="LU-9671"><del>LU-9671</del></a> but it might not be after all.</p> <p>Thanks,<br/> Stephane</p> 3.10.0-514.16.1.el7_lustre.x86_64, lustre-2.9.0_srcc7-1.el7.centos.x86_64 LU-9704

ofd_grant_check() claims GRANT, real grant 0

Bug Minor Resolved Fixed Mikhail Pershin Stephane Thiell Thu, 22 Jun 2017 19:12:35 +0000 Wed, 16 Mar 2022 23:45:35 +0000 Sat, 20 Nov 2021 15:04:27 +0000 Lustre 2.9.0 Lustre 2.12.8 Lustre 2.15.0 3 21 <p>Niu</p> <p>Could you please assist with this one?</p> <p>Thanks</p> <p>Peter</p> <p>The log has only one line of such message at the very beginning of the log file, do you have other logs which includes more messages before and after this error? What about the disk usage on this OST? Is it almost full or quite empty? Any disconnect/reconnect happened when this error showed up? Thanks.</p> <p>Hi Niu,</p> <p>The error is gone, I cannot see it anymore... I found another trace that I took at almost the same time (attaching as oak_dbg_2_oak-io1-s1.log just in case). The disk usage is pretty low on all OSTs, from 1% to 31% (we recently added more OSTs, hence the disparity in OST usage).</p> <p>I will update the case if I see this error again.</p> <p>Thanks!</p> <p>Stephane</p> <p>Looks the 'granted' value on client and server is inconsistent somehow, and it happened only on one client (with one OST server), I guess such inconsistency could be brought by some rare defect in reconnecting. Please keep updating when you see it again, thanks in advance.</p> <p>I can consistently have this happen with 2.9.59_15_g107b2cb by running <br/> ost-survey -s 10 /LUSTRE/<br/> on a client.<br/> On the system with OST0, I get:<br/> <span class="error">&#91;24348.020466&#93;</span> LustreError: 10346:0:(tgt_grant.c:733:tgt_grant_check()) lustre-OST0000: cli bafa5b19-f11b-9bfc-c44b-60ea94d9382d claims 4292608 GRANT, real grant 0<br/> <span class="error">&#91;24348.020475&#93;</span> LustreError: 10346:0:(tgt_grant.c:733:tgt_grant_check()) Skipped 2 previous similar messages</p> <p>As I increase the size option, the number of skipped messages also increases.</p> <p>Hey Niu,</p> <p>The error messages are back for us too, after having performed OST failovers (might give some hint...). We're still on 2.9 though.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [2682214.198182] LustreError: 361698:0:(ofd_grant.c:641:ofd_grant_check()) oak-OST0003: cli bf793f06-a9f4-4e4a-85c9-79e10ac88ed7 claims 32768 GRANT, real grant 0 [2682214.214014] LustreError: 361698:0:(ofd_grant.c:641:ofd_grant_check()) Skipped 1647 previous similar messages [2682817.076669] LustreError: 361674:0:(ofd_grant.c:641:ofd_grant_check()) oak-OST0003: cli bf793f06-a9f4-4e4a-85c9-79e10ac88ed7 claims 40960 GRANT, real grant 0 [2682817.092530] LustreError: 361674:0:(ofd_grant.c:641:ofd_grant_check()) Skipped 748 previous similar messages [2683417.826667] LustreError: 361684:0:(ofd_grant.c:641:ofd_grant_check()) oak-OST0011: cli 7b5cc9f0-1221-60e2-0871-2105cd878e21 claims 32768 GRANT, real grant 0 [2683417.842515] LustreError: 361684:0:(ofd_grant.c:641:ofd_grant_check()) Skipped 760 previous similar messages [2684036.203234] LustreError: 156433:0:(ofd_grant.c:641:ofd_grant_check()) oak-OST0011: cli 875f290d-d0a2-a84a-c1f4-05a972b31461 claims 847872 GRANT, real grant 0 [2684036.219161] LustreError: 156433:0:(ofd_grant.c:641:ofd_grant_check()) Skipped 736 previous similar messages [2684646.921608] LustreError: 156521:0:(ofd_grant.c:641:ofd_grant_check()) oak-OST0025: cli 0e6972c3-d679-e536-aa5f-ceff2d42a2c4 claims 28672 GRANT, real grant 0 [2684646.937477] LustreError: 156521:0:(ofd_grant.c:641:ofd_grant_check()) Skipped 917 previous similar messages </pre> </div></div> <p>Faced with this error message on one of the systems</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">claims 4218880 GRANT, real grant 0 </pre> </div></div> <p>Do you have an idea where to move investigating this issue?</p> <p>There are other credits-related messages in logs</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> kernel: LustreError: 28855:0:(ofd_grant.c:358:ofd_grant_space_left()) ... left 2232417341440 &lt; tot_grant 2232417341504 unstable 0 pending 0 dirty 29528064 </pre> </div></div> <p> </p> <p>Mike</p> <p>Can you please advise?</p> <p>Peter</p> <p>We are also seeing this issue on v2_10_2 servers with v2_7_21 client.</p> <p><span class="error">&#91;Fri Mar 2 23:12:13 2018&#93;</span> LustreError: 15798:0:(tgt_grant.c:733:tgt_grant_check()) gdata1b-OST0029: cli 6e512e04-c8a3-bdcc-f61e-bb0e3977633d claims 4849664 GRANT, real grant 0<br/> <span class="error">&#91;Fri Mar 2 23:12:13 2018&#93;</span> LustreError: 15798:0:(tgt_grant.c:733:tgt_grant_check()) Skipped 834 previous similar messages<br/> <span class="error">&#91;Fri Mar 2 23:22:18 2018&#93;</span> LustreError: 28124:0:(tgt_grant.c:733:tgt_grant_check()) gdata1b-OST0029: cli 6e512e04-c8a3-bdcc-f61e-bb0e3977633d claims 1703936 GRANT, real grant 0<br/> <span class="error">&#91;Fri Mar 2 23:22:18 2018&#93;</span> LustreError: 28124:0:(tgt_grant.c:733:tgt_grant_check()) Skipped 1150 previous similar messages<br/> <span class="error">&#91;Fri Mar 2 23:32:18 2018&#93;</span> LustreError: 28124:0:(tgt_grant.c:733:tgt_grant_check()) gdata1b-OST0029: cli 6e512e04-c8a3-bdcc-f61e-bb0e3977633d claims 5505024 GRANT, real grant 0<br/> <span class="error">&#91;Fri Mar 2 23:32:18 2018&#93;</span> LustreError: 28124:0:(tgt_grant.c:733:tgt_grant_check()) Skipped 1136 previous similar messages</p> <p>Can we also get this looked at?</p> <p>We are downgrading the client to our stable 2.5 build so we no longer have exp_grant_param_supp flag and fall back to the other behaviour of simply expanding the grant to match.</p> <p>This is a workaround but it looks like the client is having its data dropped:</p> <p><span class="error">&#91;Sat Mar 03 00:32:42 2018&#93;</span> LustreError: 11-0: gdata1b-OST000c-osc-ffff8807f41f1c00: operation ost_write to node 10.112.1.53@o2ib8 failed: rc = -107<br/> <span class="error">&#91;Sat Mar 03 00:32:42 2018&#93;</span> Lustre: 4848:0:(llite_lib.c:2818:ll_dirty_page_discard_warn()) gdata1b: dirty page discard: 10.112.1.41@o2ib8:10.112.1.42@o2ib8:/gdata1b/fj7/fid: <span class="error">&#91;0x2400101ea:0x13300:0x0&#93;</span>// may get corrupted (rc -108)<br/> <span class="error">&#91;Sat Mar 03 00:32:42 2018&#93;</span> Lustre: 4844:0:(llite_lib.c:2818:ll_dirty_page_discard_warn()) gdata1b: dirty page discard: 10.112.1.41@o2ib8:10.112.1.42@o2ib8:/gdata1b/fj7/fid: <span class="error">&#91;0x2400101ea:0x13d2a:0x0&#93;</span>// may get corrupted (rc -108)<br/> <span class="error">&#91;Sat Mar 03 00:32:42 2018&#93;</span> Lustre: 4842:0:(llite_lib.c:2818:ll_dirty_page_discard_warn()) gdata1b: dirty page discard: 10.112.1.41@o2ib8:10.112.1.42@o2ib8:/gdata1b/fj7/fid: <span class="error">&#91;0x2400101ea:0x13598:0x0&#93;</span>/ may get corrupted (rc -108)<br/> <span class="error">&#91;Sat Mar 03 00:32:42 2018&#93;</span> Lustre: 4843:0:(llite_lib.c:2818:ll_dirty_page_discard_warn()) gdata1b: dirty page discard: 10.112.1.41@o2ib8:10.112.1.42@o2ib8:/gdata1b/fj7/fid: <span class="error">&#91;0x2400101ea:0x13291:0x0&#93;</span>/ may get corrupted (rc -108)<br/> <span class="error">&#91;Sat Mar 03 00:32:42 2018&#93;</span> Lustre: 4854:0:(llite_lib.c:2818:ll_dirty_page_discard_warn()) gdata1b: dirty page discard: 10.112.1.41@o2ib8:10.112.1.42@o2ib8:/gdata1b/fj7/fid: <span class="error">&#91;0x2400101ea:0x13cfb:0x0&#93;</span>/ may get corrupted (rc -108)<br/> <span class="error">&#91;Sat Mar 03 00:32:43 2018&#93;</span> Lustre: 4840:0:(llite_lib.c:2818:ll_dirty_page_discard_warn()) gdata1b: dirty page discard: 10.112.1.41@o2ib8:10.112.1.42@o2ib8:/gdata1b/fj7/fid: <span class="error">&#91;0x2400101ea:0x13cfa:0x0&#93;</span>// may get corrupted (rc -108)</p> <p>I'm seeing this while testing on our ARM system running 2.11 and 2.11 servers on x86 boxes.</p> <p>I'm now seeing this with v2_10_2 servers and v2_5_43 client.</p> <p>Interestingly I have the same client on other nodes who aren't seeing the issue. Could this be MOFED related? It's one of the few differences between the nodes.</p> <p>I'm not sure whether or not the grant messages and the disconnect (that's what -107 is) are related or not.  I'm curious, has anyone explored a possible connection to <a href="https://jira.whamcloud.com/browse/LU-10776" title="Large grant requests still don&#39;t work, resulting in small write RPCs" class="issue-link" data-issue-key="LU-10776"><del>LU-10776</del></a></p> <p>Basically, the client (with or without OBD_GRANT_PARAM, it's a long story) has several grant bugs where it can ask for more than it is allowed <b>or</b> the grant calculation can overflow the 32 bit integer used to store it, resulting in weirdly small grant #s showing up at various points.  That patch should resolve all of them in a slightly brute force manner by fixing the arithmetic and capping the resultant request at 2 GiB - some margin.</p> <p>If the problem is relatively easy to reproduce, it would be very interesting to see the effect of that patch on the clients.  (Note that patch doesn't change anything on the server - It modifies a server-side file, but only to replace a numeric constant with a #define.  No behavioral change to the server code.)</p> <p>It would also be interesting if that patch resolved the grant message but the disconnects and lost data continued.  (In fact, that's sort of what I would expect to happen if I'm right.)</p> <p>I had a look at <a href="https://jira.whamcloud.com/browse/LU-10776" title="Large grant requests still don&#39;t work, resulting in small write RPCs" class="issue-link" data-issue-key="LU-10776"><del>LU-10776</del></a>. We are using 1MB RPC's and only 8 in flight RPC's. So I'm not sure if that is related.</p> <p>However I do think there is value in testing this patch.</p> <p>We hit similar trouble on BGI site, server is 2.10.2, client is 2.7.19.10:</p> <blockquote> <p>00002000:00020000:9.0:1526810613.427919:0:16020:0:(tgt_grant.c:733:tgt_grant_check()) zfsz2-OST0009: cli ab95b9a6-6564-3cd9-cdc8-75af4a40f6b9 claims 4849664 GRANT, real grant 0<br/> 00002000:00020000:10.0:1526810619.171036:0:16033:0:(tgt_grant.c:733:tgt_grant_check()) zfsz2-OST0009: cli ab95b9a6-6564-3cd9-cdc8-75af4a40f6b9 claims 917504 GRANT, real grant 0<br/> 00002000:00020000:0.0:1526810624.176488:0:16007:0:(tgt_grant.c:733:tgt_grant_check()) zfsz2-OST0009: cli ab95b9a6-6564-3cd9-cdc8-75af4a40f6b9 claims 917504 GRANT, real grant 0<br/> 00002000:00020000:6.0:1526810648.251551:0:16035:0:(tgt_grant.c:733:tgt_grant_check()) zfsz2-OST0009: cli ab95b9a6-6564-3cd9-cdc8-75af4a40f6b9 claims 4849664 GRANT, real grant 0<br/> 00002000:00020000:11.0:1526810649.207223:0:16007:0:(tgt_grant.c:733:tgt_grant_check()) zfsz2-OST0009: cli ab95b9a6-6564-3cd9-cdc8-75af4a40f6b9 claims 786432 GRANT, real grant 0<br/> 00002000:00020000:11.0:1526810649.655100:0:16035:0:(tgt_grant.c:733:tgt_grant_check()) zfsz2-OST0009: cli ab95b9a6-6564-3cd9-cdc8-75af4a40f6b9 claims 4849664 GRANT, real grant 0</p></blockquote> <p>I still don't know if these messages have a bad effect but they appeared again for us on a full 2.12.0 config (servers and clients) yesterday (on March 23, 20:11:59) and this is still ongoing.</p> <p>I checked and this is only on one OST with a single client uuid:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>fir-io2-s1: Mar 23 20:11:59 fir-io2-s1 kernel: LustreError: 55051:0:(tgt_grant.c:742:tgt_grant_check()) fir-OST000c: cli b312248d-5ec3-9bfd-dd3f-1b6622ac23e2 claims 2256896 GRANT, real grant 0 fir-io2-s1: Mar 23 20:11:59 fir-io2-s1 kernel: LustreError: 55051:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 68 previous similar messages fir-io2-s1: Mar 23 20:16:59 fir-io2-s1 kernel: LustreError: 54991:0:(tgt_grant.c:742:tgt_grant_check()) fir-OST000c: cli b312248d-5ec3-9bfd-dd3f-1b6622ac23e2 claims 4218880 GRANT, real grant 0 ... ... ... fir-io2-s1: Mar 24 22:47:52 fir-io2-s1 kernel: LustreError: 55075:0:(tgt_grant.c:742:tgt_grant_check()) fir-OST000c: cli b312248d-5ec3-9bfd-dd3f-1b6622ac23e2 claims 2224128 GRANT, real grant 0 fir-io2-s1: Mar 24 22:47:52 fir-io2-s1 kernel: LustreError: 55075:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 40 previous similar messages </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ cat /proc/fs/lustre/obdfilter/fir-OST000c/exports/10.8.1.31@o2ib6/uuid b312248d-5ec3-9bfd-dd3f-1b6622ac23e2 </pre> </div></div> <p>client is <tt>sh-01-31</tt>, in production, quiet / no kernel logs at all (since Mar 23 02:52:28). On this node, I have found only one job and so a single user that is accessing the filesystem at the moment, and his quota values seem ok, I guess...</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ lfs quota -u ieisenbe /fir Disk quotas for usr ieisenbe (uid 256460): Filesystem kbytes quota limit grace files quota limit grace /fir 6629070564 80530636800 80530636800 - 1844715 75000000 75000000 - $ lfs quota -g $(id -g ieisenbe) /fir Disk quotas for grp 254778 (gid 254778): Filesystem kbytes quota limit grace files quota limit grace /fir 35467077100 91268055040 91268055040 - 14727670 85000000 85000000 - </pre> </div></div> <p> </p> <p>Is this still valid? A solution landed to 2.12.</p> <p>"Vladimir Saveliev &lt;vlaidimir.saveliev@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45371" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45371</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9704" title="ofd_grant_check() claims GRANT, real grant 0" class="issue-link" data-issue-key="LU-9704"><del>LU-9704</del></a> grant: ignore grant info on read resend<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 18332a73752be688e69ead3e8d5ac8edb7d45c82</p> <p>The following scenario makes a message like "claims 28672 GRANT, real<br/> grant 0" to appear:</p> <p>1. client owns X grants and run rpcs to shrink part of those<br/> 2. server fails over so that the shrink rpc is to be resent.<br/> 3. on the clinet reconnect server and client sync on initial amount<br/> of grants for the client.<br/> 4. shrink rpc is resend, if server disk space is enough, shrink does<br/> not happen and the client adds amount of grants it was going to<br/> shrink to its newly initial amount of grants. Now, client thinks that<br/> it owns more grants than it does from server points of view.<br/> 5. the client consumes grants and sends rpcs to server. Server avoids<br/> allocating new grants for the client if the current amount of grant<br/> is big enough:<br/> static long tgt_grant_alloc(struct obd_export *exp, u64 curgrant,<br/> ...<br/> if (curgrant &gt;= want || curgrant &gt;= ted-&gt;ted_grant + chunk)<br/> RETURN(0);<br/> 6. client continues grants consuming which eventually leads to<br/> complains like "claims 28672 GRANT, real grant 0".</p> <p>"Mike Pershin &lt;mpershin@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45474" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45474</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9704" title="ofd_grant_check() claims GRANT, real grant 0" class="issue-link" data-issue-key="LU-9704"><del>LU-9704</del></a> grant: ignore grant info on read resend<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 4162f0d61b762f70e5eb6d099291fad9f836de9d</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/45474/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45474/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9704" title="ofd_grant_check() claims GRANT, real grant 0" class="issue-link" data-issue-key="LU-9704"><del>LU-9704</del></a> grant: ignore grant info on read resend<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: e1d132acf58c3e6a90a527a0a09cdd0fff7fc392</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/45371/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45371/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9704" title="ofd_grant_check() claims GRANT, real grant 0" class="issue-link" data-issue-key="LU-9704"><del>LU-9704</del></a> grant: ignore grant info on read resend<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 38c78ac2e390b30106f3e185d8c4d92b8cb19c2b</p> <p>Landed for 2.15</p> Related LU-14125 LU-14124 LU-13457 Development Rank 1|hzzfkn: Rank (Obsolete) 9223372036854775807 Severity