https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8480 Lustre <p>Error message seen in system log on at least one production server:<br/> (running lustre-2.7.2-1.1nasS_mofed32v1)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>May 20 18:38:58 nbp1-oss1 kernel: LustreError: 38115:0:(ofd_grant.c:183:ofd_grant_sanity_check()) ofd_obd_disconnect: tot_granted 98847872 != fo_tot_granted 100945024 </pre> </div></div> <p>Reproduced on test system, (running lustre-2.7.2-2nasS_mofed32v1.el67.20160517v2), by mounting just 8 clients, running IOR write from 1, IOR read from 1, mdtest on the other 6. Then upon unmount of all 8 clients simultaneously, get the messages on each (of the two) OSS nodes:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Aug 5 08:06:36 service320 kernel: LustreError: 76476:0:(ofd_grant.c:183:ofd_grant_sanity_check()) ofd_obd_disconnect: tot_granted 69347328 != fo_tot_granted 102901760 Aug 5 08:06:36 service323 kernel: LustreError: 76171:0:(ofd_grant.c:183:ofd_grant_sanity_check()) ofd_obd_disconnect: tot_granted 69347328 != fo_tot_granted 102901760 </pre> </div></div> <p>What does this message mean, and is it cause for major concern?</p> <p>We are testing in preparation for upgrade to all remaining production file systems on 8/15, so need to know whether to proceed ASAP.</p> Mostly CentOS-6.8, with 6.7 kernel 2.6.32_573.26.1.el6<br/> Lustre 2.7.2-2nasS_mofed32v1.el67.20160517v2 LU-8480

Server syslog: ofd_grant.c:183:ofd_grant_sanity_check()) ofd_obd_disconnect: tot_granted 69347328 != fo_tot_granted 102901760

Bug Major Resolved Fixed Jian Yu Nathan Dauchy Fri, 5 Aug 2016 20:57:46 +0000 Thu, 15 Jun 2017 23:44:53 +0000 Mon, 29 Aug 2016 13:02:03 +0000 Lustre 2.7.0 Lustre 2.9.0 0 7 <p>It looks like the delta between the grant amounts is about 32MB, which is the amount of grant held by a single client after it has done some writes to the OST. It looks like there is a race window between traversing all of the client exports to accumulate their grant, and the running total that is kept for the whole OST when a client is being unmounted, and is possible to be triggered when multiple clients are being unmounted. This doesn't seem harmful in itself, and for systems with more than 100 clients this check is skipped because it slows down the unmount too much (it is O(n^2) to accumulate the per-export grants as each client unmounts).</p> <p>It looks like there may be a very simple fix, for which I can push a patch if you could give it a test.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/21813" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21813</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8480" title="Server syslog: ofd_grant.c:183:ofd_grant_sanity_check()) ofd_obd_disconnect: tot_granted 69347328 != fo_tot_granted 102901760" class="issue-link" data-issue-key="LU-8480"><del>LU-8480</del></a> ofd: hold obd_dev_lock across grant comparison<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 14d7415e9048a287aabde9a5d03dd2d9b6c48ae9</p> <p>Thanks for the quick review. Since this shouldn't be harmful, and is skipped for more than 100 clients anyway, we may not rush to get a rebuild into testing prior to the upgrade planned for next week. Will test as soon as time permits though.</p> <p>NOTE: we will need a backport to 2.7.2.</p> <p>Jian</p> <p>Could you please port this fix to 2.7 FE once it has landed to master</p> <p>Thanks</p> <p>Peter</p> <p>Here is the back-ported patch for Lustre 2.7 FE: <a href="http://review.whamcloud.com/22018" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22018</a></p> <p>For some reason, I am unable to view the backported patch. Is the link correct, or is there perhaps a permissions issue with my gerrit account?</p> <p>Nathan</p> <p>It must be the latter. Please send me an email about this issue and I'll get it sorted out.</p> <p>Peter</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/21813/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21813/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8480" title="Server syslog: ofd_grant.c:183:ofd_grant_sanity_check()) ofd_obd_disconnect: tot_granted 69347328 != fo_tot_granted 102901760" class="issue-link" data-issue-key="LU-8480"><del>LU-8480</del></a> ofd: hold obd_dev_lock across grant comparison<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: cf54bda257dc3287458a9157bb4647f05d8f8469</p> <p>Landed for 2.9</p> <p>Please re-open until the backport patch lands to 2.7 FE.</p> Duplicate Related Development Rank 1|hzyjn3: Rank (Obsolete) 9223372036854775807 Severity