https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11721 Lustre <p>When issuing an "lfs df -i", the "IUse%" field shows the percentage of used blocks, instead of the percentage of used inodes.</p> LU-11721

"lfs df -i" prints used blocks percentage instead of inodes

Bug Minor Resolved Fixed Nikitas Angelinas Nikitas Angelinas Thu, 29 Nov 2018 23:13:58 +0000 Tue, 11 Jun 2019 05:35:26 +0000 Tue, 11 Jun 2019 05:35:26 +0000 Lustre 2.11.0 Lustre 2.13.0 Lustre 2.12.3 0 6 <p>Nikitas Angelinas (nangelinas@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/33758" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33758</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11721" title="&quot;lfs df -i&quot; prints used blocks percentage instead of inodes" class="issue-link" data-issue-key="LU-11721"><del>LU-11721</del></a> utils: print used inodes ratio when using "lfs df -i"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b7893ce973b899c96bc6ca88b7ddc2b8017f50f6</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33758/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33758/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11721" title="&quot;lfs df -i&quot; prints used blocks percentage instead of inodes" class="issue-link" data-issue-key="LU-11721"><del>LU-11721</del></a> utils: print used inodes ratio when using "lfs df -i"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 757403191c37db75ed35b02c971846dced5d5119</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34136" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34136</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11721" title="&quot;lfs df -i&quot; prints used blocks percentage instead of inodes" class="issue-link" data-issue-key="LU-11721"><del>LU-11721</del></a> utils: print used inodes ratio when using "lfs df -i"<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: cd944a0834b7a5f273f26ce9aad1d8c178c923aa</p> <p>Landed for 2.13</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34164" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34164</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11721" title="&quot;lfs df -i&quot; prints used blocks percentage instead of inodes" class="issue-link" data-issue-key="LU-11721"><del>LU-11721</del></a> tests: wait for statfs to update on DNE<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 96e6faefa7025d443a32cb51135ea84f83240dac</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34164/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34164/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11721" title="&quot;lfs df -i&quot; prints used blocks percentage instead of inodes" class="issue-link" data-issue-key="LU-11721"><del>LU-11721</del></a> tests: wait for statfs to update on DNE<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 263e80f4572b49044407b09f8a3e393677eafb5d</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34167" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34167</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11721" title="&quot;lfs df -i&quot; prints used blocks percentage instead of inodes" class="issue-link" data-issue-key="LU-11721"><del>LU-11721</del></a> tests: limit length of sanity test_418<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a8ff59ab1d50c5f0698ff97c2f625474c9a2215c</p> <p>This exposed a bug in <tt>lod_statfs()</tt> vs. <tt>ll_statfs_internal()</tt> because the new aggregate statfs code in <tt>lod_statfs()</tt> wasn't reducing the reported free inode counts if the OSTs report fewer total free inodes.</p> <p>This issue does not exist in 2.10.6, since it was introduced by the statfs proxy patch in 2.12, so no need to backport the patch.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34321" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34321</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11721" title="&quot;lfs df -i&quot; prints used blocks percentage instead of inodes" class="issue-link" data-issue-key="LU-11721"><del>LU-11721</del></a> tests: wait for statfs to update on DNE<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 306951f8b3907e767b8cdcf52afeea5c0c0a4dbc</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=nangelinas" class="user-hover" rel="nangelinas">nangelinas</a>, this issue is resolved, but <a href="https://review.whamcloud.com/#/c/34167/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/34167/</a> from Andreas is still marked in Gerrit as "Needs Code-Review". What should happen with that code?</p> <p>Reopening until <a href="https://review.whamcloud.com/#/c/34167/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/34167/</a> lands.</p> <p>Cory, looking at the ticket history, we seem to have marked the ticket as Resolved after the initial patch that I submitted was landed, but Andreas submitted two additional patches to address an issue with the patch I submitted in DNE, and a related issue it uncovered with statfs aggregation. I am not sure what the process is, but I took the liberty of reopening the ticket until <a href="https://review.whamcloud.com/#/c/34167/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/34167/</a> lands, which it should, imho.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=nangelinas" class="user-hover" rel="nangelinas">nangelinas</a> - exactly right. Thanks for your eagle eyes <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=spitzcor" class="user-hover" rel="spitzcor">spitzcor</a> <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34167/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34167/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11721" title="&quot;lfs df -i&quot; prints used blocks percentage instead of inodes" class="issue-link" data-issue-key="LU-11721"><del>LU-11721</del></a> lod: limit statfs ffree if less than OST ffree<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a829595add808d0fb09bab525c500d0aa6955883</p> <p>All landed for 2.13 now</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34136/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34136/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11721" title="&quot;lfs df -i&quot; prints used blocks percentage instead of inodes" class="issue-link" data-issue-key="LU-11721"><del>LU-11721</del></a> utils: print used inodes ratio when using "lfs df -i"<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 52a9419f7afaa31f8db922526ca78ef08d909ed4</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34321/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34321/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11721" title="&quot;lfs df -i&quot; prints used blocks percentage instead of inodes" class="issue-link" data-issue-key="LU-11721"><del>LU-11721</del></a> tests: wait for statfs to update on DNE<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: b319390728b1fa6402233d5cb05a1e7d5d09825b</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34453" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34453</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11721" title="&quot;lfs df -i&quot; prints used blocks percentage instead of inodes" class="issue-link" data-issue-key="LU-11721"><del>LU-11721</del></a> lod: limit statfs ffree if less than OST ffree<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 96000f6e5b8dd42dc8095aa4eff19fe56d493887</p> <p>We are still seeing sanity test 418 fail with ''df and lfs df inodes output mismatch: ..." on master branch after all these patches have landed. </p> <p>Here are a couple of recent failures:<br/> <a href="https://testing.whamcloud.com/test_sets/6e908786-4c5c-11e9-9720-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/6e908786-4c5c-11e9-9720-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/c65cfdbc-4c4f-11e9-92fe-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/c65cfdbc-4c4f-11e9-92fe-52540065bddc</a></p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34493" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34493</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11721" title="&quot;lfs df -i&quot; prints used blocks percentage instead of inodes" class="issue-link" data-issue-key="LU-11721"><del>LU-11721</del></a> tests: enable full debugging for sanity<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5a8e480e3380009f5e9fddb7d38389e2b3473d8e</p> <p>Looking at the client debug logs for the failure, it doesn't record any statfs RPCs at all during the test, and the exports are all reported IDLE. That should definitely <b>not</b> be the case for "<tt>lfs df</tt>" and "<tt>lfs df -i</tt>", since the client needs to contact all targets to get the statfs info. Only the "<tt>df</tt>" statfs can be aggregated, but it shouldn't be cached more than a second. The MDS logs show some activity, but not any of the actual statfs results, because the debug level is too low.</p> <p>It looks like the test is not running with full debugging enabled as it should for all sanity tests, so I pushed patch 34493 to ensure this is the case, and hopefully we will get some more usable debug logs after that patch lands.</p> <p>The actual problem looks mostly to be related to what the 34453 patch was trying to fix - namely that the proxy statfs data returned by "<tt>df -i</tt>" is not being limited to the number of OST objects in the filesystem as it should. This makes the total and free inodes wildly larger than what "<tt>lfs df -i</tt>" reports. That patch landed "Fri Mar 15 23:46:21 2019" so it shouldn't be seen in patches that are testing this week unless they haven't been rebased, and I confirmed at least one failure was running a patch that included this fix. </p> <p>There is a second, less common failure mode on ZFS (e.g. maybe 10% of cases) shows the "<tt>df -i</tt>" output is slightly different than the "<tt>lfs df -i</tt>". That might be due to e.g. ZFS background activity changing the number of free blocks, which affects the free inode estimate. This is not a problem for ldiskfs. We might have to allow a margin of error in the total and free inodes for ZFS.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34453/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34453/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11721" title="&quot;lfs df -i&quot; prints used blocks percentage instead of inodes" class="issue-link" data-issue-key="LU-11721"><del>LU-11721</del></a> lod: limit statfs ffree if less than OST ffree<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 805b98c8a9812a2c2bf77b54d30e1bc40aff299e</p> <p>I suspect the last issue was fixed with <a href="https://jira.whamcloud.com/browse/LU-12248" title="sanity test_418: df and lfs df inodes output mismatch" class="issue-link" data-issue-key="LU-12248"><del>LU-12248</del></a>.</p> Related LU-12248 LU-10018 Development Rank 1|i00773: Rank (Obsolete) 9223372036854775807 Severity