[LU-11721] "lfs df -i" prints used blocks percentage instead of inodes

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.13.0, Lustre 2.12.3
• Affects Version/s: Lustre 2.11.0
• Severity: 3
• Assignee: Nikitas Angelinas
• Reporter: Nikitas Angelinas

Description

When issuing an "lfs df -i", the "IUse%" field shows the percentage of used blocks, instead of the percentage of used inodes.

Issue Links

• is related to: LU-12248 sanity test_418: df and lfs df inodes output mismatch
• is related to: LU-10018 MDT as a statfs proxy

Comments

[Andreas Dilger]

I suspect the last issue was fixed with LU-12248.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34453/
Subject: LU-11721 lod: limit statfs ffree if less than OST ffree
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 805b98c8a9812a2c2bf77b54d30e1bc40aff299e

[Andreas Dilger]

The actual problem looks mostly to be related to what the 34453 patch was trying to fix - namely that the proxy statfs data returned by "df -i" is not being limited to the number of OST objects in the filesystem as it should. This makes the total and free inodes wildly larger than what "lfs df -i" reports. That patch landed "Fri Mar 15 23:46:21 2019" so it shouldn't be seen in patches that are testing this week unless they haven't been rebased, and I confirmed at least one failure was running a patch that included this fix.

There is a second, less common failure mode on ZFS (e.g. maybe 10% of cases) shows the "df -i" output is slightly different than the "lfs df -i". That might be due to e.g. ZFS background activity changing the number of free blocks, which affects the free inode estimate. This is not a problem for ldiskfs. We might have to allow a margin of error in the total and free inodes for ZFS.

[Andreas Dilger]

Looking at the client debug logs for the failure, it doesn't record any statfs RPCs at all during the test, and the exports are all reported IDLE. That should definitely not be the case for "lfs df" and "lfs df -i", since the client needs to contact all targets to get the statfs info. Only the "df" statfs can be aggregated, but it shouldn't be cached more than a second. The MDS logs show some activity, but not any of the actual statfs results, because the debug level is too low.

It looks like the test is not running with full debugging enabled as it should for all sanity tests, so I pushed patch 34493 to ensure this is the case, and hopefully we will get some more usable debug logs after that patch lands.

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34493
Subject: LU-11721 tests: enable full debugging for sanity
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5a8e480e3380009f5e9fddb7d38389e2b3473d8e

[James Nunez (Inactive)]

We are still seeing sanity test 418 fail with ''df and lfs df inodes output mismatch: ..." on master branch after all these patches have landed.

Here are a couple of recent failures:
https://testing.whamcloud.com/test_sets/6e908786-4c5c-11e9-9720-52540065bddc
https://testing.whamcloud.com/test_sets/c65cfdbc-4c4f-11e9-92fe-52540065bddc

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34453
Subject: LU-11721 lod: limit statfs ffree if less than OST ffree
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 96000f6e5b8dd42dc8095aa4eff19fe56d493887

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34321/
Subject: LU-11721 tests: wait for statfs to update on DNE
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: b319390728b1fa6402233d5cb05a1e7d5d09825b

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34136/
Subject: LU-11721 utils: print used inodes ratio when using "lfs df -i"
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 52a9419f7afaa31f8db922526ca78ef08d909ed4

[Peter Jones]

All landed for 2.13 now