[LU-11721] "lfs df -i" prints used blocks percentage instead of inodes

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.11.0
• Fix Version/s: Lustre 2.13.0, Lustre 2.12.3
• Type: Bug
• Priority: Minor
• Reporter: Nikitas Angelinas
• Assignee: Nikitas Angelinas
• Resolution: Fixed
• Severity: 3

Description

When issuing an "lfs df -i", the "IUse%" field shows the percentage of used blocks, instead of the percentage of used inodes.

Issue Links

• is related to: LU-12248 sanity test_418: df and lfs df inodes...
• is related to: LU-10018 MDT as a statfs proxy

Comments

[Gerrit Updater]

Nikitas Angelinas ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33758
Subject: LU-11721 utils: print used inodes ratio when using "lfs df -i"
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b7893ce973b899c96bc6ca88b7ddc2b8017f50f6

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33758/
Subject: LU-11721 utils: print used inodes ratio when using "lfs df -i"
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 757403191c37db75ed35b02c971846dced5d5119

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34136
Subject: LU-11721 utils: print used inodes ratio when using "lfs df -i"
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: cd944a0834b7a5f273f26ce9aad1d8c178c923aa

[Peter Jones]

Landed for 2.13

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34164
Subject: LU-11721 tests: wait for statfs to update on DNE
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 96e6faefa7025d443a32cb51135ea84f83240dac

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch https://review.whamcloud.com/34164/
Subject: LU-11721 tests: wait for statfs to update on DNE
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 263e80f4572b49044407b09f8a3e393677eafb5d

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34167
Subject: LU-11721 tests: limit length of sanity test_418
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a8ff59ab1d50c5f0698ff97c2f625474c9a2215c

[Andreas Dilger]

This exposed a bug in lod_statfs() vs. ll_statfs_internal() because the new aggregate statfs code in lod_statfs() wasn't reducing the reported free inode counts if the OSTs report fewer total free inodes.

[Andreas Dilger]

This issue does not exist in 2.10.6, since it was introduced by the statfs proxy patch in 2.12, so no need to backport the patch.

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34321
Subject: LU-11721 tests: wait for statfs to update on DNE
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 306951f8b3907e767b8cdcf52afeea5c0c0a4dbc

[Cory Spitz]

nangelinas, this issue is resolved, but https://review.whamcloud.com/#/c/34167/ from Andreas is still marked in Gerrit as "Needs Code-Review". What should happen with that code?

[Nikitas Angelinas]

Reopening until https://review.whamcloud.com/#/c/34167/ lands.

[Nikitas Angelinas]

Cory, looking at the ticket history, we seem to have marked the ticket as Resolved after the initial patch that I submitted was landed, but Andreas submitted two additional patches to address an issue with the patch I submitted in DNE, and a related issue it uncovered with statfs aggregation. I am not sure what the process is, but I took the liberty of reopening the ticket until https://review.whamcloud.com/#/c/34167/ lands, which it should, imho.

[Peter Jones]

nangelinas - exactly right. Thanks for your eagle eyes spitzcor

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34167/
Subject: LU-11721 lod: limit statfs ffree if less than OST ffree
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a829595add808d0fb09bab525c500d0aa6955883

[Peter Jones]

All landed for 2.13 now

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34136/
Subject: LU-11721 utils: print used inodes ratio when using "lfs df -i"
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 52a9419f7afaa31f8db922526ca78ef08d909ed4

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34321/
Subject: LU-11721 tests: wait for statfs to update on DNE
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: b319390728b1fa6402233d5cb05a1e7d5d09825b

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34453
Subject: LU-11721 lod: limit statfs ffree if less than OST ffree
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 96000f6e5b8dd42dc8095aa4eff19fe56d493887

[James Nunez (Inactive)]

We are still seeing sanity test 418 fail with ''df and lfs df inodes output mismatch: ..." on master branch after all these patches have landed.

Here are a couple of recent failures:
https://testing.whamcloud.com/test_sets/6e908786-4c5c-11e9-9720-52540065bddc
https://testing.whamcloud.com/test_sets/c65cfdbc-4c4f-11e9-92fe-52540065bddc

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34493
Subject: LU-11721 tests: enable full debugging for sanity
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5a8e480e3380009f5e9fddb7d38389e2b3473d8e

[Andreas Dilger]

Looking at the client debug logs for the failure, it doesn't record any statfs RPCs at all during the test, and the exports are all reported IDLE. That should definitely not be the case for "lfs df" and "lfs df -i", since the client needs to contact all targets to get the statfs info. Only the "df" statfs can be aggregated, but it shouldn't be cached more than a second. The MDS logs show some activity, but not any of the actual statfs results, because the debug level is too low.

It looks like the test is not running with full debugging enabled as it should for all sanity tests, so I pushed patch 34493 to ensure this is the case, and hopefully we will get some more usable debug logs after that patch lands.

[Andreas Dilger]

The actual problem looks mostly to be related to what the 34453 patch was trying to fix - namely that the proxy statfs data returned by "df -i" is not being limited to the number of OST objects in the filesystem as it should. This makes the total and free inodes wildly larger than what "lfs df -i" reports. That patch landed "Fri Mar 15 23:46:21 2019" so it shouldn't be seen in patches that are testing this week unless they haven't been rebased, and I confirmed at least one failure was running a patch that included this fix.

There is a second, less common failure mode on ZFS (e.g. maybe 10% of cases) shows the "df -i" output is slightly different than the "lfs df -i". That might be due to e.g. ZFS background activity changing the number of free blocks, which affects the free inode estimate. This is not a problem for ldiskfs. We might have to allow a margin of error in the total and free inodes for ZFS.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34453/
Subject: LU-11721 lod: limit statfs ffree if less than OST ffree
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 805b98c8a9812a2c2bf77b54d30e1bc40aff299e

[Andreas Dilger]

I suspect the last issue was fixed with LU-12248.