[LU-16771] add client statfs() cache for project quotas

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.17.0
• Affects Version/s: Lustre 2.15.2
• Labels: medium
• Environment: 2.12 servers with 2.15 clients
• Severity: 3
• Epic Link: MTFL: Tenant level statistics
• Assignee: Alex Zhuravlev
• Reporter: Stephane Thiell

Description

We have noticed that smbd can do lots of statfs() calls, which on our lustre filesystem with project quota enabled will gather project quota info and takes time.
I tried to play with lustre's statfs caching but it doesn't seem to help. Is that not used when project quotas are set?

For example, on Oak we have 464 HDD-based OSTs and when they are loaded, gathering quota info might take a bit of time but statfs caching doesn't seem to help:

[root@oak-h04v20 pmischel]# lctl set_param llite.oak-ffff999f20cf9800.statfs_max_age=600
llite.oak-ffff999f20cf9800.statfs_max_age=600
[root@oak-h04v20 pmischel]# time df .; sleep 1; time df .; sleep 1; time df .
Filesystem                              1K-blocks      Used    Available Use% Mounted on
10.0.2.51@o2ib5:10.0.2.52@o2ib5:/oak 250000000000 349118136 249650881864   1% /oak

real	0m1.786s
user	0m0.002s
sys	0m0.006s
Filesystem                              1K-blocks      Used    Available Use% Mounted on
10.0.2.51@o2ib5:10.0.2.52@o2ib5:/oak 250000000000 349118136 249650881864   1% /oak

real	0m1.253s
user	0m0.000s
sys	0m0.008s
Filesystem                              1K-blocks      Used    Available Use% Mounted on
10.0.2.51@o2ib5:10.0.2.52@o2ib5:/oak 250000000000 349118136 249650881864   1% /oak

real	0m0.272s
user	0m0.002s
sys	0m0.006s

We use the "get quota command" defined in smbd.conf to return filesystem quotas to SMB clients, which is a user-space program for us, that does some caching by itself, so we don't need quota info from statfs() at all. Is there a way to either disable the project quota behavior with statfs() or make statfs() caching work with 2.15 clients and project quotas?

Issue Links

• is related to: LU-17395 df -h is limited by project quota even if quota enforcement is disabled
• is related to: LU-18222 LQA: Implement quota aggregation limits by ID range
• is related to: LU-15721 projid quota limit statfs() on directories only with PROJINHERIT
• is related to: LU-17191 sanity-quota test_1b, 1d, 1f, 1i: FAIL: user write success, but expect EDQUOT
• is related to: LU-19034 'lfs df' should limit statfs by project quota limits
• is related to: LU-9555 "df /path/to/project" should return projid-specific values

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/59898/
Subject: LU-16771 llite: statfs on regular files is OK
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 7efafec4bd8944b8d5123b73583989383446fbcd

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/59898
Subject: LU-16771 tests: test statfs on regular file
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 436cd2e74550070ced7b686d2f17f456303da981

[Peter Jones]

Merged for 2.17

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/58781/
Subject: LU-16771 llite: cache statfs data for projects
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 2354f2f040d6c581bf6e523de9dd26f5467ae7c4

[Gerrit Updater]

"Alex Zhuravlev <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/58781
Subject: LU-16771 llite: cache statfs data for projects
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 3801a46723f4535da2dbdce83e0abe74a2656396

[Alex Zhuravlev]

I think there are two improvements:
1) introduce a per-inode cache like mentioned before
2) introduce async versions for osc_quotactl() and mdc_quotactl() so lov_quotactl() and lmv_quotactl() can send multiple RPCs in parallel, currently all those are serialized.

[Martin Schwenke]

I haven't read all of the comments, but...

For NSDS, we default to expecting quota info from statfs(). If someone complains about performance due to this then we might help them tune the behaviour using nostatfs_project on their mount, and use the alternative method of getting the quota info.

[Peter Jones]

As per discussion on the LWG call today, moving tickets that do not appear to be essential to fix version 2.17. If the fix lands before code freeze we will update the fix version to reflect that but we want to focus on activities on the critical path. Please speak up if you think that this issue definitely needs to be fixed before we could issue a 2.16 release.

[Andreas Dilger]

Peter, I don't think that the landed patch really fixes the issue properly. It adds a knob to disable the project quota specific statfs, but that only helps if the admin changes the parameter. It would be better to have a cache on the client for this info so that it "just works".

[Peter Jones]

Landed for 2.16