[LU-12368] concurrent statfs() calls on the client should be blocked

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.13.0
• Affects Version/s: Lustre 2.10.8, Lustre 2.12.3
• Labels: easy
• Assignee: Andreas Dilger
• Reporter: Andreas Dilger

Description

If multiple threads on a client are executing statfs() calls concurrently, and the obd_statfs() cache has expired, then each thread will send an OST_STATFS RPC to each OST. With certain statfs-heavy workloads on many-core client nodes, this can result in thousands of needless RPCs being sent from each client every few seconds.

Since all of the callers funnel through obd_statfs(), and there is no benefit to having multiple OST_STATFS or MDS_STATFS replies from the same target (they return the same data, and all threads are blocked on the reply) it makes sense to just allow one thread to execute the statfs and other threads to (interruptibly) wait for it to complete.

Issue Links

• is related to: LU-13296 statfs isn't work properly with MDT statfs proxy

Comments

[Alexey Lyashkov]

In fact - fix is incorrect. Lets look to LU-13296.

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/36874
Subject: LU-12368 ptlrpc: make DEBUG_REQ messages consistent
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: bcf0777c27333aade617e92b9cfe03877e568e4e

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35311/
Subject: LU-12368 ptlrpc: make DEBUG_REQ messages consistent
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c0fa0ba4a8efcd774f1fe27986a0217c76dedf6d

[Andreas Dilger]

Patrick, there is still patch https://review.whamcloud.com/35383 "obdclass: allow 'lfs df' to specify cache age" still open. I added that patch while working on "obdclass: don't send multiple statfs RPCs" but I'm not sure if it is actually useful to land or not.

[Patrick Farrell (Inactive)]

Andreas,

Any reason this ticket is still open?  I think we've got the patches all merged.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35485/
Subject: LU-12368 obdclass: don't send multiple statfs RPCs
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 1839debbba0e70dd5cc9f11c8bc83bcead0114d4

[Andreas Dilger]

Bob, the default statfs cache age has been 1s since the beginning of Lustre. Increasing it to 30s is pretty reasonable, as (IMHO) applications shouldn't really put too much stock in the amount of free space in the filesystem anyway, given that there may be a large number of jobs running concurrently and allocating and freeing space, space is reserved by clients, quota limits may intervene before the filesystem runs out of space, etc. Either there is lots of free space, and the application shouldn't care, or there isn't lots of space and the application may run out of space even if it checks in advance (e.g. some other process consumes the remaining space).

Is there any clear indication why the application is doing so many statfs() calls?

[Bob Hawkins]

LU-12368 testing is underway right now at the customer site, running ExaScaler 4.2 and DDN Lustre 2.10.7_ddn5. We noted the default llite.*.statfs_max_age was set to 1. (or 1 second)

When testing with the default age of 1, we observed a regular occurrence (~1 in 10) of slow app timesteps in which each timestep made numerous statfs calls.

Increasing the statfs_max_age to 30 seconds, we did see a marked, positive change in behavior in which we only saw 1 in 900 slow app timesteps.

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35485
Subject: LU-12368 obdclass: don't send multiple statfs RPCs
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: f0819a6e0b676fb80b9a612ce5c48786242da864

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35380/
Subject: LU-12368 obdclass: don't send multiple statfs RPCs
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1c41a6ac390bf74a135861efcd576a3b433d3c49