-
Improvement
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
Description
When a client holds many PR extent locks on one object, cancelling them is
quadratic in the number of locks. ldlm_extent_shift_kms() runs for every
cancel, under lock_res_and_lock(), and walks every lock in the extent tree
whenever no remaining lock protects the old KMS.
With 20000 adjacent one-page PR locks, cancelling them all takes about 5.6 s
on average on test clients, and 0.19 s with the patch below (30x faster, see
"Measurements"). The cost grows with the square of the lock count, so it
gets much worse at the lock counts seen on shared files.
The same problem was seen together with client evictions during shared-file
IOR with unaligned records ("Observed in the field", below). The link to
the evictions is likely, but it is not proven. This ticket is about the
cancel cost, which is measured.
Analysis
__osc_dlm_blocking_ast() calls ldlm_extent_shift_kms() for every cancel, under
lock_res_and_lock(). When no remaining lock protects the old KMS, the
function walks every lock in the extent tree. That is O(N) per cancel and
O(N^2) for a burst of N cancels. Other threads that need the resource lock
spin behind it.
The client cancels its unused locks from the newest to the oldest. Locks taken
from low to high offsets are therefore cancelled highest first, which is the
worst case.
Related tickets: LU-12832 and LU-12377 show the same symptoms. The
"end + 1 >= old_kms" change discussed in LU-12832 is already in the tree
(LU-13312), and the stacks below still show the scan. It is not LU-19721:
the client build has no invalidate_lock code in llite or osc.
Measurements
Seven clients sharing one striped file. Each client takes 20000 adjacent
one-page PR locks on the same object with "lfs ladvise -a lockahead", lowest
offset first, so the locks are cancelled highest first. The time is from
"lru_size=clear" to lock_count reaching 0 on that client. One run per build.
cancel 20000 locks unpatched patched ratio sles160 5574 ms 130 ms 43x sles157 6518 ms 126 ms 52x rhel98 5245 ms 158 ms 33x rhel102 6435 ms 233 ms 28x debian137 5146 ms 214 ms 24x ubuntu2404 4943 ms 220 ms 22x ubuntu2604 5359 ms 219 ms 24x mean 5603 ms 186 ms 30x
The same case on a 4 vCPU rocky9 VM (sanity 255e, new), one run each:
unpatched: lowest first 2083 ms, highest first 6630 ms -> FAIL patched: lowest first 2137 ms, highest first 2137 ms -> PASS
kunit ldlm_extent test 3, cost per iteration (one lock added, the oldest
cancelled, and one ldlm_extent_shift_kms() call), on the same VM:
locks unpatched patched 10 9.1 us 9.1 us 1,000 28.1 us 9.3 us 10,000 220.5 us 9.4 us ~27,000 ~346 us n/a 100,000 not reached 9.2 us
Workaround (from LU-12832)
lctl set_param ldlm.namespaces.*osc*.lru_max_age=30000
This shortens how long idle locks accumulate, at the cost of more lock RPCs
and more re-reads. It does not remove the cost of cancelling many live
locks at once.