-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
Hybrid IO turns a large buffered IO into an unaligned direct IO, so
a single thread's bandwidth is set by how fast the client can build
RPCs and copy the user buffer. Both are currently confined to one
CPU, and the threshold deciding when to switch predates large folio
support.
Measured with one IOR thread on a 192 core client against 16
targets, with the targets faked (fail_loc 0x238) so the numbers are
the client path alone:
1. RPC building runs on the submitting CPU. Since 776e163cf542
("LU-17180 ptlrpc: don't block ptlrpcd too long") each OSC's
writeback work is queued on the system workqueue, which is
per-CPU, so one thread's IO to a striped file is built on one
core – including the user copy of an unaligned DIO write. A
256 MiB unaligned write ran at 18 GiB/s.
2. Moving to an unbound workqueue is not sufficient. Since Linux
6.5 the default affinity scope is the last level cache, so every
OSC still builds inside the submitting CPU's CCD, 8 CPUs on this
part.
3. The unaligned copy is itself serial: one thread copies the whole
IO into or out of the bounce buffer before the RPCs go out.
With RPC building moved to a dedicated unbound workqueue, spread
over the submitting node, and the copy parallelized, the same
256 MiB unaligned write reaches 206 GiB/s.
Separately, the buffered-to-direct thresholds (2 MiB write, 8 MiB
read) were chosen before large folio support. With folios enabled
the buffered write path is substantially faster, so the size at
which direct IO wins moves out; buffered reads gain nothing from
folios in this series while hybrid reads gain a great deal, so the
read threshold moves the other way. The default should follow
whether folio support is on rather than adding a tunable.
- is related to
-
EX-16106 Loading...