Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20818

hybrid IO: single thread bandwidth limited by client-side serialization

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      Hybrid IO turns a large buffered IO into an unaligned direct IO, so
      a single thread's bandwidth is set by how fast the client can build
      RPCs and copy the user buffer. Both are currently confined to one
      CPU, and the threshold deciding when to switch predates large folio
      support.

      Measured with one IOR thread on a 192 core client against 16
      targets, with the targets faked (fail_loc 0x238) so the numbers are
      the client path alone:

      1. RPC building runs on the submitting CPU. Since 776e163cf542
      ("LU-17180 ptlrpc: don't block ptlrpcd too long") each OSC's
      writeback work is queued on the system workqueue, which is
      per-CPU, so one thread's IO to a striped file is built on one
      core – including the user copy of an unaligned DIO write. A
      256 MiB unaligned write ran at 18 GiB/s.

      2. Moving to an unbound workqueue is not sufficient. Since Linux
      6.5 the default affinity scope is the last level cache, so every
      OSC still builds inside the submitting CPU's CCD, 8 CPUs on this
      part.

      3. The unaligned copy is itself serial: one thread copies the whole
      IO into or out of the bounce buffer before the RPCs go out.

      With RPC building moved to a dedicated unbound workqueue, spread
      over the submitting node, and the copy parallelized, the same
      256 MiB unaligned write reaches 206 GiB/s.

      Separately, the buffered-to-direct thresholds (2 MiB write, 8 MiB
      read) were chosen before large folio support. With folios enabled
      the buffered write path is substantially faster, so the size at
      which direct IO wins moves out; buffered reads gain nothing from
      folios in this series while hybrid reads gain a great deal, so the
      read threshold moves the other way. The default should follow
      whether folio support is on rather than adding a tunable.

            wc-triage WC Triage
            patrickbot Patrick Bot
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: