Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-21217

socklnd: zero-copy send copies on RHEL 9 and sends one page per call

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      socklnd's zero-copy send does more work than it should, in two ways, both in ksocknal_lib_sendpage(). On RHEL 9 it is not zero copy at all. On kernels where it is, it hands TCP one 4 KB page per call. Both cost sender CPU on every bulk transfer, so they matter where the sender is the limit: an OSS serving reads to many clients, or a client writing fast.

      1. RHEL 9: the "zero-copy" send is a copy.
      socklnd picks its send method with "#ifdef MSG_SPLICE_PAGES". Upstream added that flag in 6.5 together with TCP support for it, and removed ->sendpage. RHEL 9 defines MSG_SPLICE_PAGES (include/linux/socket.h, brought in for an rxrpc backport), but its TCP never looks at the flag: nothing in net/ipv4 or net/core handles it, and tcp_sendmsg_locked() copies from the bvec iterator as for any other send. RHEL 9 still has tcp_sendpage(), which socklnd's other branch would have used. So on EL9 every bulk payload is copied, one page per call, and the sender still waits for a ZC-ACK from the peer, a protocol that only exists so the sender knows when it may reuse pages it did not copy. Checked in the Rocky 9 source, kernel 5.14.0-687.46.1.el9_8. RHEL 8, RHEL 10 and upstream 6.5 and later behave as the code assumes.
      Fix: choose the method with a configure test for 'struct proto' having 'sendpage', not with the flag.
      Expected gain: on EL9 senders whose CPU is the limit, probably 20 to 30% more bulk throughput. LU-19763 measured more than 30% lower throughput in VMs when socklnd's zero-copy send was turned off, the closest measurement of the same difference, and the EL9 path is somewhat worse than socklnd's own copy path, which at least sends a message in one call and asks for no ZC-ACK.

      2. One page per call where splicing works.
      When TCP does take MSG_SPLICE_PAGES (upstream 6.5 and later, RHEL 10), socklnd still builds the iterator over a single page per sock_sendmsg() call, a shape kept from ->sendpage when the code was ported (LU-17081). A 1 MB message takes 256 calls, each taking the socket lock and redoing TCP's per-call work, where socklnd's own copy path takes one call. MSG_SPLICE_PAGES takes the whole bio_vec array, as the NFS server and nvme-tcp use it, and a MSG_DONTWAIT call returns the bytes queued so far when the send buffer fills, which socklnd already handles.
      Fix: pass the whole bio_vec array in one call when splicing. Kernels with ->sendpage still send a page per call.
      Expected gain: smaller than (1). At a few tenths of a microsecond per call, 255 extra calls are on the order of 0.1 ms of sender CPU per MB, against roughly 0.6 ms per MB for a core moving 1.5 GB/s, so perhaps 10 to 15% of sender CPU per byte. This is an estimate; measurements will be added here.

      Found by reading the socklnd data path (lnet/klnds/socklnd/socklnd_lib.c).

            wc-triage WC Triage
            patrickbot Patrick Bot
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: