-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
socklnd's zero-copy send does more work than it should, in two ways, both in ksocknal_lib_sendpage(). On RHEL 9 it is not zero copy at all. On kernels where it is, it hands TCP one 4 KB page per call. Both cost sender CPU on every bulk transfer, so they matter where the sender is the limit: an OSS serving reads to many clients, or a client writing fast.
1. RHEL 9: the "zero-copy" send is a copy.
socklnd picks its send method with "#ifdef MSG_SPLICE_PAGES". Upstream added that flag in 6.5 together with TCP support for it, and removed ->sendpage. RHEL 9 defines MSG_SPLICE_PAGES (include/linux/socket.h, brought in for an rxrpc backport), but its TCP never looks at the flag: nothing in net/ipv4 or net/core handles it, and tcp_sendmsg_locked() copies from the bvec iterator as for any other send. RHEL 9 still has tcp_sendpage(), which socklnd's other branch would have used. So on EL9 every bulk payload is copied, one page per call, and the sender still waits for a ZC-ACK from the peer, a protocol that only exists so the sender knows when it may reuse pages it did not copy. Checked in the Rocky 9 source, kernel 5.14.0-687.46.1.el9_8. RHEL 8, RHEL 10 and upstream 6.5 and later behave as the code assumes.
Fix: choose the method with a configure test for 'struct proto' having 'sendpage', not with the flag.
Expected gain: on EL9 senders whose CPU is the limit, probably 20 to 30% more bulk throughput. LU-19763 measured more than 30% lower throughput in VMs when socklnd's zero-copy send was turned off, the closest measurement of the same difference, and the EL9 path is somewhat worse than socklnd's own copy path, which at least sends a message in one call and asks for no ZC-ACK.
2. One page per call where splicing works.
When TCP does take MSG_SPLICE_PAGES (upstream 6.5 and later, RHEL 10), socklnd still builds the iterator over a single page per sock_sendmsg() call, a shape kept from ->sendpage when the code was ported (LU-17081). A 1 MB message takes 256 calls, each taking the socket lock and redoing TCP's per-call work, where socklnd's own copy path takes one call. MSG_SPLICE_PAGES takes the whole bio_vec array, as the NFS server and nvme-tcp use it, and a MSG_DONTWAIT call returns the bytes queued so far when the send buffer fills, which socklnd already handles.
Fix: pass the whole bio_vec array in one call when splicing. Kernels with ->sendpage still send a page per call.
Expected gain: smaller than (1). At a few tenths of a microsecond per call, 255 extra calls are on the order of 0.1 ms of sender CPU per MB, against roughly 0.6 ms per MB for a core moving 1.5 GB/s, so perhaps 10 to 15% of sender CPU per byte. This is an estimate; measurements will be added here.
Found by reading the socklnd data path (lnet/klnds/socklnd/socklnd_lib.c).
- is related to
-
LU-21222 socklnd: use kernel zero-copy completions instead of ZC-ACK
-
- Open
-