-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
A bulk write whose object offset is not aligned to the larger of the two page sizes hangs indefinitely, with the writer thread stuck in D state holding a PW extent lock. The console on the client shows repeated LNet drops of the server's bulk GET:
lnet_try_match_md() ... Matching packet from <server> ... length 1015808 too big: 32768 left
Root cause: both peers build the bulk memory-descriptor (MD) list with `__ptlrpc_prep_bulk_page()`, but a new MD is started whenever `bd_iov_count` reaches a multiple of `LNET_MAX_IOV`. `bd_iov_count` advances at a different rate on each end — a 64KiB-page sender splits pages to keep each MD within `LNET_MTU` and so accumulates kiov entries faster than a 4KiB-page peer — so the two ends place MD boundaries at different byte offsets. For a full-size (16MiB) transfer that is not page-aligned, the 64KiB client builds 17 MDs where the 4KiB server builds 16. The server issues a bulk GET sized for one of its MDs against the match bits of a smaller client MD; LNet drops it as too large; the write times out and retries forever.
LU-19180 already made the byte accounting page-size-independent (`bd_iop_len`, in 4KiB interop units) but left the MD-boundary rule keyed on `bd_iov_count`, which is page-size-dependent — so the disagreement remains for large unaligned transfers.
*Fix:*
Take the MD boundary from `bd_iop_len` (the 4KiB-virtual on-wire byte count, the only quantity identical on both ends) instead of `bd_iov_count`: open the first MD when none exists, and open each later MD when the current one reaches `LNET_MTU`. Each kiov adds at least `PTLRPC_BULK_INTEROP_PAGE_SIZE` to `bd_iop_len`, so an MD reaches `LNET_MTU` at or before its `LNET_MAX_IOV`'th kiov and the per-MD iov limit is still honoured. This synchronizes segmentation for every transfer whose fragments are 4KiB-aligned (all page-cache IO, any page size). Sub-4KiB unaligned direct IO is out of scope — its fragment lengths themselves differ between page sizes, so agreement there requires exchanging the segmentation on the wire.
*Reproduction:*
`sanity` test_119s: writes one full-size RPC at an object offset that is not page-aligned and fails if the two ends disagree on the MD count. Reproduces only with a client page larger than the server's (`clientarch=aarch64 serverarch=x86_64`); the existing 56x coverage (1–4MiB, matched page sizes) never exercised it.
*Workaround:*
On large-page clients, cap the RPC below the divergence point: `lctl set_param osc.*.max_pages_per_rpc=240` (validated via https://review.whamcloud.com/67524).