Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20778

Unified I/O scope and object-offset I/O

XMLWordPrintable

    • Icon: New Feature New Feature
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      Give the whole family of "do I/O against one stripe (i.e. one OST object) inside a layout component" operations a single addressing model and a single userspace interface.

      FLR designated mirror, EC parity resync/verify, single-stripe migration (LU-20534 / LU-20541), EC rebuild and PFL/DoM merging (LU-13612) all share the same I/O-scope description and UAPI.

      Key points
      • One I/O-scope UAPI: struct ll_ioc_io_scope plus LL_IOC_SET_IO_SCOPE / LL_IOC_CLEAR_IO_SCOPE. All four ll_io_scope values — FILE / MIRROR / STRIPE / OBJECT — are defined in the first UAPI version; OBJECT is initially reserved and implemented in P2.
      • Identity and offset space are kept separate: STRIPE pins one object but keeps using file offsets; only OBJECT reinterprets pos as an object offset.
      • Object-offset I/O is a separate entry path; it does not restructure the existing file→stripe mapping. It is O_DIRECT only and serves lfs / llapi only.

      0. Terminology

      Terminology follows existing Lustre / LOV usage.

      lmm_stripe_offset in the UAPI is the starting OST index, not a byte offset within an object. The latter is always called an object offset here.

      Term Meaning
      file offset Byte offset in the parent file (LOV coordinate, lov_off)
      object offset Byte offset within one OST object (obdoff)
      obj(X) File offset X projected onto a given stripe under the component's striping geometry. obj(e_start) / obj(e_end) are the component boundaries.
      prefix hole [0, obj(e_start)) on the object: does not contribute to the parent i_size, unreachable via that component's file-offset path. This is what LU-13612 needs to write.

      1. Background

      1.1 Today: three unrelated ways to designate a target

      FLR resync / verify

      • Mechanism: LL_IOC_FLR_SET_MIRROR(long), fd_designated_mirror
      • Offset space: file offset
      • Problem: can only select a mirror, not a stripe; no layout_version / FID validation

      EC parity resync / verify (EC branch)

      • Mechanism: LL_IOC_FLR_SET_STRIPE(long), meaning "which parity within the raidset"
      • Offset space: file offset (parity relative to component e_start)
      • Problem: the interface exposes raidset-internal geometry; inconsistent with the mirror interface

      Single-stripe migration (LU-20534)

      • Mechanism: userspace converts file offset to object offset via lov_stripe_offset() and then copies; swap uses LL_IOC_LOV_SWAP_LAYOUTS_FID
      • Offset space: file offset for the copy, FID for the swap
      • Problem: the copy duplicates the conversion in userspace. PFL components, parity, and SEEK_DATA on the parent file crossing stripes / components all need separate handling.

      All three are the same operation: select one OST object identified by (component, stripe), then read or write it in some offset space.

      Today identity, offset space and failure policy are tangled inside each individual ioctl. Once such an interface ships, any later consolidation must retain compatibility paths.

      1.2 Use cases

      • EC rebuild: data on a failed OST is recovered from parity and written into a new object.
      • OST decommissioning / moving a single stripe to another OST: relocate one object as a whole, leaving the rest of the file untouched.
      • PFL/DoM merging (LU-13612): write the data of the DoM component (or of the previous component) into the prefix hole of the next component's object, then move that component's e_start down and delete the previous component.

      Whole-file DoM→OST migration (LU-10177 / LU-11421) already exists, but at the cost of rewriting the whole file; LU-13612 is the optimisation of that.

      Object bytes outside the visible range cannot be reached through that component's file-offset path, which is why OBJECT is needed.


      2. Goals

      1. One SET/CLEAR ioctl pair and one structure that can name a mirror, a parity, or a single stripe, with ios_layout_version. ios_fid is OBJECT-only. FLR mirror stays on SET_MIRROR as an alias for MIRROR + ios_mirror_id; behaviour is unchanged.
      2. Object-offset I/O is a separate entry path in the kernel: it does not restructure the existing file→stripe mapping and introduces no regression on the file-offset path.
      3. Replacing an OST object (migration, rebuild) always uses single-stripe swap. On an existing data object, object-offset writes are only opened up for the prefix hole (LU-13612), and no other range — see §3.1.

      3. Addressing model

      A single I/O selects its target by scope. The offset space of pos follows the scope:

      scope Target Space of pos
      FILE The whole file file offset
      MIRROR One FLR mirror (ios_mirror_id) file offset
      STRIPE One OST object (ios_comp_id, ios_stripe_idx) file offset, via the file→stripe mapping
      OBJECT The same OST object (same fields) object offset
      • ios_stripe_idx is the object index within the component (same ordering as lfs getstripe). On parity, "which parity within the raidset" is idx % p and the raidset is idx / p; these are derived quantities, not interface concepts.
      • Whether a failure triggers recovery is only defined under file offsets. It follows the existing read path by default and is not part of the UAPI.
      Identity fields

      Unused identity fields must be 0:

      • FILE rejects all of them
      • MIRROR rejects ios_comp_id / ios_stripe_idx / ios_fid
      • STRIPE rejects ios_mirror_id / ios_fid

      ios_fid is OBJECT-only (the same FID as LL_IOC_LOV_SWAP_LAYOUTS_FID). FILE / MIRROR / STRIPE require a zero FID. OBJECT requires a non-zero FID.

      3.1 OBJECT I/O rules

      OBJECT reads may target the selected data or parity object. Writes to an existing data object do not open up the whole object.

      The rules below define the read and write boundaries. Replacing the object is a different operation (single-stripe swap).

      1. Reads: the target object's kms determines the boundary. Glimpse first if kms is invalid or the requested position exceeds it; return -EIO if the target is unavailable.
      2. Writes to a data object: only the prefix hole.
      3. Replacing an object: independent of offset space; fill the new object, do a final data version check, then single-stripe swap.
      Object range Object-offset write
      [0, obj(e_start)) accepted — the prefix hole, LU-13612
      [obj(e_start), obj(e_end)) rejected — visible range uses file offset + designated (P1)
      [obj(e_end), ∞) -EINVAL — no consumer

      An object's size may exceed the component's visible span: that is the prefix hole, already true for PFL.

      lov_stripe_size() maps those bytes to a file offset below e_start. lov_attr_get_composite() takes the max across components, so a prefix write cannot raise i_size.

      Writes past obj(e_end) are rejected by rule 2. Under the prefix-hole-only rule, the existing size/kms merge path does not change.

      The OBJECT write path itself must not use an object offset to update the VVP inode's i_size (see §4.2).


      4. Design

      4.1 UAPI

      ll_ioc_io_scope
      enum ll_io_scope {
      	LL_IO_SCOPE_FILE	= 0,	/* whole file, file offset */
      	LL_IO_SCOPE_MIRROR	= 1,	/* one FLR mirror, file offset */
      	LL_IO_SCOPE_STRIPE	= 2,	/* one object, file offset */
      	LL_IO_SCOPE_OBJECT	= 3,	/* one object, object offset */
      };
      
      struct ll_ioc_io_scope {
      	__u32		ios_scope;		/* enum ll_io_scope */
      	__u32		ios_flags;		/* reserved; must be 0 */
      	__u32		ios_mirror_id;		/* MIRROR */
      	__u32		ios_comp_id;		/* STRIPE/OBJECT */
      	__u32		ios_stripe_idx;		/* STRIPE/OBJECT */
      	__u32		ios_layout_version;	/* 0 = skip check */
      	struct lu_fid	ios_fid;		/* OBJECT; must be 0 otherwise */
      	__u64		ios_padding[2];		/* must be 0 */
      };
      
      #define LL_IOC_SET_IO_SCOPE		_IOW('f', 180, struct ll_ioc_io_scope)
      #define LL_IOC_CLEAR_IO_SCOPE		_IO('f', 181)
      

      Enum-valued fields in UAPI structures are declared as __u32 with the real enum named in a comment next to the field. The width and signedness of a C enum vary between compilers, while ioctl structures must have fixed widths. This matches idv_flags / pccs_type in the tree.

      No flags enum is defined: object offsets are a scope value, not a bit.

      • Unknown ios_scope, ios_flags != 0, non-zero ios_padding, or unused identity fields set → -EINVAL.
      • STRIPE: pins one object by ios_comp_id / ios_stripe_idx; pos is still a file offset. ios_mirror_id and ios_fid must be 0. Object bytes outside that component's visible extent remain unreachable.
      • OBJECT: same ios_comp_id / ios_stripe_idx; pos is an object offset. Requires a non-zero ios_fid, O_DIRECT, and no mmap. Mutating operations added in P3 also require a non-zero ios_layout_version. Unavailable objects return -EIO and do not enter CIT_EC_RD. Reserved in the enum from P1, returning -EOPNOTSUPP until implemented; I/O is wired up in P2.
      • Non-FILE scopes require O_DIRECT. LL_IOC_FLR_SET_MIRROR is kept as an alias for MIRROR + ios_mirror_id. The fd stores the whole struct ll_ioc_io_scope in fd_io_scope.

      4.2 Kernel

      The fd stores struct ll_ioc_io_scope as fd_io_scope. ll_io_set_scope() copies it into ci_designated_* at I/O init:

      • MIRROR sets ci_designated_mirror
      • STRIPE sets ci_designated_comp (lsme_id) and ci_designated_stripe (stored as idx + 1; 0 = unset)
      • On FLR, STRIPE derives ci_designated_mirror from mirror_id_of(ios_comp_id)

      Identity filtering stays on ci_designated_* plus lov_io_comp_wanted() / lov_io_stripe_wanted() / lov_io_designated_pos_check(). The file→stripe algorithm is unchanged.

      STRIPE is not a resync: it must not set LU_LAYOUT_RESYNC.

      No new cl_io_type: when ci_obj_offset is set, lov_io_init_composite() attaches &lov_obj_io_ops instead of &lov_io_ops.

      Only one LOV callback is rewritten for the OBJECT path. Every other difference is a short-circuit (early-return) branch that performs the minimum object-coordinate work and returns.

      The file-offset helpers (lov_io_rw_iter_init(), lov_io_iter_init(), lov_offset.c) are not modified.

      • Rewrite — only cio_iter_init. Existing iterator callbacks split a file-offset range across components and stripes. OBJECT wants one sub-io, identity lis_pos / lis_endpos, and ci_continue = 0. It therefore uses lov_obj_io_iter_init rather than adding branches to lov_io_rw_iter_init() / lov_io_iter_init(). Writes are range-checked per rule 2 of §3.1. P2 covers READ/LSEEK; P3 covers WRITE/DV/FSYNC. Except for the dedicated short-circuit callbacks listed below, the remaining cio_{} keep pointing at the existing lov_io_.
      • Short-circuit branches — any existing path that would map pos as a file offset, cross components, or update the file's i_size returns after the OBJECT-specific identity work. For a cio_{}{}, lov_obj_io_ops points at a dedicated callback. Where that table cannot select the call (layout llo_, before the table is attached, or VVP), the existing function gets a branch.
      Function OBJECT branch does
      lov_obj_io_lseek_end Return an object offset, never crossing components
      lov_obj_io_lru_reserve Reserve on the single object lis_obj_*; do not call lov_stripe_intersects()
      lov_lock_sub_init() One identity sub-lock on lis_obj_{} (llo_, still LLT_COMP)
      lov_page_init_composite() / lov_dio_pages_init_composite() suboff = offset
      lov_io_slice_init() Before the table is attached, resolve (comp_id, idx) into lis_obj_*. Do not set LU_LAYOUT_RESYNC; that bit is only for designated resync writes.
      ll_direct_IO() / vvp_prep_size() / lseek EOF from the target object's kms, not the file's i_size
      vvp_io_write_start() Do not update the file's i_size from an object offset

      EOF has three cases:

      • OBJECT → kms of the target object (glimpse if kms is invalid or pos > kms)
      • ci_parity_io → ci_parity_eof
      • otherwise the file's i_size

      OBJECT does not set ci_parity_io and does not use ci_parity_eof (that is the projection of designated parity into file space, a different space).

      The read path must not go through generic_file_read_iter(), or a DIO short-read fallback would pull object pages into the file page cache.

      4.3 Userspace

      • llapi_io_scope_set/clear. llapi_mirror_set/clear keep their signatures and still issue LL_IOC_FLR_SET_MIRROR; the kernel handles that ioctl as an alias for MIRROR + ios_mirror_id.
      • Target resolution: by (comp_id, idx), by FID, or by OST.
      • OBJECT-scope pread / lseek (P2) and pwrite (P3, prefix hole [0, obj(e_start)) only). Use the llapi_io_object_* prefix (alongside llapi_io_scope_set, matching LL_IO_SCOPE_OBJECT).

      4.4 Where each consumer lands

      Single-stripe migration (LU-20534)

      • Source: today userspace converts file offsets to object offsets → becomes SEEK_DATA + pread on the object
      • Destination: identity pwrite on a volatile file with stripe_count = 1 + single-stripe swap (unchanged)

      EC rebuild

      • Source: recovery read by file offset (automatically entering CIT_EC_RD)
      • Destination: identity write to a volatile file + single-stripe swap

      EC resync / verify

      • Source: still operates on raidset rows; I3 may switch the read side to object offsets
      • Destination: file offset + designated / CIT_EC_WR (unchanged)

      EC dump

      • Source: linear address → (idx, object offset); geometry stays in userspace

      PFL/DoM merging (LU-13612)

      • Destination: client writes the prefix hole by object offset or MDS OUT_WRITE. Both options stand, and both need the MDS layout change.

      5. Implementation phases

      P1–P3 are core implementation phases; I1–I4 are consumer-integration phases.

      P1 ──► P2 ──► P3 ──► I4 client (LU-13612 prefix write)
       │      │
       │      ├──► I1  migration copy source uses object read
       │      └──► I3  dump/verify reads use object offset (may defer)
       └──► I2  EC rebuild (file-offset recovery; no object I/O)
      
      MDS layout change ──► I4 client
                        └──► I4 MDS OUT_WRITE
      
      Phase Content Depends on Notes
      P1 ioctl; FILE / MIRROR / STRIPE; OBJECT reserved — Land with LU-20534 or EC
      P2 OBJECT read + lseek P1  
      P3 Prefix-hole write P2  
      I1 Migration copy source uses object read P2 LU-20534 / LU-20541: move one stripe; dest + swap unchanged
      I2 File-offset recovery + volatile write + swap P1 EC rebuild; no object I/O
      I3 dump/verify reads use object offset P2 LU-20690; may defer
      I4 Prefix-hole write or MDS OUT_WRITE P3 / MDS LU-13612; LU-10177 / LU-11421 already landed as whole-file fallback

      Ordering principles: identity before offset space; reads before writes; replacing an object always goes through single-stripe swap; object-offset writes are only opened up for the prefix hole.

      I3 may be deferred independently; whether P2 itself slips depends on the I1 / I4 schedule. Neither P2 nor P3 is removed from the model.


      Appendix A. Rationale

      A.1 Why object offsets are needed

      Migration and rebuild write a new OST object: a plain pwrite on a volatile file with stripe_count = 1 (an identity mapping) followed by single-stripe swap is enough.

      The destination side does not need object-offset writes on the parent file (LU-20534 lands as it stands).

      What file offsets cannot reach, and therefore still requires object offsets:

      • Writes (LU-13612): only the DoM / previous component is touched; the next object stays in place. The prefix hole on that object does not contribute to i_size and that component's file-offset path cannot reach it, so it has to be written by object offset, or via MDS OUT_WRITE (see §4.4).
      • Reads (the source side of a single-stripe copy): on a healthy object this becomes lseek(SEEK_DATA) + pread, so userspace no longer has to convert file offsets into object offsets itself.

      Verify / dump are not strictly required: the geometry stays in userspace (raidset rows, linear address → (row, raidset, idx) → parity object offset).

      Object reads only replace the final hop, which today additionally has to be folded back into a RAID10 file offset to perform designated I/O. The user-visible dump remains a linear address.

      A.2 Why object-offset mode is O_DIRECT only

      The client page cache indexes pages by file offset. An object offset is not a file offset.

      If object offset 0 of two different stripes were cached as file page 0, they would overwrite each other, and the prefix hole has no corresponding page in the file at all.

      So object mode does not use the page cache and only supports O_DIRECT. Non-FILE scopes of SET_IO_SCOPE (and the SET_MIRROR alias) likewise force O_DIRECT.

      If buffered object I/O is ever wanted, it has to be done on the OST object's own pages, which is out of scope here.

            wc-triage WC Triage
            kxu Keguang Xu
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: