Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20564

Probe md RAID geometry so OST ldiskfs allocations align to full stripes

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      *Background*

      ldiskfs uses the underlying array's `stride` and `stripe_width` so that mballoc
      can align allocations to full RAID rows; without them, large writes arrive
      misaligned and the array services them as read-modify-write. mke2fs normally
      derives both from the blkid topology, which covers a `mkfs.lustre` over a bare
      md device. Two configurations fall outside what the topology can express:

      • *Stacked devices* — with dm layers between the OST and the array
          (dm-linear, lvmcache), the topology is not always propagated.
      • *Declustered arrays* — `optimal_io_size` describes the whole device pool
          rather than one data row, so a `stripe_width` derived from it is not the row
          mballoc should align to.

      Neither is signalled today: an OST formatted without usable geometry looks
      identical to one formatted with it, and the cost appears later as reduced
      write throughput.

      *Improvement*

      `mkfs.lustre` probes the md array beneath an OST through sysfs, descending
      stacked dm devices via the `slaves` links, and seeds `-E stride=/stripe_width=`
      from the array's own chunk size and data-disk count. The existing
      mke2fs-derived path is untouched wherever it already works, user-supplied
      values always win, and if several arrays are found with disagreeing geometry
      nothing is seeded.

      `mount.lustre` gains a matching advisory when an OST sits on a striped array
      whose filesystem carries no matching geometry, making the condition
      discoverable on filesystems that already exist. Misaligned partition starts are
      reported at mkfs.

      With the geometry in hand, `mkfs.lustre` can also report a data row
      (`data_disks * chunk`) that is not a power of two. Bulk RPC sizes are powers of
      two in practice, and only a power of two divides one, so a 320 KiB row (k=5 at
      64K) leaves a partial row in every RPC, at every chunk size. Unlike
      stride/stripe_width this cannot be corrected once the array exists, so it is
      reported while the geometry can still be chosen.

      *Scope*

      Userspace only, confined to `lustre/utils/libmount_utils_ldiskfs.c`. No on-disk
      format, wire protocol or kernel change.

      *Testing*

      Built with `-Wall -Werror`; 13 mkfs and 7 mount cases on a RHEL 9.7 OSS against
      live loop-backed md arrays, each cross-checked with `dumpe2fs`: raid0/4/5/6,
      raid10 near and far layouts, a non-power-of-two row, user-supplied values,
      dm-linear over an array, dm-linear spanning two arrays with conflicting
      geometry, and aligned vs misaligned partition starts.

      Declustered support is decode-verified against the array driver's layout
      definitions but has not yet been run against a live declustered array.

            hnishida Hiroshi Nishida
            hnishida Hiroshi Nishida
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: