Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-21043

sanity test_51d: OST object imbalance with more than 4 MDTs 'OST 1 has fewer #0 objects vs. OST 0 (50 < 257 x 4/6)'

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      [Claude] sanity test_51d fails reproducibly with more than 4 MDTs: the first (stripecount=3) pass finds a large, repeatable OST object imbalance.

      sanity test_51d: @@@@@@ FAIL: stripecount=3:  OST 1 has fewer #0 objects vs. OST 0  (50 < 257 x 4/6)
      sanity test_51d: @@@@@@ FAIL: stripecount=3:  OST 1 has fewer objects vs. OST 0  (169 < 375 x 4/5)
      

      Seen 28 times between 2024-09 and 2026-03 in the many-MDT custom sessions of https://review.whamcloud.com/56384 (LU-14058 sanity.sh MDT names), with 6, 12, 16, 24, 40 and 48 MDTs and 8 OSTs: about 28 of ~50 such runs. The same numbers recur from run to run. On review-dne-part-1 (4 MDTs) the failure rate is ~0.1% (LU-19093).

      Object counts from the 12-MDT run (test directory is a striped dir -i3 -c2 -H crush2, 800 files):

      OST0 has 376 objects, 257 are index 0
      OST1 has 375 objects, 50 are index 0
      OST2 has 376 objects, 52 are index 0
      OST3 has 170 objects, 52 are index 0
      OST4 has 376 objects, 257 are index 0
      ...
      OST7 has 169 objects, 52 are index 0
      

      Two OSTs four apart get most of the first stripes, and the OSTs before them get about 45% fewer objects, so this is a deterministic round-robin pattern rather than precreate starvation.

      The part of test_51d that depends on MDSCOUNT is the precreate trigger, which creates MDSCOUNT*2 "setstripe -c -1 wide.$i" files in the test directory before the measured pass. With 12 MDTs that is 24 full-width files on the two dir-stripe MDTs, each of which advances that MDT's round-robin state (lqr_offset_idx) before the measurement, and the wide files are also counted in the per-OST totals (824 first objects for 800 files).

      Possible fixes: create the wide precreate files in a separate directory or use a fixed number of them, and/or the LU-8933 round-robin balance changes (https://review.whamcloud.com/69305).

      sanity test_51d is currently excluded by 56384 with MDSCOUNT > 4 (under LU-9) and by https://review.whamcloud.com/69271 with OSTCOUNT > 12 (under LU-8933); the 56384 exception should reference this ticket instead.

            wc-triage WC Triage
            maloo Maloo
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: