-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
[Claude] sanity test_51d fails reproducibly with more than 4 MDTs: the first (stripecount=3) pass finds a large, repeatable OST object imbalance.
sanity test_51d: @@@@@@ FAIL: stripecount=3: OST 1 has fewer #0 objects vs. OST 0 (50 < 257 x 4/6) sanity test_51d: @@@@@@ FAIL: stripecount=3: OST 1 has fewer objects vs. OST 0 (169 < 375 x 4/5)
Seen 28 times between 2024-09 and 2026-03 in the many-MDT custom sessions of https://review.whamcloud.com/56384 (LU-14058 sanity.sh MDT names), with 6, 12, 16, 24, 40 and 48 MDTs and 8 OSTs: about 28 of ~50 such runs. The same numbers recur from run to run. On review-dne-part-1 (4 MDTs) the failure rate is ~0.1% (LU-19093).
Object counts from the 12-MDT run (test directory is a striped dir -i3 -c2 -H crush2, 800 files):
OST0 has 376 objects, 257 are index 0 OST1 has 375 objects, 50 are index 0 OST2 has 376 objects, 52 are index 0 OST3 has 170 objects, 52 are index 0 OST4 has 376 objects, 257 are index 0 ... OST7 has 169 objects, 52 are index 0
Two OSTs four apart get most of the first stripes, and the OSTs before them get about 45% fewer objects, so this is a deterministic round-robin pattern rather than precreate starvation.
The part of test_51d that depends on MDSCOUNT is the precreate trigger, which creates MDSCOUNT*2 "setstripe -c -1 wide.$i" files in the test directory before the measured pass. With 12 MDTs that is 24 full-width files on the two dir-stripe MDTs, each of which advances that MDT's round-robin state (lqr_offset_idx) before the measurement, and the wide files are also counted in the per-OST totals (824 first objects for 800 files).
Possible fixes: create the wide precreate files in a separate directory or use a fixed number of them, and/or the LU-8933 round-robin balance changes (https://review.whamcloud.com/69305).
sanity test_51d is currently excluded by 56384 with MDSCOUNT > 4 (under LU-9) and by https://review.whamcloud.com/69271 with OSTCOUNT > 12 (under LU-8933); the 56384 exception should reference this ticket instead.