-
Improvement
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
The autotest rolling-upgrade/rolling-downgrade sessions run on lustre-master and lustre-master-next builds are the last sessions to finish, about 21h after every other session of the build. They are an autotest construct, not test scripts of that name.
Each "rolling-*" Maloo session is one step of a 9-step chain run back-to-back on the same cluster (same test host and queue idx):
upgrade-init -> upgrade-oss -> upgrade-mds -> upgrade-client1 -> upgrade-client2 -> downgrade-client2 -> downgrade-client1 -> downgrade-mds -> downgrade-oss
Every step reprovisions the nodes (median 16 min), runs lustre-initialization (2 min) and then the full sanity suite (872 subtests, median 222 min), so one chain takes 9 x ~4h: median 37-38h, up to 62h. Chains are started about 2h after the build starts, so queue priority is not the issue; the chain length is.
Data from 4000 lustre-master/lustre-master-next sessions over 2026-10-23..2026-11-13, 9 builds with complete chains (medians):
rolling chains start after build start 1.9h all other sessions of the build finished 26.8h last rolling chain finished 49.2h extra time added by rolling chains only +21.4h
Variants per build cover b2_15 and b2_16 as the old version, ldiskfs and zfs, and several new-version distros; the clean-upgrade/clean-downgrade chains (3 steps, ~1.4h) are not a problem.
Options to shorten the critical path (autotest test-group configuration):
- Split each chain into an upgrade chain and a downgrade chain on separate clusters, run in parallel. The downgrade chain starts with old-version init + clean upgrade of all nodes at once (~0.9h, as clean-upgrade does), then the 3 downgrade steps. Longest chain ~37h -> ~16h, below the other sessions; costs ~1h of extra test time per chain.
- Do not run full sanity in the all-old upgrade-init step: the old version is already tested on its own branch, the step only needs to format and create data. Saves ~3.7h per chain.
- Run full sanity only in the states that matter for deployments (e.g. new servers with old clients, old servers with new clients, and the final state) and a 30-60 min interop subset in the transient mixed states (oss-only, mds-only, one client upgraded). Running the subset everywhere would bring a chain from ~37h to ~12h, with some loss of coverage.
- Run with only a single client, which avoids the rolling-upgrade/downgrade-client2 step entirely and cuts ~8h from the chain, since client2 isn't really used by sanity at all
- Do not run sessions with SLOW=yes, since the extra coverage adds only a small number of subtests that take a long time to run
Options 1+2 alone would bring the chains to ~16h and the build completion from ~49h to ~27h.
The step failures (sanity test_77c, 120e/f, 56xk, 66, 160a/i, 807, mostly already ticketed interop issues) do not stop the chain and are not the cause of the run time.
Related to LU-20869 (reduce run time of long-running tests).
- is related to
-
LU-18041 struct obd_uuid is too small for MGC"NID" uuid with IPv6
-
- Open
-
-
LU-19942 sanity test_119p: matching packet too big leads to timeout (interop)
-
- Open
-
-
LU-19357 sanity test_120f: FAIL: 10 cancel RPC occured
-
- Open
-
-
LU-20780 interop: sanity test_277: expected mb 1 got 2
-
- Open
-
-
LU-17827 sanity test_116a: Failed with "test_116a returned 1"
-
- Open
-
-
LU-18312 interop: sanity test_123ac: FAIL: statx -c "%n %i %A %h %u %g %W %X %Z" -D 100000 files is slower with statahead!
-
- Open
-
-
LU-19725 sanity test_119f: packet is too big, leading to timeout
-
- In Progress
-
-
LU-17047 interop rolling-downgrade-client2 sanity test_77c: no checksum dump file on OSS
-
- Reopened
-
-
LU-15469 Automate Lustre Upgrade/Downgrade Testing
-
- Resolved
-
-
LU-20869 reduce run time of long-running tests
-
- Reopened
-