Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-21209

autotest: shorten rolling-upgrade/downgrade chains on master builds

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      The autotest rolling-upgrade/rolling-downgrade sessions run on lustre-master and lustre-master-next builds are the last sessions to finish, about 21h after every other session of the build. They are an autotest construct, not test scripts of that name.

      Each "rolling-*" Maloo session is one step of a 9-step chain run back-to-back on the same cluster (same test host and queue idx):

      upgrade-init -> upgrade-oss -> upgrade-mds -> upgrade-client1 -> upgrade-client2
        -> downgrade-client2 -> downgrade-client1 -> downgrade-mds -> downgrade-oss
      

      Every step reprovisions the nodes (median 16 min), runs lustre-initialization (2 min) and then the full sanity suite (872 subtests, median 222 min), so one chain takes 9 x ~4h: median 37-38h, up to 62h. Chains are started about 2h after the build starts, so queue priority is not the issue; the chain length is.

      Data from 4000 lustre-master/lustre-master-next sessions over 2026-10-23..2026-11-13, 9 builds with complete chains (medians):

      rolling chains start after build start       1.9h
      all other sessions of the build finished    26.8h
      last rolling chain finished                 49.2h
      extra time added by rolling chains only    +21.4h
      

      Variants per build cover b2_15 and b2_16 as the old version, ldiskfs and zfs, and several new-version distros; the clean-upgrade/clean-downgrade chains (3 steps, ~1.4h) are not a problem.

      Options to shorten the critical path (autotest test-group configuration):

      1. Split each chain into an upgrade chain and a downgrade chain on separate clusters, run in parallel. The downgrade chain starts with old-version init + clean upgrade of all nodes at once (~0.9h, as clean-upgrade does), then the 3 downgrade steps. Longest chain ~37h -> ~16h, below the other sessions; costs ~1h of extra test time per chain.
      2. Do not run full sanity in the all-old upgrade-init step: the old version is already tested on its own branch, the step only needs to format and create data. Saves ~3.7h per chain.
      3. Run full sanity only in the states that matter for deployments (e.g. new servers with old clients, old servers with new clients, and the final state) and a 30-60 min interop subset in the transient mixed states (oss-only, mds-only, one client upgraded). Running the subset everywhere would bring a chain from ~37h to ~12h, with some loss of coverage.
      4. Run with only a single client, which avoids the rolling-upgrade/downgrade-client2 step entirely and cuts ~8h from the chain, since client2 isn't really used by sanity at all
      5. Do not run sessions with SLOW=yes, since the extra coverage adds only a small number of subtests that take a long time to run

      Options 1+2 alone would bring the chains to ~16h and the build completion from ~49h to ~27h.

      The step failures (sanity test_77c, 120e/f, 56xk, 66, 160a/i, 807, mostly already ticketed interop issues) do not stop the chain and are not the cause of the run time.

      Related to LU-20869 (reduce run time of long-running tests).

            colmstea Charlie Olmstead
            adilger Andreas Dilger
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: