Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20869

reduce run time of long-running tests

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • 3
    • 9223372036854775807

      Timing data (Maloo finished sessions, 2026-09-03 to 2026-10-02) shows CI test time is spread thin: the 50 tests with the largest total time (runs x mean) are 22% of all test time outside the scale, performance, racer, sanity-benchmark and node-provisioning suites. No single test is more than 2%, so this is a series of small fixes aimed at tests that are slow because of how they are written.

      Top 50 tests by total test-hours (runs x mean), 2026-09-03 to 2026-10-02. Mean in minutes, total in hours, share and cumulative are of the non-excluded pool (213,538 test-hours). The 50 tests total 47,115 hours, 22.1% of the pool.

      # Suite Test Runs Mean (min) Total (h) Share Action Change
      1 sanity test_255a 14432 17.1 4122 1.9% Patch 69390
      2 sanity test_133h 14452 14.5 3499 1.6% Patch 69384
      3 lustre-initialization lustre-initialization 49017 3.9 3151 1.5% Deferred - See notes  
      4 sanity test_123ac 14462 10.1 2426 1.1% Patch 69391
      5 sanity test_123ad 14055 5.9 1390 0.7% Patch 69391
      6 lnet-selftest test_smoke 10917 7.0 1270 0.6% Deferred - See notes  
      7 sanity-dom test_cleanup 14204 5.0 1189 0.6% Deferred - See notes  
      8 recovery-small test_52 4359 16.1 1168 0.5% Patch 69383
      9 sanity test_160n 14443 4.6 1106 0.5% Patch 69389
      10 sanity test_900 14355 4.5 1075 0.5% No change - See notes  
      11 sanity test_65k 14578 4.1 996 0.5% Patch TODO  
      12 sanity test_398m 14415 3.9 938 0.4% Patch TODO  
      13 sanity-sec test_31 5873 9.3 908 0.4% Deferred - See notes  
      14 sanity test_123h 14055 3.9 901 0.4% No change - See notes  
      15 sanity-sec test_27c 6696 7.6 850 0.4% Patch 69385
      16 sanity test_133g 14452 3.5 833 0.4% Patch - merged into 133f 69388
      17 sanity test_154g 14455 3.4 827 0.4% Patch 69386
      18 sanity-sec test_27a 6840 7.3 827 0.4% Patch TODO  
      19 conf-sanity test_73c 3794 12.9 817 0.4% Patch TODO  
      20 sanity test_133f 14453 3.3 805 0.4% Patch 69388
      21 sanity-pfl test_8 3998 12.0 801 0.4% Patch 69387
      22 conf-sanity test_73e 3824 12.0 767 0.4% Patch TODO  
      23 sanity test_56wa 14235 3.1 739 0.3% Investigate, then patch  
      24 sanity test_123f 14055 3.1 716 0.3% No change - See notes  
      25 sanity-quota test_49a 5426 7.6 683 0.3% Patch TODO  
      26 conf-sanity test_103 4013 10.2 680 0.3% Not reviewed  
      27 sanity test_834 13100 3.1 665 0.3% Not reviewed  
      28 sanity test_55b 14227 2.8 661 0.3% Not reviewed  
      29 conf-sanity test_46a 4085 9.7 661 0.3% Not reviewed  
      30 sanityn test_41i 3220 12.2 655 0.3% Not reviewed  
      31 sanity-sec test_27ac 6742 5.6 624 0.3% Not reviewed  
      32 sanity-sec test_27ab 6741 5.4 609 0.3% Not reviewed  
      33 sanity-sec test_81a 6555 5.6 607 0.3% Not reviewed  
      34 sanity test_51b 12656 2.8 596 0.3% Not reviewed  
      35 sanity test_276 14439 2.5 596 0.3% Not reviewed  
      36 sanity-quota test_91 5373 6.6 593 0.3% Not reviewed  
      37 sanity test_51f 14788 2.3 564 0.3% Not reviewed  
      38 replay-dual test_20 6274 5.4 563 0.3% Not reviewed  
      39 replay-single test_70c 5779 5.8 555 0.3% Not reviewed  
      40 ost-pools test_18 4901 6.8 551 0.3% Not reviewed  
      41 runtests test_1 4053 8.1 545 0.3% Not reviewed  
      42 sanity-quota test_3b 5648 5.6 526 0.2% Not reviewed  
      43 sanity test_903 14351 2.2 524 0.2% Not reviewed  
      44 insanity test_7 5656 5.5 521 0.2% Not reviewed  
      45 sanity test_813 14355 2.2 514 0.2% Not reviewed  
      46 sanityn test_15 1642 18.5 505 0.2% Not reviewed  
      47 conf-sanity test_32a 3956 7.6 503 0.2% Not reviewed  
      48 sanity-quota test_3a 5648 5.3 500 0.2% Not reviewed  
      49 conf-sanity test_32d 3935 7.6 497 0.2% Not reviewed  
      50 conf-sanity test_76a 4025 7.4 494 0.2% Not reviewed  

      Notes

      • 3 lustre-initialization (Deferred): an Autotest setup step, not in lustre-release; it needs a different fix.
      • 6 lnet-selftest test_smoke (Deferred): the batch needs about 500 s but SLOW=no caps the run at 300 s, so an early exit saves nothing; a real saving means less coverage. It also fails often (LU-16880, LU-16876).
      • 7 sanity-dom test_cleanup (Deferred): a cleanup step, not a test body; not yet reviewed.
      • 10 sanity test_900 (No change): the full unmount and remount is the test; the 20 s fail_loc pause is a small part of the time.
      • 13 sanity-sec test_31 (Deferred): reconfigures the LNet NIDs on every node and restarts the servers, so most of the time is mount cycles that the test needs; no obvious loop to remove. Revisit if a measurement shows otherwise.
      • 14 sanity test_123h (No change): 10001 file creates with touch and the statahead scan; the time is file-system speed, not a loop that grows, and createmany cannot make its zero-padded names. One baseline run failed with an I/O error (not investigated).
      • 16, 20 sanity test_133g and test_133f (Patch): both ended with a full cleanup and setup remount (60 to 150 s) after writing junk to proc files, which resets tunables the tests damage; the remount cannot simply be dropped, so 133g was folded into 133f and one remount follows both. 133g failed in 9 of about 78,000 suite runs in 90 days; its Maloo history ends. A standalone ONLY=133f now also runs the server part.
      • 17 sanity test_154g (Patch): the cost was llapi_fid_test test20, which builds 2000+ nested directories and calls fid2path on every level.
      • 24 sanity test_123f (No change): examined with the 123 series; nothing worth changing.
      • Rows marked "Not reviewed" have not been looked at yet.

      Results

      Expected CI saving = seconds saved per run x runs per 30 days, from the Rainshadow means above; the "after" times are from local ltvm runs, so CI may differ. Percentages are of the pool (213,538 test-hours) and of the 50 tests (47,115 hours).

      Change Test Before / after (local) Saving (h/30 d) % of pool % of 50 tests
      69390 sanity 255a 1089, 2017 s / 137, 252 s about 3,200 1.5% 6.8%
      69384 sanity 133h 1462 s / 25-28 s about 3,400 1.6% 7.2%
      69383 recovery-small 52 965 s / about 246 s about 870 0.4% 1.8%
      69389 sanity 160n 757, 1057 s / 43, 27 s about 1,000 0.5% 2.1%
      69385 sanity-sec 27c 112, 110 s / 45, 40 s about 126 (floor) 0.1% 0.3%
      69388 sanity 133f + 133g 115, 145 s / 64, 64 s about 815 0.4% 1.7%
      69387 sanity-pfl 8 722 s / 74 s about 720 0.3% 1.5%
      69391 sanity 123aa-ad 123aa 35, 300 s / 60, 85 s about 1,000 (550 to 2,000) 0.5% 2.1%
      69386 sanity 154g 262-358 s / 150 s about 500 (400 to 600) 0.2% 1.1%
      Total     about 11,630 5.4% 24.7%

      Changes:

            wc-triage WC Triage
            paf0186 Patrick Farrell
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: