https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1792 Lustre <p>Looking at recent review test runs to see where the time is being spent, I see that conf-sanity.sh test_53a and test_53b are sometimes taking far too long to run - over 800s and 1000s respectively, sometimes twice that. They should be able to complete in a few seconds, but the remount in the middle of the test seems to take the longest time. Given that these tests run in our test environment about 20x per day, this could be wasting 10h or more of testing time each day.</p> <p>Some investigation needs to be done to see why these tests are taking so long to run. Is it that mount and/or unmount is very slow? If so, why? Simply skipping these tests for SLOW=no is not a valid solution, since slow mounting/unmounting affects all of our users and wastes even more time for every test that is run, but it is less visible when done at the start of a test run instead of in the middle.</p> <p>See <a href="https://maloo.whamcloud.com/sub_tests/c14c1bec-ef9e-11e1-bdf7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/c14c1bec-ef9e-11e1-bdf7-52540035b04c</a> and <a href="https://maloo.whamcloud.com/sub_tests/c1559e7e-ef9e-11e1-bdf7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/c1559e7e-ef9e-11e1-bdf7-52540035b04c</a> for logs.</p> LU-1792

conf-sanity.sh test_53a/test_53b take too long to run

Bug Minor Resolved Cannot Reproduce WC Triage Andreas Dilger Mon, 27 Aug 2012 18:31:54 +0000 Sat, 10 Mar 2018 08:16:30 +0000 Sat, 10 Mar 2018 08:16:30 +0000 Lustre 2.4.0 0 1 <p>As much as I hate to beat a (way) dead (and buried) horse, ltest used to time every test and keep a history of each test's run time and use those gathered timings to apply a timeout to future test runs. The result was that any individual test's time run was bounded by it's historical run-times to prevent excessive wasting of time. Of course, any test that over-ran it's historically calculated limit was considered a failure.</p> <p>Just food for thought.</p> <p>Sure, I remember. I recall there was a 10-15% margin for variability in the test. However, looking at the test results the variability is huge. This may be related to running in a VM and potentially contending for CPU or disk bandwidth?</p> <p>The shortest passing test took 115s and the longest took 1541s so any attempt to do this for the current virtual test environment wouldn't be possible. I haven't made any attempt to correlate this to branch/arch/cluster.</p> <p>Hrm. Yeah. I suppose VMing all of these test clusters could indeed introduce a lot of variability. Perhaps too much to apply any sort of "expected run time" type of watchdogs. Pity.</p> <p>I do remember that feature being a boon to preventing test clusters from spinning for many many hours on a test that had failed in an unpredictable manner.</p> <p>I wonder if it would be worth the effort for somebody to actually do the branch/arch/cluster correlation of the wildly swinging test times to see just how unpredictable it really is.</p> <p>This is an issue of why lustre is taking so long to mount/unmount/remount the filesystem and the focus should be there for this topic.</p> <p>We do have statistics for the duration of every test ever run on autotest, so if people want to mine the data they are welcome. The problem of 'correct time' is not a vm vs !vm issue because physical hardware can vary by as much between systems and network bandwidth in particular is very dependent on what else is happening on the system.</p> <p>The vm's client #24 and below probably provide very consistent times because they are a completely closed system during test with no outside influence.</p> <p>Current subtest runs are in the 125 or 250s range.</p> Development Rank 1|hzw27j: Rank (Obsolete) 10440 Severity