https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2524 Lustre <p>Following tests depend on results of previously executed tests and cannot be executed separately. That's a regression from 2.2.</p> <p>sanity.2a<br/> sanity.3b<br/> sanity.4b<br/> sanity.5<br/> sanity.6g<br/> sanity.76<br/> sanity.7a<br/> sanity.7b<br/> sanity.8<br/> sanity.9<br/> sanity.10<br/> sanity.11<br/> sanity.12<br/> sanity.13<br/> sanity.15<br/> sanity.27i<br/> sanity.51a<br/> sanity.56a<br/> sanity.101b</p> <p>sanityn.1b<br/> sanityn.1c</p> <p>Typical log:<br/> chmod: cannot access `/mnt/lustre/d0.sanity/d5/d2': No such file or directory<br/> Can't lstat /mnt/lustre/d0.sanity/d5/d2: No such file or directory</p> 1node virtual clustrer, Xperior framework. LU-2524

Tests regressions: tests interrelation introduced.

Bug Blocker Closed Won't Fix James Nunez Alexander Lezhoev patch tests Wed, 26 Dec 2012 05:17:40 +0000 Sat, 9 Oct 2021 06:08:37 +0000 Sun, 14 Aug 2016 17:21:31 +0000 Lustre 2.4.0 Lustre 2.6.0 2 12 <p>sanity test_2a:<br/> test_mkdir $DIR/$tdir failed if $DIR wasn't created before. test_mkdir -p must be used.</p> <p>sanity test_3b:<br/> if [ ! -d $DIR/$tdir ]; then<br/> mkdir $DIR/$tdir<br/> fi<br/> failed if $DIR wasn't created before. mkdir -p must be used.</p> <p>test_4b() {<br/> if [ ! -d $DIR/$tdir ]; then<br/> test_mkdir $DIR/$tdir<br/> fi<br/> failed if $DIR wasn't created before. mkdir -p must be used.</p> <p>test_5() {<br/> test_mkdir $DIR/$tdir<br/> failed if $DIR wasn't created before. mkdir -p must be used.</p> <p>...</p> <p>test_56a() { # was test_56<br/> rm -rf $DIR/$tdir<br/> $SETSTRIPE -d $DIR<br/> test_mkdir $DIR/$tdir<br/> failed if $DIR wasn't created before. mkdir -p must be used. This test depends on test_52b</p> <p>As you are able to test this most easily, please submit a patch that resolves these issues for your test framework.</p> <p>Hi Andreas,<br/> Could you please help to understand why such complicated logic required for so simple test scenario?</p> <blockquote> <p>commit 13b269ab77daca78fdfc374986d4cc34c7e66309<br/> Author: wangdi &lt;di.wang@whamcloud.com&gt;<br/> Date: Wed Nov 14 12:02:43 2012 -0800</p> <p> <a href="https://jira.whamcloud.com/browse/LU-1187" title="DNE Phase 1: Remote Directories" class="issue-link" data-issue-key="LU-1187"><del>LU-1187</del></a> tests: Add test_mkdir in sanity for DNE.</p> <p> 1. Replace mkdir with test_mkdir, which will create local or<br/> remote directory according to its testnum, i.e. choose the<br/> mdt index by testnum % mdt_count, then create directory<br/> by lfs mkdir.</p> </blockquote> <p>The sanityn failings are related to Xyratex MRP-335, that we haven't yet landed to wc main stream.<br/> The test sanity.76, sanity.101b look not related to the issue.</p> <p>Note regarding all the rest failing tests. The problem is not that $DIR is not created, the actual problem is that<br/> $tdir variable in fact isn't directory name, but piece of path, so because following instruction is used everywhere:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">test_mkdir $DIR/$tdir </pre> </div></div> <p>and even if the $DIR exists the $tdir can't be created, see test-framework.sh line:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">export tdir=d0.${TESTSUITE}/d${base} </pre> </div></div> <p>I think it make a confusion to assign <em>path</em> to "constant" variable with name dir.<br/> In many places in tests scripts it is logically to assume that <b>tdir</b> is a dir name that is why it is easy to make a mistake. In general, when you type a command "test_mkdir $DIR/$tdir" you expect there will be only one operation.</p> <p>One option is to have test_mkdir default to "mkdir -p" internally, which will ensure that the subdirectories are always created, but I don't know if this will have any side-effect for other tests. I expect it will not.</p> <p>Another simple option is to default <tt>DIR=$MOUNT/d0.${TESTSUITE</tt>} then <tt>tdir=d.${base</tt>} and <tt>tfile=f.${testnum</tt>}. This would move all of the test files into a subdirectory, but it might also have some side-effects for tests that expect to be run in the root directory. The benefit would be that <tt>$tdir</tt> and <tt>$tfile</tt> are both "one level" names and "mkdir -p" is not needed in all of the callers, only when $DIR is first created.</p> <p>Finally, just using <tt>test_mkdir -p $DIR/$tdir</tt> directly would have minimal complexity, but does not improve the situation for the future.</p> <p>Making <em>test_mkdir</em> default to recursive creation of path may disallow for some specific scenario, that may be used further or used somewhere already, for example, we may want to check <tt>mkdir -p</tt> fails correctly while trying to create directory by not existing path.</p> <p>Regarding second option, I wonder why the <em>tdir</em> must not be in the root directory? It can be for example: </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">tdir=d.${TESTSUITE}.${base} </pre> </div></div> <p>I still have a question why the <em>test_mkdir</em> function setting stripes in such a pseudo-random way:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">test_nume % mdt_count </pre> </div></div> <p>I'm not sure that this code because it make test scenario opaque. If we want to test striping then we need special purpose tests.</p> <p>I think the original reason for using tdir=d0.${TESTSUITE} is that it puts the tests into a separate subdirectory for each test script (e.g. so that some existing "d0" with 100k files in it did not cause another test to fail during <tt>rm -rf $DIR/<span class="error">&#91;Rdfs&#93;</span><span class="error">&#91;0-9&#93;</span><b></tt> at the start of sanity.sh and sanityn.sh). However, looking at this again, since this is deleting all <tt>d<span class="error">&#91;0-9&#93;</span></b></tt> directories at the start, it would also delete <tt>d0.replay-single</tt> and such, so I suspect the extra level of indirection is not helpful.</p> <p>Looking though the <tt>git/cvs annotate</tt> commit history for test-framework.sh is always useful to find more information about why code is written in a particular way. This returns commit <tt>9975e4daa, b=13979</tt> and before that <tt>067e8dc6, b=13798</tt> has the original change (<a href="https://bugzilla.lustre.org/show_bug.cgi?id=13798#c21" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=13798#c21</a>). It seems that the original goal was to add the test name into the directory name to make it unique between test scripts, and the implementation to put it in a <em>subdirectory</em> was not the important part.</p> <blockquote> <p>I would like to comment also that tdir/tfile stuff seems needs some modification (not sure that it makes sense to discuss this here, please let me know if I have to file a new bug):</p> <p>a) Currently $tdir $tfile stuff does not provide a possibility to work each test<br/> on each own directory because sanity test_1 has the same $tdir as replay-single<br/> test_1 and sanityN test_1, etc. If this $tdir stuff was added to t-f with the<br/> goal to provide the unique dir for each test &#8211; then run_one should be changed to<br/> export unique dir name contained $TESTSUITE at the name.</p></blockquote> <p>Please submit a patch that changes to <tt>tdir=d0.${TESTSUITE}.${base</tt>} and see whether it can pass testing or not. It could then also remove all of the "-p" arguments from tests that no longer need that, which will probably speed up the tests a small amount as well.</p> <p>As for test_mkdir(), it was recently added for testing multiple MDTs in a variety of ways more easily for DNE. This is only pseudo-random, so it is definitely repeatable for the same test configuration (i.e. with the same <tt>mdt_count</tt>), and I don't think it will hurt anything.</p> <p>test_mkdir does not assume there will be single or multiple dirs needs to be created, if there are more than one dirs needs to be created, -p is definitely needed here, i.e. test_mkdir -p needs to be used. It seems some of the test_mkdir(in sanity.sh) misses -p indeed. </p> <p>"I'm not sure that this code because it make test scenario opaque. If we want to test striping then we need special purpose tests."</p> <p>If you want to test special striping, you probably need add special test for it, i.e. using lfs mkdir -i directly, instead of using test_mkdir. </p> <p>Posted patch for review:<br/> <a href="http://review.whamcloud.com/5022" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5022</a></p> <p>Andreas,<br/> after this fix we have following <tt>run_one</tt> function:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">run_one() { local testnum=$1 local message=$2 tfile=f0.${TESTSUITE}.${testnum} export tdir=d0.${TESTSUITE}.${base} export TESTNAME=test_$testnum local SAVE_UMASK=`umask` umask 0022 banner <span class="code-quote">"test $testnum: $message"</span> test_${testnum} || error <span class="code-quote">"test_$testnum failed with $?"</span> cd $SAVE_PWD reset_fail_loc check_grant ${testnum} || error <span class="code-quote">"check_grant $testnum failed with $?"</span> check_catastrophe || error <span class="code-quote">"LBUG/LASSERT detected"</span> <span class="code-keyword">if</span> [ <span class="code-quote">"$PARALLEL"</span> != <span class="code-quote">"yes"</span> ]; then ps auxww | grep -v grep | grep -q multiop &amp;&amp; error <span class="code-quote">"multiop still running"</span> fi unset TESTNAME unset tdir umask $SAVE_UMASK <span class="code-keyword">return</span> 0 } </pre> </div></div> <p>What is interesting for me is:</p> <ol> <li>Why <em>tfile</em> is not exported as <em>tdir</em> and <em>TESTNAME</em> are.</li> <li>Why <em>tfile</em> and <em>tdir</em> are lowercase if they are kind of global variables.</li> <li>The <em>tdir</em> has following value: <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> export tdir=d0.${TESTSUITE}.${base} </pre> </div></div> <p>Where <em>$base</em> is numerical id "stripped of" the letters. In this case, for example, two tests 31a and 31b with the same base 31 will work within the same working directory and will mess their working stuff.</p></li> <li>Both <em>tfile</em> and <em>tdir</em> have "magic" prefixes: <tt>f0</tt> and <tt>d0</tt> correspondingly. Why not just assign values to them as follows: <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">tfile=f${testnum}.${TESTSUITE} tdir=d${testnum}.${TESTSUITE} </pre> </div></div> <p>In this case the code </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">rm -rf $DIR/[Rdfs][0-9]*</pre> </div></div> <p> should be still working and each test case will have its unique testing directory. Notice, this will cause many failures probably, because we didn't refactor all tests that are related between each other.</p></li> </ol> <p>Kyr,</p> <ul class="alternate" type="square"> <li>I'm not sure why $tfile is not exported. Perhaps because it was only thought to be relevant to the test being run under run_one()? In any case, I'm also not fond of the inconsistency here.</li> <li>there is not much consistency for upper and lower case for environment variables in the tests. I'd prefer to just keep them lower case, since it would just case a LOT of churn in the tests for very little value if they were changed to uppercase.</li> <li>I think the $tdir being the same subdir was done on purpose so that different subtests would operate on the same files, and only the first one would be set up independently. I think we have moved away from this to some extent, but some tests definitely depend on this behaviour. Other tests (e.g. "lfs find") use it to avoid having to recreate the test files for each subtest. In any case, it cannot be changed until after all such tests are fixed.</li> <li>I'm happy if you want to remove the magic prefix and instead go to only having the $TESTNAME suffix. I don't think this will cause many failures, definitely fewer than changing $tdir to be unique per subtest.</li> </ul> <p>Since each of these changes is potentially large, please submit them as separate patches as needed. I think the #3 change could be made first as part of your current patch, then the "-p" removal, then at the end the $tdir separation, and ideally a separate patch fixing any remaining tests that hard-code the dir/file names instead of using $tdir and $tfile. </p> <p>Similar effort to combine tests that depend on each other: <a href="https://jira.whamcloud.com/browse/LU-3514" title="sanity tests 27c, 77c, 77e, 77h are skipped when previous dependent test is not ran" class="issue-link" data-issue-key="LU-3514"><del>LU-3514</del></a></p> <p>Although many of the tests and test numbers have changed, there are still interdependent many tests in out test scripts. In sanity, the following tests depend on results of previously executed tests and will fail when executed separately: 6g, 7a, 7b, 8, 9, 10, 11, 12, 13, 15, 16, 24a, 24b, 24c, 24d, 56a. In many tests, the results of test_mkdir are not checked and some of these tests would fail on the mkdir instead of on a later call if a check on mkdir was in place. For example sanity tests 14, 50, 72b. </p> <p>I’d like to move this patch/work ahead by submitting the following series of patches that, I think, accomplish what is done in the current patch and in the comments posted here. </p> <p>Patch 1 Change $tdir to a single directory with the proposed change to the name format d0.${TESTSUITE}/d${base} to d${testnum}.${TESTSUITE}. This makes the “-p” option for (test_)mkdir unnecessary in most cases. Thus, we can remove the “-p” option where possible and add the “-p” option where it is more efficient as in tests 24r, s, t, 31i and 31j. Also, check return codes of test_mkdir and have an useful error message for all calls to error(). </p> <p>Patch 2: Where possible, change test file names and directories to $tfile and $tdir, respectively. Modify tfile with the proposed change to name format ; f.${TESTSUITE}.${testnum} to f${testnum}.${TESTSUITE} . Also, export the $tfile variable.</p> <p>Patch 3: Combine tests that are directly dependent on each other. For example sanity tests 26d and 26e, 25a and 25b, 34a and 34c, 34a and 34c, 34a and 34c, sanityn 1b and 1c.</p> <p>I am open to splitting or combining the above proposed patches or modifications to the work done in them if someone feels strongly about it.</p> <p>James, I'm all in favour of this. It needs someone to move the existing patch forward. </p> <p>Patch #1 posted for review <a href="http://review.whamcloud.com/#/c/5022" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/5022</a></p> <p>Since the patch above was large and touched so many test scripts, I've broken it up in the hopes that autotest will pass and the patch can be landed. There will be a small series of patches with each one modifying a small set of test scripts instead of trying to modify all the test scripts at once. The first patch makes the proposed modifications to $tdir and $tfile and makes the proposed modifications to conf-sanity; remove the "-p" on mkdir when possible, add error messages to calls to error and changes some calls to return with calls to error. Once this patch lands, all the other test scripts can be modified with these changes (improvements?):</p> <p><a href="http://review.whamcloud.com/#/c/8123/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8123/</a></p> <p>The modification to $tdir and $tfile in patch <a href="http://review.whamcloud.com/#/c/8123/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8123/</a> have landed to master. </p> <p>I'm leaving this ticket open since there is more work to be done that is described here.</p> <p>The patch 8123 introduced a bunch of regressions for tests that depend on another test (the so called subseris - with letters.)</p> <p>The poster example is sanity test_51b and test_51ba<br/> In the past tdir would be </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>export tdir=d0.${TESTSUITE}/d${base} </pre> </div></div> <p>with the new patch it is </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>export tdir=d${testnum}.${TESTSUITE} </pre> </div></div> <p>The test_51b crates a huge bunch of dirs and then test_51ba used to delete all of them, but not anymore as they live in separate dirs now.<br/> There are more tests like this.<br/> The end result is (small test) MDS fills up real fast early on and then tests start to fail for no good reason.</p> <p>This needs to be addressed ASAP!</p> <p>It might be enough to change tdir to use "<tt>d$base</tt>" instead of "<tt>d$testnum</tt>". That would get it back to the original config where the subtests shared the same dirs. I don't know if that is good or bad, probably a but if both. </p> <p>Since sanity and other tests have several subtests that are dependent on each other, these subtests cannot be run separately. If they cannot be run alone, then it makes sense to combine the subtests. One of the past versions of this patch combined dependent subtests. So, instead of changing the tdir variable, shouldn't we either combine dependent tests or make them not depend on each other?</p> <p>EDIT: If combining tests is not desirable, then any dependent subtests need to check that the previous test was run.</p> <p>I guess at least in the case of test_51b and test_51ba it was a pretty bad idea to defer a huge cleanup of current test stuff on to the next test.<br/> As such it certainly makes sense to recombine tests into bigger more self-contained ones.<br/> We probably should not do it to all of them, some discretion needs to be excercised to see how separate do the tests actually look like.</p> <p>I pushed <a href="http://review.whamcloud.com/9021" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9021</a> to fix the sanity.sh test_51b/51ba problem.</p> <p>What is reason to keep dependencies in tests while the jira tickets is about avoiding dependencies? I think good solution should be merging deeply depended tests to one.</p> <p>I've refreshed the patch at <a href="http://review.whamcloud.com/#/c/5022" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/5022</a>. </p> <p>This patch cleans up code and checks/corrects for sub test dependencies for:<br/> conf-sanity.sh<br/> dne-sanity.sh<br/> functions.sh<br/> sanity-quota.sh<br/> sanity-scrub.sh</p> <p>I'll do the same code clean up for other tests in the next patch.</p> <p>enable conf-sanity/58 on ZFS<br/> <a href="http://review.whamcloud.com/9675" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9675</a></p> <p>I've broken up previous patches to modify just one test suite per patch. The patch cleaning up conf-sanity is at <a href="http://review.whamcloud.com/10530" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10530</a></p> <p>More patches will follow.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/10530/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10530/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2524" title="Tests regressions: tests interrelation introduced." class="issue-link" data-issue-key="LU-2524"><del>LU-2524</del></a> test: Code clean up for conf-sanity<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 648c73b50abdeeb8c6d646a7b743d748c84451d6</p> <p>James Nunez (james.a.nunez@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13170" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13170</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2524" title="Tests regressions: tests interrelation introduced." class="issue-link" data-issue-key="LU-2524"><del>LU-2524</del></a> test: Test suite clean up for replay-single<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f78ba34a6334cb5997cbdaf9b005882507aab4ea</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13170/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13170/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2524" title="Tests regressions: tests interrelation introduced." class="issue-link" data-issue-key="LU-2524"><del>LU-2524</del></a> test: Test suite clean up for replay-single<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0444a40d9838b868092c78d3bdd4c7c3a00199e6</p> <p>James Nunez (james.a.nunez@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/14680" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14680</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2524" title="Tests regressions: tests interrelation introduced." class="issue-link" data-issue-key="LU-2524"><del>LU-2524</del></a> test: Clean up sanity-quota<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: dfc1b0f4ceebd580b74c1701725d778185461f4c</p> <p>James Nunez (james.a.nunez@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/14685" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14685</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2524" title="Tests regressions: tests interrelation introduced." class="issue-link" data-issue-key="LU-2524"><del>LU-2524</del></a> test: Clean up replay-ost-single test suite<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ead20ce593e2b6b1e85a0898ed05013430189f41</p> <p>Andreas Dilger (andreas.dilger@intel.com) merged in patch <a href="http://review.whamcloud.com/14685/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14685/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2524" title="Tests regressions: tests interrelation introduced." class="issue-link" data-issue-key="LU-2524"><del>LU-2524</del></a> test: Clean up replay-ost-single test suite<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1d40519afde98abf22bf0e4998effc656f1cb1f0</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/14680/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14680/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2524" title="Tests regressions: tests interrelation introduced." class="issue-link" data-issue-key="LU-2524"><del>LU-2524</del></a> test: Clean up sanity-quota<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: cc623e9ff1a0b6e3ffcdfac086a7630f93e70b93</p> <p>Old blocker for unsupported version</p> Duplicate LU-2671 LU-2992 LU-3687 Related LU-3514 LU-5047 LU-4769 LU-4536 LU-4564 LU-4343 Development Rank 1|hzve7r: Rank (Obsolete) 5942 Severity