https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11520 Lustre <p>I already made this proposal several times that landing bugfixes in series of 2 patches of a test and a fix on top on the test is a good idea because:</p> <p> </p> <ul class="alternate" type="square"> <li>it explicitly shows you the test actually reproduces a problem, not because we trust a developer;</li> <li>you can revert only a fix at some later point and confirm the test still able to reproduce some problem;<br/>  <br/> however there were several objections:</li> <li>it will not work with git bisect;</li> <li>malloo does not (and not supposed to) set +1 if test failure occurs;<br/>  <br/> The proposal which seems to resolve these objections is to add a tag/label “mustfail” to the commit message of the test patch and the malloo is supposed to consider this particular failure as a success (as well as the successful test as a failure) and to set +1 (-1). at the same time the fix sitting on top already does not have this tag/label and therefore the test already must pass.<br/>  <br/> the proposal was discussed with Andreas o the last LAD and we agreed that it would work, thus filing the ticket for it.</li> </ul> LU-11520

separate tests and bugfixes

Improvement Minor Open Unresolved Elena Gryaznova Vitaly Fertman Mon, 15 Oct 2018 15:08:41 +0000 Thu, 25 Apr 2019 22:09:00 +0000 0 10 <p>I think this is fairly reasonable, and would work for a majority of cases, except where the test causes a node to crash or otherwise hang and not complete the test. </p> <p>Could you submit a patch pair as an example to show how this will work? I don't think there should be anything needed except e.g. <tt>Test-Parameters: envdefinitions=CONF_SANITY_MUSTFAIL=32c</tt> or similar, and machinery in test-framework to make this happen. It would need to be specific to each test script, to avoid the eventual future case where there are two different tests with the same number that have the <tt>mustfail</tt> functionality. </p> <p>Probably, something like <tt>envdefinitions=MUSTFAIL=conf-sanity/32c</tt> would be more handy. This construction can be unwrapped into internal variable like <tt>c MUSTFAIL_conf_sanity_32c</tt> to be checked before test run.</p> <blockquote> <p>I think this is fairly reasonable, and would work for a majority of cases, except where the test causes a node to crash or otherwise hang and not complete the test.</p></blockquote> <p>why? the test system while stopping the system, gathering the test results, is supposed to check if it is marked as failing and report the success.</p> <blockquote> <p>except where the test causes a node to crash or otherwise hang and not complete the test.</p></blockquote> <p>The problem is that if the client or server node crashes then the rest of that test script will be aborted. Also, the pass/fail decision is made at the test script level , there is no mechanism (AFAIK) for the external autotest environment to understand whether this specific test failure should actually be considered a pass.</p> <p>In the meantime, I think we can make progress with this and handle the majority of patches which do not induce a crash on a client or server node, and further improvements can be implemented at a later time to handle the whole-node failure case.</p> <p>the full solution seems to introduce this logic at the external level to cover those cases as well, and the t-f solution will be useless or at least redundant.</p> <p>could we estimate the amount of work needed to introduce it at the autotest environment level now, it is probably worth to follow this way immediately.</p> <p>..and by "immediately" you mean "at the start of the 2.13 release cycle", right? <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I meant just a technical approach, not planning. so no milestones <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>He ok - thanks for confirming <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>As discussed on this week's LWG call, it makes sense to implement the simple change to parse the <tt>MUSTFAIL</tt> parameter in <tt>build_test_filter()</tt> first (either as a simple list of test numbers similar to <tt>ONLY</tt>, <tt>EXCEPT</tt>, and <tt>ALWAYS_EXCEPT</tt>, or including the test name as <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=diff" class="user-hover" rel="diff">diff</a> suggests like <tt>conf-sanity/32a</tt>) to set <tt>MUSTFAIL_&lt;test_number&gt;</tt> for that script, and then checking the result in <tt>run_one_logged()</tt> or maybe <tt>pass()</tt> to consider a failure of this subtest as <tt>PASS</tt>. </p> <p>Because we run tests in multiple different configurations (e.g. ldiskfs vs. ZFS, DNE vs. none, Ubuntu vs. RHEL) and the test failures may not be applicable to all configurations, I don't think we can <b>require</b> that the test fails with <tt>MUSTFAIL</tt> set in order for it to be considered a pass. I guess one option would be to add a similar <tt>SHOULDFAIL</tt> parameter that allows a conditional failure, while <tt>MUSTFAIL</tt> should fail in all test configurations?</p> <p>Implementing this as part of the autotest framework will take more time, and since many failures are not crash/timeout failures we can still get benefits from having this support in <tt>test-framework.sh</tt>.</p> <p>MUSTFAIL can be implemented via parameter in build_test_filter(), yes (this also needs the parameter like EXPECTED_FAILURE_IS, which contains for example the particular string in stdout or in nodes logs). But this implementation does cover the cases like crashes or timeouts because t-f does not handle such cases atall and such cases can be tracked only on automation side (not t-f).</p> <p>So, I do not think that it is reasonable to split such feature into 2 parts: t-f implementation and automation implementation.</p> Related LU-11642 Development Rank 1|i0047r: Rank (Obsolete) 9223372036854775807