https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4344 Lustre <p>LU context can be found at <a href="https://jira.whamcloud.com/browse/LU-764" title="test marked as fail but all subtest pass" class="issue-link" data-issue-key="LU-764"><del>LU-764</del></a> test marked "No sub tests failed in this test set.":</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> Keith Mannthey added a comment - 11/Mar/13 12:44 PM I have seen a bit of this over the last week or so on master. This is a good example. https://maloo.whamcloud.com/test_sessions/bf361f32-8919-11e2-b643-52540035b04c There was one real error at the very end end of this test. Other than that all subtests "passed" even though 3 whole sections are marked FAILED. What logs do you look at to see the cleanup issues previously mentioned? How can we tell if it is a problem with the patch or some autotest anomaly? " </pre> </div></div> <p> Minh Diep added a comment - 21/Mar/13 9:37 AM</p> <p>I looked at the log above, sanity actually ran much longer but maloo only show a few hundreds of second<br/> == sanity test complete, duration 2584 sec == 14:03:39 (1362866619)</p> <p>I don't think this is the same problem. Please file a TT ticket.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre></pre> </div></div> <p>This has happened a bit here an there. It would be nice to always know what failed when something "fails".</p> <p>Please update the LU if this is seen as an LU issue. </p> patches pushed to git LU-4344

Test marked FAIL with "No sub tests failed in this test set"

Bug Major Resolved Cannot Reproduce WC Triage Keith Mannthey support Thu, 21 Mar 2013 17:25:45 +0000 Thu, 16 Sep 2021 21:41:09 +0000 Thu, 16 Sep 2021 21:41:09 +0000 0 8 <p>Some review-zfs runs that all subtests pass, but test is marked as failed:<br/> <a href="https://maloo.whamcloud.com/test_sets/669e7926-ad38-11e2-bd7c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/669e7926-ad38-11e2-bd7c-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/b6c9e15c-ad19-11e2-bd7c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b6c9e15c-ad19-11e2-bd7c-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/424c1ea6-ad2f-11e2-bd7c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/424c1ea6-ad2f-11e2-bd7c-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/65d3f9be-ad1d-11e2-bd7c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/65d3f9be-ad1d-11e2-bd7c-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/1a56168c-ad38-11e2-9002-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/1a56168c-ad38-11e2-9002-52540035b04c</a></p> <p>Another review-zfs run (sanity-quota):<br/> <a href="https://maloo.whamcloud.com/test_sets/4704e34c-c83c-11e2-b8c5-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/4704e34c-c83c-11e2-b8c5-52540035b04c</a></p> <p>This is a totally broken ZFS run:<br/> <a href="https://maloo.whamcloud.com/test_sessions/6fcdc280-c837-11e2-8dd9-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/6fcdc280-c837-11e2-8dd9-52540035b04c</a></p> <p>For example on lnetself test:<br/> <a href="https://maloo.whamcloud.com/test_sets/9c5751b0-c83a-11e2-8dd9-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9c5751b0-c83a-11e2-8dd9-52540035b04c</a></p> <p>Duration is 376 seconds, it is marked FAIL, 0/0 subtests have ran</p> <p>There are no lustre-initialization logs to look at as well. </p> <p><a href="https://maloo.whamcloud.com/test_sessions/9917ca1e-c832-11e2-8dd9-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/9917ca1e-c832-11e2-8dd9-52540035b04c</a> is similar. </p> <p>Right so the problem here is in the Lustre test framework.</p> <p>The results.yaml file created by the results framework looks like this (abbreviated).</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Tests: - name: sanity description: auster sanity submission: Thu Jul 25 06:26:09 PDT 2013 report_version: 2 SubTests: - name: test_0a status: PASS duration: 0 return_code: 0 error: - name: test_0b status: PASS duration: 3 return_code: 0 error: status: PASS </pre> </div></div> <p>but sometimes it looks like this</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Tests: - name: sanity description: auster sanity submission: Thu Jul 25 06:26:09 PDT 2013 report_version: 2 SubTests: - name: test_0a status: PASS duration: 0 return_code: 0 error: - name: test_0b status: PASS duration: 3 return_code: 0 error: status: FAIL </pre> </div></div> <p>Which autotest/maloo faithfully reports. </p> <p>So the framework needs to be fixed up.</p> <p>Now I think we could remove the final status from the yaml and let Maloo decide if the suite passed, but while the result is their Maloo should report the result it receives.</p> <p>My guess is that it is showing a real issue.</p> <p>This needs to be fixed in the test framework so I will change this to a lustre ticket.</p> <p>I agree with Chris. Most the time there is some issue. Normally in the unmounting of filesystems or some other cleanup phase of the test.</p> <p>Perhaps a setup and cleanup phase for each subtest could catch all these extra issues in a less confusing way. </p> <p>This needs to be investigated in the Lustre code and so needs to be fixed under an LU ticket.</p> <p>Perhaps <a href="https://jira.whamcloud.com/browse/LU-764" title="test marked as fail but all subtest pass" class="issue-link" data-issue-key="LU-764"><del>LU-764</del></a> should be reopened? </p> <p>One more time it happens: <a href="https://maloo.whamcloud.com/test_sets/9618ff94-81e2-11e3-94d9-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9618ff94-81e2-11e3-94d9-52540035b04c</a></p> <p>More instances?</p> <p><a href="https://maloo.whamcloud.com/test_sets/392d7044-a794-11e3-ba84-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/392d7044-a794-11e3-ba84-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/bb8b2514-a80b-11e3-9505-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/bb8b2514-a80b-11e3-9505-52540035b04c</a></p> <p>I don't see why this failed, is this another instance?<br/> <a href="https://maloo.whamcloud.com/test_sets/9bea8fb6-b77d-11e3-98de-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9bea8fb6-b77d-11e3-98de-52540035b04c</a></p> <p>Matt, </p> <p>I think that one failed due to TEI-1403.</p> <p>one more fail: <a href="https://testing.hpdd.intel.com/test_sets/7fc6919c-3863-11e4-b7d4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7fc6919c-3863-11e4-b7d4-5254006e85c2</a></p> <p>Another failure <a href="https://testing.hpdd.intel.com/test_sets/b42e4da6-501b-11e4-8734-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b42e4da6-501b-11e4-8734-5254006e85c2</a></p> <p>It seems like many of these failures that I can find happen when some server is being unmounted. Unfortunately, it isn't possible to diagnose this remotely because these failures do not leave any server console/dmesg/debug logs on the test. I've filed TEI-2815 to try and get our test framework to capture more logs in this kind of failure situation.</p> <p>It looks like some tests that are timed out (e.g. sanity-sec test_13 or test_14 due to <a href="https://jira.whamcloud.com/browse/LU-5847" title="sanity-sec: lctl nodemap_test_nid on test_13" class="issue-link" data-issue-key="LU-5847"><del>LU-5847</del></a>) are being marked <tt>FAIL</tt>, but checking the <tt>Set-wide logs:</tt> <tt>suite_log</tt> and/or <tt>suite_stdout</tt> shows that the test was still running, but just taking too long to complete. The <tt>suite_log</tt> shows the output of the test continues even after it is marked for timeout:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>08:50:15:== sanity-sec test 13: test nids == 08:49:23 (1409672963) : : 08:50:17:CMD: onyx-35vm3 /usr/sbin/lctl nodemap_test_nid 30.0.0.1 08:50:17:CMD: onyx-35vm3 /usr/sbin/lctl nodemap_test_nid 30.0.0.2 : : 09:51:01:CMD: onyx-35vm3 /usr/sbin/lctl nodemap_test_nid 30.7.0.190 09:51:02:CMD: onyx-35vm3 /usr/sbin/lctl nodemap_test_nid 30.7.0.191 09:51:02:********** Timeout by autotest system **********09:51:03:CMD: onyx-35vm3 /usr/sbin/lctl nodemap_test_nid 30.7.0.192 09:51:03:CMD: onyx-35vm3 /usr/sbin/lctl nodemap_test_nid 30.7.0.193 09:51:04:CMD: onyx-35vm3 /usr/sbin/lctl nodemap_test_nid 30.7.0.194 : : 09:55:56:CMD: onyx-35vm3 /usr/sbin/lctl nodemap_test_nid 30.7.2.27 09:55:56:CMD: onyx-35vm3 /usr/sbin/lctl nodemap_test_nid 30.7.2.28 </pre> </div></div> <p>In a few cases, I've even see the test complete and be marked <tt>PASS</tt> in the logs after it was "timed out". In the case of sanity-sec test_13 and test_14, the tests have been shortened to run in a couple of minutes, but I think the more important issues are:</p> <ul class="alternate" type="square"> <li>the test should have been marked as <tt>TIMEOUT</tt> instead of <tt>FAIL</tt> if a subtest is timed out</li> <li>if a subtest has started it should produce an entry for that subtest in the logs, even if it did not complete (TEI-2815)</li> </ul> Related LU-5847 LU-5738 Development Rank 1|hzupkf: Rank (Obsolete) 1792 Severity