https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1528 Lustre <p><a href="https://maloo.whamcloud.com/test_sets/5dca0c26-7543-11e1-bfc7-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/5dca0c26-7543-11e1-bfc7-5254004bbbd3</a></p> <p>The test set shows up as passing in maloo, yet it clearly has a failed test set. As this was the only test set in the test session the test session also erroneously showed up as passed.</p> LU-1528

Test set with a failing subtest is flagged as passed in maloo

Bug Blocker Resolved Fixed Minh Diep Mike Stok Fri, 23 Mar 2012 20:56:32 +0000 Wed, 18 Jul 2012 00:14:00 +0000 Wed, 18 Jul 2012 00:14:00 +0000 Lustre 2.3.0 Lustre 2.3.0 0 2 <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@maloo current]# RAILS_ENV=production bundle exec rails c Loading production environment (Rails 3.1.4) irb(main):001:0&gt; test_set = TestSet.find('5dca0c26-7543-11e1-bfc7-5254004bbbd3') TestSet Load (8.5ms) SELECT `test_sets`.* FROM `test_sets` WHERE `test_sets`.`id` = ? ORDER BY test_sets.submission DESC LIMIT 1 [["id", "5dca0c26-7543-11e1-bfc7-5254004bbbd3"]] =&gt; #&lt;TestSet id: "5dca0c26-7543-11e1-bfc7-5254004bbbd3", test_session_id: "5b0bedce-7543-11e1-bfc7-5254004bbbd3", version: 2, created_at: "2012-03-23 23:53:19", updated_at: "2012-03-23 23:53:19", duration: 1801, submission: "2012-03-23 22:46:02", status: "PASS", test_set_script_id: "f60ca966-8b52-11e0-aab9-52540025f9af", sub_tests_count: 1, sub_tests_passed_count: 0, sub_tests_skipped_count: 0, sub_tests_failed_count: 1&gt; irb(main):002:0&gt; </pre> </div></div> <p>Note that the <tt>status</tt> is <tt>PASS</tt>, yet the <tt>sub_tests_failed_count</tt> is <tt>1</tt>.</p> <p>Looking at the uploaded <tt>results.yml</tt> we see (leading line numbers added):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 1 TestGroup: 2 test_group: acc-sm-client-13vm1 3 testhost: client-13vm1 4 submission: Fri Mar 23 15:46:02 PDT 2012 5 user_name: root 6 7 Tests: 8 - 9 name: posix 10 description: auster posix 11 submission: Fri Mar 23 15:46:02 PDT 2012 12 report_version: 2 13 SubTests: 14 - 15 name: test_1 16 status: FAIL 17 duration: 1793 18 return_code: 0 19 error: "Run\ POSIX\ testsuite\ on\ /mnt/lustre\ failed" 20 duration: 1801 21 status: PASS </pre> </div></div> <p>Maloo seems to have mechanically recorded the <tt>FAIL</tt> from line 16 for the subtest and the <tt>PASS</tt> from line 21 for the test.</p> <p>We could make maloo check for this, and it seems the test harness is not getting it right.</p> <p>There's a similar problem with vbr in failover.</p> <p>Really a sub task of <a href="https://jira.whamcloud.com/browse/LU-1192" title="Make Autotest support POSIX Compliance Testing" class="issue-link" data-issue-key="LU-1192"><del>LU-1192</del></a> but I don't want to make this an LU issue.</p> <p>The log above is from posix test which hasn't landed in lustre yet. we'll revisit this when we hit it again in autotest</p> <p>This is definitely being seen in real testing. I found the following two cases when searching for lnet-selftest results, and it was obvious that "0/1" tests passed should not be considered a "PASS" result. Looking at the tests results shows the subtest reporting failure:</p> <p><a href="https://maloo.whamcloud.com/test_sets/5b53e0a0-b479-11e1-bdae-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/5b53e0a0-b479-11e1-bdae-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/42ad0e1a-b3ce-11e1-8808-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/42ad0e1a-b3ce-11e1-8808-52540035b04c</a></p> <p>This is fairly serious (hence increased priority), because it casts doubt on all of our pass results.</p> <p>I tracked the second report from above and here is the result from yaml file</p> <p>-<br/> name: lnet-selftest<br/> description: auster lnet-selftest<br/> submission: Mon Jun 11 06:49:27 PDT 2012<br/> report_version: 2<br/> SubTests:<br/> -<br/> name: test_smoke<br/> status: FAIL<br/> duration: 345<br/> return_code: 254<br/> error: "test_smoke\ returned\ 254"<br/> duration: 357<br/> status: PASS</p> <p>the overal test status should be failed. Notice that the second subtest never started nor completed.</p> <p>moved to LU since this is likely to be test-framework issue</p> <p>patch <a href="http://review.whamcloud.com/#change,3112" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3112</a></p> <p>landed in master</p> Development Project Rank 1|hzus13: Rank (Obsolete) 2203 Severity