[LU-13773] sanity-dom test failure not triggering test suite marked as FAIL

Tue, 19 Apr 2022 16:15:58 +0000

We see, in ~~LU-13759~~, that a sanityn test that is run as part of sanity-dom can fail and the failure is not propagated to a level that Maloo recognizes it as a failure. There are several problems that are also seen with sanity-dom logs; logs for the failed tests are not collected and client log for successful tests are not collected. Note that this seems like an issue with the test framework and not with Maloo.

For failure at https://testing.whamcloud.com/test_sets/08c6fa9d-e2a3-457b-a1ed-b4318dbf166a, we can see that the sanity-dom/sanityn test 20 failure is recognized, but this does not lead to the sanity-dom test suite being marked at failed. In the results file at https://testing.whamcloud.com/test_sessions/b31ca578-dd4f-4725-9a0c-5e19f3031c69/show_results, we see that the fail is registered, but

    -   name: test_sanityn
    -   name: test_1
        status: PASS
        duration: 4
        return_code: 0
        error: 
…
    -   name: test_19
        status: SKIP
        duration: 2
        return_code: 0
        error: not cache-capable obdfilter
    -   name: test_20
        status: FAIL
        duration: 5
        return_code: 1
        error: 1 page left in cache after lock cancel
    -   name: test_23
        status: PASS
        duration: 65
        return_code: 0
        error: 
…
    -   name: test_51d
        status: PASS
        duration: 435
        return_code: 0
        error: 
    duration: 1129
    status: PASS

We had a related issue where, when a sanity-dom/sanityn test failed, the failure would trigger the whole test suite to fail, but Maloo thinks the last test in sanityn failed which is false; see ~~LU-10589~~ and one example failure at https://testing.whamcloud.com/test_sets/870c7a78-467f-11e9-9646-52540065bddc. In this case it looks like all logs are not collected.

Patrick Farrell produced a patch which clears the problem with a sub test of a sub suite failure not triggering the suite to be marked as fail at https://review.whamcloud.com/#/c/34186/ . I think the no log collected/displayed issue still exists with this proposed solution.

Whamcloud Community JIRA

[LU-13773] sanity-dom test failure not triggering test suite marked as FAIL