[LU-7673] conf-sanity test failure cause multiple tests to fail

Tue, 6 Oct 2020 03:23:55 +0000

There are several examples in conf-sanity and other test suites where a test fails and that failure causes several tests that follow to fail. This ticket is to harden tests when a previous test fails.

For example, when conf-sanity test 52 fails, test 53a also fails because the MDS is already mounted. Test 52 calls cleanup() at the end of the test and cleanup() calls stop_ost, stop_mds and unloads modules. Test 53a runs right after test 52 and calls setup(), which calls start_mds, start_ost, etc. and will return an error if any of these fail. Thus, when test 52 fails, it does not call cleanup(), all servers are will remain mounted when test 53a starts. Test 53a calls setup() and returns an error. Then test 53b starts and calls setup() and fails. Then 54a fails because the OST is still mounted. Test 54b fails because the MDT is still mounted.

One example of this cascade of errors is at
https://testing.hpdd.intel.com/test_sets/3d0a4d2c-ba9d-11e5-87b4-5254006e85c2

Another example of one test failing leading to several others to fail is at https://testing.hpdd.intel.com/test_sets/df65fd10-bad8-11e5-b3d5-5254006e85c2. In this case, test 44 failed and this caused test 45 to fail for the same reason as above; not calling cleanup() due to test failure and the next test calls setup().

There are a few ways to stop these failures. A couple of possible solutions are:
1. Insert a TRAP that would unmount all servers at the end of these tests.
2. Modify setup to return a value when start_mds, start_ost, mount_client, client_up fail. Then each individual test that calls setup() can choose if, for example, an already mounted MDS is a reason to fail the test or not and continue the test.

Whamcloud Community JIRA

[LU-7673] conf-sanity test failure cause multiple tests to fail