[LU-7673] conf-sanity test failure cause multiple tests to fail

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.8.0
• Type: Improvement
• Priority: Minor
• Reporter: James Nunez (Inactive)
• Assignee: WC Triage
• Resolution: Unresolved
• Labels: tests

Description

There are several examples in conf-sanity and other test suites where a test fails and that failure causes several tests that follow to fail. This ticket is to harden tests when a previous test fails.

For example, when conf-sanity test 52 fails, test 53a also fails because the MDS is already mounted. Test 52 calls cleanup() at the end of the test and cleanup() calls stop_ost, stop_mds and unloads modules. Test 53a runs right after test 52 and calls setup(), which calls start_mds, start_ost, etc. and will return an error if any of these fail. Thus, when test 52 fails, it does not call cleanup(), all servers are will remain mounted when test 53a starts. Test 53a calls setup() and returns an error. Then test 53b starts and calls setup() and fails. Then 54a fails because the OST is still mounted. Test 54b fails because the MDT is still mounted.

One example of this cascade of errors is at
https://testing.hpdd.intel.com/test_sets/3d0a4d2c-ba9d-11e5-87b4-5254006e85c2

Another example of one test failing leading to several others to fail is at https://testing.hpdd.intel.com/test_sets/df65fd10-bad8-11e5-b3d5-5254006e85c2. In this case, test 44 failed and this caused test 45 to fail for the same reason as above; not calling cleanup() due to test failure and the next test calls setup().

There are a few ways to stop these failures. A couple of possible solutions are:
1. Insert a TRAP that would unmount all servers at the end of these tests.
2. Modify setup to return a value when start_mds, start_ost, mount_client, client_up fail. Then each individual test that calls setup() can choose if, for example, an already mounted MDS is a reason to fail the test or not and continue the test.

Comments

[James Nunez (Inactive)]

Looks like same issue when conf-sanity test 22 fails, test 23a fails because the MDT is already mounted. Logs at
2016-01-26 16:38:56 - https://testing.hpdd.intel.com/test_sets/0285473a-c47d-11e5-8866-5254006e85c2

[Saurabh Tandan (Inactive)]

Another instance found for interop tag 2.7.66 - EL7 Server/2.7.1 Client, build# 3316
https://testing.hpdd.intel.com/test_sets/c3a8632c-cc91-11e5-b80c-5254006e85c2

Another instance found for interop tag 2.7.66 - EL6.7 Server/2.7.1 Client, build# 3316
https://testing.hpdd.intel.com/test_sets/55d7aa40-cc98-11e5-b80c-5254006e85c2

Another instance found for interop tag 2.7.66 - EL6.7 Server/2.5.5 Client, build# 3316
https://testing.hpdd.intel.com/test_sets/bdea5946-cc9f-11e5-963e-5254006e85c2

Another instance found for interop tag 2.7.66 - EL7 Server/2.5.5 Client, build# 3316
https://testing.hpdd.intel.com/test_sets/79a03aac-cc46-11e5-901d-5254006e85c2

[Saurabh Tandan (Inactive)]

Another instance found for interop - EL7 Server/2.7.1 Client, tag 2.7.90.
https://testing.hpdd.intel.com/test_sessions/495aabae-d306-11e5-be5c-5254006e85c2
Another instance found for interop - EL6.7 Server/2.7.1 Client, tag 2.7.90.
https://testing.hpdd.intel.com/test_sessions/42ace612-d560-11e5-9cc2-5254006e85c2
Another instance found for interop - EL6.7 Server/2.5.5 Client, tag 2.7.90.
https://testing.hpdd.intel.com/test_sessions/f99a2d60-d567-11e5-bc47-5254006e85c2
Another instance found for interop - EL7 Server/2.5.5 Client, tag 2.7.90.
https://testing.hpdd.intel.com/test_sessions/93baffee-d2ae-11e5-8697-5254006e85c2

[James Nunez (Inactive)]

When conf-sanity test 20 fails, tests 21*, 22 and 23a typically also fail. These tests need to be more resilient to previous failures.

See https://testing.hpdd.intel.com/test_sets/afcbd7ba-79d6-11e6-b058-5254006e85c2 for one example of this cascade of failures.