[LU-9661] sanity test_57b: MDC before 382704 != after 382680

• Type: Bug
• Resolution: Duplicate
• Priority: Minor
• Affects Version/s: Lustre 2.10.0
• Environment: Servers - 2.9.58, 3591, master; Client - 2.9.0, build 22, b2_9
• Severity: 3
• Assignee: Jian Yu
• Reporter: Maloo

Description

This issue was created by maloo for Saurabh Tandan <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sessions/f37f95be-4af5-11e7-b400-5254006e85c2.

The sub-test test_57b failed with the following error:

MDC before 382704 != after 382680

test log:

== sanity test 57b: default LOV EAs are stored inside large inodes =================================== 01:33:33 (1496712813)
mcreating 100 files
total: 100 creates in 0.07 seconds: 1478.94 creates/second
Filesystem            1K-blocks   Used Available Use% Mounted on
10.2.4.49@tcp:/lustre    991128 493192    429096  54% /mnt/lustre
opening files to create objects/EAs
Filesystem            1K-blocks   Used Available Use% Mounted on
10.2.4.49@tcp:/lustre    991128 493192    429096  54% /mnt/lustre
 sanity test_57b: @@@@@@ FAIL: MDC before 382704 != after 382680 

This issue is seen only during rolling upgrade/downgrade testing. Checked results for interop testing but this issue didn't occured.

Other results with same issue.
https://testing.hpdd.intel.com/test_sessions/6729ebe2-4af9-11e7-91f4-5254006e85c2
https://testing.hpdd.intel.com/sub_tests/ebfa335c-4b05-11e7-bc6c-5254006e85c2

Issue Links

• duplicates: LU-9677 Checking OST counts in test suites where needed
• is related to: LU-9677 Checking OST counts in test suites where needed
• is related to: LU-4109 sanity test_57b failure: 'MDC before 15214812 != after 15214800'

Comments

[Jian Yu]

The test script will be improved in LU-9677.

[Jian Yu]

Thank you, Sarah.

[Sarah Liu]

We should improve the test scripts with checking OST count to avoid this kind of failure. I will open a ticket- LU-9677

[Saurabh Tandan (Inactive)]

Yeah sure, I will do that.
Thanks for looking into the issues.

[Jian Yu]

Thank you Saurabh.
Could you please re-run the tests with 2 OSTs and larger device size? (many tests in the test session failed with out of space issue)

[Saurabh Tandan (Inactive)]

I don't have them on vms and yes you are right may be that can be the issue. And I will make sure I use 2 OSTs .

[Jian Yu]

BTW, could you please run the tests with at least 2 OSTs in the future so as to avoid some configuration test failures?

[Jian Yu]

Thank you Saurabh for the info. The above passed session used physical server nodes instead of vm nodes and had much larger server target device size. Is there any successful test session also using vm server nodes?
If yes, I'm going to reproduce the failure and find out the root cause.

[Saurabh Tandan (Inactive)]

Yes, the same test did pass before for rolling upgrade/downgrade testing with the same config. Here are the results for the last successful run:
https://testing.hpdd.intel.com/test_sessions/977f2f34-1e29-11e7-9de9-5254006e85c2

[Jian Yu]

Hi Saurabh,
I saw the configuration was 1 MGT/MDT, 1 OST and 1 Client. Did the same test pass before during rolling upgrade testing with the same configuration?
Many tests required at least 2 OSTs. Some of them have codes to check the OST count, and some do not have such codes, which will fail with 1 OST (e.g., sanity test 27F failed with 1 OST) instead of skipping the test.