[LU-9661] sanity test_57b: MDC before 382704 != after 382680

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.10.0
• Type: Bug
• Priority: Minor
• Reporter: Maloo
• Assignee: Jian Yu
• Resolution: Duplicate

Environment

Servers - 2.9.58, 3591, master
Client - 2.9.0, build 22, b2_9

• Severity: 3

Description

This issue was created by maloo for Saurabh Tandan <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sessions/f37f95be-4af5-11e7-b400-5254006e85c2.

The sub-test test_57b failed with the following error:

MDC before 382704 != after 382680

test log:

== sanity test 57b: default LOV EAs are stored inside large inodes =================================== 01:33:33 (1496712813)
mcreating 100 files
total: 100 creates in 0.07 seconds: 1478.94 creates/second
Filesystem            1K-blocks   Used Available Use% Mounted on
10.2.4.49@tcp:/lustre    991128 493192    429096  54% /mnt/lustre
opening files to create objects/EAs
Filesystem            1K-blocks   Used Available Use% Mounted on
10.2.4.49@tcp:/lustre    991128 493192    429096  54% /mnt/lustre
 sanity test_57b: @@@@@@ FAIL: MDC before 382704 != after 382680 

This issue is seen only during rolling upgrade/downgrade testing. Checked results for interop testing but this issue didn't occured.

Other results with same issue.
https://testing.hpdd.intel.com/test_sessions/6729ebe2-4af9-11e7-91f4-5254006e85c2
https://testing.hpdd.intel.com/sub_tests/ebfa335c-4b05-11e7-bc6c-5254006e85c2

Issue Links

• duplicates: LU-9677 Checking OST counts in test suites wh...
• is related to: LU-4109 sanity test_57b failure: 'MDC before ...
• is related to: LU-9677 Checking OST counts in test suites wh...

Comments

[Peter Jones]

Jian

can you please investigate?

Thanks

Peter

[Jian Yu]

Hi Saurabh,
I saw the configuration was 1 MGT/MDT, 1 OST and 1 Client. Did the same test pass before during rolling upgrade testing with the same configuration?
Many tests required at least 2 OSTs. Some of them have codes to check the OST count, and some do not have such codes, which will fail with 1 OST (e.g., sanity test 27F failed with 1 OST) instead of skipping the test.

[Saurabh Tandan (Inactive)]

Yes, the same test did pass before for rolling upgrade/downgrade testing with the same config. Here are the results for the last successful run:
https://testing.hpdd.intel.com/test_sessions/977f2f34-1e29-11e7-9de9-5254006e85c2

[Jian Yu]

Thank you Saurabh for the info. The above passed session used physical server nodes instead of vm nodes and had much larger server target device size. Is there any successful test session also using vm server nodes?
If yes, I'm going to reproduce the failure and find out the root cause.

[Jian Yu]

BTW, could you please run the tests with at least 2 OSTs in the future so as to avoid some configuration test failures?

[Saurabh Tandan (Inactive)]

I don't have them on vms and yes you are right may be that can be the issue. And I will make sure I use 2 OSTs .

[Jian Yu]

Thank you Saurabh.
Could you please re-run the tests with 2 OSTs and larger device size? (many tests in the test session failed with out of space issue)

[Saurabh Tandan (Inactive)]

Yeah sure, I will do that.
Thanks for looking into the issues.

[Sarah Liu]

We should improve the test scripts with checking OST count to avoid this kind of failure. I will open a ticket- LU-9677

[Jian Yu]

Thank you, Sarah.

[Jian Yu]

The test script will be improved in LU-9677.