[LU-4805] Test failure sanity test_230c: stat /mnt/lustre/d230c.sanity/f5 failed

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.6.0
• Fix Version/s: Lustre 2.6.0
• Type: Bug
• Priority: Blocker
• Reporter: Maloo
• Assignee: Di Wang
• Resolution: Fixed
• Labels: dne
• Severity: 3

Description

This issue was created by maloo for Nathaniel Clark <[e-mail]>

This issue relates to the following test suite run:
http://maloo.whamcloud.com/test_sets/fb88e8cc-b161-11e3-a823-52540035b04c
https://maloo.whamcloud.com/test_sets/e1eb27ec-b178-11e3-9a4b-52540035b04c

The sub-test test_230c failed with the following error:

stat /mnt/lustre/d230c.sanity/f5 failed

Info required for matching: sanity 230c

Issue Links

• is related to: LU-4835 Test failure on test suite sanityn
• is related to: LU-4936 sanity test_230c: 0 != 10 during migr...

Comments

[Jodi Levi (Inactive)]

Di,
Could you please comment on this ticket to indicate what priority you think it should be?
Thank you!

[Jian Yu]

More instance on master branch with MDSCOUNT=4:
https://maloo.whamcloud.com/test_sets/0c4a8282-b465-11e3-bdd1-52540035b04c

[Jian Yu]

More instance on master branch:
https://maloo.whamcloud.com/test_sets/648d2190-b46d-11e3-bdd1-52540035b04c

sanityn was also affected by this failure:
https://maloo.whamcloud.com/test_sets/fada0724-b46f-11e3-bdd1-52540035b04c

[John Hammond]

https://maloo.whamcloud.com/test_sets/28f58bcc-b4ea-11e3-b2ed-52540035b04c

[Di Wang]

http://review.whamcloud.com/9806

[Bob Glossman (Inactive)]

maloo is currently reporting a VERY high failure rate on this:

Failure Rate: 98.00% of last 100 executions [all branches]

suspect it's blocking a lot of tests.

another:
https://maloo.whamcloud.com/test_sets/c7acb7c6-b8b5-11e3-a578-52540035b04c

[Peter Jones]

Di

Any suggestions on this one?

Thanks

Peter

[Peter Jones]

Sorry I missed the patch already listed above

[Nathaniel Clark]

Patch 9806 may also fix LU-4835

[Jodi Levi (Inactive)]

Patch landed to Master.
Nathaniel, are you able to confirm if this fixes LU-4835 as you suggested?
Thank you!

[Jodi Levi (Inactive)]

This is still being hit regularly on Master.

[Di Wang]

hmm, it seems all failures above happened before 9806 is landed. Could some one post link of failure. And I also check maloo, and did not find this problem in the recent run.

[Nathaniel Clark]

review-dne-part-1 on master:
https://maloo.whamcloud.com/test_sets/943f6a6c-d117-11e3-ae99-52540035b04c
https://maloo.whamcloud.com/test_sets/97c012a8-cc9a-11e3-bda1-52540035b04c
https://maloo.whamcloud.com/test_sets/1a74ec9e-ca81-11e3-9211-52540035b04c
https://maloo.whamcloud.com/test_sets/c05cca3c-c860-11e3-9a8f-52540035b04c

[Andreas Dilger]

Di,
it does look like most of the failures in the past two weeks are actually from patches that were based on a commit before the change 9806 landing (which was on 04-01), in particular a bunch of LFSCK test runs and DLC test runs.

There was one test failure that looked similar in one of your own patches: http://review.whamcloud.com/10041

It might be that this patch is re-introducing the same bug, or it could be a different bug with the same failure symptoms. Probably better to open a new bug for the 10041 failure and close this one.

[Di Wang]

Yes, I checked those failures, and according to the debug log, it seems they are not being tested with 9806. 10041 has different issue, I will update the patch. In the meantime, I will close this ticket for now. Please re-open the ticket, if this failures happened again. Thanks.