[LU-2602] test: recovery-small 24b: test IGNORED and test log badly corrupted

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.0, Lustre 2.1.3
• Fix Version/s: Lustre 2.4.0
• Type: Bug
• Priority: Minor
• Reporter: Jay Lan (Inactive)
• Assignee: Bob Glossman (Inactive)
• Resolution: Cannot Reproduce

Environment

Server: 2.1.3-1nasS, centos 6.3, 2.6.32_279.2.1.el6
Client: 2.1.3-SP2_1nasC_ofed154, sles11sp2, 3.0.51-0.7.9.1.20130104.
1 mds (S337), 2 oss (S361, S362), 2 clients (S331, S332)

• Severity: 3

Description

This test is IGNORED.
recovery-small test_24b: @@@@@@ IGNORE (bzmultiop didn't fail fsync: 5 or close: 0):
...

Is this result expected?
Also, the test log was badly corrupted.

Test logs tarball is attached.

Comments

[Peter Jones]

Bob

Could you please look into this one?

Thanks

Peter

[Bob Glossman (Inactive)]

Jay, I'm a little confused. You say your client is 2.1.3, but as far as I can see the 2.1.3 version of recovery-small doesn't even have test 24b. It only has test_24.

[Bob Glossman (Inactive)]

I'm guessing that the bad log comes from the error_ignore function being called with the wrong number of arguments. It's supposed to be 'error_ignore bug-number error-string', but instead it seems to be just 'error_ignore string' in test 24b. This could also account for weird stuff in the log.

I suspect this isn't seen a lot as the multiop command usually fails as it's supposed to in this test.

To check this out could you just edit your recovery-small.sh script, add any number (12345 will do) as the first argument to the error_ignore line near the end of test_24b, run the test again and see what happens?

[Jay Lan (Inactive)]

Bob, I hit this problem with lustre client 2.3.0-2nasC, sles11sp2, 3.0.42_0.7.3 also.

Please ignore the 2.1.3-SP2_1nasC_ofed154 client for sles11sp2. It is not a supported version.

[Bob Glossman (Inactive)]

That makes more sense. definitely a test 24b in 2.3 and newer versions of recovery-small. looks like the bogus error_ignore call is there too.

[Bob Glossman (Inactive)]

Sorry, spoke too soon. Just checked the tree. The 2.3 release version of the script still only has test 24. Only newer builds like current master will have 24b. The relevant change went in ~10/30/2012

[Jay Lan (Inactive)]

Bob, add '12345' as the first argument did not help. However, if I change '12345' to '5494', the number used in 24a, then I got a clean log. I guess probably 12345 not a valid bug number?

[Jay Lan (Inactive)]

My git repo is at https://github.com/jlan/lustre-nas,
with nas-2.3.0 branch and 2.3.0-2nasC tag. It is newer than wc-2.1.3 release by cherry-picking some commits from b2_1 branch.

[Bob Glossman (Inactive)]

Thanks for going the extra mile. I think you have verified that the form of the error_ignore call is causing the problem. Knowing that I can do a 1 line fix.

Not sure why it didn't like 12345. At first blush it seems like any number would be OK. There are examples of error_ignore calls with other 5 digit numbers in various scripts.

[Jay Lan (Inactive)]

Blame on commit cb102ce, which I picked up while picking up fix for LU-921 we requested.

[Bob Glossman (Inactive)]

fix in http://review.whamcloud.com/4995

[Sarah Liu]

Hit this issue in lustre-master tag-2.3.59 ofd build testing
https://maloo.whamcloud.com/test_sets/c092e582-62c6-11e2-982f-52540035b04c

server/client lustre-master build# 1176 ofd build

[Jay Lan (Inactive)]

Was the failure reported by Sarah Liu reproducible? It not, it is not caused by commit checked in on http://review.whamcloud.com/4995.

If this did not cause side effect, can we close this ticket?

[Bob Glossman (Inactive)]

Jay, Looking back in the failure history of this subtest I don't see any similar failures since 1/26. I'm willing to close the ticket if it's OK with you.

[Jay Lan (Inactive)]

Yes, let's close it. Thanks!

[Jinshan Xiong (Inactive)]

Test case 24b should be excluded for ldiskfs as well. It has tons of race and timing issue to make it fail. For example, the evict can happen before processes write the data; and "dirty page discard" can be skipped from dmesg occasionally; and the write can succeed before eviction if there is no grant on client because it will be finished with SYNC write.

[John Fuchs-Chesney (Inactive)]

Customer indicated that we can 'close' this issue.
~ jfc.