[LU-5079] conf-sanity test_47 timeout

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.7.0, Lustre 2.5.4
• Affects Version/s: Lustre 2.6.0
• Labels: llnl patch
• Severity: 3
• Assignee: Jian Yu
• Reporter: Maloo

Description

This issue was created by maloo for Nathaniel Clark <[e-mail]>

This issue relates to the following test suite run:
http://maloo.whamcloud.com/test_sets/7f09a2f6-dd9d-11e3-9262-52540035b04c
https://maloo.whamcloud.com/test_sets/99ea9712-dc88-11e3-9450-52540035b04c

The sub-test test_47 failed with the following error:

test failed to respond and timed out

Info required for matching: conf-sanity 47

Issue Links

• is duplicated by: LU-5773 obdfilter-survey test 1c: oom occurred on OSS
• is related to: LU-5077 insanity test_1: out of memory on MDT in crypto_create_tfm()
• is related to: LU-5900 replay-dual test_11: rm: cannot remove `/mnt/lustre/f11.replay-dual-[1-5]': No such file or directory
• is related to: LU-5901 replay-dual test_15a: import is not in FULL state
• is related to: LU-5902 replay-dual test_20: FAIL: recovery time is growing 215 > 107
• is related to: LU-5803 This server is not able to keep up with request traffic
• is related to: LU-5724 IR recovery doesn't behave properly with Lustre 2.5
• is related to: LU-5077 insanity test_1: out of memory on MDT in crypto_create_tfm()
• is related to: LU-5805 tgt_recov blocked and "waking for gap in transno"
• is related to: LU-4578 Early replies do not honor at_max
• is related to: LU-5358 Failure on test suite replay-vbr test_7e

Comments

[Andreas Dilger]

Alexander, could you please take a look at LU-5900, LU-5901, LU-5902 failures? This looks to be caused by the http://review.whamcloud.com/11213 patch backported to b2_5 (http://review.whamcloud.com/12365). Is this something that is just causing the tests to fail that won't affect real users, or is there something bad in b2_5 that will cause recovery problems for users also?

[Jian Yu]

The patches caused regression failures LU-5900, LU-5901 and LU-5902 on master and b2_5 branches.

[Peter Jones]

It has landed to master now

[Andreas Dilger]

Reopen this bug until the replay-vbr test patch has been landed to master, otherwise it will not be tracked properly.

[Jian Yu]

Here is the patch for master branch to speed up replay-vbr test 7*: http://review.whamcloud.com/12490
And here is the test result: https://testing.hpdd.intel.com/test_sets/0ae4ab72-5fcb-11e4-895a-5254006e85c2

With the above patch, total run time for replay-vbr test 7* was reduced from 18796s to 4742s.

[Jodi Levi (Inactive)]

Patch landed to Master.

[Jian Yu]

patch http://review.whamcloud.com/#/c/11213/

I tried to manually run replay-vbr test and found it finally passed and took about 8 hours:
https://testing.hpdd.intel.com/test_sets/18fb6cc0-5eb1-11e4-a2a3-5254006e85c2

Among the sub-tests, test 7e took 3876s, which exceeded the 3600s timeout value set by autotest system. That was why the test was stopped in autotest runs.

[Christopher Morrone (Inactive)]

I tried http://review.whamcloud.com/12365 on top of LLNL's 2.5.3-1chaos tag, and saw lots of problems. I don't know if they were actually cause by it or not. Experience is recorded in LU-5805.

[Jian Yu]

patch http://review.whamcloud.com/#/c/11213/

The above patch introduced regression failure in replay-vbr test 7e.

On master branch:
https://testing.hpdd.intel.com/test_sets/92fe44e0-5b80-11e4-a35f-5254006e85c2

On b2_5 branch:
https://testing.hpdd.intel.com/test_sets/4d02235e-59c2-11e4-aa32-5254006e85c2
https://testing.hpdd.intel.com/test_sets/25213bfe-59c2-11e4-aa32-5254006e85c2
https://testing.hpdd.intel.com/test_sets/3238224a-59bc-11e4-816e-5254006e85c2
https://testing.hpdd.intel.com/test_sets/166cc778-59bc-11e4-816e-5254006e85c2

[Jian Yu]

Here is the back-ported patch for Lustre b2_5 branch: http://review.whamcloud.com/12365

[James A Simmons]

This appears to be related to the LU-5077 problems. Since the patch for LU-4578 is in the b2_5 branch I believe this might be the source of our recovery problems. I was seeing recovery issues on 2.5 in my test bed until I applied the patch for this ticket here.