[LU-7103] ost-pools test_7a: test failed to respond and timed out

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.8.0
• Affects Version/s: Lustre 2.8.0
• Environment: client and server: lustre-master build # 3167 RHEL7.1
• Severity: 3
• Assignee: Bob Glossman (Inactive)
• Reporter: Maloo

Description

This issue was created by maloo for sarah_lw <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/30ff2db8-5288-11e5-920d-5254006e85c2.

The sub-test test_7a failed with the following error:

test failed to respond and timed out

test hang but cannot find useful information

Issue Links

• is duplicated by: LU-7208 ost-pools test_7a: test failed to respond and timed out
• is related to: LU-6234 lfs computes pool name length incorrectly

Comments

[Joseph Gmitter (Inactive)]

Landed for 2.8.0

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/17350/
Subject: LU-7103 test: avoid cat of /dev/urandom
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: cddbef5fae44d09756639c4cd9de62f28f5b34cf

[Gerrit Updater]

Bob Glossman ([e-mail]) uploaded a new patch: http://review.whamcloud.com/17350
Subject: LU-7103 test: avoid cat of /dev/urandom
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 72bf3691eb56e3b187b73ec1bb4a173feab81dd1

[Andreas Dilger]

I would suspect that cat /dev/urandom is getting blocked for some reason in the VM. Please add a patch that removes the use of this and instead use something like echo $$$RANDOM$RANDOM which should generate a string between 3 and 15 characters long that can be cut shorter if needed.

[Zhenyu Xu]

The debug patch hit two instances of the failure (https://testing.hpdd.intel.com/test_sessions/3eeb4454-6bb8-11e5-b85e-5254006e85c2), while only client2 test log shows following info

== ost-pools test 7a: create various pool name ======================================================= 22:18:26 (1444083506)
'[' 8 -lt 2 ']'
mkdir -p /mnt/lustre/d7a.ost-pools
for i in 1 9 15
cat /dev/urandom
tr -dc a-zA-Z0-9
fold -w 1
head -n 1

No other logs show any trace of the shell command echo, don't understand it.

[Zhenyu Xu]

Bob,

I meant other TIMEOUT failure of ost-pools tests, since test_7a was added in LU-6234 patch, so you would not find test_7a failure before that time.

[Bob Glossman (Inactive)]

bobijam, you say in an earlier comment that there are failures from before the LU-6234 patch landed. I can't find any. Searching https://testing.hpdd.intel.com/sub_tests/query?commit=Update+results&page=2&sub_test%5Bquery_bugs%5D=&sub_test%5Bstatus%5D=TIMEOUT&sub_test%5Bsub_test_script_id%5D=e1c3310c-da20-11e4-b0f3-5254006e85c2&test_node%5Barchitecture_type_id%5D=&test_node%5Bdistribution_type_id%5D=&test_node%5Bfile_system_type_id%5D=&test_node%5Blustre_branch_id%5D=&test_node%5Bos_type_id%5D=&test_node_network%5Bnetwork_type_id%5D=&test_session%5Bquery_date%5D=&test_session%5Bquery_recent_period%5D=&test_session%5Btest_group%5D=&test_session%5Btest_host%5D=&test_session%5Buser_id%5D=&test_set%5Btest_set_script_id%5D=6bea3250-3db2-11e0-80c0-52540025f9af&utf8=✓

reports the earliest instance as 8/20. That is after the LU-6234 patch landed.

[Zhenyu Xu]

I've pushed the verbose echo shell debug patch at http://review.whamcloud.com/16676

[Gerrit Updater]

Bobi Jam ([e-mail]) uploaded a new patch: http://review.whamcloud.com/16676
Subject: LU-7103 debug: add verbose shell echo
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: bb1f19082b2a227daa7eee3ba2840b84e479e3b0

[Andreas Dilger]

Bobijam, I'm not so sure I agree that this timeout wasn't added by test_7a itself. There are a bunch of test_11 timeouts from 2015-07-25 to 2015-08-05, but those all related to a single patch (version 3 of http://review.whamcloud.com/15767) which was no longer hit for any of the later versions of that patch.