Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-7103] ost-pools test_7a: test failed to respond and timed out https://jira.whamcloud.com/browse/LU-7103 Lustre <p>This issue was created by maloo for sarah_lw <wei3.liu@intel.com></p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/30ff2db8-5288-11e5-920d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/30ff2db8-5288-11e5-920d-5254006e85c2</a>.</p> <p>The sub-test test_7a failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test failed to respond and timed out </pre> </div></div> <p>test hang but cannot find useful information</p> client and server: lustre-master build # 3167 RHEL7.1 LU-7103

ost-pools test_7a: test failed to respond and timed out

Bug Major Resolved Fixed Bob Glossman Maloo Fri, 4 Sep 2015 06:23:46 +0000 Wed, 2 Dec 2015 16:02:26 +0000 Wed, 2 Dec 2015 16:02:26 +0000 Lustre 2.8.0 Lustre 2.8.0 0 8 <p>Bob</p> <p>You could please try and identify which commit introduced this recent regression?</p> <p>Thanks</p> <p>Peter</p> <p>This first started failing on 2015-08-20 13:07:43 so is likely related to a patch that landed just before that time. </p> <p>I'm quite suspicious of the fix for "<a href="https://jira.whamcloud.com/browse/LU-6234" title="lfs computes pool name length incorrectly" class="issue-link" data-issue-key="LU-6234"><del>LU-6234</del></a> util: check fsname and pool name". That went in on 8/18 and added test 7a. No fails before then since the test didn't exist.</p> <p>commit 7c25eb1ba2b1db3009a0e88b3ecf229134f8ac92</p> <p>Bobijam</p> <p>Do you think that this issue could be related to the <a href="https://jira.whamcloud.com/browse/LU-6234" title="lfs computes pool name length incorrectly" class="issue-link" data-issue-key="LU-6234"><del>LU-6234</del></a> patch?</p> <p>Thanks</p> <p>Peter</p> <p>no, from <a href="https://testing.hpdd.intel.com/test_sets/query?utf8=%E2%9C%93&test_set%5Btest_set_script_id%5D=6bea3250-3db2-11e0-80c0-52540025f9af&test_set%5Bstatus%5D=TIMEOUT&test_set%5Bquery_bugs%5D=&test_session%5Btest_host%5D=&test_session%5Btest_group%5D=&test_session%5Buser_id%5D=&test_session%5Bquery_date%5D=&test_session%5Bquery_recent_period%5D=&test_node%5Bos_type_id%5D=&test_node%5Bdistribution_type_id%5D=&test_node%5Barchitecture_type_id%5D=&test_node%5Bfile_system_type_id%5D=&test_node%5Blustre_branch_id%5D=&test_node_network%5Bnetwork_type_id%5D=&commit=Update+results" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/query?utf8=%E2%9C%93&test_set%5Btest_set_script_id%5D=6bea3250-3db2-11e0-80c0-52540025f9af&test_set%5Bstatus%5D=TIMEOUT&test_set%5Bquery_bugs%5D=&test_session%5Btest_host%5D=&test_session%5Btest_group%5D=&test_session%5Buser_id%5D=&test_session%5Bquery_date%5D=&test_session%5Bquery_recent_period%5D=&test_node%5Bos_type_id%5D=&test_node%5Bdistribution_type_id%5D=&test_node%5Barchitecture_type_id%5D=&test_node%5Bfile_system_type_id%5D=&test_node%5Blustre_branch_id%5D=&test_node_network%5Bnetwork_type_id%5D=&commit=Update+results</a></p> <p>there are ost-pools test TIMEOUT failure before <a href="https://jira.whamcloud.com/browse/LU-6234" title="lfs computes pool name length incorrectly" class="issue-link" data-issue-key="LU-6234"><del>LU-6234</del></a> patch, and also has test_7a pass (<a href="https://testing.hpdd.intel.com/test_sets/02cddbde-4e9f-11e5-b0b8-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/02cddbde-4e9f-11e5-b0b8-5254006e85c2</a>), I tend to think that there is a hidden issue other than <a href="https://jira.whamcloud.com/browse/LU-6234" title="lfs computes pool name length incorrectly" class="issue-link" data-issue-key="LU-6234"><del>LU-6234</del></a> in these TIMEOUT failure.</p> <p>This is now the top failing test in autotest.</p> <p>Bob, can you please push a patch to add <tt>set -vx</tt> at the start of test_7a and <tt>set +vx</tt> at the end, and add <tt>Test-Parameters: testlist=ost-pools,ost-pools,...</tt> to run this test maybe 10 times (if you put them on two lines they would run in parallel). Then we might see what the test is doing when it is failing. The current logs are not helpful. </p> <p>Bobijam, I'm not so sure I agree that this timeout wasn't added by test_7a itself. There are a bunch of test_11 timeouts from 2015-07-25 to 2015-08-05, but those all related to a single patch (version 3 of <a href="http://review.whamcloud.com/15767" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15767</a>) which was no longer hit for any of the later versions of that patch. </p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/16676" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16676</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7103" title="ost-pools test_7a: test failed to respond and timed out" class="issue-link" data-issue-key="LU-7103"><del>LU-7103</del></a> debug: add verbose shell echo<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bb1f19082b2a227daa7eee3ba2840b84e479e3b0</p> <p>I've pushed the verbose echo shell debug patch at <a href="http://review.whamcloud.com/16676" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16676</a></p> <p>bobijam, you say in an earlier comment that there are failures from before the <a href="https://jira.whamcloud.com/browse/LU-6234" title="lfs computes pool name length incorrectly" class="issue-link" data-issue-key="LU-6234"><del>LU-6234</del></a> patch landed. I can't find any. Searching <a href="https://testing.hpdd.intel.com/sub_tests/query?commit=Update+results&page=2&sub_test%5Bquery_bugs%5D=&sub_test%5Bstatus%5D=TIMEOUT&sub_test%5Bsub_test_script_id%5D=e1c3310c-da20-11e4-b0f3-5254006e85c2&test_node%5Barchitecture_type_id%5D=&test_node%5Bdistribution_type_id%5D=&test_node%5Bfile_system_type_id%5D=&test_node%5Blustre_branch_id%5D=&test_node%5Bos_type_id%5D=&test_node_network%5Bnetwork_type_id%5D=&test_session%5Bquery_date%5D=&test_session%5Bquery_recent_period%5D=&test_session%5Btest_group%5D=&test_session%5Btest_host%5D=&test_session%5Buser_id%5D=&test_set%5Btest_set_script_id%5D=6bea3250-3db2-11e0-80c0-52540025f9af&utf8=" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/query?commit=Update+results&page=2&sub_test%5Bquery_bugs%5D=&sub_test%5Bstatus%5D=TIMEOUT&sub_test%5Bsub_test_script_id%5D=e1c3310c-da20-11e4-b0f3-5254006e85c2&test_node%5Barchitecture_type_id%5D=&test_node%5Bdistribution_type_id%5D=&test_node%5Bfile_system_type_id%5D=&test_node%5Blustre_branch_id%5D=&test_node%5Bos_type_id%5D=&test_node_network%5Bnetwork_type_id%5D=&test_session%5Bquery_date%5D=&test_session%5Bquery_recent_period%5D=&test_session%5Btest_group%5D=&test_session%5Btest_host%5D=&test_session%5Buser_id%5D=&test_set%5Btest_set_script_id%5D=6bea3250-3db2-11e0-80c0-52540025f9af&utf8=</a>✓</p> <p>reports the earliest instance as 8/20. That is after the <a href="https://jira.whamcloud.com/browse/LU-6234" title="lfs computes pool name length incorrectly" class="issue-link" data-issue-key="LU-6234"><del>LU-6234</del></a> patch landed.</p> <p>Bob,</p> <p>I meant other TIMEOUT failure of ost-pools tests, since test_7a was added in <a href="https://jira.whamcloud.com/browse/LU-6234" title="lfs computes pool name length incorrectly" class="issue-link" data-issue-key="LU-6234"><del>LU-6234</del></a> patch, so you would not find test_7a failure before that time.</p> <p>The debug patch hit two instances of the failure (<a href="https://testing.hpdd.intel.com/test_sessions/3eeb4454-6bb8-11e5-b85e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/3eeb4454-6bb8-11e5-b85e-5254006e85c2</a>), while only client2 test log shows following info</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== ost-pools test 7a: create various pool name ======================================================= 22:18:26 (1444083506) '[' 8 -lt 2 ']' mkdir -p /mnt/lustre/d7a.ost-pools for i in 1 9 15 cat /dev/urandom tr -dc a-zA-Z0-9 fold -w 1 head -n 1 </pre> </div></div> <p>No other logs show any trace of the shell command echo, don't understand it.</p> <p>I would suspect that <tt>cat /dev/urandom</tt> is getting blocked for some reason in the VM. Please add a patch that removes the use of this and instead use something like <tt>echo $$$RANDOM$RANDOM</tt> which should generate a string between 3 and 15 characters long that can be cut shorter if needed.</p> <p>Bob Glossman (bob.glossman@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17350" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17350</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7103" title="ost-pools test_7a: test failed to respond and timed out" class="issue-link" data-issue-key="LU-7103"><del>LU-7103</del></a> test: avoid cat of /dev/urandom<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 72bf3691eb56e3b187b73ec1bb4a173feab81dd1</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17350/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17350/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7103" title="ost-pools test_7a: test failed to respond and timed out" class="issue-link" data-issue-key="LU-7103"><del>LU-7103</del></a> test: avoid cat of /dev/urandom<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: cddbef5fae44d09756639c4cd9de62f28f5b34cf</p> <p>Landed for 2.8.0</p> Duplicate LU-7208 Related LU-6234 Development Rank 1|hzxmhr: Rank (Obsolete) 9223372036854775807 Severity