https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4349 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/1d5f9fd8-5c6a-11e3-9d08-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/1d5f9fd8-5c6a-11e3-9d08-52540035b04c</a>.</p> <p>The sub-test test_47 failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>Info required for matching: conf-sanity 47</p> <p>This test took 3600s and failed with timeout, cannot find error logs. I checked<br/> a PASS conf-sanity run on zfs, the same sub test takes about 134s to complete:</p> <p><a href="https://maloo.whamcloud.com/sub_tests/b9351066-5d53-11e3-956b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/b9351066-5d53-11e3-956b-52540035b04c</a> </p> lustre-master build # 1791 RHEL6 zfs LU-4349

conf-sanity test_47: test failed to respond and timed out

Bug Blocker Resolved Fixed Mikhail Pershin Maloo Thu, 5 Dec 2013 18:22:47 +0000 Sat, 11 Oct 2014 04:00:16 +0000 Fri, 18 Apr 2014 13:09:29 +0000 Lustre 2.6.0 Lustre 2.6.0 Lustre 2.5.4 0 12 <p>also seen this error in SLES11 SP3 client:</p> <p><a href="https://maloo.whamcloud.com/test_sets/fb3c1e54-5de5-11e3-8f3c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fb3c1e54-5de5-11e3-8f3c-52540035b04c</a></p> <p>This is also timing out on ldiskfs test runs:</p> <p><a href="https://maloo.whamcloud.com/sub_tests/e2c36e9c-5eab-11e3-b298-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/e2c36e9c-5eab-11e3-b298-52540035b04c</a></p> <p>This failure looks like it was introduced by <a href="http://review.whamcloud.com/4960" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4960</a> since it first appeared on 2013-11-26 on two tests for this patch (review-zfs and review-dne-zfs so it was not flagged as a failure) and then after that patch was landed on 2013-12-02 it started failing on other test runs starting 2013-12-03. It does fail more often on ZFS than ldiskfs, but maybe only because ZFS writes take longer?</p> <p>Test failed due to recovery problems, for some reason the recovery timers grows up and never stops. I am not 100% sure why but I found problem with <a href="http://review.whamcloud.com/4960" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4960</a> that may be related. Patch is there <a href="http://review.whamcloud.com/8582" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8582</a></p> <p>Another failure instance:</p> <p><a href="https://maloo.whamcloud.com/test_sets/b089e1d4-6e89-11e3-9de0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b089e1d4-6e89-11e3-9de0-52540035b04c</a></p> <p>failed at: <a href="https://maloo.whamcloud.com/test_sessions/187b6dee-6f07-11e3-ad93-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/187b6dee-6f07-11e3-ad93-52540035b04c</a></p> <p>Patch landed to master on 2013-12-27 19:13 so please verify any failures on tests after this time do not contain this patch. </p> <p>This failed more recently than patch #8582 was landed:<br/> <a href="https://maloo.whamcloud.com/test_sets/325bb0ac-7317-11e3-92ab-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/325bb0ac-7317-11e3-92ab-52540035b04c</a></p> <p>I had opened bug <a href="https://jira.whamcloud.com/browse/LU-4426" title="Test failure on test suite conf-sanity, subtest test_47" class="issue-link" data-issue-key="LU-4426"><del>LU-4426</del></a>, I'll close that as duplicate of this one.</p> <p>Recovery timer goes around again, it seems to be stuck:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00010000:00080000:0.0:1388959703.814015:0:25116:0:(ldlm_lib.c:1010:target_handle_connect()) lustre-OST0000: connection from lustre-MDT0000-mdtlov_UUID@10.10.17.33@tcp recovering/t0 exp (null) cur 1388959703 last 0 00010000:02000400:0.0:1388959703.814020:0:25116:0:(ldlm_lib.c:1054:target_handle_connect()) lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 10.10.17.33@tcp), waiting for all 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:03 ... 00010000:00080000:0.1:1388959707.256838:0:0:0:(ldlm_lib.c:2152:target_recovery_expired()) lustre-OST0000: recovery timed out; 1 clients are still in recovery after 2076s (1 clients connected) 00010000:02000400:0.0:1388959707.256873:0:24824:0:(ldlm_lib.c:1803:target_recovery_overseer()) lustre-OST0000: recovery is timed out, evict stale exports 00000020:00000001:0.0:1388959707.256876:0:24824:0:(genops.c:1354:class_disconnect_stale_exports()) Process entered 00000020:00000001:0.0:1388959707.256879:0:24824:0:(genops.c:1281:class_disconnect_export_list()) Process entered 00000020:00000001:0.0:1388959707.256880:0:24824:0:(genops.c:1319:class_disconnect_export_list()) Process leaving 00000020:00000001:0.0:1388959707.256881:0:24824:0:(genops.c:1394:class_disconnect_stale_exports()) Process leaving 00010000:00080000:0.0:1388959707.256883:0:24824:0:(ldlm_lib.c:1607:extend_recovery_timer()) lustre-OST0000: recovery timer will expire in 30 seconds ... 00010000:00080000:0.0:1388959708.813899:0:25116:0:(ldlm_lib.c:1010:target_handle_connect()) lustre-OST0000: connection from lustre-MDT0000-mdtlov_UUID@10.10.17.33@tcp recovering/t0 exp (null) cur 1388959708 last 0 00010000:02000400:0.0:1388959708.813904:0:25116:0:(ldlm_lib.c:1054:target_handle_connect()) lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 10.10.17.33@tcp), waiting for all 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:28 </pre> </div></div> <p>Mike, any progress on this problem? This test failed 30 times in the past 4 days.</p> <p>Patch to count disconnected clients in stale counter: <a href="http://review.whamcloud.com/8785" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8785</a><br/> Not sure this is exactly the reason, but looks like stale counter is wrong after normal export disconnect comparing with eviction.</p> <p>Another instance on master branch: <a href="https://maloo.whamcloud.com/test_sets/601c8390-7bbe-11e3-8a96-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/601c8390-7bbe-11e3-8a96-52540035b04c</a></p> <p>Yu Jian,<br/> Did the latest instance happen with Mike's patch? Or do we think his patch will fix this issue once it lands?</p> <p>Hi Jodi,<br/> The latest instance happened without Mike's patch. The test results in <a href="http://review.whamcloud.com/8785" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8785</a> showed that conf-sanity test 47 passed.</p> <p>It is hard to know whether the patch fixes the problem without repeated runs, since it also passes without the patch.</p> <p>That said, there were almost 100 failures of this test in the past 4 weeks, so this is failing pretty often.</p> <p>Yu Jian,<br/> Since Change, 8785 has landed, can you verify this is fixed and close the ticket or comment?<br/> Thank you!</p> <p>By searching on Maloo, I found that after patch #8785 landed on master branch 3 days ago, conf-sanity test 47 all passed (more than 60 test runs, except several failed ones because the patches were not rebased). So, the issue was fixed. Let's close this ticket.</p> <p>I think I met the same failure with #8785 applied.</p> <p><a href="https://maloo.whamcloud.com/test_sets/fdfb4102-8355-11e3-bedf-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fdfb4102-8355-11e3-bedf-52540035b04c</a></p> <p>I don't see evidences that this is the same failure, in original bug there was recovery stuck with message like this:<br/> kernel: Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 10.10.18.247@tcp), waiting for all 5 known clients (3 recovered, 1 in progress, and 0 evicted) to recover in 0:12</p> <p>In your log there are no such messages, there is nothing useful in log actually. If there will be another test failures with recovery stuck let me know. I'd wait for more instances of this bug in master to say for sure this bug still exists, until that this looks more related to patch applied. </p> <p>Here is the summary for recent conf-sanity test_47 failure. I am not sure they are really the same as <a href="https://jira.whamcloud.com/browse/LU-4349" title="conf-sanity test_47: test failed to respond and timed out" class="issue-link" data-issue-key="LU-4349"><del>LU-4349</del></a>, but it is true that we are still trouble with conf-sanity failure. If it is not the same, I will open a new ticket.</p> <p><a href="https://maloo.whamcloud.com/sub_tests/query?utf8=" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/query?utf8=</a>✓&amp;test_set%5Btest_set_script_id%5D=7f66aa20-3db2-11e0-80c0-52540025f9af&amp;sub_test%5Bsub_test_script_id%5D=8392c2be-3db2-11e0-80c0-52540025f9af&amp;sub_test%5Bstatus%5D=TIMEOUT&amp;sub_test%5Bquery_bugs%5D=&amp;test_session%5Btest_host%5D=&amp;test_session%5Btest_group%5D=&amp;test_session%5Buser_id%5D=&amp;test_session%5Bquery_date%5D=&amp;test_session%5Bquery_recent_period%5D=604800&amp;test_node%5Bos_type_id%5D=&amp;test_node%5Bdistribution_type_id%5D=&amp;test_node%5Barchitecture_type_id%5D=&amp;test_node%5Bfile_system_type_id%5D=&amp;test_node%5Blustre_branch_id%5D=&amp;test_node_network%5Bnetwork_type_id%5D=&amp;commit=Update+results</p> <p>Hm… with the patch <a href="http://review.whamcloud.com/8997" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8997</a> (for <a href="https://jira.whamcloud.com/browse/LU-4413" title="Test failure on test suite conf-sanity, subtest test_56" class="issue-link" data-issue-key="LU-4413"><del>LU-4413</del></a>) landed to master, I did not met conf-sanity failure. So I prefer to close this bug.</p> <p>Please reopen it when hit it again in the future.</p> <p>If this was fixed by landing the <a href="https://jira.whamcloud.com/browse/LU-4413" title="Test failure on test suite conf-sanity, subtest test_56" class="issue-link" data-issue-key="LU-4413"><del>LU-4413</del></a> patch, then it should be linked to that bug. </p> <p>Andreas, while this was happened in the same test, the failure in <a href="https://jira.whamcloud.com/browse/LU-4413" title="Test failure on test suite conf-sanity, subtest test_56" class="issue-link" data-issue-key="LU-4413"><del>LU-4413</del></a> looked different than in <a href="https://jira.whamcloud.com/browse/LU-4349" title="conf-sanity test_47: test failed to respond and timed out" class="issue-link" data-issue-key="LU-4349"><del>LU-4349</del></a>. There was no stuck in recovery but after. So I doubt it is related actually, it looks more like coincidence. </p> <p>Hit this issue again during tag-2.5.56 testing in DNE mode</p> <p><a href="https://maloo.whamcloud.com/test_sets/92cc87f8-a02c-11e3-947c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/92cc87f8-a02c-11e3-947c-52540035b04c</a></p> <p>may be the same issue, seen in b2_5:<br/> <a href="https://maloo.whamcloud.com/test_sets/550c0fde-a521-11e3-9fee-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/550c0fde-a521-11e3-9fee-52540035b04c</a></p> <p>Last two issues were happened in branches without <a href="https://jira.whamcloud.com/browse/LU-4413" title="Test failure on test suite conf-sanity, subtest test_56" class="issue-link" data-issue-key="LU-4413"><del>LU-4413</del></a> fix mentioned in Fan Yong comment. We have to port <a href="http://review.whamcloud.com/8997" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8997</a> to the b2_5 it seems</p> <p>set ticket as replated to <a href="https://jira.whamcloud.com/browse/LU-4413" title="Test failure on test suite conf-sanity, subtest test_56" class="issue-link" data-issue-key="LU-4413"><del>LU-4413</del></a> just because of the similar symptoms </p> <p>Sarah,<br/> Did you hit this on the 2.5.57 tag testing?</p> <p>Hi Jodi,<br/> No, I didn't see this issue in tag-2.5.57 testing</p> <p>Somebody needs to backport the patch from lu-4413 as it could not be cleanly cherry-picked to b2_5</p> <p>We are talking about back port of <a href="http://review.whamcloud.com/8997" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8997</a>, right? I can do that.</p> <p><a href="http://review.whamcloud.com/9875" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9875</a></p> <p>Minh retriggered the test of Change, 9875 on Shadow today.</p> <p>Landed for 2.6</p> Duplicate LU-4413 LU-4426 Related LU-793 LU-4413 LU-3895 Development Rank 1|hzwaqn: Rank (Obsolete) 11914 Severity