https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13514 Lustre <p>This issue was created by maloo for Chris Horn &lt;hornc@cray.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/4960ea6e-2914-4b3d-a77d-e0e5a0a4c9a6" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/4960ea6e-2914-4b3d-a77d-e0e5a0a4c9a6</a></p> <p>test_32a failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Timeout occurred after 143 mins, last suite running was conf-sanity </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> conf-sanity test_32a - Timeout occurred after 143 mins, last suite running was conf-sanity</p> LU-13514

conf-sanity test_32a: Timeout occurred after 143 mins

Bug Major Resolved Fixed Yang Sheng Maloo Mon, 4 May 2020 15:05:16 +0000 Thu, 7 Dec 2023 21:45:44 +0000 Sat, 29 Jan 2022 07:52:35 +0000 Lustre 2.14.0 Lustre 2.12.6 Lustre 2.12.6 Lustre 2.15.0 0 11 <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/c0c4e2b5-ac3b-4739-b5fe-e317575a774a" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/c0c4e2b5-ac3b-4739-b5fe-e317575a774a</a></p> <p>I got multiple occurrences of this problem after rebase of my patches on top of master branch, eg:<br/> <a href="https://testing.whamcloud.com/test_sets/0952e3ff-a20e-4461-afba-6042dd35d042" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/0952e3ff-a20e-4461-afba-6042dd35d042</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sessions/cc02ea29-8465-4511-ac15-7690c947d11e" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/cc02ea29-8465-4511-ac15-7690c947d11e</a></p> <p>It seems that no recent testing managed to pass conf-sanity because of this problem.<br/> Raising ticket priority.</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/38615" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38615</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13514" title="conf-sanity test_32a: Timeout occurred after 143 mins" class="issue-link" data-issue-key="LU-13514"><del>LU-13514</del></a> tests: test conf-sanity test_32a - base<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: de0d8d26b4ab6deb4618f04c0d82f6489d9b8ab9</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/38616" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38616</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13514" title="conf-sanity test_32a: Timeout occurred after 143 mins" class="issue-link" data-issue-key="LU-13514"><del>LU-13514</del></a> tests: test conf-sanity test_32a - 1<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 631c2ad1096597af7b8a99db0fef5e55283246c3</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/38617" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38617</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13514" title="conf-sanity test_32a: Timeout occurred after 143 mins" class="issue-link" data-issue-key="LU-13514"><del>LU-13514</del></a> tests: test conf-sanity test_32a - 2<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 90693299e21be79621a4891313b3c1a712ed434a</p> <p>The patches above show that conf-sanity test_32a fails with tip of master branch (results of <a href="https://review.whamcloud.com/38615" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38615</a>), and passes when we revert commit 6b979daaff "<a href="https://jira.whamcloud.com/browse/LU-11643" title="create disk images for Lustre 2.10 and 2.12 for ldiskfs" class="issue-link" data-issue-key="LU-11643"><del>LU-11643</del></a> tests: add new images and tests for upgrade tests" (results of <a href="https://review.whamcloud.com/3861" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/3861</a>).</p> <p>So I think commit 6b979daaff "<a href="https://jira.whamcloud.com/browse/LU-11643" title="create disk images for Lustre 2.10 and 2.12 for ldiskfs" class="issue-link" data-issue-key="LU-11643"><del>LU-11643</del></a> tests: add new images and tests for upgrade tests" needs to be reverted for now, as it blocks testing of all new patches.</p> <p>Sebastien, do you know what is broken in the tests, and why/how that patch passed testing before it landed? Is it just because the testing is now slower and timing out, or is there a code defect (hang)?</p> <p>Well, commit 6b979daaff "<a href="https://jira.whamcloud.com/browse/LU-11643" title="create disk images for Lustre 2.10 and 2.12 for ldiskfs" class="issue-link" data-issue-key="LU-11643"><del>LU-11643</del></a> tests: add new images and tests for upgrade tests" explicitly adds stuff that conf-sanity test_32c goes through. 2 more patches landed after this one, but one is adding a new test script, and the other changes the file lustre/osd-zfs/osd_scrub.c. So they cannot be responsible for failure in review-dne-part-3 test group, which runs on ldiskfs.</p> <p>The explanation I see for the test failure in master now is that patch <a href="https://review.whamcloud.com/35049" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35049</a> (6b979daaff "<a href="https://jira.whamcloud.com/browse/LU-11643" title="create disk images for Lustre 2.10 and 2.12 for ldiskfs" class="issue-link" data-issue-key="LU-11643"><del>LU-11643</del></a> tests: add new images and tests for upgrade tests") was tested on a too old branch. I can see that patchset 16 was based on commit a83c820f89 "<a href="https://jira.whamcloud.com/browse/LU-12312" title="sanity-sec: test_31: &#39;network&#39; mount option cannot be taken into account" class="issue-link" data-issue-key="LU-12312">LU-12312</a> lnet: handle no discovery flag", that dates back from April, 23rd.</p> <p>I see that conf-sanity test_32a is still failing with this same error even for a patch based on the latest master commit <tt>v2_13_53-165-gebaf3b1b9980</tt> "<tt><a href="https://jira.whamcloud.com/browse/LU-11643" title="create disk images for Lustre 2.10 and 2.12 for ldiskfs" class="issue-link" data-issue-key="LU-11643"><del>LU-11643</del></a> tests: revert new images and tests for upgrade patch</tt>":<br/> <a href="https://testing.whamcloud.com/test_sets/a4c8f3d3-b8e5-4e7e-9192-2b0bc22279b4" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/a4c8f3d3-b8e5-4e7e-9192-2b0bc22279b4</a></p> <p>+1 on Master: <a href="https://testing.whamcloud.com/sub_tests/e3575182-057a-4057-965e-fd0c293e939b" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/e3575182-057a-4057-965e-fd0c293e939b</a></p> <p>more on master: <br/> <a href="https://testing.whamcloud.com/test_sets/b52e5506-8e0a-48c7-8c97-3666e4df460e" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/b52e5506-8e0a-48c7-8c97-3666e4df460e</a><br/> <a href="https://testing.whamcloud.com/test_sets/3c7736eb-a501-445a-ad51-4e4d2e004212" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/3c7736eb-a501-445a-ad51-4e4d2e004212</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sessions/807df025-8fd7-45d1-8961-b7fbaf84cdc2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/807df025-8fd7-45d1-8961-b7fbaf84cdc2</a></p> <p>It seems that almost no patch managed to pass conf-sanity test_32a in the last couple of days.</p> <p>Looking at the test results, it seems that <tt>review-dne-part-3</tt> (ldiskfs) is the only session that is timing out, and never <tt>review-dne-zfs-part-3</tt>, so the failure must be related to one of the ldiskfs test images.</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39108" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39108</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13514" title="conf-sanity test_32a: Timeout occurred after 143 mins" class="issue-link" data-issue-key="LU-13514"><del>LU-13514</del></a> tests: stop running conf-sanity test 32a ldiskfs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 440654a0768c4bebec20fa99571318e2bf429ccc</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39109" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39109</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13514" title="conf-sanity test_32a: Timeout occurred after 143 mins" class="issue-link" data-issue-key="LU-13514"><del>LU-13514</del></a> tests: remove upgrade images for conf-sanity<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b33b1a85c843be4ffffd181605d8ae2ac07c3bac</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39109/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39109/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13514" title="conf-sanity test_32a: Timeout occurred after 143 mins" class="issue-link" data-issue-key="LU-13514"><del>LU-13514</del></a> tests: remove upgrade images for conf-sanity<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d574a55778f035691bd3bed621cfcdb8200a9785</p> <p>Yang Sheng (ys@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40492" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40492</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13514" title="conf-sanity test_32a: Timeout occurred after 143 mins" class="issue-link" data-issue-key="LU-13514"><del>LU-13514</del></a> tests: remove upgrade images for conf-sanity<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: d50c28405f107249b65623be6580e20b311e0bef</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40492/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40492/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13514" title="conf-sanity test_32a: Timeout occurred after 143 mins" class="issue-link" data-issue-key="LU-13514"><del>LU-13514</del></a> tests: remove upgrade images for conf-sanity<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 1d4c0620b41bf2bee63cf3eb40a6c78fa503645c</p> <p>Is turning off this test a solution or the equivalent of adding something to the always accept list?</p> <p>For master, future 2.14.0, for the past 4 weeks, the only conf-sanity test 32a hangs/timeouts are for interop testing:<br/> 2.13.0 server/2.13.56.40 clients - <a href="https://testing.whamcloud.com/test_sets/b032d2d1-f0bf-4952-91ca-060a69f7d1ab" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/b032d2d1-f0bf-4952-91ca-060a69f7d1ab</a><br/> 2.12.5 servers/2.13.56.45 clients - <a href="https://testing.whamcloud.com/test_sets/896bf228-69d7-4417-b163-58a75fe97c23" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/896bf228-69d7-4417-b163-58a75fe97c23</a></p> <p> For the b2_12 branch, we see this test hang in non-interop testing:<br/> 2.12.5.82 server/client - <a href="https://testing.whamcloud.com/test_sets/f8ec6569-d632-41c9-8d6b-21dd1ac9c635" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/f8ec6569-d632-41c9-8d6b-21dd1ac9c635</a></p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40537/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40537/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13514" title="conf-sanity test_32a: Timeout occurred after 143 mins" class="issue-link" data-issue-key="LU-13514"><del>LU-13514</del></a> tests: replace nid in conf-sanity test_32<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 327c8b77694bb0796f168df26e0c543d9610691e</p> <p>Yang Sheng, is patch <a href="https://review.whamcloud.com/40537" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40537</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-13514" title="conf-sanity test_32a: Timeout occurred after 143 mins" class="issue-link" data-issue-key="LU-13514"><del>LU-13514</del></a> tests: replace nid in conf-sanity test_32</tt>" needed on master?</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/46354" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46354</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13514" title="conf-sanity test_32a: Timeout occurred after 143 mins" class="issue-link" data-issue-key="LU-13514"><del>LU-13514</del></a> tests: replace nid in conf-sanity test_32<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 98e3e1018f57293061d4d9dd515507577c4e2cae</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/46354/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46354/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13514" title="conf-sanity test_32a: Timeout occurred after 143 mins" class="issue-link" data-issue-key="LU-13514"><del>LU-13514</del></a> tests: replace nid in conf-sanity test_32<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4d2efa87abda3c177ca5a2178cd5b2eafa7f99af</p> Related LU-11643 LU-11643 Development Rank 1|i00zhb: Rank (Obsolete) 9223372036854775807 Severity