https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14824 Lustre <p>This issue was created by maloo for S Buisson &lt;sbuisson@ddn.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/a8d333cd-1b69-4f00-9829-2590702b0c0e" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/a8d333cd-1b69-4f00-9829-2590702b0c0e</a></p> <p>test_413a failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Timeout occurred after 492 mins, last suite running was sanity </pre> </div></div> <p>The test is blocked for an unknown reason, as nothing is visible in the console of the client or server nodes. Last message in test log is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Mkdir (stripe_count 3) roundrobin: </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity test_413a - Timeout occurred after 492 mins, last suite running was sanity</p> LU-14824

sanity test_413a: timeout

Bug Minor Resolved Fixed Lai Siyao Maloo Wed, 7 Jul 2021 06:25:56 +0000 Fri, 22 Dec 2023 23:29:18 +0000 Wed, 8 Mar 2023 04:10:10 +0000 Lustre 2.16.0 Lustre 2.15.4 0 10 <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/9255ef4d-66b3-44ce-aafd-7dd654d0508a" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/9255ef4d-66b3-44ce-aafd-7dd654d0508a</a></p> <p>This looks to be on zfs backend only, and the possible reason is slow striped directory mkdir. I'll look into the test scripts to see how to improve this.</p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/014222ff-aefc-4677-8545-f8b4bf0975c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/014222ff-aefc-4677-8545-f8b4bf0975c2</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/712a4fd4-1460-412c-a436-f648b3e0fc3d" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/712a4fd4-1460-412c-a436-f648b3e0fc3d</a></p> <p>Proposing for 2.15.0 given the recent activity with master.</p> <p>Andreas, all the failures are zfs system, and I don't see anything special in test logs, this test creates lots of files/directories, and unlinks them after test, it's stuck in unlink time. Should we just disable this test on zfs system?</p> <p>I would prefer not to disable it if possible. I think one option to speed up the test for ZFS is to use larger DoM files, since this <b>also</b> reduces free inodes on a ZFS MDT, and should give the same behavior as creating a large number of inodes, unlike on ldiskfs. </p> <p>"Lai Siyao &lt;lai.siyao@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45955" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45955</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14824" title="sanity test_413a: timeout" class="issue-link" data-issue-key="LU-14824"><del>LU-14824</del></a> test: collect debug logs on zfs system<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1d583433b1a4ad23a99ecb85fe4dc6858edaef20</p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/3720ae52-c898-40bd-9bb0-f41ea075568c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/3720ae52-c898-40bd-9bb0-f41ea075568c</a></p> <p>Currently failing about 2.5% of runs, but 7.5% of ZFS runs.</p> <p><del>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch:</del> <a href="https://review.whamcloud.com/46774" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46774</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-14824" title="sanity test_413a: timeout" class="issue-link" data-issue-key="LU-14824"><del>LU-14824</del></a> tests: reduce sanity test_413 ZFS test time</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: 502cc9bb0ee94537a77e56f3888c25f11f8790a0</del></p> <p>It is worthwhile to note that patch <a href="https://review.whamcloud.com/46734" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46734</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-15528" title="downgrade remote PW/EX lock taken in distributed transactions to COS mode " class="issue-link" data-issue-key="LU-15528">LU-15528</a> mdt: enqueue newly created object locks in TXN mode</tt>" and the later patch <a href="https://review.whamcloud.com/46733" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46733</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-15526" title="PDO lock for object on remote MDT" class="issue-link" data-issue-key="LU-15526"><del>LU-15526</del></a> mdt: enable remote PDO lock</tt>" are about 10x faster (~150-170s vs. ~1100-3000s) when running sanity test_413a compared to unpatched systems:</p> <p><a href="https://testing.whamcloud.com/search?server_file_system_type_id=00437f32-318d-11e1-9c6d-5254004bbbd3&amp;test_set_script_id=f9516376-32bc-11e0-aaee-52540025f9ae&amp;sub_test_script_id=44d5fa14-70d0-11e9-a6f2-52540065bddc&amp;start_date=2022-03-07&amp;end_date=2022-03-09&amp;source=sub_tests#redirect" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/search?server_file_system_type_id=00437f32-318d-11e1-9c6d-5254004bbbd3&amp;test_set_script_id=f9516376-32bc-11e0-aaee-52540025f9ae&amp;sub_test_script_id=44d5fa14-70d0-11e9-a6f2-52540065bddc&amp;start_date=2022-03-07&amp;end_date=2022-03-09&amp;source=sub_tests#redirect</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/46d8f39d-da3a-4039-bd02-d7c90196d7f9" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/46d8f39d-da3a-4039-bd02-d7c90196d7f9</a></p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/45955/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/45955/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14824" title="sanity test_413a: timeout" class="issue-link" data-issue-key="LU-14824"><del>LU-14824</del></a> test: sanity 413a/b unlink timeout<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5ff3e400f1a74ea49b7eb9cf19715f0fae08c3f5</p> <p>Landed for 2.16</p> <p>Patch is being reverted due to many timeouts in ldiskfs since landing. </p> <p>The patch 45955 was pushed and tested on 2022-03-06, but it looks like another patch may have landed in this same code in between and caused the failure. </p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49646" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49646</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14824" title="sanity test_413a: timeout" class="issue-link" data-issue-key="LU-14824"><del>LU-14824</del></a> Revert "test: sanity 413a/b unlink timeout"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 46277a6a1affb8b21abb28941ff4b471d2b3bd32</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49646/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49646/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14824" title="sanity test_413a: timeout" class="issue-link" data-issue-key="LU-14824"><del>LU-14824</del></a> Revert "test: sanity 413a/b unlink timeout"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 601ed56575a304c15ccb6d98a252162e64ef95e9</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49799" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49799</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14824" title="sanity test_413a: timeout" class="issue-link" data-issue-key="LU-14824"><del>LU-14824</del></a> test: sanity 413a/b unlink timeout v2<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 09dd0a091defa34ed7de2402b64f0e38dbefd9a6</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49799/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49799/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14824" title="sanity test_413a: timeout" class="issue-link" data-issue-key="LU-14824"><del>LU-14824</del></a> test: sanity 413a/b unlink timeout v2<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 09fc9ccb1acc534b3fb074433dcbcecebb175384</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51561" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51561</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14824" title="sanity test_413a: timeout" class="issue-link" data-issue-key="LU-14824"><del>LU-14824</del></a> test: sanity 413a/b unlink timeout v2<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 6c3655aa62aadbed870cc9a1b8ec559156238aac</p> Related LU-14659 LU-16507 Development Rank 1|i01ynj: Rank (Obsolete) 9223372036854775807 Severity