Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-4108] Failure on test suite performance-sanity test_4 https://jira.whamcloud.com/browse/LU-4108 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/daabb6e4-3505-11e3-b76d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/daabb6e4-3505-11e3-b76d-52540035b04c</a>.</p> <p>The sub-test test_4 failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>Info required for matching: performance-sanity 4</p> client and server: lustre-b2_5 RHEL6 build #2 LU-4108 Failure on test suite performance-sanity test_4 Bug Blocker Resolved Fixed Nathaniel Clark Maloo Tue, 15 Oct 2013 20:11:09 +0000 Thu, 29 May 2014 13:58:48 +0000 Wed, 26 Mar 2014 15:24:30 +0000 Lustre 2.5.0 Lustre 2.4.2 Lustre 2.5.1 Lustre 2.4.3 Lustre 2.6.0 Lustre 2.5.2 0 7 <p>Cannot find useful logs, it looks like just a slow run caused the timeout. In the following link, similar situation of parallel-scale, parallel-scale-nvsv3/4 and obdfilter-survey</p> <p><a href="https://maloo.whamcloud.com/test_sessions/3f307b78-3500-11e3-b76d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/3f307b78-3500-11e3-b76d-52540035b04c</a></p> <p>This is almost certainly caused by slowness due to many ZFS pools sharing the same underlying disk.</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/70/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/70/</a> (2.4.2 RC2)<br/> Distro/Arch: RHEL6.4/x86_64</p> <p>FSTYPE=zfs<br/> MDSCOUNT=1<br/> MDSSIZE=2097152<br/> OSTCOUNT=2<br/> OSTSIZE=8388608</p> <p>performance-sanity test 8 timed out in 28800s:<br/> <a href="https://maloo.whamcloud.com/test_sets/37e26e00-6b4f-11e3-99ba-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/37e26e00-6b4f-11e3-99ba-52540035b04c</a></p> <p>parallel-scale test metabench timed out in 14400s:<br/> <a href="https://maloo.whamcloud.com/test_sets/92f82460-6b4f-11e3-99ba-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/92f82460-6b4f-11e3-99ba-52540035b04c</a></p> <p>conf-sanity test 69 timed out in 3600s:<br/> <a href="https://maloo.whamcloud.com/test_sets/d2e9712c-6b4b-11e3-99ba-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/d2e9712c-6b4b-11e3-99ba-52540035b04c</a></p> <p>sanity-benchmark test iozone timed out in 14400s:<br/> <a href="https://maloo.whamcloud.com/test_sets/3935574e-6b4b-11e3-99ba-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/3935574e-6b4b-11e3-99ba-52540035b04c</a></p> <p>Nothing abnormal in the console logs.</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/5/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/5/</a><br/> Distro/Arch: RHEL6.4/x86_64</p> <p>FSTYPE=zfs<br/> MDSCOUNT=1<br/> MDSSIZE=2097152<br/> OSTCOUNT=2<br/> OSTSIZE=8388608</p> <p>parallel-scale test metabench timed out in 14400s:<br/> <a href="https://maloo.whamcloud.com/test_sets/628b4e78-73c5-11e3-b4ff-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/628b4e78-73c5-11e3-b4ff-52540035b04c</a></p> <p>conf-sanity test 69 timed out in 3600s:<br/> <a href="https://maloo.whamcloud.com/test_sets/93e12716-73c2-11e3-b4ff-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/93e12716-73c2-11e3-b4ff-52540035b04c</a></p> <p>performance-sanity/4 hasn't timed out on master since 2013-07-25 (where it was <a href="https://jira.whamcloud.com/browse/LU-1357" title="Test failure on test suite performance-sanity, subtest test_4" class="issue-link" data-issue-key="LU-1357"><del>LU-1357</del></a>)<br/> This bug has happened several times on b2_5 and many times b2_4</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/20/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/20/</a><br/> Distro/Arch: RHEL6.4/x86_64</p> <p>FSTYPE=zfs<br/> MDSCOUNT=1<br/> MDSSIZE=2097152<br/> OSTCOUNT=2<br/> OSTSIZE=8388608</p> <p>parallel-scale-nfsv4 test compilebench timed out in 7200s:<br/> <a href="https://maloo.whamcloud.com/test_sets/863a9f90-91a7-11e3-ba94-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/863a9f90-91a7-11e3-ba94-52540035b04c</a></p> <p>The following sub-tests timed out in 3600s:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>sanity-benchmark test bonnie replay-ost-single test 8a metadata-updates ost-pools test 23a obdfilter-survey test 1a </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sessions/d343de6e-91a2-11e3-ba94-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/d343de6e-91a2-11e3-ba94-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/73/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/73/</a> (2.4.3 RC1)<br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs</p> <p><a href="https://maloo.whamcloud.com/test_sets/7183f11a-ac5e-11e3-81d7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7183f11a-ac5e-11e3-81d7-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/ef26bbce-ac5f-11e3-81d7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ef26bbce-ac5f-11e3-81d7-52540035b04c</a></p> <p>This bug is basically the same issue as <a href="https://jira.whamcloud.com/browse/LU-2600" title="lustre metadata performance is very slow on zfs" class="issue-link" data-issue-key="LU-2600"><del>LU-2600</del></a> (poor metadata performance on ZFS). The NUM_FILES run should either be decreased (similar to parallel-scale.sh) or the test should be marked as SLOW for zfs</p> <p><a href="http://review.whamcloud.com/9725" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9725</a></p> <p>Patch Landed to master</p> Duplicate LU-1357 LU-2887 Related LU-2600 Development Rank 1|hzw5s7: Rank (Obsolete) 11060 Severity