https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1897 Lustre <p>This issue was created by maloo for jay &lt;jay@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/c4620b6c-fc31-11e1-a4a6-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c4620b6c-fc31-11e1-a4a6-52540035b04c</a>.</p> <p>The sub-test test_70b failed with the following error:</p> <blockquote> <p>dbench not found on some of client-28vm1,client-28vm2.lab.whamcloud.com !</p></blockquote> <p>Info required for matching: replay-single 70b</p> LU-1897

Failure on test suite replay-single, test_70b: dbench not found on some of the test nodes

Bug Blocker Closed Cannot Reproduce Keith Mannthey Maloo MB mn1 Tue, 11 Sep 2012 15:21:19 +0000 Thu, 24 Dec 2015 17:09:53 +0000 Tue, 15 Dec 2015 19:40:08 +0000 Lustre 2.4.0 Lustre 2.1.6 Lustre 2.8.0 Lustre 2.4.0 1 11 <p><a href="https://maloo.whamcloud.com/test_sets/8702cb68-0eeb-11e2-8f7c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/8702cb68-0eeb-11e2-8f7c-52540035b04c</a></p> <p>Another failure instance:</p> <p><a href="https://maloo.whamcloud.com/test_sets/2e20ce44-26da-11e2-b938-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2e20ce44-26da-11e2-b938-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/sub_tests/51c6d29e-26da-11e2-b938-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/51c6d29e-26da-11e2-b938-52540035b04c</a></p> <p>another failure: <a href="https://maloo.whamcloud.com/test_sets/c2d37650-2819-11e2-aa14-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c2d37650-2819-11e2-aa14-52540035b04c</a></p> <p>rundbench load on fat-intel-1vm2,fat-intel-1vm6.lab.whamcloud.com failed!</p> <p>also seen in:<br/> <a href="https://maloo.whamcloud.com/sub_tests/e0d319ee-3093-11e2-9075-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/e0d319ee-3093-11e2-9075-52540035b04c</a></p> <p>client-10vm2: dbench: no process found<br/> replay-single test_70b: @@@@@@ FAIL: dbench not found on some of client-10vm1,client-10vm2!</p> <p>This is happening quite often <a href="https://maloo.whamcloud.com/test_sets/d11835ae-3c71-11e2-82b5-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/d11835ae-3c71-11e2-82b5-52540035b04c</a></p> <p><a href="https://maloo.whamcloud.com/test_sets/query?utf8=" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/query?utf8=</a>✓&amp;test_set%5Btest_set_script_id%5D=f6a12204-32c3-11e0-a61c-52540025f9ae&amp;test_set%5Bstatus%5D=FAIL&amp;test_set%5Bquery_bugs%5D=&amp;test_session%5Btest_host%5D=&amp;test_session%5Btest_group%5D=&amp;test_session%5Buser_id%5D=&amp;test_session%5Bquery_date%5D=&amp;test_session%5Bquery_recent_period%5D=&amp;test_node%5Bos_type_id%5D=&amp;test_node%5Bdistribution_type_id%5D=&amp;test_node%5Barchitecture_type_id%5D=&amp;test_node%5Bfile_system_type_id%5D=&amp;test_node%5Blustre_branch_id%5D=24a6947e-04a9-11e1-bb5f-52540025f9af&amp;test_node_network%5Bnetwork_type_id%5D=&amp;commit=Update+results</p> <p>Hi Chris</p> <p>Is this a TT issue?</p> <p>Peter</p> <p><a href="https://maloo.whamcloud.com/test_sets/c2c1a95c-405b-11e2-a16b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c2c1a95c-405b-11e2-a16b-52540035b04c</a></p> <p>test log looks very odd. Although it reports</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>replay-single test_70b: @@@@@@ FAIL: dbench not found on some of client-10vm1.lab.whamcloud.com,client-10vm2! </pre> </div></div> <p>the log has output of dbench actually running just a few lines before the error report.</p> <p>It looks like dbench ends right before test checks that it is running.</p> <p>Another failure instance:</p> <p><a href="https://maloo.whamcloud.com/test_sets/19e6b2fe-513c-11e2-b56e-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/19e6b2fe-513c-11e2-b56e-52540035b04c</a></p> <p>another instance:<br/> <a href="https://maloo.whamcloud.com/test_sets/927c0d00-5717-11e2-8b17-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/927c0d00-5717-11e2-8b17-52540035b04c</a></p> <p>another instance:<br/> <a href="https://maloo.whamcloud.com/test_sets/72d446e2-5943-11e2-9b54-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/72d446e2-5943-11e2-9b54-52540035b04c</a></p> <p>So Bob and I looked into the problem a bit today.</p> <p>In general dbench is just a background load for the test and as mentioned Dbench is ending before the test finishes. </p> <p>I observed the Dbench finishing at 62s and the test finishing in and checking in on it at a little under 90's. </p> <p>I have submitted at patch to move the dbench duration out a bit to see it help the issue. The test could stand to be rewritten to make the back ground load a non-issue. </p> <p><a href="http://review.whamcloud.com/4973" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4973</a> is simple patch that adds a comment and increases a duration that should allow dbench to run longer. </p> <p>Looks like I suffered from this over the weekend: <a href="https://maloo.whamcloud.com/test_sessions/b2e3bc58-5c73-11e2-ab3b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/b2e3bc58-5c73-11e2-ab3b-52540035b04c</a></p> <p>It might be nice as part of this fix to change the error message to "dbench is no longer running on some of the test nodes", which would not give the false impression that it is not installed on the test nodes...</p> <p>I will change the comment and resubmit the patch. </p> <p><a href="http://review.whamcloud.com/4973" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4973</a> has been merged. Please reopen if the problem sill occurs.</p> <p>I think this failed here:<br/> <a href="https://maloo.whamcloud.com/test_sets/43757b76-8eb0-11e2-81eb-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/43757b76-8eb0-11e2-81eb-52540035b04c</a></p> <p>It looks like dbench didn't start on client-26vm5 fast enough for the first check?<br/> <a href="https://maloo.whamcloud.com/test_logs/6cc6de7a-8eb0-11e2-81eb-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_logs/6cc6de7a-8eb0-11e2-81eb-52540035b04c</a></p> <p>12 seconds isn't long enough to start dbench on a remote client... Fun. I will submit a patch. </p> <p>This is the console from the main client. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>15:52:52:Lustre: DEBUG MARKER: /usr/sbin/lctl mark Started rundbench load pid=1931 ... 15:52:52:Lustre: DEBUG MARKER: Started rundbench load pid=1931 ... 15:53:03:Lustre: DEBUG MARKER: killall -0 dbench 15:53:04:Lustre: DEBUG MARKER: /usr/sbin/lctl mark replay-single test_70b: @@@@@@ FAIL: dbench not running on some of client-26vm5,client-26vm6.lab.whamcloud.com! </pre> </div></div> <p>In the autotest long you see dbench start some time after this on the other client. It does run just not at the right time. </p> <p><a href="http://review.whamcloud.com/5761" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5761</a> version 1 of the patch has been submitted. </p> <p>Landed for 2.4</p> <p>The issue also exists on Lustre b2_1 branch:<br/> <a href="https://maloo.whamcloud.com/test_sets/a0806aa0-c5ce-11e3-9255-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a0806aa0-c5ce-11e3-9255-52540035b04c</a></p> <p>master, build# 3264, 2.7.64 tag<br/> Hard Failover: EL6.7 Server/Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/80a20678-9edd-11e5-87a9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/80a20678-9edd-11e5-87a9-5254006e85c2</a></p> <p>Instance found in recent tags 2.7.63, 2.7.64</p> <p>master, build# 3264, 2.7.64 tag<br/> Hard Failover: EL7 Server/Client - ZFS<br/> <a href="https://testing.hpdd.intel.com/test_sets/5ba6d7bc-9e20-11e5-91b0-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5ba6d7bc-9e20-11e5-91b0-5254006e85c2</a></p> <p>master, build# 3266, 2.7.64 tag<br/> Hard Failover:EL7 Server/SLES11 SP3 Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/a8b3fb9e-a077-11e5-8d69-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a8b3fb9e-a077-11e5-8d69-5254006e85c2</a></p> <p>close this ticket as the recent instances on master (tag-2.7.63 and tag-2.7.64) are caused by the "no space left on device"</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>shadow-52vm6: [769] open ./clients/client0/~dmtmp/PWRPNT/PPTC112.TMP failed for handle 10033 (No space left on device) shadow-52vm6: (770) ERROR: handle 10033 was not found shadow-52vm6: Child failed with status 1 shadow-52vm6: touch: missing file operand shadow-52vm6: Try 'touch --help' for more information. shadow-52vm6: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 21 sec shadow-52vm1: 1 733 0.17 MB/sec warmup 124 sec latency 53560.001 ms shadow-52vm5: 1 791 0.17 MB/sec warmup 124 sec latency 54097.359 ms shadow-52vm1: 1 733 0.17 MB/sec warmup 125 sec latency 1340.522 ms shadow-52vm5: 1 794 0.17 MB/sec warmup 125 sec latency 54380.844 ms CMD: shadow-52vm1.shadow.whamcloud.com,shadow-52vm5,shadow-52vm6 killall -0 dbench shadow-52vm6: dbench: no process found replay-single test_70b: @@@@@@ FAIL: dbench stopped on some of shadow-52vm1.shadow.whamcloud.com,shadow-52vm5,shadow-52vm6! shadow-52vm1: 1 733 0.17 MB/sec warmup 126 sec latency 2340.713 ms shadow-52vm5: 1 794 0.17 MB/sec warmup 126 sec latency 1715.056 ms </pre> </div></div> <p>Sarah,<br/> Is there a different ticket tracking the "no space left on device" failure? If this problem is still happening, but this bug is closed, then either this bug needs to be linked to an existing ticket or a new ticket opened up to track these failures. </p> <p>Hi Andreas,</p> <blockquote><p>Is there a different ticket tracking the "no space left on device" failure?</p></blockquote> <p>Yes, it's <a href="https://jira.whamcloud.com/browse/LU-4846" title="Failover test failure on test suite replay-single test_26: No space left" class="issue-link" data-issue-key="LU-4846">LU-4846</a>. And the original issue in this ticket is not related to "No space left on device", so we do not need link this ticket to <a href="https://jira.whamcloud.com/browse/LU-4846" title="Failover test failure on test suite replay-single test_26: No space left" class="issue-link" data-issue-key="LU-4846">LU-4846</a>.</p> Duplicate LU-2120 Development Rank 1|hzvd7b: Rank (Obsolete) 5743 Severity