https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10616 Lustre <p>replay-single test_70b fails with two error messages</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>replay-single test_70b: @@@@@@ FAIL: dbench stopped on some of onyx-31vm1.onyx.hpdd.intel.com,onyx-31vm2! </pre> </div></div> <p>and later</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>replay-single test_70b: @@@@@@ FAIL: rundbench load on onyx-31vm1.onyx.hpdd.intel.com,onyx-31vm2 failed! </pre> </div></div> <p>Looking at the suite_log, we see </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: onyx-31vm1.onyx.hpdd.intel.com,onyx-31vm2 killall -0 dbench onyx-31vm1: [3] open ./clients/client0 failed for handle 16385 (No such file or directory) onyx-31vm1: (4) ERROR: handle 16385 was not found onyx-31vm1: Child failed with status 1 onyx-31vm1: dbench: no process found onyx-31vm1: dbench: no process found replay-single test_70b: @@@@@@ FAIL: dbench stopped on some of onyx-31vm1.onyx.hpdd.intel.com,onyx-31vm2! </pre> </div></div> <p>The only thing that looks suspicious in the console logs is on the MDS1, 3</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 5354.241985] Lustre: DEBUG MARKER: Started rundbench load pid=3403 ... [ 5354.488828] LustreError: 12371:0:(osd_oi.c:978:osd_idc_find_or_init()) lustre-MDT0000: can't lookup: rc = -2 [ 5354.753146] Lustre: DEBUG MARKER: /usr/sbin/lctl mark replay-single test_70b: @@@@@@ FAIL: dbench stopped on some of onyx-31vm1.onyx.hpdd.intel.com,onyx-31vm2! </pre> </div></div> <p>This test has failed in this way many times, so far, for only full test sessions with DNE configured and ZFS:<br/> 2.10.57 el7 build 3703 – <a href="https://testing.hpdd.intel.com/test_sets/46a0b60a-078f-11e8-bd00-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/46a0b60a-078f-11e8-bd00-52540065bddc</a><br/> 2.10.57 el7 build 3702 – <a href="https://testing.hpdd.intel.com/test_sets/13cdeb9e-0352-11e8-a10a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/13cdeb9e-0352-11e8-a10a-52540065bddc</a><br/> 2.10.57 el7 build 3700 - <a href="https://testing.hpdd.intel.com/test_sets/fa0a850e-014f-11e8-a6ad-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fa0a850e-014f-11e8-a6ad-52540065bddc</a><br/> 2.10.57 el7 build 3697 - <a href="https://testing.hpdd.intel.com/test_sets/ebd4b25e-fd83-11e7-a7cd-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ebd4b25e-fd83-11e7-a7cd-52540065bddc</a><br/> 2.10.57 el7 patchless build 59 – <a href="https://testing.hpdd.intel.com/test_sets/dee6191a-ffaf-11e7-a6ad-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/dee6191a-ffaf-11e7-a6ad-52540065bddc</a><br/> 2.10.57 el7 patchless build 58 – <a href="https://testing.hpdd.intel.com/test_sets/16fa9310-fe7c-11e7-a6ad-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/16fa9310-fe7c-11e7-a6ad-52540065bddc</a><br/> 2.10.56 el7 build 3693 – <a href="https://testing.hpdd.intel.com/test_sets/d309f58a-f77b-11e7-bd00-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d309f58a-f77b-11e7-bd00-52540065bddc</a><br/> 2.10.56 el7 patchless build 53 – <a href="https://testing.hpdd.intel.com/test_sets/38f48bae-f636-11e7-94c7-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/38f48bae-f636-11e7-94c7-52540065bddc</a><br/> 2.10.56 el7 patchless build 50 – <a href="https://testing.hpdd.intel.com/test_sets/c46aeb7c-f228-11e7-8c43-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c46aeb7c-f228-11e7-8c43-52540065bddc</a><br/> 2.10.56 el7 build 3685 – <a href="https://testing.hpdd.intel.com/test_sets/6c00afc0-e7c0-11e7-8027-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6c00afc0-e7c0-11e7-8027-52540065bddc</a><br/> 2.10.56 el7 patchless build 44 – <a href="https://testing.hpdd.intel.com/test_sets/53f8d684-e674-11e7-a066-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/53f8d684-e674-11e7-a066-52540065bddc</a></p> LU-10616

replay-single test_70b fails with 'rundbench load on <hostname(s)> failed!'

Bug Major Open Unresolved Lai Siyao James Nunez dne zfs Tue, 6 Feb 2018 18:45:24 +0000 Tue, 24 Jan 2023 22:50:58 +0000 Lustre 2.11.0 Lustre 2.12.0 Lustre 2.13.0 Lustre 2.10.6 Lustre 2.10.7 Lustre 2.12.3 Lustre 2.14.0 Lustre 2.12.5 Lustre 2.15.1 0 8 <p>From John Hammond, it looks like there is an issue with dbench start up as seen in the suite_log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>trevis-11vm1: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/trevis-11vm1.trevis.hpdd.intel.com at Thu Feb 1 01:34:50 UTC 2018 trevis-11vm1: dbench PID=30955 trevis-11vm1: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 trevis-11vm1: trevis-11vm1: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs trevis-11vm1: failed to create barrier semaphore trevis-11vm1: 0 of 1 processes prepared for launch 0 sec trevis-11vm2: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 trevis-11vm2: trevis-11vm2: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs trevis-11vm2: failed to create barrier semaphore trevis-11vm2: 0 of 1 processes prepared for launch 0 sec CMD: trevis-11vm1.trevis.hpdd.intel.com,trevis-11vm2 killall -0 dbench trevis-11vm1: 1 of 1 processes prepared for launch 0 sec trevis-11vm1: releasing clients trevis-11vm2: 1 of 1 processes prepared for launch 0 sec trevis-11vm2: releasing clients trevis-11vm1: [3] open ./clients/client0 failed for handle 16385 (No such file or directory) trevis-11vm1: (4) ERROR: handle 16385 was not found trevis-11vm1: Child failed with status 1 trevis-11vm1: dbench: no process found trevis-11vm1: dbench: no process found </pre> </div></div> <p>+1 on master 2.11.51 failover</p> <p><a href="https://testing.hpdd.intel.com/test_sets/7d85d5ce-492f-11e8-960d-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7d85d5ce-492f-11e8-960d-52540065bddc</a></p> <p>+1 <a href="https://testing.whamcloud.com/sub_tests/13bce654-fc76-11e9-98f1-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/13bce654-fc76-11e9-98f1-52540065bddc</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/9333fec4-2406-11ea-b1e8-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/9333fec4-2406-11ea-b1e8-52540065bddc</a></p> <p>Hello,</p> <p>I have the same kind of messages on: <a href="https://testing.whamcloud.com/sub_tests/a7c49599-b0e2-49e1-a4af-9111f676fdcf" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/a7c49599-b0e2-49e1-a4af-9111f676fdcf</a></p> <p>Except the message:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> trevis-9vm4: [341] open ./clients/client0/~dmtmp/PARADOX/COURSES.DB failed <span class="code-keyword">for</span> handle 9977 (Stale file handle) </pre> </div></div> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/b5f87cba-d087-45dc-85ef-e1005ef15186" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/b5f87cba-d087-45dc-85ef-e1005ef15186</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/1c452361-2846-41bd-af35-995e1de3fd99" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/1c452361-2846-41bd-af35-995e1de3fd99</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/ee775d6f-00db-41b2-ad02-d4ae7e31ce6c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/ee775d6f-00db-41b2-ad02-d4ae7e31ce6c</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/c154d88e-a784-4023-9c59-f40662559bea" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/c154d88e-a784-4023-9c59-f40662559bea</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/d3c778e5-e533-4a1d-8dce-263b64809701" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/d3c778e5-e533-4a1d-8dce-263b64809701</a></p> <p>+1 <a href="https://testing.whamcloud.com/test_sets/15d4b4b3-8a48-4743-b935-bf96afb0e27d" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/15d4b4b3-8a48-4743-b935-bf96afb0e27d</a></p> <p>Lai, should replay-single test_70b be updated to add "<tt>stack_trap fail_abort_cleanup</tt>" so that it can clean up afterward? However, while the test is doing failover (via <tt>test-framework.sh::fail()-&gt;facet_failover()</tt>) it doesn't look like this subtest is actually aborting recovery, so it <b>shouldn't</b> be seeing this kind of problem.</p> <p>This subtest is failing pretty regularly, could you please investigate why it is having problems during recovery? It should be possible to use "<tt>Test-Parameters: fortestonly testlist=replay-single env=ONLY=70b,ONLY_REPEAT=100 livedebug</tt>" to run 70b until it is hit and then leave the node in that state to log in and debug.</p> Duplicate LU-14791 LU-14813 Related LU-15624 LU-16336 LU-16065 Development Rank 1|hzzsbb: Rank (Obsolete) 9223372036854775807 Severity