https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6669 Lustre <p>This issue was created by maloo for sarah_lw &lt;wei3.liu@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/f446d474-007b-11e5-9650-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f446d474-007b-11e5-9650-5254006e85c2</a>.</p> <p>The sub-test test_failover_mds failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test_failover_mds returned 3 </pre> </div></div> <p>test log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: shadow-21vm1,shadow-21vm2 cat /tmp/client-load.pid shadow-21vm2: cat: /tmp/client-load.pid: No such file or directory shadow-21vm1: cat: /tmp/client-load.pid: No such file or directory </pre> </div></div> LU-6669

Hard Failover recovery-mds-scale test_failover_mds: test_failover_mds returned 3

Bug Minor Resolved Cannot Reproduce WC Triage Maloo Mon, 1 Jun 2015 18:54:06 +0000 Tue, 14 Dec 2021 22:36:49 +0000 Tue, 14 Dec 2021 22:36:49 +0000 Lustre 2.8.0 0 5 <p>The failure is due to failed writing pid into LOAD_PID_FILE in run_*.sh</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># recovery-*-scale scripts use this to signal the client loads to die echo $$ &gt;$LOAD_PID_FILE </pre> </div></div> <p>To mee it looks like something wiped all or most of /tmp.<br/> This is supported by lack of various log files that would have been there otherwise.</p> <p>This could be some external force or a bug in one of the test scripts.</p> <p>recovery-*-scale.sh are all affected by this failure</p> <p>lustre-master build 3118</p> <p><a href="https://testing.hpdd.intel.com/test_sets/057df5c4-35c4-11e5-8c30-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/057df5c4-35c4-11e5-8c30-5254006e85c2</a></p> <p>master, build# 3264, 2.7.64 tag<br/> Hard Failover: EL6.7 Server/Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/7b412132-9edd-11e5-87a9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7b412132-9edd-11e5-87a9-5254006e85c2</a></p> <p>This test has failed with the same error around 14 times in the past month.<br/> <a href="https://testing.hpdd.intel.com/sub_tests/5c07fe72-9e2d-11e5-87a9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/5c07fe72-9e2d-11e5-87a9-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/cf49392e-9e9a-11e5-b163-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/cf49392e-9e9a-11e5-b163-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/7b4609e0-9edd-11e5-87a9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/7b4609e0-9edd-11e5-87a9-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/2c618474-9ebc-11e5-98a4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/2c618474-9ebc-11e5-98a4-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/0c64a484-9d37-11e5-8e88-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/0c64a484-9d37-11e5-8e88-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/3b5e3922-9c64-11e5-9866-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/3b5e3922-9c64-11e5-9866-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/ab3c95d2-9b7d-11e5-9930-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/ab3c95d2-9b7d-11e5-9930-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/c1059ce0-9a8a-11e5-8b28-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/c1059ce0-9a8a-11e5-8b28-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/d5af25c2-99b2-11e5-9bd2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/d5af25c2-99b2-11e5-9bd2-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/fae19aac-9938-11e5-802b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/fae19aac-9938-11e5-802b-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/b2f329fe-98fc-11e5-8079-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/b2f329fe-98fc-11e5-8079-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/7d5e3812-9909-11e5-aeec-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/7d5e3812-9909-11e5-aeec-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/e1399bd6-95f0-11e5-a6d2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/e1399bd6-95f0-11e5-a6d2-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/f7c7dc36-8a70-11e5-ba42-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/f7c7dc36-8a70-11e5-ba42-5254006e85c2</a></p> <p>master, build# 3264, 2.7.64 tag<br/> Hard Failover: EL6.7 Server/Client - ZFS<br/> <a href="https://testing.hpdd.intel.com/test_sets/2c5961c2-9ebc-11e5-98a4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2c5961c2-9ebc-11e5-98a4-5254006e85c2</a></p> <p>master, build# 3264, 2.7.64 tag<br/> Hard Failover: EL6.7 Server/Client - ZFS<br/> recovery-double-scale test_pairwise_fail failed with the the same issue.<br/> <a href="https://testing.hpdd.intel.com/test_sets/2ce07428-9ebc-11e5-98a4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2ce07428-9ebc-11e5-98a4-5254006e85c2</a></p> <p>master, build# 3264, 2.7.64 tag<br/> Hard Failover: EL7 Server/Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/cf43bf1c-9e9a-11e5-b163-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/cf43bf1c-9e9a-11e5-b163-5254006e85c2</a></p> <p>master, build# 3264, 2.7.64 tag<br/> Hard Failover: EL7 Server/Client - ZFS<br/> <a href="https://testing.hpdd.intel.com/test_sets/572d25ba-9e20-11e5-91b0-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/572d25ba-9e20-11e5-91b0-5254006e85c2</a></p> <p>master, build# 3266, 2.7.64 tag<br/> Hard Failover: EL6.7 Server/SLES11 SP3 Clients<br/> <a href="https://testing.hpdd.intel.com/test_sets/b71e8f10-a080-11e5-85ed-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b71e8f10-a080-11e5-85ed-5254006e85c2</a><br/> Hard Failover:EL7 Server/SLES11 SP3 Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/a39034e8-a077-11e5-8d69-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a39034e8-a077-11e5-8d69-5254006e85c2</a><br/> Hard Failover: EL7 Server/SLES11 SP3 Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/a403d43e-a077-11e5-8d69-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a403d43e-a077-11e5-8d69-5254006e85c2</a></p> <p>Also, recovery-mds-scale test_failover_ost failed with same error.<br/> <a href="https://testing.hpdd.intel.com/test_sets/b6b0f202-a080-11e5-85ed-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b6b0f202-a080-11e5-85ed-5254006e85c2</a></p> <p>recovery-double-scale test_pairwise_fail failing with same issue.<br/> Hard Failover: EL7 Server/SLES11 SP3 Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/a428828e-a077-11e5-8d69-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a428828e-a077-11e5-8d69-5254006e85c2</a></p> <p>Another instance found for hardfailover: EL6.7 Server/SLES11 SP3 Clients<br/> <a href="https://testing.hpdd.intel.com/test_sets/762762d0-ba4c-11e5-9a07-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/762762d0-ba4c-11e5-9a07-5254006e85c2</a></p> Development Rank 1|hzxemn: Rank (Obsolete) 9223372036854775807 Severity