https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11366 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;adilger@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/5b970932-b5dc-11e8-9df3-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5b970932-b5dc-11e8-9df3-52540065bddc</a></p> <p>test_80f failed review-dne-zfs-part-4 timed out with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rm: cannot remove '/mnt/lustre/d80f.replay-single/remote_dir': Input/output error rmdir failed </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> replay-single test_80f - rmdir failed</p> LU-11366

replay-single timeout test 80f: rm: cannot remove '/mnt/lustre/d80f.replay-single/remote_dir': Input/output error

Bug Minor Resolved Duplicate Lai Siyao Maloo Tue, 11 Sep 2018 20:31:30 +0000 Tue, 26 Feb 2019 07:47:27 +0000 Tue, 26 Feb 2019 07:47:27 +0000 Lustre 2.12.0 Lustre 2.13.0 0 8 <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33153" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33153</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11366" title="replay-single timeout test 80f: rm: cannot remove &#39;/mnt/lustre/d80f.replay-single/remote_dir&#39;: Input/output error" class="issue-link" data-issue-key="LU-11366"><del>LU-11366</del></a> tests: disable tests for DNE/ZFS testing<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ca1b8b35ff7ec0df64180fcfb19fcd56ef890e6d</p> <p>Lai, could you please take a look into this. The replay-single test_80f is failing 100% of the time with review-dne-zfs-part-4 and causing a lot of test failures. We don't currently enforce this test session, but we'd like to get that test passing and enable it. James tried disabling that subtest, but then test_80g and the next tests are failing in a similar manner, so it is likely that the same issue is affecting all of those tests.</p> <p>Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33515" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33515</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11366" title="replay-single timeout test 80f: rm: cannot remove &#39;/mnt/lustre/d80f.replay-single/remote_dir&#39;: Input/output error" class="issue-link" data-issue-key="LU-11366"><del>LU-11366</del></a> test: test replay-single with full debug<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6606aca76ec770a27a6498fc75079eafd8d66517</p> <p>Lai, I saw that you have abandoned the debug patch. Does that mean you understand what is causing the test failures here?</p> <p>We'd like to enforce <tt>review-dne-zfs-part-4</tt> and remove the non-DNE versions of the ZFS tests, to reduce the testing load and to get better test coverage.</p> <p>no, full debug mode failed to dump logs because of crash, I'll turn to this when I have spare time.</p> <p>+1 at <a href="https://testing.whamcloud.com/test_sessions/86a18ffa-c1c0-4f89-99d9-bef2ad60e1ac" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/86a18ffa-c1c0-4f89-99d9-bef2ad60e1ac</a></p> <p>+1 at <a href="https://testing.whamcloud.com/test_sessions/d7f0ec38-7b1d-4cba-8e5f-ca45754a694d" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/d7f0ec38-7b1d-4cba-8e5f-ca45754a694d</a></p> <p>+1 at <a href="https://testing.whamcloud.com/test_sessions/456887b0-6287-497f-8357-d511966ac54c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/456887b0-6287-497f-8357-d511966ac54c</a></p> <p>This has suddenly become brutally common:</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <th class='confluenceTh'>test_80f</th> <td class='confluenceTd'><ul> <li>Error: 'rmdir failed' <br/> Failure Rate: 34.15% of most recent 41 runs, 59 skipped (all branches)</li> </ul> </td> </tr> </tbody></table> </div> <p>That's review-dne-zfs-part-4</p> <p><a href="https://testing.whamcloud.com/test_sets/e1b9590e-1ac0-11e9-9ed8-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/e1b9590e-1ac0-11e9-9ed8-52540065bddc</a></p> <p>I was hitting this locally very often, but not with <a href="https://review.whamcloud.com/#/c/34069/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/34069/</a></p> <p>few successful replay-single with ZFS/DNE using the patch above:<br/> <a href="https://testing.whamcloud.com/test_sets/40843630-1dea-11e9-9ed8-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/40843630-1dea-11e9-9ed8-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/07b11042-1e5a-11e9-8388-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/07b11042-1e5a-11e9-8388-52540065bddc</a></p> <p>I think 2.13 should be fine given <a href="https://jira.whamcloud.com/browse/LU-10143" title="LBUG dt_object.h:2166:dt_declare_record_write" class="issue-link" data-issue-key="LU-10143"><del>LU-10143</del></a> landed</p> <p>I checked with Maloo - all tests passed since <a href="https://jira.whamcloud.com/browse/LU-10143" title="LBUG dt_object.h:2166:dt_declare_record_write" class="issue-link" data-issue-key="LU-10143"><del>LU-10143</del></a> landing.<br/> <a href="https://testing.whamcloud.com/test_sets/query?utf8=" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/query?utf8=</a>✓&amp;warn%5Bnotice%5D=&amp;test_set_script_id=f6a12204-32c3-11e0-a61c-52540025f9ae&amp;query_bugs=&amp;builds=&amp;hosts=&amp;commit_id=&amp;test_groups%5B%5D=review-dne-zfs-part-4&amp;horizon=1123200&amp;window%5Bstart_date%5D=&amp;window%5Bend_date%5D=&amp;os_type_id=&amp;distribution_type_id=&amp;architecture_type_id=&amp;file_system_type_id=&amp;branch_type_id=24a6947e-04a9-11e1-bb5f-52540025f9af&amp;network_type_id=&amp;commit=Update+results&amp;num_results=250</p> <p>Issue was fixed via patch <a href="https://review.whamcloud.com/34069" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34069</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-10143" title="LBUG dt_object.h:2166:dt_declare_record_write" class="issue-link" data-issue-key="LU-10143"><del>LU-10143</del></a> osd-zfs: allocate sequence in advance</tt>".</p> Related LU-9157 LU-11330 LU-10740 LU-11538 LU-10143 LU-10589 LU-11748 Development Rank 1|i00287: Rank (Obsolete) 9223372036854775807 Severity