https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6006 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/53deabf8-7ec4-11e4-ab67-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/53deabf8-7ec4-11e4-ab67-5254006e85c2</a>.</p> <p>The sub-test test_22a failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Remote creation failed 1 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: shadow-23vm5 mkdir /mnt/lustre2/d22a.replay-dual/remote_dir/dir shadow-23vm5: mkdir: cannot create directory `/mnt/lustre2/d22a.replay-dual/remote_dir/dir': No such file or directory replay-dual test_22a: @@@@@@ FAIL: Remote creation failed 1 </pre> </div></div> <p>Info required for matching: replay-dual 22a</p> server and client: lustre-master build # 2770 LU-6006

replay-dual test_22a: Remote creation failed 1

Bug Minor Resolved Fixed Alex Zhuravlev Sarah Liu Mon, 8 Dec 2014 22:27:30 +0000 Wed, 22 Dec 2021 09:04:36 +0000 Wed, 20 May 2020 14:01:50 +0000 Lustre 2.7.0 Lustre 2.8.0 Lustre 2.9.0 Lustre 2.10.0 Lustre 2.10.1 Lustre 2.11.0 Lustre 2.10.2 Lustre 2.10.3 Lustre 2.10.7 Lustre 2.12.1 Lustre 2.14.0 0 10 <p>This problem may be fixed with <a href="https://jira.whamcloud.com/browse/LU-5759" title="replay-dual test_21b: Restart of mds0 failed" class="issue-link" data-issue-key="LU-5759"><del>LU-5759</del></a></p> <p>It looks like all the test_22&#42; subtests are failing after <a href="https://jira.whamcloud.com/browse/LU-5759" title="replay-dual test_21b: Restart of mds0 failed" class="issue-link" data-issue-key="LU-5759"><del>LU-5759</del></a> has been hit, so this may just be a configuration error at this point (e.g. MDS not mounted). There may not be anything to fix here once <a href="http://review.whamcloud.com/12363" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12363</a> fixes that problem.</p> <p>replay-dual 22* and 23* all failed with "Remote creation failed 1" for tests run on the OpenSFS cluster. Results at <a href="https://testing.hpdd.intel.com/test_sets/78dc0abe-861b-11e4-ac52-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/78dc0abe-861b-11e4-ac52-5254006e85c2</a> .</p> <p>Prior to this, test 21b failed with "Restart of mds0 failed!" So, maybe the MDS was never remouted?</p> <p>Hit this problem again with lustre-master tag 2.6.93 with results at <a href="https://testing.hpdd.intel.com/test_sets/9c692176-adec-11e4-a0b6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9c692176-adec-11e4-a0b6-5254006e85c2</a></p> <p>Another instance failing with the same error as above for tag 2.7.66 for FULL - EL6.7 Server/EL6.7 Client - DNE , master build# 3314.<br/> <a href="https://testing.hpdd.intel.com/test_sets/86ca0268-ca83-11e5-9215-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/86ca0268-ca83-11e5-9215-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 - EL6.7 Server/EL6.7 Client - DNE, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/86ca0268-ca83-11e5-9215-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/86ca0268-ca83-11e5-9215-5254006e85c2</a></p> <p>from the previous comments, this issue affects 2.8.0 and 2.9<br/> another instance seen on DNE mode, lustre-master/#3324<br/> <a href="https://testing.hpdd.intel.com/test_sets/5e000c36-e7db-11e5-afa2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5e000c36-e7db-11e5-afa2-5254006e85c2</a></p> <p>2.9.57, b3575:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/edde2a3e-9ae8-434a-8170-b64e9e85529c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/edde2a3e-9ae8-434a-8170-b64e9e85529c</a></p> <p>2.10.57 <br/> SLES 12 SP3 Server/DNE/ldiskfs<br/> SLES 12 SP3 Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/ac6fcf5c-fd4e-11e7-a7cd-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ac6fcf5c-fd4e-11e7-a7cd-52540065bddc</a></p> <p>+1 on b2_10<br/> <a href="https://testing.hpdd.intel.com/test_sets/9f8beb02-1169-11e8-bd00-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9f8beb02-1169-11e8-bd00-52540065bddc</a></p> <p>replay-dual tests 22a, 22b, 22c, 22d, 23a, 23c seem to have all stopped failing with this error over all branches.</p> <p>test 22a last failed on 2018-01-19 06:11:08 UTC</p> <p>test 22b last failed on 2018-03-13 14:44:45 UTC</p> <p>test 22c last failed on 2018-01-10 06:03:24 UTC</p> <p>test 22d last failed on 2018-01-10 06:03:24 UTC</p> <p>test 23a last failed on 2018-01-10 06:03:24 UTC</p> <p>test 23c last failed on 2018-01-10 06:03:24 UTC</p> <p> </p> <p>Test 23d last failed on 2018-03-23 01:16:02 UTC for b2_10. Please see <a href="https://jira.whamcloud.com/browse/LU-10729" title="replay-dual test_23d: FAIL: Remote creation failed 1 : mkdir: cannot create directory&#39;: File exists" class="issue-link" data-issue-key="LU-10729"><del>LU-10729</del></a>.</p> <p> </p> <p>We are still seeing replay-dual test_23b failing with this error message. See <a href="https://testing.hpdd.intel.com/test_sets/0c24b7cc-35e5-11e8-8f8a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0c24b7cc-35e5-11e8-8f8a-52540065bddc</a> . When test 23b fails with 'Remote creation failed 1', replay-vbr, insanity, sanity-quota, sanity-sec, sanity-pfl, lustre-rsync-test, metadata-updates, mds-survey, mmp, large-scale, lnet-selftest, and obdfilter-survey all fail with some form of</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rm: cannot remove '/mnt/lustre/d23b.replay-dual': Directory not empty  insanity : @@@@@@ FAIL: remove sub-test dirs failed  </pre> </div></div> <p>+1 on 2.10 <a href="https://testing.hpdd.intel.com/test_sessions/33663a7e-8f39-4015-95ec-ba0769bf55d5" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/33663a7e-8f39-4015-95ec-ba0769bf55d5</a></p> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37343" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37343</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6006" title="replay-dual test_22a: Remote creation failed 1" class="issue-link" data-issue-key="LU-6006"><del>LU-6006</del></a> tests: add sleep 1 after command in background<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8572b72a75f1baf700dc480d8af04b682acdf33e</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37343/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37343/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6006" title="replay-dual test_22a: Remote creation failed 1" class="issue-link" data-issue-key="LU-6006"><del>LU-6006</del></a> tests: add sleep 1 after command in background<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 2d57999401072a034650d00a37fb59ef9b3f53d0</p> <p>Landed for 2.14</p> <p>+1 <a href="https://testing.whamcloud.com/test_sets/e910ae4b-f45d-4680-b16b-b2f4b7ce3b05" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/e910ae4b-f45d-4680-b16b-b2f4b7ce3b05</a></p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45916" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45916</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6006" title="replay-dual test_22a: Remote creation failed 1" class="issue-link" data-issue-key="LU-6006"><del>LU-6006</del></a> tests: add sleep 1 after command in background<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 98f14023cb3a2f8b69180c33b555ad2be44894a2</p> Related LU-10137 LU-5759 LU-14406 LU-10729 Development Rank 1|hzx25b: Rank (Obsolete) 16742 Severity