[LU-6006] replay-dual test_22a: Remote creation failed 1 Created: 08/Dec/14  Updated: 22/Dec/21  Resolved: 20/May/20

Status: Resolved
Project: Lustre
Component/s: None
Affects Version/s: Lustre 2.7.0, Lustre 2.8.0, Lustre 2.9.0, Lustre 2.10.0, Lustre 2.10.1, Lustre 2.11.0, Lustre 2.10.2, Lustre 2.10.3, Lustre 2.10.7, Lustre 2.12.1
Fix Version/s: Lustre 2.14.0

Type: Bug Priority: Minor
Reporter: Sarah Liu Assignee: Alex Zhuravlev
Resolution: Fixed Votes: 0
Labels: None
Environment:

server and client: lustre-master build # 2770

Issue Links:
Related
is related to LU-10137 sanity-sec test_23b: setfacl /mnt/lus... Open
is related to LU-5759 replay-dual test_21b: Restart of mds0... Resolved
is related to LU-14406 replay-dual test 22d fails with “Remo... Open
is related to LU-10729 replay-dual test_23d: FAIL: Remote cr... Resolved
Severity: 3
Rank (Obsolete): 16742

 Description   

This issue was created by maloo for sarah <sarah@whamcloud.com>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/53deabf8-7ec4-11e4-ab67-5254006e85c2.

The sub-test test_22a failed with the following error:

Remote creation failed 1

CMD: shadow-23vm5 mkdir /mnt/lustre2/d22a.replay-dual/remote_dir/dir
shadow-23vm5: mkdir: cannot create directory `/mnt/lustre2/d22a.replay-dual/remote_dir/dir': No such file or directory
 replay-dual test_22a: @@@@@@ FAIL: Remote creation failed 1

Info required for matching: replay-dual 22a

 Comments   
Comment by Jodi Levi (Inactive) [ 09/Dec/14 ]

This problem may be fixed with LU-5759

Comment by Andreas Dilger [ 09/Dec/14 ]

It looks like all the test_22* subtests are failing after LU-5759 has been hit, so this may just be a configuration error at this point (e.g. MDS not mounted). There may not be anything to fix here once http://review.whamcloud.com/12363 fixes that problem.

Comment by James Nunez (Inactive) [ 18/Dec/14 ]

replay-dual 22* and 23* all failed with "Remote creation failed 1" for tests run on the OpenSFS cluster. Results at https://testing.hpdd.intel.com/test_sets/78dc0abe-861b-11e4-ac52-5254006e85c2 .

Prior to this, test 21b failed with "Restart of mds0 failed!" So, maybe the MDS was never remouted?

Comment by James Nunez (Inactive) [ 12/Feb/15 ]

Hit this problem again with lustre-master tag 2.6.93 with results at https://testing.hpdd.intel.com/test_sets/9c692176-adec-11e4-a0b6-5254006e85c2

Comment by Saurabh Tandan (Inactive) [ 03/Feb/16 ]

Another instance failing with the same error as above for tag 2.7.66 for FULL - EL6.7 Server/EL6.7 Client - DNE , master build# 3314.
https://testing.hpdd.intel.com/test_sets/86ca0268-ca83-11e5-9215-5254006e85c2

Comment by Saurabh Tandan (Inactive) [ 10/Feb/16 ]

Another instance found for Full tag 2.7.66 - EL6.7 Server/EL6.7 Client - DNE, build# 3314
https://testing.hpdd.intel.com/test_sets/86ca0268-ca83-11e5-9215-5254006e85c2

Comment by Sarah Liu [ 14/Mar/16 ]

from the previous comments, this issue affects 2.8.0 and 2.9
another instance seen on DNE mode, lustre-master/#3324
https://testing.hpdd.intel.com/test_sets/5e000c36-e7db-11e5-afa2-5254006e85c2

Comment by James Casper [ 24/May/17 ]

2.9.57, b3575:
https://testing.hpdd.intel.com/test_sessions/edde2a3e-9ae8-434a-8170-b64e9e85529c

Comment by Saurabh Tandan (Inactive) [ 30/Jan/18 ]

2.10.57
SLES 12 SP3 Server/DNE/ldiskfs
SLES 12 SP3 Client
https://testing.hpdd.intel.com/test_sets/ac6fcf5c-fd4e-11e7-a7cd-52540065bddc

Comment by Minh Diep [ 14/Feb/18 ]

+1 on b2_10
https://testing.hpdd.intel.com/test_sets/9f8beb02-1169-11e8-bd00-52540065bddc

Comment by James Nunez (Inactive) [ 03/Apr/18 ]

replay-dual tests 22a, 22b, 22c, 22d, 23a, 23c seem to have all stopped failing with this error over all branches.

test 22a last failed on 2018-01-19 06:11:08 UTC

test 22b last failed on 2018-03-13 14:44:45 UTC

test 22c last failed on 2018-01-10 06:03:24 UTC

test 22d last failed on 2018-01-10 06:03:24 UTC

test 23a last failed on 2018-01-10 06:03:24 UTC

test 23c last failed on 2018-01-10 06:03:24 UTC

 

Test 23d last failed on 2018-03-23 01:16:02 UTC for b2_10. Please see LU-10729.

 

We are still seeing replay-dual test_23b failing with this error message. See https://testing.hpdd.intel.com/test_sets/0c24b7cc-35e5-11e8-8f8a-52540065bddc . When test 23b fails with 'Remote creation failed 1', replay-vbr, insanity, sanity-quota, sanity-sec, sanity-pfl, lustre-rsync-test, metadata-updates, mds-survey, mmp, large-scale, lnet-selftest, and obdfilter-survey all fail with some form of

rm: cannot remove '/mnt/lustre/d23b.replay-dual': Directory not empty
 insanity : @@@@@@ FAIL: remove sub-test dirs failed 
Comment by Minh Diep [ 11/Apr/18 ]

+1 on 2.10 https://testing.hpdd.intel.com/test_sessions/33663a7e-8f39-4015-95ec-ba0769bf55d5

Comment by Gerrit Updater [ 28/Jan/20 ]

Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/37343
Subject: LU-6006 tests: add sleep 1 after command in background
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8572b72a75f1baf700dc480d8af04b682acdf33e

Comment by Gerrit Updater [ 20/May/20 ]

Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/37343/
Subject: LU-6006 tests: add sleep 1 after command in background
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 2d57999401072a034650d00a37fb59ef9b3f53d0

Comment by Peter Jones [ 20/May/20 ]

Landed for 2.14

Comment by Artem Blagodarenko (Inactive) [ 11/Dec/20 ]

+1 https://testing.whamcloud.com/test_sets/e910ae4b-f45d-4680-b16b-b2f4b7ce3b05

Comment by Gerrit Updater [ 22/Dec/21 ]

"Etienne AUJAMES <eaujames@ddn.com>" uploaded a new patch: https://review.whamcloud.com/45916
Subject: LU-6006 tests: add sleep 1 after command in background
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 98f14023cb3a2f8b69180c33b555ad2be44894a2

Generated at Sat Feb 10 01:56:23 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.