Details
-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
Lustre 2.14.0, Lustre 2.15.0
-
None
-
DNE/ZFS
-
3
-
9223372036854775807
Description
replay-dual test_22d started failing on 30 SEPT 2020 when testing the patch for LU-13417 ‘mdd: default DNE MDT balance on new filesystems’ https://review.whamcloud.com/38553 (which has not itself landed, and cannot be the source of the test failures) with logs at https://testing.whamcloud.com/test_sets/b4dc132e-a4d6-4abb-a81c-753d8f23a18e. Since that time, this test has failed 10 times during review/patch testing. On 03 FEB 2021, we see this test fail with the same error message for branch/full testing; for DNE and ZFS with logs at https://testing.whamcloud.com/test_sets/17948bab-e647-4f32-874a-0fe07a464353.
Looking at this DNE/ZFS failure, we see the following in the suite_log:
CMD: trevis-66vm8 /usr/sbin/lctl --device lustre-MDT0000 notransno CMD: trevis-66vm8 /usr/sbin/lctl --device lustre-MDT0000 readonly CMD: trevis-66vm8 /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 CMD: trevis-66vm6 mkdir /mnt/lustre2/d22d.replay-dual/remote_dir/dir trevis-66vm6: mkdir: cannot create directory '/mnt/lustre2/d22d.replay-dual/remote_dir/dir': No such file or directory pdsh@trevis-66vm5: trevis-66vm6: ssh exited with exit code 1 replay-dual test_22d: @@@@@@ FAIL: Remote creation failed 1 Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:6273:error() = /usr/lib64/lustre/tests/replay-dual.sh:725:test_22d()
Looking at replay-dual test 22d, we see that the error is in create_remote_dir_files_22() in mkdir
607 create_remote_dir_files_22() {
608 do_node $CLIENT2 mkdir ${MOUNT2}/$remote_dir/dir || return 1
609 do_node $CLIENT1 createmany -o $MOUNT1/$remote_dir/dir/$tfile- 2 ||
610 return 2
611 do_node $CLIENT2 createmany -o $MOUNT2/$remote_dir/$tfile- 2 ||
612 return 3
613 return 0
614 }
Logs for more failures are at
https://testing.whamcloud.com/test_sets/260d3237-ec78-46c1-88a4-f5455a9265ce
https://testing.whamcloud.com/test_sets/4b7ef1a1-cb32-42b5-a2c2-9a2c7604900b
https://testing.whamcloud.com/test_sets/3e23af01-fb55-4bcf-8667-ea706fa084b3
