[LU-15553] replay-vbr test 12a fails with 'test_12a failed with 4'

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.16.0, Lustre 2.15.0, Lustre 2.15.4, Lustre 2.15.5, Lustre 2.15.6
• Labels: zfs
• Severity: 3
• Assignee: Lai Siyao
• Reporter: James Nunez (Inactive)

Description

replay-vbr test_12a started failing with 'test_12a failed with 4' on August 4, 2021 for Lustre 2.14.53.7 with logs at https://testing.whamcloud.com/test_sets/17efe0ba-7e4a-4e7f-b7f5-02383e1314c5. We’ve seen this test fail for ZFS and ldiskfs, but, so far, always DNE.

Looking at a recent failure at https://testing.whamcloud.com/test_sets/014ce4c3-c654-47f9-9333-1c58ebf545c3, the suite_log shows

CMD: onyx-24vm7 e2label /dev/mapper/mds1_flakey 2>/dev/null
Started lustre-MDT0000
CMD: onyx-55vm7.onyx.whamcloud.com unlinkmany /mnt/lustre/f12a.replay-vbr- 25
 - unlinked 0 (time 1643080125 ; total 0 ; last 0)
total: 25 unlinks in 0 seconds: inf unlinks/second
CMD: onyx-55vm7.onyx.whamcloud.com unlinkmany /mnt/lustre/f12a.replay-vbr-3- 25
 - unlinked 0 (time 1643080125 ; total 0 ; last 0)
total: 25 unlinks in 0 seconds: inf unlinks/second
CMD: onyx-55vm7.onyx.whamcloud.com checkstat -v /mnt/lustre/d12a.replay-vbr/f12a.replay-vbr
 replay-vbr test_12a: @@@@@@ FAIL: test_12a failed with 4 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:6391:error()
  = /usr/lib64/lustre/tests/test-framework.sh:6695:run_one()

Looking at the code for this test,

1152     # All 50 files should have been replayed
1153     do_node $CLIENT1 unlinkmany $DIR/$tfile- 25 || return 2
1154     do_node $CLIENT1 unlinkmany $DIR/$tfile-3- 25 || return 3
1155     do_node $CLIENT1 $CHECKSTAT $DIR/$tdir/$tfile && return 4
1156 
1157     return 0
1158 }
1159 run_test 12a "lost data due to missed REMOTE client during replay"

The call to checkstat is what produces this error.

Issue Links

• is related to: LU-9096 sanity test_253: File creation failed after rm

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/56943/
Subject: LU-15553 test: mkdir_on_mdt0 in replay-vbr.sh
Project: fs/lustre-release
Branch: b2_15
Current Patch Set:
Commit: 5f0b7d1950615fc227b0a4aa0002f00046ae01f1

[Jian Yu]

Lustre 2.15.6 RC1: https://testing.whamcloud.com/test_sets/7abc37df-fdfe-46c8-9a6b-e18930aec48c

[Gerrit Updater]

"Jian Yu <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/56943
Subject: LU-15553 test: mkdir_on_mdt0 in replay-vbr.sh
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: 202791be5bc3d3e6c32474ec537db6b9de040fae

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51665/
Subject: LU-15553 test: mkdir_on_mdt0 in replay-dual
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: fc8042c7c80b79cb6c5578a7f2598fdb9d955e6e

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/56539/
Subject: LU-15553 test: mkdir_on_mdt0 in conf-sanity.sh
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: ed016772eaf87201904b27cf630699728ca5d2d4

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/56540/
Subject: LU-15553 test: mkdir_on_mdt0 in replay-vbr.sh
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: f6c733c422eae64cea93c33fb14e6adb2eed81d0

[Jian Yu]

This failure occurred consistently on the following test sessions:
lustre-master-el9.4-x86_64-full-dne-zfs-part-2
lustre-master-el9.3-x86_64-full-dne-zfs-part-2
lustre-master-el8.10-x86_64-full-dne-zfs-part-2
lustre-master-el8.9-x86_64-full-dne-zfs-part-2
lustre-master-el8.8-x86_64-full-dne-zfs-part-2

[Jian Yu]

+1 on master branch: https://testing.whamcloud.com/test_sets/996381e8-642f-4995-b156-75be0da63bb9

[Gerrit Updater]

"Lai Siyao <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/56540
Subject: LU-15553 test: mkdir_on_mdt0 in replay-vbr.sh
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: cc2608d647dfd25d5e79cbe013610b0152d9ddf9

[Gerrit Updater]

"Lai Siyao <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/56539
Subject: LU-15553 test: mkdir_on_mdt0 in conf-sanity.sh
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ebeb04ba6fcecdd69bbff66d9cff755f95156e34