[LU-1060] Test failure on test suite replay-vbr, subtest test_7c

• Type: Bug
• Resolution: Duplicate
• Priority: Blocker
• Fix Version/s: Lustre 2.2.0
• Affects Version/s: Lustre 2.2.0, Lustre 2.1.1
• Severity: 3
• Assignee: nasf (Inactive)
• Reporter: Maloo

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/c95fc5c2-4c42-11e1-bd50-5254004bbbd3.

The sub-test test_7c failed with the following error:

Test 7c.2 failed

Info required for matching: replay-vbr 7c

Issue Links

• is related to: LU-966 post-fsck MDS LBUG during recovery due to missing FID
• Trackbacks: Lustre 2.1.1 release testing tracker Lustre 2.1.1 RC2 Tag: v2110RC2 Build:
• Trackbacks: Changelog 2.2 version 2.2.0 Support for networks: o2iblnd OFED 1.5.4 Server support for kernels: 2.6.32220.4.2.el6 (RHEL6) Client support for unpatched kernels: 2.6.18274.18.1.el5 (RHEL5) 2.6.32220.4.2.el6 (RHEL6) 2.6.32.360....

Comments

[Mikhail Pershin]

VBR detects ENOENT cases already and fails if some object is missing. For such object VBR count its version as ENOENT_VERSION and compare it with version in replay, FID_002 in your example. So there must be version mismatch. If that doesn't work for some reason, we need to find that reason. Keep in mind that this worked well for a long time but failed in 2.1<->2.1.55 case, so probably this is compatibility issue between lustre versions. First of all I'd check where that -ENOENT came from and why VBR checks missed it.

[nasf (Inactive)]

The patch for above issue:

http://review.whamcloud.com/#change,2144

[nasf (Inactive)]

There are some defects in current VBR implementation. For example in replay-vbr.sh test_7c

test_7c() {
...
    first="createmany -o $DIR/$tdir/$tfile- 2"
    lost="rm $MOUNT2/$tdir/$tfile-0; mkdir $MOUNT2/$tdir/$tfile-0"
    last="mv $DIR/$tdir/$tfile-1 $DIR/$tdir/$tfile-0"
    test_7_cycle "$first" "$lost" "$last" || error "Test 7c.2 failed"
...
}

The operations sequence in test_7_cycle() is as following:

(step 0) replay barrier
(step 1) client1 "first"
(step 2) client2 "lost"
(step 3) client1 "last"
(step 4) umount client2
(step 5) MDS failover
(step 6) client1 try to recover

Since client2 umount before MDS failover, and client1's "last" operation depends on client2's "lost" operation, client1 is expected to fail to replay the "last" operation.

But now we found client1 was not evicted after MDS failover. The reason is as following:

The original $tfile-0 FID was "FID_001" when created by client1 step 1, then the client2 unlink $tfile-0, and mkdir with the same name, the new FID for $tfile-0 is "FID_002" by step 2. When client1 performed step 3, it used the $tfile-0's new "FID_002", and such FID also be used when client1 replayed "last" during MDS failover. But during the MDS failover, client2 missed, then nobody re-created $tfile-0 with new "FID_002", so when client1 tried to replay "last" during MDS failover, it could not find the target $tfile-0 with "FID_002". Under such case, client1's recovery should be regarded as failure, and client1 should be evicted. But in current implementation, during the VBR phase, only evict the client with VBR failure (for object version unmatched cases). test_7c failed for "-ENOENT", no chance to compare the versions yet, so client1 was not evicted.

The simplest way to resolve the issue is to regard all target missed cases during recovery as VBR failure, then evict related client.

[Jian Yu]

Lustre Clients:
Tag: 2.1.0
Distro/Arch: RHEL5/x86_64 (kernel version: 2.6.18-238.19.1.el5)
Network: IB (in-kernel OFED)
ENABLE_QUOTA=yes

Lustre Servers:
Tag: v2_1_1_0_RC1
Distro/Arch: RHEL5/x86_64 (kernel version: 2.6.18-274.12.1.el5_lustre)
Build: http://build.whamcloud.com/job/lustre-b2_1/37/arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/
Network: IB (in-kernel OFED)

The same issue occurred:
https://maloo.whamcloud.com/test_sets/68802da0-54bc-11e1-9fd4-5254004bbbd3

[Jian Yu]

Lustre Tag: v2_1_1_0_RC1
Distro/Arch: RHEL5.7 (server), SLES11SP1 (client)
Lustre Build: http://build.whamcloud.com/job/lustre-b2_1/37/
Network: TCP

The same issue occurred:
https://maloo.whamcloud.com/test_sets/e2334988-5300-11e1-81a2-5254004bbbd3

[Peter Jones]

Fanyong

Could you please look into this 2.2 blocker?

Thanks

Peter

[Sarah Liu]

dmesg and debug log from servers