[LU-4442] Failure on test suite replay-vbr test_7g: Test 7g.3 failed

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.6.0, Lustre 2.5.1
• Affects Version/s: Lustre 2.1.6, Lustre 2.6.0, Lustre 2.5.1, Lustre 2.4.3
• Labels: mn1 mn4
• Environment: client and server: lustre-master build 1823 RHEL6 ldiskfs
• Severity: 3
• Assignee: Emoly Liu
• Reporter: Maloo

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: http://maloo.whamcloud.com/test_sets/930d9194-74df-11e3-96b0-52540035b04c.

The sub-test test_7g failed with the following error:

Test 7g.3 failed

test log shows:

CMD: client-31vm7 /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null
osp.lustre-OST0000-osc-MDT0000.old_sync_processed
osp.lustre-OST0001-osc-MDT0000.old_sync_processed
osp.lustre-OST0002-osc-MDT0000.old_sync_processed
osp.lustre-OST0003-osc-MDT0000.old_sync_processed
osp.lustre-OST0004-osc-MDT0000.old_sync_processed
osp.lustre-OST0005-osc-MDT0000.old_sync_processed
osp.lustre-OST0006-osc-MDT0000.old_sync_processed
wait mds1 secs maximumly for client-31vm7 mds-ost sync done.
/usr/lib64/lustre/tests/test-framework.sh: line 2135: [: mds1: integer expression expected
CMD: client-31vm7 /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed
1
1
1
1
1
1
1
 recovery node iozone not done in mds1 sec. 
 replay-vbr test_7g: @@@@@@ FAIL: Test 7g.3 failed 

Comments

[Jian Yu]

Back-ported patch for Lustre b2_4 branch: http://review.whamcloud.com/9987
Back-ported patch for Lustre b2_1 branch: http://review.whamcloud.com/9988

[Jian Yu]

Also hit on interop test between 2.5.1 server and master client:

This is because http://review.whamcloud.com/9213 was landed on Lustre b2_5 branch. We need change Lustre version number 2.5.52 to 2.5.1 in replay-vbr test_7g().
Patch for master branch: http://review.whamcloud.com/9986

[Sarah Liu]

Also hit on interop test between 2.5.1 server and master client:

https://maloo.whamcloud.com/test_sets/8e4f6b3a-b244-11e3-a93f-52540035b04c

[Jian Yu]

Landing http://review.whamcloud.com/8371 on Lustre b2_5 build #25 also caused this regression failure on Lustre b2_5 branch:
https://maloo.whamcloud.com/test_sets/0103e6d0-977c-11e3-b941-52540035b04c
https://maloo.whamcloud.com/test_sets/c79df196-97b8-11e3-acb5-52540035b04c

[Jian Yu]

Patches for Lustre b2_5 branch:
http://review.whamcloud.com/9289
http://review.whamcloud.com/9290

[Emoly Liu]

Both patches have been landed to 2.6.

[Emoly Liu]

Thanks, Tappro, I saw your choice of http://review.whamcloud.com/8973 .

[Emoly Liu]

The root cause of this failure is that since mdt_object_exists() was added to mdt_reint_link() in http://review.whamcloud.com/#/c/8371, if the child object doesn't exist, there is no chance to do object version check and client1 will not be evicted.

I create the following two patches to fix this problem, and I am not sure which is better:

• http://review.whamcloud.com/8972 is to do mdt_version_get_check() before calling mdt_object_exists(), so that the client can be evicted due to (EOVERFLOW)"Version mismatch" as before;
• http://review.whamcloud.com/8973 is to change test_7g script to check the different return value for different MDS version.

Tappro, could you please give any advice? Thanks.

[Emoly Liu]

By searching maloo, I notice this error has occurred since Dec. 21, and finally I find it is related to LU-3528 http://review.whamcloud.com/8371 .

I am working on the patch.

[Emoly Liu]

The maloo test report https://maloo.whamcloud.com/test_logs/2a98cbc0-7bfa-11e3-a7b6-52540035b04c/show_text shows that test_7g has another problem besides test script issue to be fixed by http://review.whamcloud.com/8796.

I will investigate and provide another patch for it.