[LU-8333] replay-dual test_21b: can't check if COS works: rename replied w/o COS

• Status: Reopened
• Project: Lustre
• Affects Version/s: Lustre 2.7.0, Lustre 2.9.0, Lustre 2.10.0, Lustre 2.11.0
• Type: Bug
• Priority: Critical
• Reporter: Maloo
• Assignee: Hongchao Zhang
• Resolution: Unresolved
• Labels: always_except

Environment

Hard failover EL7 Server/Client
master, build# 3399

• Severity: 3

Description

This issue was created by maloo for Saurabh Tandan <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/31581902-3ad4-11e6-bbf5-5254006e85c2.

The sub-test test_21b failed with the following error:

can't check if COS works: rename replied w/o COS

Test log:

Started clients trevis-56vm5: 
CMD: trevis-56vm5 mount | grep /mnt/lustre' '
10.9.6.124@tcp:10.9.6.120@tcp:/lustre on /mnt/lustre type lustre (rw,flock,user_xattr)
CMD: trevis-56vm5 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/qt-3.3/bin:/usr/lib64/compat-openmpi16/bin:/usr/bin:/bin:/usr/sbin:/sbin::/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh set_default_debug \"-1\" \"all -lnet -lnd -pinger\" 4 
 replay-dual test_21b: @@@@@@ FAIL: can't check if COS works: rename replied w/o COS 

In past 15 days this issue has occurred around 36 times.

Issue Links

• is duplicated by: LU-2230 replay-dual test_21b: @@@@@@ FAIL: Th...
• is duplicated by: LU-4104 Failure on test suite replay-dual tes...
• is related to: LU-4470 replay-dual test_21b: FAIL: lustre-MD...
• is related to: LU-9586 replay-dual test cases 15c 21b remov...

Comments

[Peter Jones]

Hongchao

Could you please advise on this one?

Thanks

Peter

[Gerrit Updater]

Hongchao Zhang ([e-mail]) uploaded a new patch: http://review.whamcloud.com/21924
Subject: LU-8333 test: make sure COS is cleared
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: d0f614f780182e3ea56290e215cf5da73faf2cf1

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/21924/
Subject: LU-8333 test: make sure COS is cleared
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: ae2ccbc6da32def014463ee98fab76a93835d85d

[Peter Jones]

Landed for 2.9

[Saurabh Tandan (Inactive)]

This issue is still seen on master regularly.
https://testing.hpdd.intel.com/test_sets/c4a7f648-a1c0-11e6-8ed2-5254006e85c2
https://testing.hpdd.intel.com/sub_tests/4cab1870-a2d6-11e6-8986-5254006e85c2
https://testing.hpdd.intel.com/sub_tests/ca5e0b42-a2ae-11e6-8b77-5254006e85c2
https://testing.hpdd.intel.com/sub_tests/05adec2a-a274-11e6-8986-5254006e85c2

[Saurabh Tandan (Inactive)]

Reopening this ticket as the issue is still seen on master consistently. Please refer previous message above for latest failures.

[Gerrit Updater]

Hongchao Zhang ([e-mail]) uploaded a new patch: http://review.whamcloud.com/23830
Subject: LU-8333 test: debug log
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4df7fb623fb09fe078d86d51361e1f39e2db792d

[Hongchao Zhang]

I have checked the recent failures and found the transactions of the client2 (intended to introduce the transaction gap)
has been committed before failing over the MDT, which cause the client1 to recover successfully.

I'll consider some new means to test the COS feature.

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/25848
Subject: LU-8333 test: disable replay-dual 21b
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9d046935201bf2ea6b0e788f16c2d1bdaf843577

[Joseph Gmitter (Inactive)]

Hi Mike,

Can you comment on the validity of this test case in relation to commit on share? Is it a valid test case, providing useful testing?

Thanks.
Joe

[Gerrit Updater]

Hongchao Zhang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/26030
Subject: LU-8333 test: add more conflict operations
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: dad5c6ded7336ee7dba2a1cf79833840032b7306

[Gerrit Updater]

James Casper ([e-mail]) uploaded a new patch: https://review.whamcloud.com/26631
Subject: LU-8333: Excepts replay-dual test_21b while a new COS check test can be developed
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 95fde52018764c4a88c1d5ff479063631cb9cb6b

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/26631/
Subject: LU-8333 test: Add replay-dual 21b to ALWAYS_EXCEPT
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 83296931009c8bdc423a2374c921e610786870c3

[Peter Jones]

Landed for 2.10

[Saurabh Tandan (Inactive)]

Re-opening the ticket. Recently, replay-dual test 21b was run removing it from Always_Except list and the issue still persists.
https://testing.hpdd.intel.com/test_sets/3dc43c8a-7d49-11e7-9ce0-5254006e85c2
Looking at the comments history on this ticket it appears that Hongchao's proposed solution https://review.whamcloud.com/#/c/26030/ never got landed.

[Hongchao Zhang]

the patch https://review.whamcloud.com/#/c/26030/ has been updated.

[Gerrit Updater]

Hongchao Zhang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31538
Subject: LU-8333 test: use async_commit_count to test COS
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: c78f0c429012757b1b5d0c6bd5a368557f1d929a