[LU-12261] sanity test 43b fails with 'expected error, got success'

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.13.0
• Fix Version/s: Lustre 2.13.0
• Type: Bug
• Priority: Critical
• Reporter: James Nunez (Inactive)
• Assignee: Patrick Farrell (Inactive)
• Resolution: Fixed

Environment

ARM clients

• Severity: 3

Description

sanity test_43b fails with 'expected error, got success' primarily for ARM architecture client testing. This test started failing on 2019-04-30.

Looking at one of the recent failures, logs at https://testing.whamcloud.com/test_sets/d1626222-6b7d-11e9-aeec-52540065bddc, we see the following in the suite_log

== sanity test 43b: truncate of file being executed should return -ETXTBSY =========================== 17:02:16 (1556643736)
 sanity test_43b: @@@@@@ FAIL: expected error, got success 
/usr/lib64/lustre/tests/sanity.sh: line 4172: /mnt/lustre/d43b.sanity/sleep: Text file busy

There are no errors in any of the node console logs.

There are several instances of this failure. Logs for a few are at
https://testing.whamcloud.com/test_sets/ea9ca12c-6b72-11e9-aeec-52540065bddc
https://testing.whamcloud.com/test_sets/1c74df3c-6b8e-11e9-8bb1-52540065bddc

There are a small number of x86_64 client failures
https://testing.whamcloud.com/test_sets/376809d8-6b78-11e9-a6f2-52540065bddc

Issue Links

• is related to: LU-12195 sanity/43 and sanityn/14 fail on loca...

Comments

[James A Simmons]

That is bizarre. truncate is returning ETXTBUSY as it should but the test doesn't see that result.

[Patrick Farrell (Inactive)]

Culprit is almost certainly:

commit 9a1f327a76f72c7713e53d8b354ff7f0e32be870
Author: Alex Zhuravlev <[e-mail]>
Date: Fri Apr 19 15:01:12 2019 +0300

LU-12195 tests: use sleep instead of wrapped multiop

in sanity/43* and sanity/14* tests as multiop is not a binary,
but libtool-wrapped script. the tests fail when started from a
build tree.

Change-Id: Iaec3433f03aab23583052373e5f0252d9eac7f04
Signed-off-by: Alex Zhuravlev <[e-mail]>
Reviewed-on: https://review.whamcloud.com/34721
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <[e-mail]>
Tested-by: Maloo <[e-mail]>
Reviewed-by: James Simmons <[e-mail]>
Reviewed-by: Oleg Drokin <[e-mail]>

diff --git a/lustre/tests/sanity.sh b/lustre/tests/sanity.sh

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34791
Subject: LU-12261 tests: Race between exec and truncate
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: cd40a13a6e7a2efa4f967b597ed6c5aa0b7d7435

[James A Simmons]

Thank you Patrick for fixing this.

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch https://review.whamcloud.com/34791/
Subject: LU-12261 tests: Race between exec and truncate
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c64855fca1504bddcb0fc7ad7316d8d6b20a9c6f

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35189
Subject: LU-12261 tests: Race between exec and truncate
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 23f71de911fd2a8e9004322a772529da5a64a47f