[LU-10589] sanity-dom test_251: test_sanity failed with 1

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.14.0
• Severity: 3
• Assignee: James Nunez (Inactive)
• Reporter: Maloo

Description

sanity-dom test_251 - test_sanity failed with 1
^^^^^^^^^^^^^ DO NOT REMOVE LINE ABOVE ^^^^^^^^^^^^^

This issue was created by maloo for liuying <[e-mail]>

This issue relates to the following test suite run:

test_251 failed with the following error:

test_sanity failed with 1

More reports:
https://testing.hpdd.intel.com/sub_tests/49fa4b46-05fe-11e8-bd00-52540065bddc
https://testing.hpdd.intel.com/sub_tests/9cda340a-05ab-11e8-bd00-52540065bddc
https://testing.hpdd.intel.com/sub_tests/131a816a-0359-11e8-bd00-52540065bddc
https://testing.hpdd.intel.com/sub_tests/da9864e8-0280-11e8-a7cd-52540065bddc

Issue Links

• is related to: LU-10496 sanity-dom sanity test_39k: FAIL: mtime is lost on close: 1515705427, should be 1484169395
• is related to: LU-11366 replay-single timeout test 80f: rm: cannot remove '/mnt/lustre/d80f.replay-single/remote_dir': Input/output error

Comments

[Peter Jones]

Landed for 2.14

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/36369/
Subject: LU-10589 tests: insulate sanity-dom from test failures
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1140d209f0fb7a9200cc9c6d03553d02035ad6b8

[Gerrit Updater]

James Nunez ([e-mail]) uploaded a new patch: https://review.whamcloud.com/36369
Subject: LU-10589 tests: insulate sanity-dom from test failures
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: cdd29ccbfd4a3a5214f2e9c8aa812cba2b921608

[Wang Shilong (Inactive)]

This is consistently failing..

https://testing.whamcloud.com/test_sets/870c7a78-467f-11e9-9646-52540065bddc

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34186
Subject: LU-10589 tests: Log nested tests differently
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: d8f0ba7811db825d6b0befec85ab2cbae5c1a0eb

[Qian Yingjin (Inactive)]

another on master:

https://testing.whamcloud.com/test_sets/9665c524-920c-11e8-b0aa-52540065bddc

[Bob Glossman (Inactive)]

another on master:
https://testing.hpdd.intel.com/test_sets/f44be61c-106d-11e8-a10a-52540065bddc

[James Nunez (Inactive)]

In all of these cases, it looks like sanity-dom test_251 (sanity test 251) actually passes:

== sanity test 251: Handling short read and write correctly ========================================== 12:24:21 (1517574261)
fail_loc=0xa0001407
fail_val=1
fail_loc=0xa0001407
fail_val=1
Resetting fail_loc on all nodes...CMD: onyx-50vm1.onyx.hpdd.intel.com,onyx-50vm2,onyx-50vm3,onyx-50vm4,onyx-50vm5 lctl set_param -n fail_loc=0 	    fail_val=0 2>/dev/null
done.

…

CMD: onyx-50vm1.onyx.hpdd.intel.com,onyx-50vm2,onyx-50vm3,onyx-50vm4,onyx-50vm5 dmesg
PASS 251 (3s)
== sanity test complete, duration 386 sec ============================================================ 12:24:24 (1517574264)

Yet, there is one or more previous tests that does fail and, thus, the sanity suite fails. I think the sanity suite failure is getting assigned to the last test run for sanity:

== sanity test complete, duration 390 sec ============================================================ 07:55:23 (1516953323)
sanity: FAIL: test_39k mtime is lost on close: 1516953063, should be 1485417029
…
Stopping client onyx-32vm5.onyx.hpdd.intel.com /mnt/lustre2 opts:
Stopping client onyx-32vm6.onyx.hpdd.intel.com /mnt/lustre2 opts:
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck
 sanity-dom test_sanity: @@@@@@ FAIL: test_sanity failed with 1 

Note that “test_sanity failed with 1” is the failure message for the suite and it is the error message for test 251.

[Andreas Dilger]

Test_2 and test_3 first failed with truncate errors:

== sanity-dom test 2: Write with a seek, append, read from a single mountpoint ======================= 18:40:18 (1517337618)
dd: failed to truncate to 4194304 bytes in output file '/mnt/lustre/f2.sanity-dom': File too large
 sanity-dom test_2: @@@@@@ FAIL: test_2 failed with 1