[LU-4341] Failure on test suite sanity test_170: expected 31 bad lines, but got 34

• Type: Bug
• Resolution: Unresolved
• Priority: Blocker
• Affects Version/s: Lustre 2.5.0, Lustre 2.6.0, Lustre 2.5.1, Lustre 2.7.0, Lustre 2.5.3, Lustre 2.8.0
• Labels: always_except
• Environment: server and client: lustre-master build # 1784 ; client is running SLES11 SP3
• Severity: 3
• Assignee: WC Triage
• Reporter: Maloo

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: http://maloo.whamcloud.com/test_sets/7756e5f2-5bb9-11e3-8d79-52540035b04c.

The sub-test test_170 failed with the following error:

expected 31 bad lines, but got 34

== sanity test 170: test lctl df to handle corrupted log ============================================= 00:50:22 (1385974222)
 sanity test_170: @@@@@@ FAIL: expected 31 bad lines, but got 34 

Comments

[Jian Yu]

Just narrowed down that it was the following operation in sanity test 150 which caused test 170 fail:

remount_client $MOUNT -> zconf_mount `hostname` $1 -> set_default_debug_nodes $client

After commenting out "set_default_debug_nodes $client", the failure disappeared.

[Jian Yu]

Finally, I found that it was sanity test 150 which caused test 170 fail on SLES11SP3 client:

run_test 150 "truncate/append tests"

I've tried several ways to fix the issue but failed. Still digging.

[Jian Yu]

There is a defect in sanity test_170(), and here is a patch for master branch to fix it: http://review.whamcloud.com/10296
Since the failure cannot be reproduced by only running sanity test 170, I'm checking the previous sub-tests to see which one is the culprit.

[Bob Glossman (Inactive)]

another sles11sp3 client in master:
https://maloo.whamcloud.com/test_sets/fd386500-d167-11e3-91ff-52540035b04c

[Bob Glossman (Inactive)]

another sles11sp3 client in master:
https://maloo.whamcloud.com/test_sets/a9add412-d0ac-11e3-b9d4-52540035b04c

[Bob Glossman (Inactive)]

starting to wonder if this is a high rate failure, maybe even 100%, in any sles client.

[Bob Glossman (Inactive)]

I think this is another, but says: Error: 'expected 24 bad lines, but got 27' instead of expected 31 bad lines, but got 34,

sles11sp3 client in b2_5:
https://maloo.whamcloud.com/test_sets/63de2e74-cf07-11e3-a250-52540035b04c

[Jian Yu]

The failure occurred on the following test sessions on Lustre b2_5 and master branches:

SLES11SP2 client + RHEL6.5 server
SLES11SP3 client + RHEL6.5 server
SLES11SP3 client + SLES11SP3 server (only on master branch)

I'll look into the failure.

[Peter Jones]

Yu, Jian

This seems to be occurring sporadically and when it does it causes review failures. Could you please look into what kind of circumstances trigger these failures?

Thanks

Peter

[Bob Glossman (Inactive)]

another in b2_5
https://maloo.whamcloud.com/test_sessions/bb218674-cc67-11e3-bda1-52540035b04c

[Jian Yu]

Lustre Build: http://build.whamcloud.com/job/lustre-b2_5/40/ (2.5.1 RC2)
Distro/Arch: RHEL6.5/x86_64(server), SLES11SP3/x86_64(client)

The same failure occurred:
https://maloo.whamcloud.com/test_sets/97d92b6a-a663-11e3-aac5-52540035b04c