[LU-1897] Failure on test suite replay-single, test_70b: dbench not found on some of the test nodes

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Blocker
• Fix Version/s: Lustre 2.4.0
• Affects Version/s: Lustre 2.4.0, Lustre 2.1.6, Lustre 2.8.0
• Labels: MB mn1
• Severity: 3
• Assignee: Keith Mannthey (Inactive)
• Reporter: Maloo

Description

This issue was created by maloo for jay <[e-mail]>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/c4620b6c-fc31-11e1-a4a6-52540035b04c.

The sub-test test_70b failed with the following error:

dbench not found on some of client-28vm1,client-28vm2.lab.whamcloud.com !

Info required for matching: replay-single 70b

Issue Links

• is duplicated by: LU-2120 Test failure on test suite replay-single, subtest test_70b

Comments

[Saurabh Tandan (Inactive)]

master, build# 3266, 2.7.64 tag
Hard Failover:EL7 Server/SLES11 SP3 Client
https://testing.hpdd.intel.com/test_sets/a8b3fb9e-a077-11e5-8d69-5254006e85c2

[Saurabh Tandan (Inactive)]

master, build# 3264, 2.7.64 tag
Hard Failover: EL7 Server/Client - ZFS
https://testing.hpdd.intel.com/test_sets/5ba6d7bc-9e20-11e5-91b0-5254006e85c2

[Saurabh Tandan (Inactive)]

Instance found in recent tags 2.7.63, 2.7.64

[Saurabh Tandan (Inactive)]

master, build# 3264, 2.7.64 tag
Hard Failover: EL6.7 Server/Client
https://testing.hpdd.intel.com/test_sets/80a20678-9edd-11e5-87a9-5254006e85c2

[Jian Yu]

The issue also exists on Lustre b2_1 branch:
https://maloo.whamcloud.com/test_sets/a0806aa0-c5ce-11e3-9255-52540035b04c

[Peter Jones]

Landed for 2.4

[Keith Mannthey (Inactive)]

http://review.whamcloud.com/5761 version 1 of the patch has been submitted.

[Keith Mannthey (Inactive)]

It looks like dbench didn't start on client-26vm5 fast enough for the first check?
https://maloo.whamcloud.com/test_logs/6cc6de7a-8eb0-11e2-81eb-52540035b04c

12 seconds isn't long enough to start dbench on a remote client... Fun. I will submit a patch.

This is the console from the main client.

15:52:52:Lustre: DEBUG MARKER: /usr/sbin/lctl mark Started rundbench load pid=1931 ...
15:52:52:Lustre: DEBUG MARKER: Started rundbench load pid=1931 ...
15:53:03:Lustre: DEBUG MARKER: killall -0 dbench
15:53:04:Lustre: DEBUG MARKER: /usr/sbin/lctl mark  replay-single test_70b: @@@@@@ FAIL: dbench not running on some of client-26vm5,client-26vm6.lab.whamcloud.com! 

In the autotest long you see dbench start some time after this on the other client. It does run just not at the right time.

[Nathaniel Clark]

I think this failed here:
https://maloo.whamcloud.com/test_sets/43757b76-8eb0-11e2-81eb-52540035b04c

[Keith Mannthey (Inactive)]

http://review.whamcloud.com/4973 has been merged. Please reopen if the problem sill occurs.

[Keith Mannthey (Inactive)]

I will change the comment and resubmit the patch.