[LU-1897] Failure on test suite replay-single, test_70b: dbench not found on some of the test nodes

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Blocker
• Fix Version/s: Lustre 2.4.0
• Affects Version/s: Lustre 2.4.0, Lustre 2.1.6, Lustre 2.8.0
• Labels: MB mn1
• Severity: 3
• Assignee: Keith Mannthey (Inactive)
• Reporter: Maloo

Description

This issue was created by maloo for jay <[e-mail]>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/c4620b6c-fc31-11e1-a4a6-52540035b04c.

The sub-test test_70b failed with the following error:

dbench not found on some of client-28vm1,client-28vm2.lab.whamcloud.com !

Info required for matching: replay-single 70b

Issue Links

• is duplicated by: LU-2120 Test failure on test suite replay-single, subtest test_70b

Comments

[Saurabh Tandan (Inactive)]

Instance found in recent tags 2.7.63, 2.7.64

[Saurabh Tandan (Inactive)]

master, build# 3264, 2.7.64 tag
Hard Failover: EL6.7 Server/Client
https://testing.hpdd.intel.com/test_sets/80a20678-9edd-11e5-87a9-5254006e85c2

[Jian Yu]

The issue also exists on Lustre b2_1 branch:
https://maloo.whamcloud.com/test_sets/a0806aa0-c5ce-11e3-9255-52540035b04c

[Peter Jones]

Landed for 2.4

[Keith Mannthey (Inactive)]

http://review.whamcloud.com/5761 version 1 of the patch has been submitted.

[Keith Mannthey (Inactive)]

It looks like dbench didn't start on client-26vm5 fast enough for the first check?
https://maloo.whamcloud.com/test_logs/6cc6de7a-8eb0-11e2-81eb-52540035b04c

12 seconds isn't long enough to start dbench on a remote client... Fun. I will submit a patch.

This is the console from the main client.

15:52:52:Lustre: DEBUG MARKER: /usr/sbin/lctl mark Started rundbench load pid=1931 ...
15:52:52:Lustre: DEBUG MARKER: Started rundbench load pid=1931 ...
15:53:03:Lustre: DEBUG MARKER: killall -0 dbench
15:53:04:Lustre: DEBUG MARKER: /usr/sbin/lctl mark  replay-single test_70b: @@@@@@ FAIL: dbench not running on some of client-26vm5,client-26vm6.lab.whamcloud.com! 

In the autotest long you see dbench start some time after this on the other client. It does run just not at the right time.

[Nathaniel Clark]

I think this failed here:
https://maloo.whamcloud.com/test_sets/43757b76-8eb0-11e2-81eb-52540035b04c

[Keith Mannthey (Inactive)]

http://review.whamcloud.com/4973 has been merged. Please reopen if the problem sill occurs.

[Keith Mannthey (Inactive)]

I will change the comment and resubmit the patch.

[Andreas Dilger]

It might be nice as part of this fix to change the error message to "dbench is no longer running on some of the test nodes", which would not give the false impression that it is not installed on the test nodes...

[Prakash Surya (Inactive)]

Looks like I suffered from this over the weekend: https://maloo.whamcloud.com/test_sessions/b2e3bc58-5c73-11e2-ab3b-52540035b04c