[LU-1897] Failure on test suite replay-single, test_70b: dbench not found on some of the test nodes

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Blocker
• Fix Version/s: Lustre 2.4.0
• Affects Version/s: Lustre 2.4.0, Lustre 2.1.6, Lustre 2.8.0
• Labels: MB mn1
• Severity: 3
• Assignee: Keith Mannthey (Inactive)
• Reporter: Maloo

Description

This issue was created by maloo for jay <[e-mail]>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/c4620b6c-fc31-11e1-a4a6-52540035b04c.

The sub-test test_70b failed with the following error:

dbench not found on some of client-28vm1,client-28vm2.lab.whamcloud.com !

Info required for matching: replay-single 70b

Issue Links

• is duplicated by: LU-2120 Test failure on test suite replay-single, subtest test_70b

Comments

[Keith Mannthey (Inactive)]

It looks like dbench didn't start on client-26vm5 fast enough for the first check?
https://maloo.whamcloud.com/test_logs/6cc6de7a-8eb0-11e2-81eb-52540035b04c

12 seconds isn't long enough to start dbench on a remote client... Fun. I will submit a patch.

This is the console from the main client.

15:52:52:Lustre: DEBUG MARKER: /usr/sbin/lctl mark Started rundbench load pid=1931 ...
15:52:52:Lustre: DEBUG MARKER: Started rundbench load pid=1931 ...
15:53:03:Lustre: DEBUG MARKER: killall -0 dbench
15:53:04:Lustre: DEBUG MARKER: /usr/sbin/lctl mark  replay-single test_70b: @@@@@@ FAIL: dbench not running on some of client-26vm5,client-26vm6.lab.whamcloud.com! 

In the autotest long you see dbench start some time after this on the other client. It does run just not at the right time.

[Nathaniel Clark]

I think this failed here:
https://maloo.whamcloud.com/test_sets/43757b76-8eb0-11e2-81eb-52540035b04c

[Keith Mannthey (Inactive)]

http://review.whamcloud.com/4973 has been merged. Please reopen if the problem sill occurs.

[Keith Mannthey (Inactive)]

I will change the comment and resubmit the patch.

[Andreas Dilger]

It might be nice as part of this fix to change the error message to "dbench is no longer running on some of the test nodes", which would not give the false impression that it is not installed on the test nodes...

[Prakash Surya (Inactive)]

Looks like I suffered from this over the weekend: https://maloo.whamcloud.com/test_sessions/b2e3bc58-5c73-11e2-ab3b-52540035b04c

[Keith Mannthey (Inactive)]

So Bob and I looked into the problem a bit today.

In general dbench is just a background load for the test and as mentioned Dbench is ending before the test finishes.

I observed the Dbench finishing at 62s and the test finishing in and checking in on it at a little under 90's.

I have submitted at patch to move the dbench duration out a bit to see it help the issue. The test could stand to be rewritten to make the back ground load a non-issue.

http://review.whamcloud.com/4973 is simple patch that adds a comment and increases a duration that should allow dbench to run longer.

[Bob Glossman (Inactive)]

another instance:
https://maloo.whamcloud.com/test_sets/72d446e2-5943-11e2-9b54-52540035b04c

[Bob Glossman (Inactive)]

another instance:
https://maloo.whamcloud.com/test_sets/927c0d00-5717-11e2-8b17-52540035b04c

[nasf (Inactive)]

Another failure instance:

https://maloo.whamcloud.com/test_sets/19e6b2fe-513c-11e2-b56e-52540035b04c

[Mikhail Pershin]

It looks like dbench ends right before test checks that it is running.