[LU-4441] sanity test_60a: Module llog_test is in use

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.6.0, Lustre 2.5.3
• Fix Version/s: Lustre 2.7.0, Lustre 2.5.4
• Type: Bug
• Priority: Major
• Reporter: Maloo
• Assignee: Emoly Liu
• Resolution: Fixed
• Severity: 3

Description

This issue was created by maloo for Bob Glossman <[e-mail]>

This issue relates to the following test suite run: http://maloo.whamcloud.com/test_sets/b44af13a-761f-11e3-a7a8-52540035b04c.

Even though maloo reports a 0% failure, search on subtest 60a of sanity shows a bunch of FAILs

The sub-test test_60a failed with the following error:

wtm-20vm3: ERROR: Module llog_test is in use
test_60a failed with 3

Info required for matching: sanity 60a

Comments

[Andreas Dilger]

Bob, can this be correlated to some particular patch landing? For example, are the failures starting on a particular patch (maybe intermittently), and then moving to master after that patch landed? Or do the failures start with a common parent patch?

[Andreas Dilger]

This still seems to be failing on a regular basis - more than once a day.

[Bob Glossman (Inactive)]

another instance:
https://maloo.whamcloud.com/test_sessions/92085f0c-9a4d-11e3-baa9-52540035b04c

[James Nunez (Inactive)]

Another failure at https://maloo.whamcloud.com/test_sessions/0936d8b4-a17a-11e3-a08f-52540035b04c

[Bob Glossman (Inactive)]

another
https://maloo.whamcloud.com/test_sessions/a2060ea0-a3b1-11e3-b259-52540035b04c

[John Hammond]

https://maloo.whamcloud.com/test_sets/c58bc76e-aa61-11e3-a4db-52540035b04c

[Andreas Dilger]

This is still being hit about twice a day on average over the past four weeks.

[Bob Glossman (Inactive)]

another
https://maloo.whamcloud.com/test_sets/596ef978-bb1b-11e3-b43b-52540035b04c

[James Nunez (Inactive)]

Hit this one again: https://maloo.whamcloud.com/test_sessions/f1d4efe4-bafb-11e3-8ec1-52540035b04c

[Emoly Liu]

This error happens very occasionally recently and it's hard to reproduce it locally.

I create a patch http://review.whamcloud.com/9966/ to improve run-llog.sh so that when this error happens again, it can print more information.

[nasf (Inactive)]

Another failure instance:

https://maloo.whamcloud.com/test_sets/307cd854-cc56-11e3-8180-52540035b04c

[Andreas Dilger]

This was hit several times in the past week - 13 failures in total, of 377 test runs, which is about 3.4%. Not fatal, but enough to keep our eye on.

[Andreas Dilger]

Adding this to the 2.6.0 tracking list, since this test is causing quite a lot of failures.

Emoly, are you actually working on this, or should it be assigned back to HPDD Triage?

[Emoly Liu]

Andreas, the patch http://review.whamcloud.com/9966/ is waiting for the second review. But, that is not a fix but a debugging patch to print more information.

I don't mind reassigning this ticket to others if the issue can be fixed soon.

[Peter Jones]

Landed for 2.6

[Peter Jones]

Bob correctly pointed out that only the debug patch landed. I have reopened the ticket and moved it out of scope for 2.6 as we still do not understand the scope of this issue

[Jian Yu]

Lustre Build: https://build.hpdd.intel.com/job/lustre-b2_5/84/
Distro/Arch: RHEL6.5/x86_64
MDSCOUNT=2

The same failure occurred:
https://testing.hpdd.intel.com/test_sets/b4faa3be-2a8d-11e4-b21b-5254006e85c2

[Emoly Liu]

Hi Yujian, the patch http://review.whamcloud.com/9966/ has not been backported to b2_5. Although it is a debug patch, the problem has not been seen any more on master with it.

So, do you mind if I backport it to b2_5?

[Jian Yu]

No, of course not. Thank you, Emoly.

[Emoly Liu]

backport to b2_5: http://review.whamcloud.com/11573

[Peter Jones]

As per Andreas there is a change that could have fixed this issue so marking as fixed in 2.7

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/11573/
Subject: LU-4441 test: improve run-llog.sh to print more information
Project: fs/lustre-release
Branch: b2_5
Current Patch Set:
Commit: 623efd2fb272269d3b82b73bbdf759c8d3b20ae0