[LU-10734] sanity test_160g: User cl8 still found in changelog_users

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.12.0
• Affects Version/s: Lustre 2.11.0
• Labels: sles12 suse
• Severity: 3
• Assignee: Bruno Faccini (Inactive)
• Reporter: Maloo

Description

sanity test_160g - User cl8 still found in changelog_users
^^^^^^^^^^^^^ DO NOT REMOVE LINE ABOVE ^^^^^^^^^^^^^

This issue was created by maloo for Bob Glossman <[e-mail]>

This issue relates to the following test suite run:
https://testing.hpdd.intel.com/test_sets/5a8495f4-1bfa-11e8-a6ad-52540065bddc
https://testing.hpdd.intel.com/test_sets/34e243bc-1be3-11e8-a7cd-52540065bddc

test_160g failed with the following error:

User cl8 still found in changelog_users

This may be a dup of LU-9624
I can't tell if it is so I am raising a fresh ticket.
Will let somebody else decide if it's a dup or not.

Issue Links

• is related to: LU-11161 sanity test_160g fails for DNE with ''mds2: User cl9 still registered''
• is related to: LU-9624 enable sanity.sh test_160a failures for DNE
• is related to: LU-10680 MDT becoming unresponsive in 2.10.3

Comments

[Bruno Faccini (Inactive)]

Eh eh, after taking some time to think about it, I was wondering if the only regression/side effect coming from patch https://review.whamcloud.com/27535 ("a37134d LU-9624 tests: fix pre-DNE test exceptions/llog usage"), that we strongly suspect to be the cause of these failures, is to have slightly reduced the execution/elapsed time of sanity/test_160g script's beginning/prologue that may now take less than the 2 seconds of delay interval between 2x garbage-collection thread runs (as it has just run in sanity/test_160f, when sanity.sh is being fully executed during auto-tests) being used/configured ("changelog_min_gc_interval=2").
And this seems to be confirmed during my reproducer testing.

So a simple "sleep 2" at the beginning of sanity/test_160g should fix this problem.

[Peter Jones]

> It didn't fail during normal testing, but I guess SLES is not part of regular testing.

Well, it is tested regularly, but due to the round robin system used for pre-landing review test runs, it is not guaranteed to run before everything lands unless people proactively request this with test parameters.

[Andreas Dilger]

Note also that with patch https://review.whamcloud.com/31552 "LU-10680 mdd: disable changelog garbage collection by default" test_160f and test_160g need to be modified to set changelog_gc=1 at the start of each test, and remove the tests from ALWAYS_EXCEPT so that the tests will run properly.

[Andreas Dilger]

It looks like this failure relates to the landing of patch https://review.whamcloud.com/27535 "LU-9624 tests: fix pre-DNE test exceptions/llog usage". It didn't fail during normal testing, but I guess SLES is not part of regular testing.

[Bruno Faccini (Inactive)]

Having a better look to the recent changes that may have introduced this regression, I think that "a37134d LU-9624 tests: fix pre-DNE test exceptions/llog usage" could better be the cause of it.

Hope to get more about this soon now.

[Mikhail Pershin]

+1 on master, all with DNE
testing.hpdd.intel.com/test_sessions/7a5adbc7-2d4b-425a-9e71-a4674823a0df
testing.hpdd.intel.com/test_sessions/a9ae8e29-d45d-49b6-a639-a6fba84f5dfc

[Bob Glossman (Inactive)]

here is a similar fail seen on el7, not on SLES at all.
proof this problem isn't SLES only.

https://testing.hpdd.intel.com/test_sets/b3cb95da-20dd-11e8-a4b1-52540065bddc

[Bob Glossman (Inactive)]

after many similar fails here is a SLES test run that did NOT hit the failure:
https://testing.hpdd.intel.com/test_sessions/ba2847b6-e445-448d-882d-356fca02b96e

Don't know what the diff is between runs that fail and those that don't.
Noting this instance that didn't fail in the hope that it may be of some use.

[Bob Glossman (Inactive)]

more on master:
https://testing.hpdd.intel.com/test_sets/f9311b56-209a-11e8-a4b1-52540065bddc
https://testing.hpdd.intel.com/test_sets/c68848ae-20b4-11e8-9ec4-52540065bddc
https://testing.hpdd.intel.com/test_sets/50a68744-20d3-11e8-a6ca-52540065bddc
https://testing.hpdd.intel.com/test_sets/9d2125f0-2157-11e8-9ec4-52540065bddc
https://testing.hpdd.intel.com/test_sets/bbef9e12-2184-11e8-9ec4-52540065bddc

[Bruno Faccini (Inactive)]

Well, it is strange that it seems to only fail running with SLES, and looks like it started to fail after my 1st patch for LU-10680 (https://review.whamcloud.com/31347/) has landed, when it should fix an issue with my previous patch for LU-7340 which has introduced sanity/test_160[f,g] tests !!... I think I need to reproduce and debug it this way.

[Bob Glossman (Inactive)]

These fails only seen on master. Probably because sanity, 160g only exists on master.