[LU-4178] Test failure on test suite sanity-hsm, subtest test_200

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.6.0, Lustre 2.7.0, Lustre 2.8.0, Lustre 2.5.4
• Affects Version/s: Lustre 2.5.0
• Labels: HSM
• Severity: 3
• Assignee: Bruno Faccini (Inactive)
• Reporter: Maloo

Description

This issue was created by maloo for Bob Glossman <[e-mail]>

This issue relates to the following test suite run: http://maloo.whamcloud.com/test_sets/4c3bcdec-4025-11e3-bfaf-52540035b04c.

test_200 was only recently enabled by commit 38695729d61958ab10e9e108175298f8a7d40536. before that is was always skipped due to being in ALWAYS_EXCEPT. I'm wondering if it was a mistake to turn this test on at all. maloo reports:

Failure Rate: 66.00% of last 100 executions [all branches]

This failure looks not at all related to the change under test, at least in this case.

The sub-test test_200 failed with the following error:

request on sanity-hsm is not @@@@@@

Info required for matching: sanity-hsm 200

Issue Links

• is related to: LU-3852 sanity-hsm test_251: client26-vm "dd: no space left on device"

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13173/
Subject: LU-4178 tests: Wait requests to reach CDT before Cancel
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 6a31cf92555182a23f14d3385c8c14266887070a

[James Nunez (Inactive)]

Closing ticket because sanity-hsm tests 200, 201 and 202 are passing on master for the past month. If any more work needs to be done for this ticket, please open a new ticket and we'll track the work there.

[Gerrit Updater]

James Nunez ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13826
Subject: LU-4178 tests: increase sanity-hsm wait_request_state tiemout
Project: fs/lustre-release
Branch: b2_5
Current Patch Set: 1
Commit: 9c56a8f64d2ab4b8db8b3f38dff2d019b8cd3e40

[Gerrit Updater]

James Nunez ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13825
Subject: LU-4178 tests: add messages to sanity-hsm
Project: fs/lustre-release
Branch: b2_5
Current Patch Set: 1
Commit: 75d6d35bcc48eefe490e8b4efd673c58b3373507

[James Nunez (Inactive)]

Reopening ticket because there is one more patch for this ticket that has not landed. The patch is at: http://review.whamcloud.com/#/c/13173/

[Jodi Levi (Inactive)]

Patches landed to Master.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13206/
Subject: LU-4178 tests: increase sanity-hsm wait_request_state tiemout
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4cb51c76ed2afa168f19e999190a315803580258

[John Hammond]

The cancel action does not succeed until the CT reports that the archive is complete. In test_200 we use make_large_for_cancel() which gives us a 100MB file. Because of the 1MB/s bandwidth limit the CT will take at least 100s to archive the file. Since wait_request_state() uses a 100s timeout this make for a very racy test. And since most of these tests still use NFS for the archive there can be additional delays.

I suggest that we double the timeout in wait_request_state(). Please see http://review.whamcloud.com/13206.

[Gerrit Updater]

John L. Hammond ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13206
Subject: LU-4178 tests: increase sanity-hsm wait_request_state tiemout
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 278c1fb845c2cdd7905f717435176e94a4ad7057

[jacques-charles lafoucriere]

The analysis and patch look good but I am surprised because the CDT command registration should be synchronous. So after hsm_archive the CDT entry should be recorded.

[Bruno Faccini (Inactive)]

John,
I think that sub-tests 200-202 verify that requests can be canceled when CDT operations have been started but disabled.
BTW, your finding that sometime (very likely due to some threading cause at Agent and/or MDS side!) the Cancel requests is treated by CDT before the action it targets needs to be addressed and also explains why these failures are not solid.

Patch that adds verification that the operation has already been registered at CDT before to send the Cancel in sanity-hsm/test_[200-202], is at http://review.whamcloud.com/13173.