[LU-4178] Test failure on test suite sanity-hsm, subtest test_200

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.6.0, Lustre 2.7.0, Lustre 2.8.0, Lustre 2.5.4
• Affects Version/s: Lustre 2.5.0
• Labels: HSM
• Severity: 3
• Assignee: Bruno Faccini (Inactive)
• Reporter: Maloo

Description

This issue was created by maloo for Bob Glossman <[e-mail]>

This issue relates to the following test suite run: http://maloo.whamcloud.com/test_sets/4c3bcdec-4025-11e3-bfaf-52540035b04c.

test_200 was only recently enabled by commit 38695729d61958ab10e9e108175298f8a7d40536. before that is was always skipped due to being in ALWAYS_EXCEPT. I'm wondering if it was a mistake to turn this test on at all. maloo reports:

Failure Rate: 66.00% of last 100 executions [all branches]

This failure looks not at all related to the change under test, at least in this case.

The sub-test test_200 failed with the following error:

request on sanity-hsm is not @@@@@@

Info required for matching: sanity-hsm 200

Issue Links

• is related to: LU-3852 sanity-hsm test_251: client26-vm "dd: no space left on device"

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13173/
Subject: LU-4178 tests: Wait requests to reach CDT before Cancel
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 6a31cf92555182a23f14d3385c8c14266887070a

[James Nunez (Inactive)]

Closing ticket because sanity-hsm tests 200, 201 and 202 are passing on master for the past month. If any more work needs to be done for this ticket, please open a new ticket and we'll track the work there.

[Gerrit Updater]

James Nunez ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13826
Subject: LU-4178 tests: increase sanity-hsm wait_request_state tiemout
Project: fs/lustre-release
Branch: b2_5
Current Patch Set: 1
Commit: 9c56a8f64d2ab4b8db8b3f38dff2d019b8cd3e40

[Gerrit Updater]

James Nunez ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13825
Subject: LU-4178 tests: add messages to sanity-hsm
Project: fs/lustre-release
Branch: b2_5
Current Patch Set: 1
Commit: 75d6d35bcc48eefe490e8b4efd673c58b3373507

[James Nunez (Inactive)]

Reopening ticket because there is one more patch for this ticket that has not landed. The patch is at: http://review.whamcloud.com/#/c/13173/

[Jodi Levi (Inactive)]

Patches landed to Master.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13206/
Subject: LU-4178 tests: increase sanity-hsm wait_request_state tiemout
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4cb51c76ed2afa168f19e999190a315803580258

[John Hammond]

The cancel action does not succeed until the CT reports that the archive is complete. In test_200 we use make_large_for_cancel() which gives us a 100MB file. Because of the 1MB/s bandwidth limit the CT will take at least 100s to archive the file. Since wait_request_state() uses a 100s timeout this make for a very racy test. And since most of these tests still use NFS for the archive there can be additional delays.

I suggest that we double the timeout in wait_request_state(). Please see http://review.whamcloud.com/13206.

[Gerrit Updater]

John L. Hammond ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13206
Subject: LU-4178 tests: increase sanity-hsm wait_request_state tiemout
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 278c1fb845c2cdd7905f717435176e94a4ad7057

[jacques-charles lafoucriere]

The analysis and patch look good but I am surprised because the CDT command registration should be synchronous. So after hsm_archive the CDT entry should be recorded.