[LU-16904] initial test script to run sanity.sh and sanityn.sh with PFL layout

• Type: Task
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0
• Severity: 3
• Assignee: Sarah Liu
• Reporter: Sarah Liu

Description

This is the ticket to track progress and problems of running sanity.sh and sanityn.sh with PFL as filesystem layout. subtests and test-framework fixes should be in separate patches.

Issue Links

• is related to: LU-17655 lfs setstripe --stripe-count 0 --stripe-size 0 cannot change the pfl file/dir to use default attribute
• is related to: LU-17748 sanity-compr test_34h: New size wrong 10485760 != 1000
• is related to: LU-17755 sanity-compr test_101i: FAIL: expected 5 misses but got 6
• is related to: LU-17772 sanity-compr test_270h: Invalid layout, unrecognized error: -1
• is related to: LU-17798 sanity-compr test_27D: llapi_layout_test failed
• is related to: LU-7723 few tests in sanity.sh fail because of STRIPES_PER_OBJ=-1
• is related to: LU-17694 sanity-compr test_184d: last component index number got assigned even if it was not used after layout swap
• is related to: LU-16924 sanity-compr test_56ba: lfs find --component-count 0 - 0 != 10 files
• is related to: LU-11918 Allow setting default file layout on root directory at mkfs time

Comments

[Peter Jones]

Merged for 2.16

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/54918/
Subject: LU-16904 tests: Fix sanity test 248c and 360 when PFL layout is used
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 10c113a8d806530f498804c69b4a0022c3edf515

[Gerrit Updater]

"Sarah Liu <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/54918
Subject: LU-16904 tests: Fix sanity test 248c and 360 when PFL layout is used
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6436c90376bef9a2acaa0a46f2cd3f5339375629

[Gerrit Updater]

"Sarah Liu <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/54797
Subject: LU-16904 tests: Fix sanity test 34h, 184d and 101i when PFL layout is used
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 37279b125a312733a1ac2df13a140deba2a14241

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/54644/
Subject: LU-16904 tests: Fix sanity test 56a and 65a when PFL layout is used
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: ae68041334e410a9bed297e0bd2dd5bcb8d90b8c

[Gerrit Updater]

"Sarah Liu <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/54644
Subject: LU-16904 tests: Fix sanity test 56a and 65a when PFL layout is used
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 7bf4005f6caf354499b8e45b16e972b550456371

[Gerrit Updater]

"Sarah Liu <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53651
Subject: LU-16904 tests: Set 2-stripe layout on the system for sanity and sanityn
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 176d454ac31690426c19620cfd7e2609b958e82b

[Sarah Liu]

Thanks Andreas, I will take a look

[Andreas Dilger]

I noticed in LU-11918 that there is already support for setting a different filesystem-wide default layout using the "fs_STRIPEPARAMS" environment variable, and per-test default layouts using _testname_STRIPEPARAMS. It probably makes sense to change sanity-compr.sh over to using compr_STRIPEPARAMS instead of COMPR_EXTRA_LAYOUT, and similarly change RACER_EXTRA_LAYOUT over to racer_STRIPEPARAMS for consistency.

[Andreas Dilger]

The version of sanity-compr.sh currently in the tree is only running with the most basic 1-stripe PFL file.

This needs to be updated to include a more complex PFL file layout so that we are exercising more of the code, and the corresponding test script breakage fixed.

There are a series of increasingly complex PFL layouts that could be used to discover and fix layout assumptions in the test scripts, that should be tried in separate patches, so that we can finally enable PFL layouts in the test framework by default. This would include running with 2- and 4-stripe files by default, 2- and 3- component files, and different stripe sizes:

Test-Parameters: testlist=sanity-compr env=COMPR_EXTRA_LAYOUT="-E 64k -c 1 -E eof"
Test-Parameters: testlist=sanity-compr env=COMPR_EXTRA_LAYOUT="-E 64k -c 1 -E eof -c 2"
Test-Parameters: testlist=sanity-compr env=COMPR_EXTRA_LAYOUT="-E 64k -c 1 -E 1M -c 2 -E eof -c 4 -S 4M"

This shows all of the Test-Parameters at once, but they should probably be put into separate patches so that the failures can be isolated and fixed more easily.

Once the last 3-component PFL patch can pass testing, then this should be set as the default layout directly in the sanity-compr.sh patch, and/or move this directly into sanity.sh and sanityn.sh and finally start on a patch to allow setting the PFL layout on the filesystem directly from mkfs.lustre.

[Peter Jones]

Merged for 2.16