Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-16904

initial test script to run sanity.sh and sanityn.sh with PFL layout

Details

    • Task
    • Resolution: Fixed
    • Minor
    • Lustre 2.16.0
    • None
    • None
    • 3
    • 9223372036854775807

    Description

      This is the ticket to track progress and problems of running sanity.sh and sanityn.sh with PFL as filesystem layout. subtests and test-framework fixes should be in separate patches.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-17655 lfs setstripe --stripe-count 0 --stripe-size 0 cannot change the pfl file/dir to use default attribute

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. LU-17748 sanity-compr test_34h: New size wrong 10485760 != 1000

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. LU-17755 sanity-compr test_101i: FAIL: expected 5 misses but got 6

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. LU-17772 sanity-compr test_270h: Invalid layout, unrecognized error: -1

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. LU-17798 sanity-compr test_27D: llapi_layout_test failed

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. LU-7723 few tests in sanity.sh fail because of STRIPES_PER_OBJ=-1

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Open
          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-17694 sanity-compr test_184d: last component index number got assigned even if it was not used after layout swap

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. LU-16924 sanity-compr test_56ba: lfs find --component-count 0 - 0 != 10 files

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. LU-11918 Allow setting default file layout on root directory at mkfs time

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Activity

            [LU-16904] initial test script to run sanity.sh and sanityn.sh with PFL layout
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/54644/
            Subject: LU-16904 tests: Fix sanity test 56a and 65a when PFL layout is used
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: ae68041334e410a9bed297e0bd2dd5bcb8d90b8c

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/54644/ Subject: LU-16904 tests: Fix sanity test 56a and 65a when PFL layout is used Project: fs/lustre-release Branch: master Current Patch Set: Commit: ae68041334e410a9bed297e0bd2dd5bcb8d90b8c
            gerrit Gerrit Updater added a comment -

            "Sarah Liu <sarah@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/54644
            Subject: LU-16904 tests: Fix sanity test 56a and 65a when PFL layout is used
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 7bf4005f6caf354499b8e45b16e972b550456371

            gerrit Gerrit Updater added a comment - "Sarah Liu <sarah@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/54644 Subject: LU-16904 tests: Fix sanity test 56a and 65a when PFL layout is used Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 7bf4005f6caf354499b8e45b16e972b550456371
            gerrit Gerrit Updater added a comment -

            "Sarah Liu <sarah@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53651
            Subject: LU-16904 tests: Set 2-stripe layout on the system for sanity and sanityn
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 176d454ac31690426c19620cfd7e2609b958e82b

            gerrit Gerrit Updater added a comment - "Sarah Liu <sarah@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53651 Subject: LU-16904 tests: Set 2-stripe layout on the system for sanity and sanityn Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 176d454ac31690426c19620cfd7e2609b958e82b
            sarah Sarah Liu added a comment -

            Thanks Andreas, I will take a look

            sarah Sarah Liu added a comment - Thanks Andreas, I will take a look
            adilger Andreas Dilger added a comment -

            I noticed in LU-11918 that there is already support for setting a different filesystem-wide default layout using the "fs_STRIPEPARAMS" environment variable, and per-test default layouts using _testname_STRIPEPARAMS. It probably makes sense to change sanity-compr.sh over to using compr_STRIPEPARAMS instead of COMPR_EXTRA_LAYOUT, and similarly change RACER_EXTRA_LAYOUT over to racer_STRIPEPARAMS for consistency.

            adilger Andreas Dilger added a comment - I noticed in LU-11918 that there is already support for setting a different filesystem-wide default layout using the " fs_STRIPEPARAMS " environment variable, and per-test default layouts using _testname_STRIPEPARAMS . It probably makes sense to change sanity-compr.sh over to using compr_STRIPEPARAMS instead of COMPR_EXTRA_LAYOUT , and similarly change RACER_EXTRA_LAYOUT over to racer_STRIPEPARAMS for consistency.
            adilger Andreas Dilger added a comment - - edited

            The version of sanity-compr.sh currently in the tree is only running with the most basic 1-stripe PFL file.

            This needs to be updated to include a more complex PFL file layout so that we are exercising more of the code, and the corresponding test script breakage fixed.

            There are a series of increasingly complex PFL layouts that could be used to discover and fix layout assumptions in the test scripts, that should be tried in separate patches, so that we can finally enable PFL layouts in the test framework by default. This would include running with 2- and 4-stripe files by default, 2- and 3- component files, and different stripe sizes:

            Test-Parameters: testlist=sanity-compr env=COMPR_EXTRA_LAYOUT="-E 64k -c 1 -E eof"
Test-Parameters: testlist=sanity-compr env=COMPR_EXTRA_LAYOUT="-E 64k -c 1 -E eof -c 2"
Test-Parameters: testlist=sanity-compr env=COMPR_EXTRA_LAYOUT="-E 64k -c 1 -E 1M -c 2 -E eof -c 4 -S 4M"

            This shows all of the Test-Parameters at once, but they should probably be put into separate patches so that the failures can be isolated and fixed more easily.

            Once the last 3-component PFL patch can pass testing, then this should be set as the default layout directly in the sanity-compr.sh patch, and/or move this directly into sanity.sh and sanityn.sh and finally start on a patch to allow setting the PFL layout on the filesystem directly from mkfs.lustre.

            adilger Andreas Dilger added a comment - - edited The version of sanity-compr.sh currently in the tree is only running with the most basic 1-stripe PFL file. This needs to be updated to include a more complex PFL file layout so that we are exercising more of the code, and the corresponding test script breakage fixed. There are a series of increasingly complex PFL layouts that could be used to discover and fix layout assumptions in the test scripts, that should be tried in separate patches, so that we can finally enable PFL layouts in the test framework by default. This would include running with 2- and 4-stripe files by default, 2- and 3- component files, and different stripe sizes: Test-Parameters: testlist=sanity-compr env=COMPR_EXTRA_LAYOUT="-E 64k -c 1 -E eof" Test-Parameters: testlist=sanity-compr env=COMPR_EXTRA_LAYOUT="-E 64k -c 1 -E eof -c 2" Test-Parameters: testlist=sanity-compr env=COMPR_EXTRA_LAYOUT="-E 64k -c 1 -E 1M -c 2 -E eof -c 4 -S 4M" This shows all of the Test-Parameters at once, but they should probably be put into separate patches so that the failures can be isolated and fixed more easily. Once the last 3-component PFL patch can pass testing, then this should be set as the default layout directly in the sanity-compr.sh patch, and/or move this directly into sanity.sh and sanityn.sh and finally start on a patch to allow setting the PFL layout on the filesystem directly from mkfs.lustre .
            pjones Peter Jones added a comment -

            Merged for 2.16

            pjones Peter Jones added a comment - Merged for 2.16
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51371/
            Subject: LU-16904 test: Add sanity-compr.sh to run sanity and sanityn with PFL layout
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: b4e5faf551189f3e391a7ef268abcf0d584206bd

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51371/ Subject: LU-16904 test: Add sanity-compr.sh to run sanity and sanityn with PFL layout Project: fs/lustre-release Branch: master Current Patch Set: Commit: b4e5faf551189f3e391a7ef268abcf0d584206bd
            adilger Andreas Dilger added a comment -

            Looking at the other test failures in https://testing.whamcloud.com/test_sets/24922969-cde5-4abd-8dfd-88bc1571d1e4 I am surprised that there are only 7 failed subtests out of all the subtests in sanity.sh, so I think it would be reasonable to just fix them all in a single patch, possibly even as part of the original 51371 patch? Then, move on to a follow-on patch that adds sanityn.sh testing to sanity-compr.sh, and allow the initial patch to land.

            adilger Andreas Dilger added a comment - Looking at the other test failures in https://testing.whamcloud.com/test_sets/24922969-cde5-4abd-8dfd-88bc1571d1e4 I am surprised that there are only 7 failed subtests out of all the subtests in sanity.sh, so I think it would be reasonable to just fix them all in a single patch, possibly even as part of the original 51371 patch? Then, move on to a follow-on patch that adds sanityn.sh testing to sanity-compr.sh , and allow the initial patch to land.
            sarah Sarah Liu added a comment - - edited

            this is a fortestonly patch only to run sanity with setting "lfs setstripe -E EOF -c 1 $MOUNT"

            the Maloo result is at
            https://testing.whamcloud.com/test_sets/24922969-cde5-4abd-8dfd-88bc1571d1e4

            related issue:
            LU-16924

            sarah Sarah Liu added a comment - - edited this is a fortestonly patch only to run sanity with setting "lfs setstripe -E EOF -c 1 $MOUNT" the Maloo result is at https://testing.whamcloud.com/test_sets/24922969-cde5-4abd-8dfd-88bc1571d1e4 related issue: LU-16924
            sarah Sarah Liu added a comment - - edited

            Sarah Liu (sarah@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/51371
            Subject: LU-16904 test: Add sanity-compr.sh to run sanity with PFL layout
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: b862fa1c3f3af2c944395301e19b798048064570

            sarah Sarah Liu added a comment - - edited Sarah Liu (sarah@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/51371 Subject: LU-16904 test: Add sanity-compr.sh to run sanity with PFL layout Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: b862fa1c3f3af2c944395301e19b798048064570

            People

              sarah Sarah Liu
              sarah Sarah Liu
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: