Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-16938

"lfs setstripe -C -1" stripes too widely, should be limited to OST_COUNT

Details

    • Improvement
    • Resolution: Fixed
    • Major
    • Lustre 2.16.0
    • Lustre 2.15.0
    • None
    • 3
    • 9223372036854775807

    Description

      I am reaching out to seek clarification regarding the expected behavior of the "lfs setstripe" command when using the -C -1 option.

      Currently, it appears that this command is creating a higher stripe count than anticipated. For instance, on my test system, it generated a stripe count of 2727 for a single file. This count exceeds the allowed limit of LOV_MAX_STRIPE_COUNT. 

      I am uncertain about the appropriate solution to address this issue related to the "-1" argument. I have contemplated the following options:

      1.    Consider making the option -1 illegal, preventing its usage altogether.

      2.    Implement a mechanism to automatically set the stripe count to the maximum allowed value (LOV_MAX_STRIPE_COUNT) if the count exceeds this limit.

      I would greatly appreciate your input and guidance in this matter. It is worth noting that setting the stripe count higher than LOV_MAX_STRIPE_COUNT leads to other problems, such as the failure of the "llapi_layout_get_by_fd" API to open the file.

      Please let me know your input.

      Attachments

        Issue Links

          Activity

            [LU-16938] "lfs setstripe -C -1" stripes too widely, should be limited to OST_COUNT
            pjones Peter Jones added a comment -

            Seems like we are going to fix in place with LU-17925 rather than revert

            pjones Peter Jones added a comment - Seems like we are going to fix in place with LU-17925 rather than revert

            "Andreas Dilger <adilger@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/55543
            Subject: LU-16938 revert: utils: setstripe overstripe multiple OST count"
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 964f07f84eb67f221793a3deb7954a46d1157255

            gerrit Gerrit Updater added a comment - "Andreas Dilger <adilger@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/55543 Subject: LU-16938 revert: utils: setstripe overstripe multiple OST count" Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 964f07f84eb67f221793a3deb7954a46d1157255
            pjones Peter Jones added a comment -

            Given that this is an enhancement and not time-sensitive and we are trying to finalize the 2.16 release ATM, let's please have a revert patch ready as a contingency...

            pjones Peter Jones added a comment - Given that this is an enhancement and not time-sensitive and we are trying to finalize the 2.16 release ATM, let's please have a revert patch ready as a contingency...

            This is broken due to storing the negative stripe count as "stripe_count - 32" instead of just "-stripe_count" and needs to be fixed before 2.16.0 is released.

            adilger Andreas Dilger added a comment - This is broken due to storing the negative stripe count as "stripe_count - 32" instead of just "-stripe_count" and needs to be fixed before 2.16.0 is released.
            pjones Peter Jones added a comment -

            Merged for 2.16

            pjones Peter Jones added a comment - Merged for 2.16

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/54192/
            Subject: LU-16938 utils: setstripe overstripe multiple OST count
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 1a6ef725c285dd5c25c976956ba754dc470f6c1c

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/54192/ Subject: LU-16938 utils: setstripe overstripe multiple OST count Project: fs/lustre-release Branch: master Current Patch Set: Commit: 1a6ef725c285dd5c25c976956ba754dc470f6c1c

            "Rajeev Mishra <rajeevm@hpe.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/54192
            Subject: LU-16938 utils: enabling setstripe n multiple of ost count
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 15396dfe08e29fa02233264f44fa6861a171510e

            gerrit Gerrit Updater added a comment - "Rajeev Mishra <rajeevm@hpe.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/54192 Subject: LU-16938 utils: enabling setstripe n multiple of ost count Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 15396dfe08e29fa02233264f44fa6861a171510e

            But just to be clear, the inconsistency I'm concerned about can ultimately affect files, e.g. by ending up with a MUCH larger overstripe count than perhaps was intended if one accidentally does something like this:

            jupiter-p2:/lus/kjcf08 # mkdir test
            jupiter-p2:/lus/kjcf08 # lfs setstripe --overstripe-count 10 --stripe-count -1 test
            jupiter-p2:/lus/kjcf08 # touch test/foo
            jupiter-p2:/lus/kjcf08 # lfs getstripe test | head
            test
            stripe_count:  -1 stripe_size:   1048576 pattern:       raid0,overstriped stripe_offset: -1
            
            test/foo
            lmm_stripe_count:  2727
            lmm_stripe_size:   1048576
            lmm_pattern:       raid0,overstriped
            lmm_layout_gen:    0
            lmm_stripe_offset: 0
            	obdidx		 objid		 objid		 group
            

            (note the 2727 value is because I don't have Rajeev's other fix on this system, but on the latest code this would be 2000... still probably not what was expected on a system with 2 OSTs).

            jschwartz Josh Schwartz added a comment - But just to be clear, the inconsistency I'm concerned about can ultimately affect files, e.g. by ending up with a MUCH larger overstripe count than perhaps was intended if one accidentally does something like this: jupiter-p2:/lus/kjcf08 # mkdir test jupiter-p2:/lus/kjcf08 # lfs setstripe --overstripe-count 10 --stripe-count -1 test jupiter-p2:/lus/kjcf08 # touch test/foo jupiter-p2:/lus/kjcf08 # lfs getstripe test | head test stripe_count: -1 stripe_size: 1048576 pattern: raid0,overstriped stripe_offset: -1 test/foo lmm_stripe_count: 2727 lmm_stripe_size: 1048576 lmm_pattern: raid0,overstriped lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group (note the 2727 value is because I don't have Rajeev's other fix on this system, but on the latest code this would be 2000... still probably not what was expected on a system with 2 OSTs).

            OK, that's good, then.  The user interface is important but I was more concerned that the server might be marking the layout incorrectly.  Obviously default layouts are a different case.

            paf0186 Patrick Farrell added a comment - OK, that's good, then.  The user interface is important but I was more concerned that the server might be marking the layout incorrectly.  Obviously default layouts are a different case.
            jschwartz Josh Schwartz added a comment - - edited

            I don't think that is coming into play here because I'm just showing the default striping on a directory. If I actually create a file within the directory I believe it is behaving as you suggest:

            jupiter-p2:/lus/kjcf08 # mkdir test
            jupiter-p2:/lus/kjcf08 # lfs setstripe --overstripe-count 1024 --stripe-count 10 test
            jupiter-p2:/lus/kjcf08 # touch test/foo
            jupiter-p2:/lus/kjcf08 # lfs getstripe test | head
            test
            stripe_count:  10 stripe_size:   1048576 pattern:       raid0,overstriped stripe_offset: -1
            
            test/foo
            lmm_stripe_count:  10
            lmm_stripe_size:   1048576
            lmm_pattern:       raid0,overstriped
            lmm_layout_gen:    0
            lmm_stripe_offset: 1
            	obdidx		 objid		 objid		 group
            

            here the file is overstriped because I only have 2 OSTs.

            This is a bit of a degenerative example, but if I just set the --overstripe-count 2 the directory will have a default of overstriped with a stripe count of 2, but files that are created are not overstriped (and have a stripe count of 2):

            jupiter-p2:/lus/kjcf08 # lfs setstripe --overstripe-count 2 test
            jupiter-p2:/lus/kjcf08 # lfs getstripe -d test
            stripe_count:  2 stripe_size:   1048576 pattern:       raid0,overstriped stripe_offset: -1
            jupiter-p2:/lus/kjcf08 # touch test/foo
            jupiter-p2:/lus/kjcf08 # lfs getstripe test/foo
            test/foo
            lmm_stripe_count:  2
            lmm_stripe_size:   1048576
            lmm_pattern:       raid0
            lmm_layout_gen:    0
            lmm_stripe_offset: 1
            	obdidx		 objid		 objid		 group
            	     1	     116959791	    0x6f8aa2f	             0
            	     0	     117253333	    0x6fd24d5	             0
            

            so I think that part of it is working OK.

            jschwartz Josh Schwartz added a comment - - edited I don't think that is coming into play here because I'm just showing the default striping on a directory. If I actually create a file within the directory I believe it is behaving as you suggest: jupiter-p2:/lus/kjcf08 # mkdir test jupiter-p2:/lus/kjcf08 # lfs setstripe --overstripe-count 1024 --stripe-count 10 test jupiter-p2:/lus/kjcf08 # touch test/foo jupiter-p2:/lus/kjcf08 # lfs getstripe test | head test stripe_count: 10 stripe_size: 1048576 pattern: raid0,overstriped stripe_offset: -1 test/foo lmm_stripe_count: 10 lmm_stripe_size: 1048576 lmm_pattern: raid0,overstriped lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group here the file is overstriped because I only have 2 OSTs. This is a bit of a degenerative example, but if I just set the --overstripe-count 2 the directory will have a default of overstriped with a stripe count of 2, but files that are created are not overstriped (and have a stripe count of 2): jupiter-p2:/lus/kjcf08 # lfs setstripe --overstripe-count 2 test jupiter-p2:/lus/kjcf08 # lfs getstripe -d test stripe_count: 2 stripe_size: 1048576 pattern: raid0,overstriped stripe_offset: -1 jupiter-p2:/lus/kjcf08 # touch test/foo jupiter-p2:/lus/kjcf08 # lfs getstripe test/foo test/foo lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 116959791 0x6f8aa2f 0 0 117253333 0x6fd24d5 0 so I think that part of it is working OK.

            People

              rajeevm Rajeev Mishra
              rajeevm Rajeev Mishra
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: