[LU-16938] "lfs setstripe -C -1" stripes too widely, should be limited to OST_COUNT

• Type: Improvement
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.16.0
• Affects Version/s: Lustre 2.15.0
• Severity: 3
• Assignee: Rajeev Mishra
• Reporter: Rajeev Mishra

Description

I am reaching out to seek clarification regarding the expected behavior of the "lfs setstripe" command when using the -C -1 option.

Currently, it appears that this command is creating a higher stripe count than anticipated. For instance, on my test system, it generated a stripe count of 2727 for a single file. This count exceeds the allowed limit of LOV_MAX_STRIPE_COUNT. 

I am uncertain about the appropriate solution to address this issue related to the "-1" argument. I have contemplated the following options:

1.    Consider making the option -1 illegal, preventing its usage altogether.

2.    Implement a mechanism to automatically set the stripe count to the maximum allowed value (LOV_MAX_STRIPE_COUNT) if the count exceeds this limit.

I would greatly appreciate your input and guidance in this matter. It is worth noting that setting the stripe count higher than LOV_MAX_STRIPE_COUNT leads to other problems, such as the failure of the "llapi_layout_get_by_fd" API to open the file.

Please let me know your input.

Issue Links

• is related to: LU-17925 sanity-flr test_0b: lfs mirror create: cannot create composite file: Numerical result out of range
• is related to: LU-17199 'lfs setstripe -C -1' can be set beyond overstripe count > 2000
• is related to: LU-18244 add 'lfs mkdir -C -N' overstriping support
• is related to: LU-16623 lod_statfs_and_check() does not skip unusable OSTs
• is related to: LU-13748 'lfs setstripe -C -1' stripes too widely

Comments

[Peter Jones]

Seems like we are going to fix in place with LU-17925 rather than revert

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/55543
Subject: LU-16938 revert: utils: setstripe overstripe multiple OST count"
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 964f07f84eb67f221793a3deb7954a46d1157255

[Peter Jones]

Given that this is an enhancement and not time-sensitive and we are trying to finalize the 2.16 release ATM, let's please have a revert patch ready as a contingency...

[Andreas Dilger]

This is broken due to storing the negative stripe count as "stripe_count - 32" instead of just "-stripe_count" and needs to be fixed before 2.16.0 is released.

[Peter Jones]

Merged for 2.16

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/54192/
Subject: LU-16938 utils: setstripe overstripe multiple OST count
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1a6ef725c285dd5c25c976956ba754dc470f6c1c

[Gerrit Updater]

"Rajeev Mishra <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/54192
Subject: LU-16938 utils: enabling setstripe n multiple of ost count
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 15396dfe08e29fa02233264f44fa6861a171510e

[Josh Schwartz]

But just to be clear, the inconsistency I'm concerned about can ultimately affect files, e.g. by ending up with a MUCH larger overstripe count than perhaps was intended if one accidentally does something like this:

jupiter-p2:/lus/kjcf08 # mkdir test
jupiter-p2:/lus/kjcf08 # lfs setstripe --overstripe-count 10 --stripe-count -1 test
jupiter-p2:/lus/kjcf08 # touch test/foo
jupiter-p2:/lus/kjcf08 # lfs getstripe test | head
test
stripe_count:  -1 stripe_size:   1048576 pattern:       raid0,overstriped stripe_offset: -1

test/foo
lmm_stripe_count:  2727
lmm_stripe_size:   1048576
lmm_pattern:       raid0,overstriped
lmm_layout_gen:    0
lmm_stripe_offset: 0
	obdidx		 objid		 objid		 group

(note the 2727 value is because I don't have Rajeev's other fix on this system, but on the latest code this would be 2000... still probably not what was expected on a system with 2 OSTs).

[Patrick Farrell]

OK, that's good, then.  The user interface is important but I was more concerned that the server might be marking the layout incorrectly.  Obviously default layouts are a different case.

[Josh Schwartz]

I don't think that is coming into play here because I'm just showing the default striping on a directory. If I actually create a file within the directory I believe it is behaving as you suggest:

jupiter-p2:/lus/kjcf08 # mkdir test
jupiter-p2:/lus/kjcf08 # lfs setstripe --overstripe-count 1024 --stripe-count 10 test
jupiter-p2:/lus/kjcf08 # touch test/foo
jupiter-p2:/lus/kjcf08 # lfs getstripe test | head
test
stripe_count:  10 stripe_size:   1048576 pattern:       raid0,overstriped stripe_offset: -1

test/foo
lmm_stripe_count:  10
lmm_stripe_size:   1048576
lmm_pattern:       raid0,overstriped
lmm_layout_gen:    0
lmm_stripe_offset: 1
	obdidx		 objid		 objid		 group

here the file is overstriped because I only have 2 OSTs.

This is a bit of a degenerative example, but if I just set the --overstripe-count 2 the directory will have a default of overstriped with a stripe count of 2, but files that are created are not overstriped (and have a stripe count of 2):

jupiter-p2:/lus/kjcf08 # lfs setstripe --overstripe-count 2 test
jupiter-p2:/lus/kjcf08 # lfs getstripe -d test
stripe_count:  2 stripe_size:   1048576 pattern:       raid0,overstriped stripe_offset: -1
jupiter-p2:/lus/kjcf08 # touch test/foo
jupiter-p2:/lus/kjcf08 # lfs getstripe test/foo
test/foo
lmm_stripe_count:  2
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 1
	obdidx		 objid		 objid		 group
	     1	     116959791	    0x6f8aa2f	             0
	     0	     117253333	    0x6fd24d5	             0

so I think that part of it is working OK.