[LU-16938] "lfs setstripe -C -1" stripes too widely, should be limited to OST_COUNT

• Type: Improvement
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.16.0
• Affects Version/s: Lustre 2.15.0
• Severity: 3
• Assignee: Rajeev Mishra
• Reporter: Rajeev Mishra

Description

I am reaching out to seek clarification regarding the expected behavior of the "lfs setstripe" command when using the -C -1 option.

Currently, it appears that this command is creating a higher stripe count than anticipated. For instance, on my test system, it generated a stripe count of 2727 for a single file. This count exceeds the allowed limit of LOV_MAX_STRIPE_COUNT. 

I am uncertain about the appropriate solution to address this issue related to the "-1" argument. I have contemplated the following options:

1.    Consider making the option -1 illegal, preventing its usage altogether.

2.    Implement a mechanism to automatically set the stripe count to the maximum allowed value (LOV_MAX_STRIPE_COUNT) if the count exceeds this limit.

I would greatly appreciate your input and guidance in this matter. It is worth noting that setting the stripe count higher than LOV_MAX_STRIPE_COUNT leads to other problems, such as the failure of the "llapi_layout_get_by_fd" API to open the file.

Please let me know your input.

Issue Links

• is related to: LU-17925 sanity-flr test_0b: lfs mirror create: cannot create composite file: Numerical result out of range
• is related to: LU-17199 'lfs setstripe -C -1' can be set beyond overstripe count > 2000
• is related to: LU-18244 add 'lfs mkdir -C -N' overstriping support
• is related to: LU-16623 lod_statfs_and_check() does not skip unusable OSTs
• is related to: LU-13748 'lfs setstripe -C -1' stripes too widely

Comments

[Patrick Farrell]

Interesting, OK!  Happy to defer.  I wasn't familiar with "last option takes precedence".

[Andreas Dilger]

Actually, I will disagree with Patrick here. While "-C M" enables overstriping, it works with a stripe count that is less than the number of OSTs < M, equivalent to "-c M" in that case. I don't think it "conflicts" with a later "-c N" option. Like many utilities, the last option specified will take precedence.

[Patrick Farrell]

It would be OK to fix that as a separate patch on this ticket, it's small enough it doesn't need a new ticket.  But it should be a separate patch, that's all.

[Rajeev Mishra]

Ok I will just take care of n*ostcount as part of this ticket. Thanks Patrick

[Patrick Farrell]

Interesting!  I think we should make them mutually exclusive, yes.  I would do that in a separate patch from the one which adds '-2, -3' functionality.

[Rajeev Mishra]

I'm currently reviewing the options -c, -C, overstripe-count, and stripe-count to gain a better understanding of their behavior.

I've noticed that there's a bug where the command accepts all of these options simultaneously, even though it appears that they should be mutually exclusive. Presently, all flags can be used together, as demonstrated in the following example:

[root@test2-rocky8 jbs]# lfs setstripe --overstripe-count 1024 --stripe-count -1 -c 10 -C -1 test

[root@test2-rocky8 jbs]# lfs getstripe test | more

test

lmm_stripe_count:  2000

lmm_stripe_size:   1048576

lmm_pattern:       raid0,overstriped

lmm_layout_gen:    0

 

The documentation for these options are as follows:

 

    -c, --stripe-count <stripe_count>: Specifies the nu    mber of OSTs to stripe a file over. A value of 0 means to use the filesystem-wide default stripe count (default is 1), and -1 means to stripe over all available OSTs.

 

    -C, --overstripe-count <stripe_count>: Specifies the number of stripes to create, creating more than one stripe per OST if the count exceeds the number of OSTs in the file system. Similar to -c, 0 uses the filesystem-wide default stripe count (default is 1), and -1 means to stripe over all available OSTs.

    

Now, the question arises: should we consider making these options mutually exclusive? The reason for this consideration is that the behavior of -c is essentially the same as using -C,. If we allow mutual inclusion, we would need to define the precedence of these options when used together.

Your feedback and input on whether or not we should make these options mutually exclusive would be greatly appreciated.

The fix will stick with the pattern mentioned in the comment above, which means it will use -1, -2, -3, and so on. In simpler terms, the stripe count will be calculated as a multiple of the OST count, up to the maximum stripe count allowed.

*~ *                                                                                                     

[Andreas Dilger]

Since the core "-C -1" issues were already fixed by LU-13748 and LU-16623, I changed this issue to track the improvement for mapping "-C -1" to use OST_COUNT like "-c -1", and "-C -2" to use "2 * OST_COUNT", etc.

[Patrick Farrell]

Definitely not.  That can cause crashes, or at least errors (or it should).

[Rajeev Mishra]

@Andreas I will try to add the functionality as suggested. I assume max in any case should not cross LOV_MAX_STRIPE_COUNT that is 2000 ?

[Andreas Dilger]

Rajeev, if you have the cycles, it would be good to implement the "-C -1/-2/-3/..." option to specify 1x/2x/3x/... overstriping of the OSTs, maybe up to 32x the OST count, or at least "-C -1" limiting to OST count? There may be a couple of tests using "-C -1" that need to be changed to e.g. "-C 2000".

[Rajeev Mishra]

With the patch it works good as shown below

lfs setstripe -C -1 /mnt/lustre/rajeev

1. lfs getstripe /mnt/lustre/rajeev 

/mnt/lustre/rajeev

lmm_stripe_count:  2000

lmm_stripe_size:   1048576

lmm_pattern:       raid0,overstriped

lmm_layout_gen:    0

lmm_stripe_offset: 0