https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16938 Lustre <p>I am reaching out to seek clarification regarding the expected behavior of the <b>"lfs setstripe"</b> command when using the -C -<b>1</b> option.</p> <p>Currently, it appears that this command is creating a higher stripe count than anticipated. For instance, on my test system, it generated a stripe count of <b>2727</b> <b>for</b> a single file. This count exceeds the allowed limit of LOV_MAX_STRIPE_COUNT. </p> <p>I am uncertain about the appropriate solution to address this issue related to the <b>"-1"</b> argument. I have contemplated the following options:</p> <p>1.    Consider making the option -<b>1</b> illegal, preventing its usage altogether.</p> <p>2.    Implement a mechanism to automatically set the stripe count to the maximum allowed value (LOV_MAX_STRIPE_COUNT) <b>if</b> the count exceeds this limit.</p> <p>I would greatly appreciate your input and guidance in this matter. It is worth noting that setting the stripe count higher than LOV_MAX_STRIPE_COUNT leads to other problems, such as the failure of the <b>"llapi_layout_get_by_fd"</b> API to open the file.</p> <p>Please let me know your input.</p> LU-16938

"lfs setstripe -C -1" stripes too widely, should be limited to OST_COUNT

Improvement Major Open Unresolved Rajeev Mishra Rajeev Mishra Mon, 3 Jul 2023 18:51:23 +0000 Fri, 15 Dec 2023 23:34:24 +0000 Lustre 2.15.0 0 8 <p>This should limit the stripe count of the component to LOV_MAX_STRIPE_COUNT, or whatever will fit into the remaining 64KiB xattr space. This was also discussed in <a href="https://jira.whamcloud.com/browse/LU-13748" title="&#39;lfs setstripe -C -1&#39; stripes too widely" class="issue-link" data-issue-key="LU-13748"><del>LU-13748</del></a>. I made patch <a href="https://review.whamcloud.com/50532" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/50532</a> "{{ <a href="https://jira.whamcloud.com/browse/LU-13748" title="&#39;lfs setstripe -C -1&#39; stripes too widely" class="issue-link" data-issue-key="LU-13748"><del>LU-13748</del></a> mdt: remove LASSERT in mdt_dump_lmm()}}" to fix the crash from overstriping too widely, but didn't fix the stripe_count limit setting. </p> <p>Andreas,<br/> "-C -1" mimics "-c -1" but unlike the wide striping across all OSTs , overstripe count of 2000 is not a sane thing to do. I bet nobody but only Lustre testers really want 2000 stripes per file. That brings a question about dropping support of a special value of "-1" for -C.</p> <p>+1 Zam's comment. 2000 overstripes will more likely hurt more than it might help. We should protect the users from doing stupid things. I think if someone wants the stripe limit, then they can specify an appropriate value themselves.</p> <p>Patrick, any comments on this? At one time I was thinking that we might use "<tt>-C -2</tt>" to mean 2xOST_COUNT overstriping, etc. up to some reasonable maximum. That would make "<tt>-C -1</tt>" behave the same as "<tt>-c -1</tt>".</p> <p>Yeah, I think you're both right - The current behavior doesn't make sense and should go, and the suggested behavior from Andreas is reasonable.  My plate is full at the moment, though, so if anyone wants it... heh</p> <p>By the way, I believe <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=scherementsev" class="user-hover" rel="scherementsev">scherementsev</a> fixed the core bug reported here (where -C -1 leads to bad behavior and even possible crashes) in a patch he was doing reworking part of stripe allocation?  Hopefully he remembers which one.  He took option '2', since we should do that regardless.</p> <p>Rajeev, I would support a patch to make -C -1 do the same as '-c -1', or just to return -EINVAL.  Either one is fine with me.  The improved behavior Andreas is suggesting for -2, etc would be neat as well but would involve at least a little work.  (Documentation, tests, etc, even if the implementation is easy)</p> <p>I think Patrick is speaking about "<a href="https://jira.whamcloud.com/browse/LU-16623" title="lod_statfs_and_check() does not skip unusable OSTs" class="issue-link" data-issue-key="LU-16623"><del>LU-16623</del></a> lod: handle object allocation consistently".<br/> Checking master with mentioned patch I don't see any problem:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@vm2 tests]# lfs setstripe -C -1 /mnt/lustre/foo [root@vm2 tests]# lfs getstripe /mnt/lustre/foo | head -n 4 /mnt/lustre/foo lmm_stripe_count:  2000 lmm_stripe_size:   1048576 lmm_pattern:       raid0,overstriped </pre> </div></div> <p>Guys, does version where you reproduced the issue include above patch? Or am I doing something wrong to reproduce it? If so, please give more details.</p> <p>Thanks Sergey.</p> <p>Patrick and Sergey I do not have the LU 16623 in my workspace. Will update my workspace and let you know if the problem still persist.</p> <p>Thanks for your help.</p> <p>With the patch it works good as shown below</p> <p>lfs setstripe -C -1 /mnt/lustre/rajeev</p> <ol> <li>lfs getstripe /mnt/lustre/rajeev </li> </ol> <p>/mnt/lustre/rajeev</p> <p>lmm_stripe_count:  2000</p> <p>lmm_stripe_size:   1048576</p> <p>lmm_pattern:       raid0,overstriped</p> <p>lmm_layout_gen:    0</p> <p>lmm_stripe_offset: 0</p> <p>Rajeev, if you have the cycles, it would be good to implement the "<tt>&#45;C &#45;1/&#45;2/&#45;3/...</tt>" option to specify 1x/2x/3x/... overstriping of the OSTs, maybe up to 32x the OST count, or at least "<tt>&#45;C &#45;1</tt>" limiting to OST count? There may be a couple of tests using "<tt>&#45;C &#45;1</tt>" that need to be changed to e.g. "<tt>&#45;C 2000</tt>".</p> <p>@Andreas I will try to add the functionality as suggested. I assume max in any case should not cross LOV_MAX_STRIPE_COUNT that is 2000 ?</p> <p>Definitely not.  That can cause crashes, or at least errors (or it should).</p> <p>Since the core "<tt>&#45;C &#45;1</tt>" issues were already fixed by <a href="https://jira.whamcloud.com/browse/LU-13748" title="&#39;lfs setstripe -C -1&#39; stripes too widely" class="issue-link" data-issue-key="LU-13748"><del>LU-13748</del></a> and <a href="https://jira.whamcloud.com/browse/LU-16623" title="lod_statfs_and_check() does not skip unusable OSTs" class="issue-link" data-issue-key="LU-16623"><del>LU-16623</del></a>, I changed this issue to track the improvement for mapping "<tt>&#45;C &#45;1</tt>" to use <tt>OST_COUNT</tt> like "<tt>&#45;c &#45;1</tt>", and "<tt>&#45;C &#45;2</tt>" to use "<tt>2 &#42; OST_COUNT</tt>", etc.</p> <p>I'm currently reviewing the options -c, -C, overstripe-count, and stripe-count to gain a better understanding of their behavior.</p> <p>I've noticed that there's a bug where the command accepts all of these options simultaneously, even though it appears that they should be mutually exclusive. Presently, all flags can be used together, as demonstrated in the following example:</p> <p><span class="error">&#91;root@test2-rocky8 jbs&#93;</span># lfs setstripe --overstripe-count 1024 --stripe-count -1 -c 10 -C -1 test</p> <p><span class="error">&#91;root@test2-rocky8 jbs&#93;</span># lfs getstripe test | more</p> <p>test</p> <p>lmm_stripe_count:  2000</p> <p>lmm_stripe_size:   1048576</p> <p>lmm_pattern:       raid0,overstriped</p> <p>lmm_layout_gen:    0</p> <p> </p> <p>The documentation for these options are as follows:</p> <p> </p> <p>    -c, --stripe-count &lt;stripe_count&gt;: Specifies the nu    mber of OSTs to stripe a file over. A value of 0 means to use the filesystem-wide default stripe count (default is 1), and -1 means to stripe over all available OSTs.</p> <p> </p> <p>    -C, --overstripe-count &lt;stripe_count&gt;: Specifies the number of stripes to create, creating more than one stripe per OST if the count exceeds the number of OSTs in the file system. Similar to -c, 0 uses the filesystem-wide default stripe count (default is 1), and -1 means to stripe over all available OSTs.</p> <p>    </p> <p>Now, the question arises: should we consider making these options mutually exclusive? The reason for this consideration is that the behavior of -c is essentially the same as using -C,. If we allow mutual inclusion, we would need to define the precedence of these options when used together.</p> <p>Your feedback and input on whether or not we should make these options mutually exclusive would be greatly appreciated.</p> <p>The fix will stick with the pattern mentioned in the comment above, which means it will use -1, -2, -3, and so on. In simpler terms, the stripe count will be calculated as a multiple of the OST count, up to the maximum stripe count allowed.</p> <p>*~ *                                                                                                     </p> <p>Interesting!  I think we should make them mutually exclusive, yes.  I would do that in a separate patch from the one which adds '-2, -3' functionality.</p> <p>Ok I will just take care of n*ostcount as part of this ticket. Thanks Patrick</p> <p>It would be OK to fix that as a separate patch on this ticket, it's small enough it doesn't need a new ticket.  But it should be a separate patch, that's all.</p> <p>Actually, I will disagree with Patrick here. While "-C M" enables overstriping, it works with a stripe count that is less than the number of OSTs &lt; M, equivalent to "-c M" in that case. I don't think it "conflicts" with a later "-c N" option. Like many utilities, the last option specified will take precedence.</p> <p>Interesting, OK!  Happy to defer.  I wasn't familiar with "last option takes precedence".</p> <p>&gt; Like many utilities, the last option specified will take precedence.</p> <p>I would be fine with either (mutually exclusive or last takes precedence <em>in its entirety</em>) but this bothers me:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>jupiter-p2:/lus/kjcf08 # mkdir test jupiter-p2:/lus/kjcf08 # lfs setstripe --overstripe-count 1024 --stripe-count 10 test jupiter-p2:/lus/kjcf08 # lfs getstripe -d test test stripe_count: 10 stripe_size: 1048576 pattern: raid0,overstriped stripe_offset: -1 </pre> </div></div> <p>Note that we got (and kept) overstriped from the first param, but then picked up the count from the second. If the last option truly took precedence I would expect a stripe count of 10 <em>without</em> overstriped (just like if the first one took precedence I would expect a stripe count of 1024 <em>with</em> overstriped).</p> <p>It is inconsistent that the behavior is different if you issue them individually, but in the same order:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>jupiter-p2:/lus/kjcf08 # lfs setstripe --overstripe-count 1024 test jupiter-p2:/lus/kjcf08 # lfs getstripe -d test stripe_count: 1024 stripe_size: 1048576 pattern: raid0,overstriped stripe_offset: -1 jupiter-p2:/lus/kjcf08 # lfs setstripe --stripe-count 10 test jupiter-p2:/lus/kjcf08 # lfs getstripe -d test stripe_count: 10 stripe_size: 1048576 pattern: raid0 stripe_offset: -1 </pre> </div></div> <p>Here each command does as I would expect; <tt>&#45;&#45;overstripe-count 1024</tt> by itself yields overstriped and stripe count 1024, and <tt>&#45;&#45;stripe-count 10</tt> by itself on the same directory <em>removes overstriped</em> (which is what I would expect) yielding stripe count 10 without overstriped.</p> <p>The fact that combining them causes it to take the overstriped from the first param and the stripe count from the second is surprising. <tt>&#45;&#45;stripe-count</tt> explicitly means not-overstriped and if the rule is that the last one takes precedence, then it should be like the <tt>&#45;&#45;overstripe-count</tt> wasn't there at all instead of the <tt>&#45;&#45;stripe-count</tt> acting as a modifier.</p> <p>Josh,</p> <p>Makes sense to me.  There's also another possible bug here - how many OSTs do you have on that system?  If it's &gt;= 10, then overstriped shouldn't be set by the <b>server</b> code either, which is also a concern.  Overstriping should only be set on the file when the <b>actual file striping</b> exceeds the number of available OSTs.  (Or at least that was the intent...)</p> <p>So there may be two things to fix there - proper overriding by later parameters in userspace, so the overstriping flag isn't passed along, and then - if you have &gt;= 10 OSTs, then the server shouldn't set the overstriping pattern regardless of what userspace asked for.  If you have 20 OSTs and give -C 10, overstriping shouldn't be set, because the file is not actually overstriped.  Overstriping set on a not-overstriped file isn't fatal, but it's definitely wrong.</p> <p>There is code in <tt>lod_ost_alloc_rr()</tt> in the MDS object allocation that <em>should</em> be removing the <tt>LOV_PATTERN_OVERSTRIPING</tt> flag if it is set unnecessarily:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /* If there are enough OSTs, a component with overstriping requested * will not actually end up overstriped. The comp should reflect <span class="code-keyword">this</span>. */ <span class="code-keyword">if</span> (!overstriped) lod_comp-&gt;llc_pattern &amp;= ~LOV_PATTERN_OVERSTRIPING; </pre> </div></div> <p>If this isn't being applied consistently, then that would be a bug.</p> <p>I don't think that is coming into play here because I'm just showing the default striping on a directory. If I actually create a file within the directory I believe it is behaving as you suggest:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>jupiter-p2:/lus/kjcf08 # mkdir test jupiter-p2:/lus/kjcf08 # lfs setstripe --overstripe-count 1024 --stripe-count 10 test jupiter-p2:/lus/kjcf08 # touch test/foo jupiter-p2:/lus/kjcf08 # lfs getstripe test | head test stripe_count: 10 stripe_size: 1048576 pattern: raid0,overstriped stripe_offset: -1 test/foo lmm_stripe_count: 10 lmm_stripe_size: 1048576 lmm_pattern: raid0,overstriped lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group </pre> </div></div> <p>here the file is overstriped because I only have 2 OSTs.</p> <p>This is a bit of a degenerative example, but if I just set the <tt>--overstripe-count 2</tt> the directory will have a default of overstriped with a stripe count of 2, but files that are created are not overstriped (and have a stripe count of 2):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>jupiter-p2:/lus/kjcf08 # lfs setstripe --overstripe-count 2 test jupiter-p2:/lus/kjcf08 # lfs getstripe -d test stripe_count: 2 stripe_size: 1048576 pattern: raid0,overstriped stripe_offset: -1 jupiter-p2:/lus/kjcf08 # touch test/foo jupiter-p2:/lus/kjcf08 # lfs getstripe test/foo test/foo lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 116959791 0x6f8aa2f 0 0 117253333 0x6fd24d5 0 </pre> </div></div> <p>so I think that part of it is working OK.</p> <p>OK, that's good, then.  The user interface is important but I was more concerned that the server might be marking the layout incorrectly.  Obviously default layouts are a different case.</p> <p>But just to be clear, the inconsistency I'm concerned about can ultimately affect files, e.g. by ending up with a MUCH larger overstripe count than perhaps was intended if one accidentally does something like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>jupiter-p2:/lus/kjcf08 # mkdir test jupiter-p2:/lus/kjcf08 # lfs setstripe --overstripe-count 10 --stripe-count -1 test jupiter-p2:/lus/kjcf08 # touch test/foo jupiter-p2:/lus/kjcf08 # lfs getstripe test | head test stripe_count: -1 stripe_size: 1048576 pattern: raid0,overstriped stripe_offset: -1 test/foo lmm_stripe_count: 2727 lmm_stripe_size: 1048576 lmm_pattern: raid0,overstriped lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group </pre> </div></div> <p>(note the 2727 value is because I don't have Rajeev's other fix on this system, but on the latest code this would be 2000... still probably not what was expected on a system with 2 OSTs).</p> Related LU-16623 LU-13748 Development Rank 1|i03phr: Rank (Obsolete) 9223372036854775807 Severity