https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13058 Lustre <p>This is a simple extension to the SEL functionality for PFL files, originally suggested by Andreas.<br/> One of the features of SEL is that if there is an intermediate layout component (Consider a PFL layout like: DOM-&gt;SSD-&gt;HDD, in this case SSD is the intermediate component), it will not be instantiated if that tier is low on space, instead, the next component (HDD) is extended downwards. This allows us to skip the SSD tier if it's full.</p> <p>This is a nice feature, and there's no particular reason it has to be limited to SEL layouts. It's easy to do this for normal layouts, where the SSD component has a normally defined length.</p> <p>So, this patch adds that functionality. The canonical case is a DOM-&gt;SSD-&gt;HDD layout where the SSD tier is low on space (or even out of space entirely). Currently, when the first write happens to the SSD component, it's simply instantiated. If there is absolutely no space, an error results. With this feature, in the low on space condition*, that intermediate component is removed.</p> <p>*the same low on space condition as used in SEL, basically if one of the chosen OSTs is below the threshold value for striping. The stripe allocator will only stripe to these OSTs in absence of a better choice, so this indicates we're very low on space.</p> <p>There is one detail: The SEL code uses the "extension size" as a way to estimate how much space this component might use, so it's factored in to the "low on space" calculation. There is no obvious substitute for this with a regular file, which leads to two options:</p> <p>1. Act like the file will consume (effectively) zero space and only act if the OSTs are already low on space<br/> 2. Pick some amount of data to assume it will use - The most logical guess seems to be a multiple of stripe size, but perhaps an absolute value would be better, as stripe sizes can vary widely.</p> <p>It's not clear that 1 isn't fine, and in either case, this is just an optimization.</p> <p>Patch forthcoming.</p> LU-13058

Intermediate component removal (PFL/SEL)

Improvement Major Open Unresolved Patrick Farrell Patrick Farrell Sun, 8 Dec 2019 20:19:37 +0000 Tue, 30 May 2023 15:18:00 +0000 Lustre 2.14.0 Lustre 2.12.5 Lustre 2.16.0 0 5 <p>Patrick Farrell (farr0186@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/36953" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36953</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13058" title="Intermediate component removal (PFL/SEL)" class="issue-link" data-issue-key="LU-13058">LU-13058</a> lod: Intermediate component removal<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ac5638e83a8436b3d46a2cd74634b2a8578dadb3</p> <p>It would make sense to use the size of the intermediate component as the threshold for whether there is enough space on the OST(s).</p> <p>Ah, of course, yeah.  I'm stuck in the mindset of self extending layouts, where the component can change later.  These are fixed from the beginning, so, yeah, component size kinda makes sense.</p> <p>Unfortunately, this will likely break a bunch of tests and may introduce some usability issues for developers...?  Because if the size of your second component is (for example) 1 GiB, but you're running on the default llmount.sh config, that will <b>never</b> show as having enough space for that.  So essentially, creating a three component PFL layout on that test config and trying to instantiate that second component won't work, unless the components are very small.</p> <p>I'll give it a shot and see how many tests it breaks.  Let me know if that adjusts your thinking or if you've got an idea for coping with that.</p> <p>I agree that it is likely that some tests will fail if the default layout is changing. I think in many cases the failures can be mitigated by small/sane changes to the layout used for a particular test, or by making the test smart enough to handle this.</p> <p>I think the first thing to do would be getting regular testing to pass with a default PFL layout, starting with patch <a href="https://review.whamcloud.com/26576" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26576</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-11918" title="Allow setting default file layout on root directory at mkfs time" class="issue-link" data-issue-key="LU-11918">LU-11918</a> tests: modify file system layout in testing</tt>". The results for that patch show there are already a number of subtests failing because that have built-in assumptions of <tt>stripe_count=1</tt> or <tt>stripe-size=1MB</tt> as the default filesystem layout.</p> <p>I'd recommend to approach this in a systematic manner, first changing the filesystem default <tt>stripe_count=3</tt> or similar and fixing subtests to handle the new default and/or explicitly specify the layout that they require for the test, then the default <tt>stripe_size=3MB</tt> or whatever and repeat, then PFL layout with <tt>stripe_count=1, stripe_size=1MB</tt> as the first component, etc.</p> <p>Without first addressing the hidden assumptions in the existing tests, I think that this will be a very large patch that conflates existing issues with potential new issues that are added with this additional change.</p> <p>That said, I <b>do</b> like the idea you are proposing here. In some respects, it would be nice if all components were treated like SEL components by default and users didn't have to explicitly set extension components, or worry about if some OST is going to run out of space. </p> <p>OK, that makes sense.  I can try to take a quick look at some of that at some point - I'm doing this in my spare time, so it's uncertain how much I'll dig in to the other test stuff.</p> <p>"In some respects, it would be nice if all components were treated like SEL components by default and users didn't have to explicitly set extension components, or worry about if some OST is going to run out of space."</p> <p>I think this is intriguing - It would be doable.  Very doable, in fact, though the effects would be wide ranging.  It would be a matter of converting the normal PFL component expression (with setstripe) to have an implicit -z, basically, and then I guess converting regular setstripe -c to make an SEL file rather than a plain file.</p> <p>So all first components (DOM excluded, since it has a fixed size) would start out small, and all other components would start out zero length, and all followed by extension space.</p> <p> </p> <p>There would be lots of ripple effects, but the only obvious question (to me) is how to set the extension size (the amount of space given out at a time).  Perhaps something like 1% or 100 GiB, whichever is larger?  I'm not sure - Layout lock changes could be pretty disruptive for a large file being written in parallel.  It seems like it would be important to allow <b>not</b> doing this for that case.  (Though I suppose if a file is striped widely as well, the data per stripe might not be much different from a single writer file being written quickly by one client, so the issue might be roughly the same.)</p> <p>Hm.  The number of ripple effects and the complexity it introduces to regular layouts make me nervous.</p> <p> </p> <p>"That said, I <b>do</b> like the idea you are proposing here."<br/> As I alluded to, you originally suggested it (the "PFL could remove intermediate components" bit) during SEL review. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I also thought of another important use case for this - skipping the PFL/SEL components for OST pools in which the user has no quota. That depends on the OST pool quota feature (<a href="https://jira.whamcloud.com/browse/LU-11023" title="OST Pool Quotas" class="issue-link" data-issue-key="LU-11023"><del>LU-11023</del></a>) to be available, but seems like a logical extension of this work.</p> <p>I agree, the only downside is that it seems like it would require a little bit of plumbing - Handling quotas was rejected as part of the SEL work (at least initially) for that reason.</p> <p>Although now that I think about it, my position at the time (in the design discussions within Cray) was based on the idea of integrating quota levels in to the stripe allocator decisions, which really would be kind of terrible.</p> <p>But if we assume that quota pools and OST tiers are arranged sanely (ie, the pools used for quota match up with the pools used in the layout/tiering), which I think is fair (since things won't <b>break</b> if they are not - it will just give suboptimal behavior), then we could just make quota checking part of the "are these selected OSTs OK" step*, since the quota itself is split across the OSTs evenly.</p> <p>*ie, when we check the OSTs selected by the stripe allocator to verify space levels</p> <p> </p> <p>It's not <b>quite</b> as good as integrating quota in to the stripe allocator decisions, but that would be a huge amount of work and I think definitely overkill.</p> <p>So, yeah, that would be manageable I think.  Just some extra plumbing to check quotas from the LOD context.</p> <p>But as you noted, pool quotas required first.</p> <p>Mike, this is very similar to the DoM component shrinking/removal that you implemented. Would you be able to finish off Patrick's patch in time for 2.14?</p> <p>This may be <em>mostly</em> unnecessary in the presence of the pool spill mechanism added in patch <a href="https://review.whamcloud.com/43989" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43989</a> "<tt>LU-14825 lod: pool spilling</tt>". Rather than drop the component with the full pool and move to the next one, pool spill reassigns that component to its specified <tt>spill_target</tt> pool.</p> <p>That allows the admin to "fix" all layouts that are targeting a specific pool, including cases where removing a component wouldn't help because the <b>next</b> (...) component is also on the same pool, but with a larger stripe count. Also, it avoids unnecessarily inflating the stripe count when an early component is dropped.</p> <p>The one drawback of pool spill is that it is a single global parameter and does not allow the fine-grained control of the layout that SEL does (i.e. which pool to use for each component), but that is not (IMHO) going to be a common use case, since most users don't know how to set the layout.</p> <p>In summary, I'm not against keeping this open to eventually land this patch, but I don't think it is as useful/important as it once was.</p> Related LU-10070 LU-11918 LU-15011 LU-11023 LU-16857 LU-12785 Development Rank 1|i00qmv: Rank (Obsolete) 9223372036854775807 Severity