https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10808 Lustre <p>If the DoM component <tt>extent_end</tt> is set larger than the MDT <tt>dom_stripesize</tt> (via <tt>lfs setstripe &#45;Eextent&#95;end &#45;L mdt</tt> ) then this currently generates an error.  Since users do not have any easy way to determine the <tt>dom_stripesize</tt> on the MDT, and it may in fact be different on a per-MDT basis (e.g. if MDT0000 sets <tt>dom_stripesize=0</tt> because it was formatted before DoM, or it is adjusted automatically by the MDS when the MDT is nearly full).  This complicates DoM usage for users (imagine a striped directory that has different DoM size limits for the MDTs it is striped over).</p> <p>The MDS should automatically adjust the component <tt>extent_end</tt> to match the MDT <tt>dom_stripesize</tt>, and if <tt>dom_stripesize=0</tt> then the DoM component should be removed.</p> LU-10808

DoM: component end should align with dom_stripesize

Technical task LU-10176 Minor Resolved Fixed Mikhail Pershin Andreas Dilger DoM2 Mon, 12 Mar 2018 20:08:09 +0000 Wed, 5 Feb 2020 20:20:05 +0000 Wed, 22 Aug 2018 18:48:30 +0000 Lustre 2.11.0 Lustre 2.12.0 0 10 <p> Just curious here and hence few questions</p> <p>&gt; The MDS should automatically adjust the component extent_end to match the MDT dom_stripesize,<br/> &gt; and if dom_stripesize=0 then the DoM component should be removed.</p> <p>Does this mean that if mdt component size value greater than dom_stripesize<br/> then it will be set to dom_stripesize</p> <p>also if dom_stripesize=0 then even if user mention DoM component it wont be considered and <br/> only PFL will be considered ?</p> <p>It seems we are doing kind of similar stuff (in lod_fix_desc_stripe_size()), while we are setting dom_stripesize value.</p> <p>i.e. even if the dom_stripesize is set to value less than LOV_MIN_STRIPE_SIZE, then it is set to default<br/> value and if try to set value say 65K then it will be set as 64K as multiple of LOV_MIN_STRIPE_SIZE</p> <p>Also needing consideration here is that DoM components should be skipped entirely if there is no (or very little) space on the MDT. It might make sense to shrink <tt>dom_stripesize</tt> as the MDT begins to get full, but at least a simple fix is to just go directly to zero once the MDT free space is below some threshold.</p> <p>I tested out setting <tt>dom_maxstripesize</tt> and if this is less than <tt>LOV_MIN_STRIPE_SIZE</tt> sets it to the <b>default</b> stripe size, rather than the <b>minimum</b>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">void lod_fix_desc_stripe_size(__u64 *val) { <span class="code-keyword">if</span> (*val &lt; LOV_MIN_STRIPE_SIZE) { <span class="code-keyword">if</span> (*val != 0) LCONSOLE_INFO(<span class="code-quote">"Increasing <span class="code-keyword">default</span> stripe size to "</span> <span class="code-quote">"minimum value %u\n"</span>, LOV_DESC_STRIPE_SIZE_DEFAULT); *val = LOV_DESC_STRIPE_SIZE_DEFAULT; } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (*val &amp; (LOV_MIN_STRIPE_SIZE - 1)) { *val &amp;= ~(LOV_MIN_STRIPE_SIZE - 1); LCONSOLE_WARN(<span class="code-quote">"Changing <span class="code-keyword">default</span> stripe size to %llu (a "</span> <span class="code-quote">"multiple of %u)\n"</span>, *val, LOV_MIN_STRIPE_SIZE); } } </pre> </div></div> <p>Since this is also used for regular file striping (typically with <tt>stripe_size=0</tt> to indicate the default) then it may be difficult to change this behaviour. One option is to pass in the "<tt>default</tt>" value from the caller, so that LOV can use <tt>LOV_DESC_STRIPE_SIZE_DEFAULT</tt> but DoM can use <tt>LOV_MIN_STRIPE_SIZE</tt>. Alternately, <tt>lod_dom_stripesize_seq_write()</tt> could check for a value below <tt>LOV_MIN_STRIPE_SIZE</tt> and return an error code to the caller.</p> <p>yes, I found that too and already fixed this in a patch, it sets limit to the minimum rather than default value.</p> <p>Mike Pershin (mike.pershin@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32073" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32073</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10808" title="DoM: component end should align with dom_stripesize" class="issue-link" data-issue-key="LU-10808"><del>LU-10808</del></a> lod: align wrong DoM stripe values with defaults<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 33206489d8b3252d96ca4d23d7774d54d8b085fc</p> <p>Andreas wrote in patch comments:</p> <blockquote> <p>What happens if we have a 16-way striped directory with a default layout like:</p> <p> lfs setstripe -E 1M -L mdt -E 64M -c 1 -E -1 -c 128 /dir</p> <p>and then one of the MDTs runs out of space? Ideally, it <em>wouldn't</em> run out of space (the MDS should reduce lod_dom_max_stripesize automatically as it gets full so the DoM components get smaller) but it will eventually run out of space. In case like this, we don't really want 1/16 of the files to return -EFBIG.</p> <p>There is no way (currently) to disable file creation on a single MDT in the striped directory, so we would want to remove the DoM component from the file to reduce MDT space usage rather than failing file creation randomly based on the filename.</p> <p>Is it possible to check if the layout is a default layout, or explicitly set by a user? It would be confusing if the user specifies a DoM component and it just silently disappears, but this is somewhat less of an issue if it is coming from a default setting (parent directory or filesystem root).</p></blockquote> <p>Put it here to don't forget and prepare workaround in later patches</p> <p>Mike Pershin (mike.pershin@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32482" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32482</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10808" title="DoM: component end should align with dom_stripesize" class="issue-link" data-issue-key="LU-10808"><del>LU-10808</del></a> lod: remove DoM component is disabled<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6b1cc1f9b42e8cacab90099c9c56f0ab96d9a471</p> <p>This question has likely been answered elsewhere, but yes, it's possible to know if a layout is a default layout.  At file creation time, there's a moment where you inherit from a template.  But after that there's no way to know, currently.</p> <p>It seems like your suggestion is if any MDT in the set is low on space, then remove the DOM component from the layout (presumably at file creation time rather than permanently, since ideally out of space is transient).  So we stop doing DOM entirely because we can't skip that one MDT.  That will require some interrogation of the MDTs but it should be possible, if a little awkward.</p> <p>An alternate possibility occurs to me, sort of adjacent to the self-extending PFL work and exploiting similar ideas.  DOM is always the first component &amp; so is always instantiated.  But we could do the "stripe to target, then check space on target before really making objects" trick I'm using in the SEPFL work (posting soon, sorry).</p> <p>If there's insufficient space on a particular MDT, we could either try another MDT (...not so sure about this one...) or we could rewrite the layout dynamically to remove the DoM component, assuming it's followed by a regular OST component.  (Which it doesn't have to be.)</p> <p>If we did the "try another MDT" solution, we'd either need to try all MDTs and fail once they were out of space, or we'd <b>still</b> have to do the "rewrite layout to replace DoM with regular OST layout" thing.</p> <p>Just stop doing DoM once one MDT is full does seem like the easiest (and perhaps most comprehensible) behavior.  But there's another route if we wanted to take it.</p> <p>I'm not suggesting to disable DoM permanently if one MDT is full, only on a per-file basis. Firstly, MDT is the one that selects the file layout and should decide if it is full, and this can be done on a per-MDT basis. </p> <p>Secondly, the hash of the filename is what decides which MDT is used, so we can't try different MDTs to find space for the DoM data. </p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/32073/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32073/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10808" title="DoM: component end should align with dom_stripesize" class="issue-link" data-issue-key="LU-10808"><del>LU-10808</del></a> lod: align wrong DoM stripe values with defaults<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9146d261f35b394e10afde3eec2d5895425261e0</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/32482/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32482/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10808" title="DoM: component end should align with dom_stripesize" class="issue-link" data-issue-key="LU-10808"><del>LU-10808</del></a> lod: remove DoM component if DoM is disabled<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3fd758ed4f40f235d26df9fd7c6b459590fbe0cf</p> <p>Seems like all patches have landed for 2.12</p> Related LU-10465 LU-10786 LU-10917 LU-11608 LU-10070 Development Rank 1|hzzubb: Rank (Obsolete) 9223372036854775807