https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12434 Lustre <p>Filesystem default dir layout stored in ROOT is not used in new directory creation, this is a technical debt of DNE code. It should follow the same semantic as filesystem default file layout:<br/> 1. if user specified layout, use the input to create new directories.<br/> 2. else if parent directory has default dir layout, check whether this default dir layout is sufficient (stripe_count is not 0 and hash_type is not 'none'), if so, use it to create.<br/> 3. else if filesystem has default dir layout, and fill missing fields with it, then check whether it's sufficient, if so, use it to create.<br/> 4. else fill missing fields with default values (stripe_count is "1", hash_type is "fnv16"), and create.</p> LU-12434

use filesystem default dir layout for new directories

Bug Major Resolved Duplicate Lai Siyao Lai Siyao Thu, 13 Jun 2019 03:43:11 +0000 Fri, 26 Nov 2021 21:49:21 +0000 Fri, 26 Nov 2021 21:48:59 +0000 Upstream Lustre 2.12.0 Lustre 2.10.8 Tue, 13 Aug 2019 00:00:00 +0000 0 3 <p>A tricky problem here is that the new "<tt>hash=space</tt>" code is implemented using the default directory layout.  If this is set on the root directory, then suddenly every directory in the filesystem will use space-balanced <tt>mkdir()</tt> without any way to turn it off.</p> <p>If we want this to work, either the <tt>hash=space</tt> code needs to be implemented as a regular LMV layout (maybe just a flag in the layout type), or some other rule needs to be implemented for inheritance.</p> <p>This is not an issue for ROOT because its default layout is not inherited. And if its hash type is set to 'space', this value will only be used in its subdir creation (checked in LMV), and for other directories, the default value 'fnv16' will be used if a proper value is not provided.</p> <p>But it may become an issue for normal plain directories, we may add a inheritance count like you suggested before. Or we can leave it like this, and suggest user not to set this on normal plain directories, and let dir restripe (TDB) to balance MDT usage.</p> <p>There is a major difference between default dir layout and default file layout: ROOT default layout doesn't affect master MDT choosing for directories not under it, because it's chosen by lmv_locate_tgt() on client, and it's decided by its parent layout and default layout. This is why 'space' hash is not an issue for ROOT.</p> <p>I don't think it makes sense to wait for dir restripe, since that will be at least one release away, and will only balance large directories. Being able to balance small directories on a regular basis is more important for regular usage. I don't think it makes sense to limit space-hashed directories to only be used on ROOT, since they can be very useful for e.g. top-level user directories (to distribute their files across MDTs).</p> <p>I think the main issue is if new subdirectories below a space-hash directory are always created as striped:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /* * <span class="code-quote">"space"</span> hash type can only be set in <span class="code-keyword">default</span> LMV, child use * FNV_1A_64 in <span class="code-keyword">case</span> it's striped. */ <span class="code-keyword">if</span> (lo-&gt;ldo_dir_hash_type == LMV_HASH_TYPE_SPACE) lo-&gt;ldo_dir_hash_type = LMV_HASH_TYPE_FNV_1A_64; </pre> </div></div> <p>I think it makes sense to move the <tt>lum_stripe_count</tt> check into <tt>lod_striping_from_default()</tt> so <tt>lum_stripe_count=0</tt> or <tt>lum_stripe_count=1</tt> would create a plain subdirectory, otherwise, the directory would be created with the specified parameters in <tt>lmv_user_md_v1</tt>., which makes the code more clear:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (lds-&gt;lds_dir_def_striping_set &amp;&amp; S_ISDIR(mode)) { : : <span class="code-comment">/* <span class="code-keyword">if</span> stripe count is 1 or 0, create normal directory */</span> <span class="code-keyword">if</span> (lo-&gt;ldo_dir_stripe_count &lt;= 1) { lo-&gt;ldo_dir_hash_type = LMV_HASH_TYPE_NONE; lo-&gt;ldo_dir_stripe_count = 0; } /* * <span class="code-quote">"space"</span> hash type can only be set in <span class="code-keyword">default</span> LMV, child use * FNV_1A_64 in <span class="code-keyword">case</span> it's striped. */ <span class="code-keyword">if</span> (lo-&gt;ldo_dir_hash_type == LMV_HASH_TYPE_SPACE) lo-&gt;ldo_dir_hash_type = LMV_HASH_TYPE_FNV_1A_64; </pre> </div></div> <p>However, this only allows setting the hash type to <tt>FNV_1A_64</tt>, which may be limiting in the future.</p> <p>One option would be to change <tt>lum_hash_type</tt> to be like <tt>LOV_PATTERN_&#42;</tt> and have a 16-bit mask in the high bits and a 16-bit type, and put <tt>LMV_HASH_FLAG_SPACE</tt> in the mask part, and let the low bits store <tt>LMV_HASH_TYPE_FNV_1A_64</tt> or <tt>LMV_HASH_TYPE_ALL_CHARS</tt> or <tt>LMV_HASH_TYPE_UNKNOWN</tt> (which should be renamed to <tt>LMV_HASH_TYPE_NONE</tt>) to indicate what type of subdirectory should be created.</p> <p>For the inherited layout, the <tt>lum_hash_type</tt> would be inherited if <tt>lum_inherit_count &gt; 0</tt> and cleared once <tt>lum_inherit_count == 0</tt> to only create plain directories below that. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /* * "space" hash type can only be set in default LMV, child use * FNV_1A_64 in case it's striped. */ if (lo-&gt;ldo_dir_hash_type == LMV_HASH_TYPE_SPACE) lo-&gt;ldo_dir_hash_type = LMV_HASH_TYPE_FNV_1A_64; </pre> </div></div> <p>This code only changes hash type to default hash type, it doesn't mean subdirectories will be created as striped. This is just in case user set directory default layout hash to 'space' and stripe count &gt; 1.</p> <p>I'll add a patch to implement inherit count first. </p> <p>OK, good. I was a bit concerned about that, but it seems OK if it falls back to plain directory type for <tt>lum_stripe_count &lt;= 1</tt>.</p> <p>PS: it is confusing to me that the <tt>struct lmv_user_md</tt> fields are named "<tt>lum_&#42;</tt>" the same as <tt>struct lov_user_md</tt>. It would maybe be better to name them "<tt>luv_&#42;</tt>" or something to make them different, though I understand that would also be a lot of lines changed...</p> <p>It looks like there is already a mask <tt>LMV_HASH_TYPE_MASK</tt> declared in <tt>lustre_idl.h</tt> along with several <tt>LMV_HASH_FLAG_&#42;</tt> values, while the actual <tt>LMV_HASH_TYPE_&#42;</tt> values are declared in <tt>lustre_user.h</tt>.</p> <p>To allow us to potentially allow inheriting different hash types from "space" hashed directories, we should consider moving <tt>LMV_HASH_TYPE_SPACE</tt> to be <tt>LMV_HASH_FLAG_SPACE</tt> and change the value to be in the high bits. This could easily be done now, but will be much more difficult after 2.13 is released. The <tt>LMV_HASH_FLAG_&#42;</tt> values should also be moved into <tt>lustre_user.h</tt> so that they are together into <tt>enum lmv_hash_type</tt> , so we don't accidentally declare conflicting values.</p> <p>This was largely implemented in patch <a href="https://review.whamcloud.com/44090" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44090</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-14792" title="DNE3: enable filesystem-wide default LMV" class="issue-link" data-issue-key="LU-14792"><del>LU-14792</del></a> llite: enable filesystem-wide default LMV</tt>" using "<tt>lum_stripe_offset = -1</tt>" and remote 1-stripe directories to do space balancing.</p> <p><tt>LMV_HASH_TYPE_SPACE</tt> was renamed to <tt>LMV_HASH_FLAG_SPACE</tt> in patch <a href="https://review.whamcloud.com/35318" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35318</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-11213" title="DNE3: remote mkdir() in ROOT/ by default" class="issue-link" data-issue-key="LU-11213"><del>LU-11213</del></a> uapi: change 'space' hash type to hash flag</tt>" and removed entirely in patch <a href="https://review.whamcloud.com/35825" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35825</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-12624" title="DNE3: striped directory allocate stripes by QoS" class="issue-link" data-issue-key="LU-12624"><del>LU-12624</del></a> lod: alloc dir stripes by QoS</tt>"</p> Related LU-12624 LU-14792 LU-11213 Development Rank 1|i00i67: Rank (Obsolete) 9223372036854775807 Severity