https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9435 Lustre <p>In current implementation, when a file is created, the file's inode must be in the same MDT where the name entry locates. It's more desirable to allocate MDT objects with the QoS policies like what we do for OST objects.</p> LU-9435

DNE2 - object placement QoS policy

Improvement Minor Resolved Won't Fix Lai Siyao Jinshan Xiong dne3 Tue, 2 May 2017 21:21:00 +0000 Wed, 6 Mar 2019 13:49:35 +0000 Sat, 4 Aug 2018 19:07:43 +0000 0 7 <p>I don't think it is as straight forward as always creating the name on one MDT and allocating the inode on another arbitrary MDT. One of the major problems that would arise is that having remote directory entries for every file would hurt file creation performance, as well as every lookup or unlink of that file in the future. With a remote entry, the client first has to do name-&gt;FID lookup in the parent directory, and then separately do FID-&gt;MDT lookup in the FID Location Database (FLDB, typically very fast since it is compact and cached on the client), and then fetch attributes/layout/xattrs for the FID from the second MDT. This would double the number of RPCs needed to access the majority of files. Keeping the directory entries and inodes on the same MDT is far more efficient for creation, lookup, and deletion.</p> <p>Instead, there are several mechanisms that can be used to distribute metadata loads/allocations across MDTs while keeping names/inodes mostly local to a single MDT.</p> <ul class="alternate" type="square"> <li><b>automatic MDT selection for striped directories</b>: if the shards of a DNE2 directory are load balanced across MDTs then the names created in those shards will also be balanced. Currently (AFAIK) the shards are allocated sequentially from the master MDT index unless otherwise specified, which is not ideal: <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: fnv_1a_64 mdtidx FID[seq:oid:ver] 0 [0x200000400:0x2:0x0] 1 [0x240000401:0x2:0x0] lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: fnv_1a_64 mdtidx FID[seq:oid:ver] 0 [0x200000400:0x4:0x0] 1 [0x240000401:0x4:0x0] lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: fnv_1a_64 mdtidx FID[seq:oid:ver] 0 [0x200000400:0x6:0x0] 1 [0x240000401:0x6:0x0] </pre> </div></div> <p>This should be relatively easy to implement when striped directories are explicitly created, since all of this is decided on the MDS, and it can do <tt>MDS_STATFS</tt> RPCs to the other MDTs (as we already do with OSTs) to select MDTs based on free space, if the number of stripes is less than the number of MDTs.</p></li> </ul> <ul class="alternate" type="square"> <li><b>automatic MDT selection for remote directories</b>: is a bit more tricky, since the client specifies the FID for the remote directory, but one possibility is to have "<tt>lfs mkdir</tt>" get the MDT space usage on the client to decide which MDT to use, if it is not specified by the user. Another alternative would be for the MDS to just ignore the FID supplied by the client, and allocate its own remote directory and return the new FID to the client (this is already handled by clients, in case the file/directory already exists).</li> <li><b>automatic remote MDT selection for new directories</b>: once the above MDT selection mechanism exists, it would be possible to automatically create some subset of new directories on remote MDTs in order to balance the load across MDS nodes.</li> <li><b>automatic restriping of large directories</b>: this is related to <a href="https://jira.whamcloud.com/browse/LU-4684" title="DNE3: allow migrating DNE striped directory" class="issue-link" data-issue-key="LU-4684"><del>LU-4684</del></a> "allow migrating DNE striped directory". Basically, when a directory grows too large (e.g. over 5000 entries), the LMV layout is changed to a striped directory so that it is automatically load balanced across MDS nodes. Either a PFL-like layout that keeps existing entries in the "master" directory and new entries are inserted into the shards (lower overhead at split time, higher overhead during later lookups), and/or migrating existing entries to the new shards (higher overhead at split, lower overhead during later lookups), or a combination of both (delayed migration from master to shards some arbitrary time after split). The benefit of automatically sharding large directories is that any subdirectories will also be distributed, and space used by Data-on-MDT objects will also be balanced naturally.</li> </ul> <p>All of these options allow the majority of entries to remain local to the MDT where the inode is created, while distributing load across MDTs more evenly without user interaction.</p> <blockquote> <p>One of the major problems that would arise is that having remote directory entries for every file would hurt file creation performance, as well as every lookup or unlink of that file in the future. With a remote entry, the client first has to do name-&gt;FID lookup in the parent directory, and then separately do FID-&gt;MDT lookup in the FID Location Database (FLDB, typically very fast since it is compact and cached on the client), and then fetch attributes/layout/xattrs for the FID from the second MDT. This would double the number of RPCs needed to access the majority of files.</p></blockquote> <p>Indeed, so we only split name-entry and object for the directory (remote directory), that probably means we only do QOS thing for directory creation. </p> <blockquote> <p>This should be relatively easy to implement when striped directories are explicitly created, since all of this is decided on the MDS, and it can do MDS_STATFS RPCs to the other MDTs (as we already do with OSTs) to select MDTs based on free space, if the number of stripes is less than the number of MDTs.</p> <p>Another alternative would be for the MDS to just ignore the FID supplied by the client, and allocate its own remote directory and return the new FID to the client (this is already handled by clients, in case the file/directory already exists).</p></blockquote> <p>This really makes sense to me, and it probably also means we only need put MD QOS into LOD, (no need in LMV). which will allow us easily share MDT/OST QOS code.</p> <blockquote> <p>automatic restriping of large directories: this is related to <a href="https://jira.whamcloud.com/browse/LU-4684" title="DNE3: allow migrating DNE striped directory" class="issue-link" data-issue-key="LU-4684"><del>LU-4684</del></a> "allow migrating DNE striped directory".</p></blockquote> <p>Even curent migrate tool (rebalance the objects over MDTs) will suit a lot of QOS needs, though it is not automatic. Btw: we also need a ticket for migrating data-on-MDT objects.</p> <p>This is probably best handled at the directory level, rather than making every file remote by default. </p> <p>This will be handled via other tickets and per-directory balancing. </p> Related LU-10277 LU-11213 LU-7827 LU-10784 Development Rank 1|hzzbpj: Rank (Obsolete) 9223372036854775807