https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10329 Lustre <p>For DNE filesystems where there are large numbers of remote entries, for example once <a href="https://jira.whamcloud.com/browse/LU-4684" title="DNE3: allow migrating DNE striped directory" class="issue-link" data-issue-key="LU-4684"><del>LU-4684</del></a> is landed to restripe directories, or DNE2 striped directories with many renames, the size of the <tt>REMOTE_PARENT_DIR</tt> may become very large.</p> <p>In order to limit contention and scaling issues in <tt>REMOTE_PARENT_DIR</tt> it makes sense to have multiple such directories. As a starting point, one <tt>REMOTE_PARENT_DIR_MDTxxxx</tt> for each remote MDT would be useful, but it may be necessary to have a tree of directories similar to the <tt>O/&lt;seq&gt;/dN</tt> object directories. Having a separate <tt>REMOTE_PARENT_DIR_MDTxxxx</tt> per MDT would also allow LFSCK to efficiently scan remote entries for a given MDT, if there was a problem (e.g. MDT was marked offline and returned into the namespace later).</p> LU-10329

DNE3: REMOTE_PARENT_DIR scalability

Improvement Critical Open Unresolved Lai Siyao Andreas Dilger LMR dne3 medium Tue, 5 Dec 2017 05:55:07 +0000 Thu, 10 Aug 2023 02:50:44 +0000 Lustre 2.16.0 0 12 <p>This is needed before <a href="https://jira.whamcloud.com/browse/LU-11025" title="DNE3: directory restripe" class="issue-link" data-issue-key="LU-11025"><del>LU-11025</del></a> or <a href="https://jira.whamcloud.com/browse/LU-10784" title="DNE3: mkdir() automatically create remote directory on MDS which has more space" class="issue-link" data-issue-key="LU-10784"><del>LU-10784</del></a> or implements automatic directory restriping or remote directories. Otherwise, the large number of remote entries in the filesystem will exceed the per-directory limit (unless <a href="https://jira.whamcloud.com/browse/LU-11546" title="enable large_dir support for MDTs" class="issue-link" data-issue-key="LU-11546"><del>LU-11546</del></a> is implemented first, and that is a sub-optimal solution).</p> <p>It seems that the large directory patches have landed, so the need for this is reduced.</p> <p>However, it would still be useful to have a per-MDT <tt>REMOTE_PARENT_DIR_MDTxxxx</tt> directory, since that would simplify things like LFSCK checking if one MDT was offline/corrupted/removed. It also provides a point of scaling so that there can be different updates between pairs of MDTs that do not contend for the same locks.</p> <p>In addition to improving performance by having multiple <tt>REMOTE_PARENT_DIR_MDTxxxx</tt> directories, another significant benefit would be reduced risk from filesystem corruption. An MDT with 60M entries in <tt>REMOTE_PARENT_DIR</tt> is currently running e2fsck because of a problem with that directory, with an unknown ETA, so having multiple independent directories would reduce risk and the number of entries that need to be repaired significantly, and potentially improve the scalability of parallel pass2/pass3 operations as well.</p> <p>For upgrading MDTs that are currently using a single <tt>REMOTE_PARENT_DIR</tt>, I think a two-stage approach could be used. Initially, add "read-only" support for such directories into 2.16.0 to allow for downgrade:</p> <ul class="alternate" type="square"> <li>accept <tt>OBD_INCOMPAT_REMOTE_PARENTS = 0x00000800</tt> in <tt>tgt_scd</tt>, but do not set it</li> <li>open <tt>REMOTE_PARENT_DIR_MDTxxxx</tt> directories when a remote MDT connection is made, using <tt>local_file_find()</tt>, if available</li> <li>do agent FID lookups in <tt>REMOTE_PARENT_DIR</tt> first, and fall back to <tt>REMOTE_PARENT_DIR_MDTxxxx</tt> if it exists and FID was not found</li> <li>create new entries in <tt>REMOTE_PARENT_DIR</tt> only (as before)</li> </ul> <p>Then in 2.17+ start using those directories (which would be incompatible for pre-2.16 servers, or servers not patched as above):</p> <ul class="alternate" type="square"> <li>set <tt>OBD_INCOMPAT_REMOTE_PARENTS = 0x00000800</tt> in <tt>tgt_scd</tt> so that unpatched MDS cannot mount the MDT</li> <li>create/open <tt>REMOTE_PARENT_DIR_MDTxxxx</tt> directories when a remote MDT connection is made, using <tt>local_file_find_or_create()</tt></li> <li>do agent FID lookups in <tt>REMOTE_PARENT_DIR_MDTxxxx</tt> first, and fall back to <tt>REMOTE_PARENT_DIR</tt> if it exists and FID was not found</li> <li>start linking new agent inodes into <tt>REMOTE_PARENT_DIR_MDTxxxx</tt> based on remote parent FID</li> <li>update LFSCK to do the same when linking or looking up remote FIDs on the MDT</li> <li>(optionally?) start a background thread to move entries from <tt>REMOTE_PARENT_DIR</tt> to the proper <tt>..._MDTxxxx</tt> subdir based on the parent FID, then remove the old <tt>REMOTE_PARENT_DIR</tt> when it is empty.</li> </ul> <p>Having a two-stage update process like this, and minimizing the "read-only" patches to simplify backport (e.g. to b2_15) will allow upgrade/downgrade without breaking the whole filesystem.</p> <p>One minor drawback of having separate directories would be that renaming files/subdirs from one remote MDT directory to another would also mean renaming the agent locally from one <tt>REMOTE_PARENT_DIR_MDTxxxx</tt> to another, but that is not worse than a local cross-directory rename, and only a small fraction of the overhead of the distributed rename itself (BFL, multiple MDT transactions, etc.). That local filesystem overhead is probably offset by not having a single huge <tt>REMOTE_PARENT_DIR</tt> to hold all of the remote links.</p> <p>REMOTE_PARENT_DIR is used in osd layer only, which means it doesn't know which MDT its parent is located, but only knows that whether parent is on local MDT. So we can't create a subdir for each MDT, do you think it's okay to create 64 subdirs for all remote MDTs?</p> <p>There are definitely benefits to splitting the entries up by remote MDT. This could be used to quickly find/check/remove remote links for a specific MDT if needed (eg. MDT removed/lost with metadata redundancy).</p> <p>Does it make sense to lift REMOTE_PARENT_DIR out of the OSD and into MDD?</p> <p>"Lai Siyao &lt;lai.siyao@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51910" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51910</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10329" title="DNE3: REMOTE_PARENT_DIR scalability" class="issue-link" data-issue-key="LU-10329">LU-10329</a> obdclass: tidy up linkEA code<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b27fd4006b7e763da72eb9d834fd4423293e5349</p> <p>"Lai Siyao &lt;lai.siyao@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51911" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51911</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10329" title="DNE3: REMOTE_PARENT_DIR scalability" class="issue-link" data-issue-key="LU-10329">LU-10329</a> obdclass: add linkEA version 2<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8ea62123db1a9bbbb2e36b511c03fbd368f80b75</p> <p>"Lai Siyao &lt;lai.siyao@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51912" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51912</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10329" title="DNE3: REMOTE_PARENT_DIR scalability" class="issue-link" data-issue-key="LU-10329">LU-10329</a> osd-ldiskfs: add osd_obj_map for remote objects<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ffbc774053bec88e1aacb16831b6f0fa632529d5</p> Gantt End to Start LU-11025 LU-10784 Related LU-12892 LU-11546 LU-10192 LU-4684 LU-15383 LU-4876 LU-11446 LU-11502 LU-15330 LU-12310 LU-15307 LU-13440 Development Epic Link EX-4481 Rank 1|hzzorb: Rank (Obsolete) 9223372036854775807