https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12125 Lustre <p>In theory it should be possible to relax renaming rules and allow parallel renaming of regular (or non-directory) files. </p> <ul> <li>the easiest and most common case would be rename of files within the same parent directory, like "<tt>mv foo foo.old</tt>" where we would only need to lock the DLM resources of the source and target name hashes</li> <li>the next case would be to allow rename of files between directories</li> </ul> <p>Probably the easiest is to order such locks by DLM resource (FID+hash) to avoid deadlocks between multiple threads doing renames. For directory renames, they still need to be serialized by the BFL and also take the name hash locks in the source and target dirs to avoid conflict with regular file renames. In theory it would also be possible to allow concurrent rename of directories within the same parent, but that could be done in a separate step. </p> <p>For renames across shards within a single DNE striped directory they would be treated as renames across directories. </p> LU-12125

Allow parallel rename of regular files

Improvement Minor Resolved Fixed Andreas Dilger Andreas Dilger Fri, 28 Aug 2015 07:51:25 +0000 Tue, 16 Jan 2024 00:50:35 +0000 Wed, 21 Apr 2021 04:04:17 +0000 Lustre 2.15.0 0 9 <p>Di, Lai, what do you think of this?</p> <p>How hard would this be to implement? A few lines of code to fix up the locking for regular files? A bunch of code cleanup required first? A major feature to rewrite large parts of the MDS code?</p> <p>In some sense the rename of a regular file could be treated the same as the creation of a file in the target dir and deletion of the file from the source dir, so pdirop locking within the directory would be enough, unlike rename of directories which need full tree locks to avoid creating loops or orphan trees. </p> <p>Yes, file rename under the same parent should be easy to implement, (only need fix mdt_rename_lock and order locks of source and target). Hmm, even rename files between directories might be easy to fix, probably we only need consider to order locks of the source and target parents, though with current implementation, we need check the parent/child relationship to order the locks, which will probably still need BFL lock unless we can order the locks by FID + hash, but then we will need change the order of locks everywhere (i.e. from parent-child to FID+hash order), which probably need some time to implement.</p> <p>As you said, directories rename might not be easy, unless we have subtree lock or similar thing.</p> <p>There are still a few simply fixes we can do</p> <ul> <li>rename lock can be released early in some cases, maybe after we get LDLM locks, though I am not sure if we can do that for directories rename.</li> <li>For DNE, there can be BFL locks on each MDT, we probably only needs to enqueue BFL locks on MDTs where source and target parent locate.</li> </ul> <p>This might also need the patch <a href="https://review.whamcloud.com/14375" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/14375</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-6864" title="DNE3: Support multiple modify RPCs in flight for MDT-MDT connection" class="issue-link" data-issue-key="LU-6864"><del>LU-6864</del></a> osp: manage number of modify RPCs in flight</tt>" to allow multiple OUT RPCs in flight at one time when renaming between shards of striped directories or across MDTs.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41184" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41184</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12125" title="Allow parallel rename of regular files" class="issue-link" data-issue-key="LU-12125"><del>LU-12125</del></a> llite: send file mode with rename RPC<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d7eb3ea257299e5942a8aac77d98ddb6132f2014</p> <p>Lai, do you have a chance to look into this to see how hard it would be to handle same-dir renames (for striped directories only if source and target stripe are the same parent FID) with just a lock on the parent FID? Given that we don't really support interop between different MDS versions, there isn't much concern for protocol interoperability.</p> <p>We <em>might</em> consider an <tt>OBD_CONNECT2_RENAME</tt> flag to ensure that all of the MDTs have support for this, but even that may not be necessary. At worst, if any of the other MDTs don't support same-dir renames they will still request the <tt>LUSTRE_BFL_FID</tt> lock from MDT0000 before getting the parent FID locks, which is sub-optimal but is not problematic. Since the same-dir rename will also lock the parent FID, if other MDSes allow this but MDT0000 does not, then at worst the same-dir rename will only be blocked on the directory lock and not <tt>LUSTRE_BFL_FID</tt>, but this does not affect correctness since there is no lock ordering problem when only renaming regular files.</p> <p>It may be that all is needed is to skip the call to <tt>mdt_rename_lock()</tt> call if <tt>msrcdir == mtgtdir</tt> and the renamed object is a regular file. If the client sends the file type with the rename RPC then this is relatively easily done. Unfortunately, that is not the case today:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">static</span> <span class="code-object">int</span> ll_rename(struct inode *src, struct dentry *src_dchild, struct inode *tgt, struct dentry *tgt_dchild) { : op_data = ll_prep_md_op_data(NULL, src, tgt, NULL, 0, mode=0, LUSTRE_OPC_ANY, NULL); struct md_op_data *ll_prep_md_op_data(struct md_op_data *op_data, struct inode *i1, struct inode *i2, <span class="code-keyword">const</span> <span class="code-object">char</span> *name, size_t namelen, __u32 mode, <span class="code-keyword">enum</span> md_op_code opc, void *data) { : op_data-&gt;op_mode = mode; </pre> </div></div> <p>I've pushed a simple patch to send the file mode along with the rename RPC so that the locking decision can be made easily on the MDS. Hopefully this can be included into 2.14 before it is released.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41186" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41186</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12125" title="Allow parallel rename of regular files" class="issue-link" data-issue-key="LU-12125"><del>LU-12125</del></a> mds: allow parallel regular file rename<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bc1e773b4364dde14df17645dd4d27b1143d1ccf</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41229" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41229</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12125" title="Allow parallel rename of regular files" class="issue-link" data-issue-key="LU-12125"><del>LU-12125</del></a> mds: allow parallel directory rename<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8ba0a64c75dbc69bf9de774b67a7e49419f3ba7d</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41230" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41230</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12125" title="Allow parallel rename of regular files" class="issue-link" data-issue-key="LU-12125"><del>LU-12125</del></a> mds: allow parallel directory rename<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8f50962fb02463d620163f9444a430ceca693368</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41231" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41231</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12125" title="Allow parallel rename of regular files" class="issue-link" data-issue-key="LU-12125"><del>LU-12125</del></a> tests: allow racer to specify extra tasks<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4a12916a0375d97fa67bdcdcf860fcf9b88f4192</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/41184/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41184/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12125" title="Allow parallel rename of regular files" class="issue-link" data-issue-key="LU-12125"><del>LU-12125</del></a> llite: send file mode with rename RPC<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 47576dc68cd90068cd5cbd5c889f04531fe6e682</p> <p>Further improvements in this area beyond the patches already pushed for same-dir rename operations:</p> <ul class="alternate" type="square"> <li>instead of locking whole parent directory, just lock parent FID+source/target hash to allow parallel file (or directory) rename within a single directory</li> <li>allow rename between shards of a striped directory, given that we know source and target shards are not child/ancestor of each other (though there would still need to ordering for locking of individual shards)</li> <li>allow regular file rename between different source/target directories, which will need to get a fixed locking order to avoid deadlocks. That code already exists, so may be "only" a matter of dropping the BFL once the source/target are locked. With only regular file rename, there is no strong requirement to know the parent/child relationship of the parent directories, but the consistent locking order is still required to avoid AB/BA deadlocks with other concurrent operations.</li> </ul> <p>I think the benefit of allowing arbitrary parallel directory rename is pretty small, and probably not worthwhile pursuing in the context of this ticket.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/41186/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41186/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12125" title="Allow parallel rename of regular files" class="issue-link" data-issue-key="LU-12125"><del>LU-12125</del></a> mds: allow parallel regular file rename<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d76cc65d5d68ed3e04bfbd9b7527f64ab0ee0ca7</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/41230/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41230/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12125" title="Allow parallel rename of regular files" class="issue-link" data-issue-key="LU-12125"><del>LU-12125</del></a> mds: allow parallel directory rename<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 90979ab390a72a084f6a77cf7fdc29a4329adb41</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/41231/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41231/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12125" title="Allow parallel rename of regular files" class="issue-link" data-issue-key="LU-12125"><del>LU-12125</del></a> tests: allow racer to specify extra tasks<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: fe2663f18e50023ad5cfe5e07b695378dd27a68e</p> <p>Looks like everything has landed for 2.15</p> Related LU-12037 LU-6864 LU-15913 LU-15285 LU-17426 Development Rank 1|hzxlk7: Rank (Obsolete) 9223372036854775807