-
Improvement
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
A cross-directory rename of a regular file still takes the filesystem-wide rename lock (BFL) whenever the source parent or the source object is on a different MDT from the one serving the rename. The BFL is an exclusive lock on MDT0000's root, held for the whole distributed transaction, so all such renames in the filesystem run one at a time. Throughput is capped at 1 / hold time regardless of MDT or client count. With DNE3 spreading directory trees over many MDTs, most cross-directory renames in a tree are now cross-MDT.
The BFL exists to prevent directory loops in the namespace. Renaming a non-directory cannot create a loop. Earlier work has already taken file renames out from under the BFL step by step:
LU-12125- parallel rename of regular files within one directoryLU-17426- parallel cross-directory rename of regular files on one MDTLU-17427- shorter BFL hold time (trylock after taking the child locks)
This ticket finishes that for regular files: a cross-MDT file rename skips the BFL and uses the same FID-ordered parent locking that LU-17426 uses on a single MDT. It is controlled by a new tunable, mdt.*.enable_parallel_rename_remote (default on). The client's mode is only a hint, so the decision is made again once the source object is found, and falls back to the BFL if the object is a directory. Directory renames are unchanged.
We have been using a formal model to analyse rename locking, and it shows that relaxing the BFL for regular files introduces no consistency gaps.
This includes file renames between stripes of a striped directory, which LU-15913 moved back under the BFL.
The patch complements LU-20808 (skip the subdirectory check for non-directory renames). With the BFL gone, that walk becomes a larger share of the remaining cost.
Results on a 4-MDT VM cluster (renames/s, mean latency):
| case | master | LU-20808 | LU-20808 + this |
|---|---|---|---|
| cross-MDT, 8 tasks | 142 (56 ms) | 177 (45 ms) | 866 (9.2 ms) |
| 10 tasks, 1000 files each moved once between directories spread over MDTs | 124 (81 ms) | 145 (69 ms) | 895 (11 ms) |
In earlier runs of this patch on its own, cross-MDT renames kept scaling with the task count: 194 to 1852/s at 16 tasks. What remains per rename is the DNE distributed transaction itself, about 16-18 cross-MDT RPCs.
- is related to
-
LU-18843 allow parallel rename inside a single directory
-
- Resolved
-
- is related to
-
LU-15913 rename stress test leads to REMOTE_PARENT_DIR corruption
-
- Resolved
-
-
LU-20808 Reduce DNE Cross-MDT RPCs for DNE operation
-
- Open
-
-
LU-12125 Allow parallel rename of regular files
-
- Resolved
-
-
LU-17426 parallel cross-directory rename of regular files on single MDT
-
- Resolved
-
-
LU-17427 reduce hold time for BFL rename lock
-
- Resolved
-