Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20831

mdt: parallel cross-MDT rename of regular files

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      A cross-directory rename of a regular file still takes the filesystem-wide rename lock (BFL) whenever the source parent or the source object is on a different MDT from the one serving the rename. The BFL is an exclusive lock on MDT0000's root, held for the whole distributed transaction, so all such renames in the filesystem run one at a time. Throughput is capped at 1 / hold time regardless of MDT or client count. With DNE3 spreading directory trees over many MDTs, most cross-directory renames in a tree are now cross-MDT.

      The BFL exists to prevent directory loops in the namespace. Renaming a non-directory cannot create a loop. Earlier work has already taken file renames out from under the BFL step by step:

      • LU-12125 - parallel rename of regular files within one directory
      • LU-17426 - parallel cross-directory rename of regular files on one MDT
      • LU-17427 - shorter BFL hold time (trylock after taking the child locks)

      This ticket finishes that for regular files: a cross-MDT file rename skips the BFL and uses the same FID-ordered parent locking that LU-17426 uses on a single MDT. It is controlled by a new tunable, mdt.*.enable_parallel_rename_remote (default on). The client's mode is only a hint, so the decision is made again once the source object is found, and falls back to the BFL if the object is a directory. Directory renames are unchanged.

      We have been using a formal model to analyse rename locking, and it shows that relaxing the BFL for regular files introduces no consistency gaps.

      This includes file renames between stripes of a striped directory, which LU-15913 moved back under the BFL.

      The patch complements LU-20808 (skip the subdirectory check for non-directory renames). With the BFL gone, that walk becomes a larger share of the remaining cost.

      Results on a 4-MDT VM cluster (renames/s, mean latency):

      case master LU-20808 LU-20808 + this
      cross-MDT, 8 tasks 142 (56 ms) 177 (45 ms) 866 (9.2 ms)
      10 tasks, 1000 files each moved once between directories spread over MDTs 124 (81 ms) 145 (69 ms) 895 (11 ms)

      In earlier runs of this patch on its own, cross-MDT renames kept scaling with the task count: 194 to 1852/s at 16 tasks. What remains per rename is the DNE distributed transaction itself, about 16-18 cross-MDT RPCs.

            wc-triage WC Triage
            paf0186 Patrick Farrell
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: