Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-13417

DNE3: "lfs setdirstripe -i -1" not active on new/empty filesystem

    Details

    • Type: Bug
    • Status: Reopened
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: Lustre 2.13.0
    • Fix Version/s: Lustre 2.14.0
    • Labels:
    • Severity:
      3
    • Rank (Obsolete):
      9223372036854775807

      Description

      Since the patches from LU-11213 landed for 2.13.0, I thought "lfs setdirstripe -i -1 /mnt/lustre" on the root or other existing directory would allow creation of remote directories on other MDTs using plain "mkdir" commands. This is different than the case of "lfs setdirstripe -i -1 -c N /mnt/lustre" selecting stripes on less-full MDTs that was landed via patch https://review.whamcloud.com/35825 "LU-12624 lod: alloc dir stripes by QoS", but this patch also removed the "space" hash, so I thought that regular mkdir of a directory could be allowed to balance across MDTs?

      However, I can't seem to get this to work. On current master (2.13.52-259, just before 2.13.53) I'm not able to use "lfs setdirstripe -i -1 /path/to/dir" on an existing directory. It seems to select the less-full MDT if I explicitly run "lfs mkdir -i -1" for a new directory, but that was also true in 2.12 using patch https://review.whamcloud.com/30598 "LU-10277 utils: 'lfs mkdir -i -1' pick the less full MDTs", so it isn't clear how to enable the LU-11213 functionality to balance directories across MDTs?

      There should be a way for "mkdir(2)" from a normal application (not "lfs mkdir -i -1") to be able to create remote (1-stripe) directories in the filesystem, and it should be possible to set this by default on the root directory (per LU-11213). This is critical for being able to use multiple MDTs effectively without users knowing the details of how to configure striped/remote directories manually, or be forced to set all directories as striped (unwelcome due to performance overhead).

      The default mdt_qos_threshold_rr value should be reduced significantly (e.g. 1-2% and/or modified so that some amount of MDT balancing is active when the filesystem is balanced, at least in the root directory by default. Otherwise, without users understanding the details of DNE MDT0000 will hold all of the inodes, when it would be better if the top 1 or 2 levels of directories should be distributed across MDTs.

      Maybe this is mostly a documentation issue, and the "lfs-setdirstripe.1" man page needs to be updated to be more clear so I can understand what needs to be done to enable this? (also the usage message for setdirstripe/mkdir should remove the "This can only be done on MDT0 with the right of administrator" message.)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hongchao.zhang Hongchao Zhang
                Reporter:
                adilger Andreas Dilger
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: