[LU-13417] DNE3: mkdir() automatically create remote directory on MDS which has more space

• Type: Improvement
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.15.0
• Affects Version/s: Lustre 2.13.0
• Labels: dne3
• Severity: 3
• Assignee: Lai Siyao
• Reporter: Andreas Dilger

Description

Since the patches from LU-11213 landed for 2.13.0, I thought "lfs setdirstripe -i -1 /mnt/lustre" on the root or other existing directory would allow creation of remote directories on other MDTs using plain "mkdir" commands. This is different than the case of "lfs setdirstripe -i -1 -c N /mnt/lustre" selecting stripes on less-full MDTs that was landed via patch https://review.whamcloud.com/35825 "LU-12624 lod: alloc dir stripes by QoS", but this patch also removed the "space" hash, so I thought that regular mkdir of a directory could be allowed to balance across MDTs?

However, I can't seem to get this to work. On current master (2.13.52-259, just before 2.13.53) I'm not able to use "lfs setdirstripe -i -1 /path/to/dir" on an existing directory. It seems to select the less-full MDT if I explicitly run "lfs mkdir -i -1" for a new directory, but that was also true in 2.12 using patch https://review.whamcloud.com/30598 "LU-10277 utils: 'lfs mkdir -i -1' pick the less full MDTs", so it isn't clear how to enable the LU-11213 functionality to balance directories across MDTs?

There should be a way for "mkdir(2)" from a normal application (not "lfs mkdir -i -1") to be able to create remote (1-stripe) directories in the filesystem, and it should be possible to set this by default on the root directory (per LU-11213). This is critical for being able to use multiple MDTs effectively without users knowing the details of how to configure striped/remote directories manually, or be forced to set all directories as striped (unwelcome due to performance overhead).

The default mdt_qos_threshold_rr value should be reduced significantly (e.g. 1-2% and/or modified so that some amount of MDT balancing is active when the filesystem is balanced, at least in the root directory by default. Otherwise, without users understanding the details of DNE MDT0000 will hold all of the inodes, when it would be better if the top 1 or 2 levels of directories should be distributed across MDTs.

Maybe this is mostly a documentation issue, and the "lfs-setdirstripe.1" man page needs to be updated to be more clear so I can understand what needs to be done to enable this? (also the usage message for setdirstripe/mkdir should remove the "This can only be done on MDT0 with the right of administrator" message.)

Issue Links

• is duplicated by: LU-10784 DNE3: mkdir() automatically create remote directory on MDS which has more space
• is related to: LU-17300 Avoid creating new dir/file/object on newly added MDT/OST
• is related to: LU-13439 DNE3: MDT QOS tuning to avoid full MDTs completely
• is related to: LU-14898 sanity test_413a: (max - min) * 100 / min: division by 0 (error token is "min")
• is related to: LU-14909 LU-13417 patch breaks few recovery tests
• is related to: LU-17334 Client should handle dir/file/object created on newly added MDT/OST
• is related to: LU-14792 DNE3: enable filesystem-wide default LMV
• is related to: LU-15856 "lfs setdirstripe -D ... <dir>" should create directory if missing
• is related to: LU-13440 DNE3: limit directory default layout inheritance
• is related to: LU-13560 'lfs mkdir -i N' should be 'sticky' on a particular MDT
• is related to: LU-15216 improve MDT QOS space balance
• is related to: LU-11025 DNE3: directory restripe
• is related to: LU-12624 DNE3: striped directory allocate stripes by QoS
• is related to: LU-11213 DNE3: remote mkdir() in ROOT/ by default
• is related to: LU-15850 MDT QOS should always be used for round-robin directories.