[LU-11213] DNE3: remote mkdir() in ROOT/ by default

• Type: Improvement
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.13.0
• Labels: dne3
• Assignee: Lai Siyao
• Reporter: Andreas Dilger

Description

In order to start using multiple MDTs on a filesystem automatically, I wonder if it makes sense to start creating any directories under the filesystem ROOT/ directory remotely by default? This could be a client parameter that can be disabled if desired, but as can be seen in LU-11211 it is not very clear to users how to get multiple MDTs active by default. With OST object allocation we already do this by default for all objects.

While there is some overhead to creating remote directories, this top-level directories are best placed to allow maximal distribution across all of the MDTs. This would also work as new MDTs are added to the filesystem.

Using remote directories at the top level is a lot easier than trying to create a striped directory for ROOT/, since that would also need to change as new MDTs are added to the filesystem. It also allows more dynamic load balancing across MDTs.

As for implementation, the client tunable might be something like "llite.*.mdt_distribution_enabled" to turn it on/off, and "llite.*.mdt_distribution_seconds", which is how often the client will do obd_statfs() to the MDTs to get space/inode info, so that it doesn't need to do this for every create. Once per 5s would be the minimum, maybe as long as 60s?

The ROOT/ directory would get a flag like LUSTRE_TOPDIR_FL/LMAI_TOPDIR set when it is created that indicates the directory is a top-level directory and should have automatic remote directories created in it. This can be set manually by users on other directories with the "chattr +T" command, but it would not be inherited by subdirectories.

Using the same algorithms and tunables as LOV QOS for MDT selection would help leverage the same code, and allow us to improve the code together in the future.

Issue Links

• has to be finished together with: LUDOC-442 Documentation needed for LU-11213 DNE work
• is blocked by: LUDOC-439 Documentation update for DNE remote mkdir() in ROOT/ by default
• is related to: LU-12434 use filesystem default dir layout for new directories
• is related to: LU-9309 Add ldiskfs 64-bit inode number support
• is related to: LU-12624 DNE3: striped directory allocate stripes by QoS
• is related to: LU-13417 DNE3: mkdir() automatically create remote directory on MDS which has more space
• is related to: LU-9435 DNE2 - object placement QoS policy
• is related to: LU-10784 DNE3: mkdir() automatically create remote directory on MDS which has more space
• is related to: LU-11211 Performance degradation in mdtest
• is related to: LU-7827 DNE3: automatically select MDT for lfs mkdir

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35218/
Subject: LU-11213 lmv: use lu_tgt_descs to manage tgts
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 59fc1218fccf1a826182ff7cd52321e3efbb1eab

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/36008/
Subject: LU-11213 uapi: Remove unused CONNECT flag
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 11eba11fe0457b735194e5924e7bb1882a5b31b8

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/36008
Subject: LU-11213 uapi: Remove unused CONNECT flag
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e06df46d294b87e28071708f2f21e7bca26d0e8c

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35318/
Subject: LU-11213 uapi: change "space" hash type to hash flag
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c605ef1dbeb401bdff5ab3cfa1c407ea87a7b95d

[Gerrit Updater]

Lai Siyao ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35318
Subject: LU-11213 uapi: change "space" hash type to hash flag
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a8bb3df4d894340e58859270bfd516cc3aabd73c

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35207/
Subject: LU-11213 lod: default LMV can't be deleted
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 36243c75bd55c9e4d24f4e592fe03c92b56baedd

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35208/
Subject: LU-11213 doc: update lfs-setdirstripe man 'space' hash
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 8c914986bf415a535153f77e53e630707f8e1a1c

[Gerrit Updater]

Lai Siyao ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35219
Subject: LU-11213 lod: share object alloc QoS code with LMV
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2b42209a3abb44180938eafd6502e430bbba775b

[Gerrit Updater]

Lai Siyao ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35218
Subject: LU-11213 lmv: use lu_tgt_descs to manage tgts
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 57227c73a9f891ac6f44e3bf5eb578208c323040

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34657/
Subject: LU-11213 lmv: reuse object alloc QoS code from LOD
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b601eb35e97ae74d3de448e07a28ce41afb4adef

[Andreas Dilger]

I was thinking about how we might best use this feature in the field. It definitely makes sense to set it on the root directory (at some point by default, but not yet). However, one concern that I have is that (AFAIK) remote directories created by this feature will have fnv-1a hash type set on them (ie. they will be striped directories)? I don't think that is something that we want, but rather to make the new directories be unstriped.

It may also be useful to allow specifying an "inherit_depth=N", so that if set on the root directory it will create the next N subdirectories with space-hash (eg. inherit_depth=1 on root sets space-hash for the top-level user directories, but lower-level directories revert to plain dirs). Using this avoids the situation where one user has a lot of files on their initial MDT, but their subdirectories don't get distributed further across MDTs, but we don't want the overhead of remote directories at every level. If we want all directories to be remote, then set "inherit_depth=-1" (probably an 8-bit field would be enough, given PATH_MAX limits), and maybe special-case the "-1" value so it is never decremented. This would also be useful for striped directories I think.