[LU-11213] DNE3: remote mkdir() in ROOT/ by default

• Type: Improvement
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.13.0
• Labels: dne3
• Assignee: Lai Siyao
• Reporter: Andreas Dilger

Description

In order to start using multiple MDTs on a filesystem automatically, I wonder if it makes sense to start creating any directories under the filesystem ROOT/ directory remotely by default? This could be a client parameter that can be disabled if desired, but as can be seen in LU-11211 it is not very clear to users how to get multiple MDTs active by default. With OST object allocation we already do this by default for all objects.

While there is some overhead to creating remote directories, this top-level directories are best placed to allow maximal distribution across all of the MDTs. This would also work as new MDTs are added to the filesystem.

Using remote directories at the top level is a lot easier than trying to create a striped directory for ROOT/, since that would also need to change as new MDTs are added to the filesystem. It also allows more dynamic load balancing across MDTs.

As for implementation, the client tunable might be something like "llite.*.mdt_distribution_enabled" to turn it on/off, and "llite.*.mdt_distribution_seconds", which is how often the client will do obd_statfs() to the MDTs to get space/inode info, so that it doesn't need to do this for every create. Once per 5s would be the minimum, maybe as long as 60s?

The ROOT/ directory would get a flag like LUSTRE_TOPDIR_FL/LMAI_TOPDIR set when it is created that indicates the directory is a top-level directory and should have automatic remote directories created in it. This can be set manually by users on other directories with the "chattr +T" command, but it would not be inherited by subdirectories.

Using the same algorithms and tunables as LOV QOS for MDT selection would help leverage the same code, and allow us to improve the code together in the future.

Issue Links

• has to be finished together with: LUDOC-442 Documentation needed for LU-11213 DNE work
• is blocked by: LUDOC-439 Documentation update for DNE remote mkdir() in ROOT/ by default
• is related to: LU-12434 use filesystem default dir layout for new directories
• is related to: LU-9309 Add ldiskfs 64-bit inode number support
• is related to: LU-12624 DNE3: striped directory allocate stripes by QoS
• is related to: LU-13417 DNE3: mkdir() automatically create remote directory on MDS which has more space
• is related to: LU-9435 DNE2 - object placement QoS policy
• is related to: LU-10784 DNE3: mkdir() automatically create remote directory on MDS which has more space
• is related to: LU-11211 Performance degradation in mdtest
• is related to: LU-7827 DNE3: automatically select MDT for lfs mkdir

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/46476
Subject: LU-11213 llite: Remove old compat code
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: f73bd88bf81f6ab92648716217b638cc793fbb53

[Peter Jones]

Landed for 2.13

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35219/
Subject: LU-11213 lod: share object alloc QoS code with LMV
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: d3090bb2b4860e997730e90426e11fc51ee27c0c

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35218/
Subject: LU-11213 lmv: use lu_tgt_descs to manage tgts
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 59fc1218fccf1a826182ff7cd52321e3efbb1eab

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/36008/
Subject: LU-11213 uapi: Remove unused CONNECT flag
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 11eba11fe0457b735194e5924e7bb1882a5b31b8

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/36008
Subject: LU-11213 uapi: Remove unused CONNECT flag
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e06df46d294b87e28071708f2f21e7bca26d0e8c

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35318/
Subject: LU-11213 uapi: change "space" hash type to hash flag
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c605ef1dbeb401bdff5ab3cfa1c407ea87a7b95d

[Gerrit Updater]

Lai Siyao ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35318
Subject: LU-11213 uapi: change "space" hash type to hash flag
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a8bb3df4d894340e58859270bfd516cc3aabd73c

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35207/
Subject: LU-11213 lod: default LMV can't be deleted
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 36243c75bd55c9e4d24f4e592fe03c92b56baedd

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35208/
Subject: LU-11213 doc: update lfs-setdirstripe man 'space' hash
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 8c914986bf415a535153f77e53e630707f8e1a1c