[LU-11025] DNE3: directory restripe

• Type: New Feature
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.14.0
• Affects Version/s: Lustre 2.11.0
• Labels: dne3

Epic/Theme

dne3

• Assignee: Lai Siyao
• Reporter: Lai Siyao

Description

In DNE system, MDTs are likely to become imbalanced over time, and user may also add/remove MDTs. So there is a need to move load from one MDT to anothe, this is what dir restripe can do.

Directory restripe should meet below requirements:

• After directory restripe, the load (disk usage, incoming requests) should be shared fairly between MDTs.
• In the process of directory restriping, the directory should be accessed without problem.
• Move as little data as necessary to make restripe quick and to minimize system impact.
• Keep object fid unchanged to support NFS on Lustre.

Functional spec:

• A new hash type will be introduced, directory with this hash type should move as little data as necessary in each restripe. To achieve this, directory with this hash type doesn't hash(name) mod stripe_count, but a fixed number like maximum MDT count, so that each restripe won't cause file hash change. And unlike old striped directory, each stripe contains a range of numbers of between [0, maximum_MDT_count], e.g, [0, 99], and each restripe will split this stripe into two, and each with half the range, i.e, [0, 49] and [50, 99], that means only half of the files needs to be moved.
• Directory restripe will be done automatically, when MDT finds a directory or stripe is growing fast, it will put this directory or stripe into a global list, and a dedicated thread will scan this list and split directories or stripes in this list. Stripe merge should be similar, when a stripe size is shrinking below a limit, it will be merged to its previous stripe.
• Unlike directory migration, restripe doesn't move file inode, but only dirent. So after restripe, half of the files under this stripe will become remote objects.

Issue Links

• has to be done after: LU-10329 DNE3: REMOTE_PARENT_DIR scalability
• is related to: LU-7749 DNE3: migrated orphan survive till next reboot
• is related to: LU-15692 performance regressions for files in stripe directory
• is related to: LU-15720 imbalanced file creation in 'crush' striped directory
• is related to: LU-13424 unable to migrate mirrored files
• is related to: LU-13406 set_param: param_path 'lod/*/mdt_hash': No such file or dir
• is related to: LU-8698 DNE3: check unknown hash type properly in lmv
• is related to: LU-13522 mdt.mdt_mds_mds_conns is not counted upon MDT stop and start
• is related to: LU-14459 DNE3: directory auto split during create
• is related to: LU-13691 Allow for lfs migrate between MDTs to include DOM
• is related to: LUDOC-462 Add feature documentation for directory restripe
• is related to: LU-13417 DNE3: mkdir() automatically create remote directory on MDS which has more space
• is related to: LU-12867 DNE3: new DNE2 hash function to handle rsync temporary filename
• is related to: LU-17307 osd_dirent_count() keeps multiple threads busy
• is related to: LU-13481 sanity test_33h: MDT index mismatch 5 times
• is related to: LU-7607 Preserve inode number after MDT migration
• is related to: LU-4684 DNE3: allow migrating DNE striped directory

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/38801/
Subject: LU-11025 mdt: remove unused code
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: f3fef81c7d7656160d6db6f542dd5178974fab78

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/37284/
Subject: LU-11025 dne: directory restripe and auto split
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a336d7c7c1cd62a5a5213835aa85b8eaa87b076a

[Gerrit Updater]

Lai Siyao ([e-mail]) uploaded a new patch: https://review.whamcloud.com/38801
Subject: LU-11025 mdt: remove unused code
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a35065a10025b6f41e2ae6f8336f1ab984b4e0f6

[Andreas Dilger]

Lai, I think it makes sense to move the FIDMAP patches over to their own LU ticket. I don't think that this functionality is critical for 2.14, and it will simplify their tracking. I don't think that DNE directory auto-split will move all of the inodes by default, only the directory entries, and FIDMAP is only needed in the case of inode migration (which should not be the default).

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/36898/
Subject: LU-11025 dne: support directory restripe
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 2e2b16c28bcf4048ba4f34129b7fb91c36b55a71

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/38097/
Subject: LU-11025 osd: osd_attr_get() returns dirent count
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 03a4431dac1c59fa2b98501fc7dfb8451a0a2af8

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/37282/
Subject: LU-11025 obdclass: add lu_device_operations::ldo_fid_alloc()
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 736d2d62ab1f00926000f0c3aa31fcb6aa53050f

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/38232/
Subject: LU-11025 uapi: add OBD_CONNECT2_FIDMAP
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c96fa612d5b0f3218642052c8ae1918883267c61

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/37281/
Subject: LU-11025 mdt: don't save remote lock if op not from client
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e496dbf7ed9eb64d347ac203873713531a3fbe59

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/36865/
Subject: LU-11025 lod: refactor lod_mdt_alloc_qos/rr()
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e284bfbe78d008ebfaae14878893bc7200338f83