[LU-4684] DNE3: allow migrating DNE striped directory

• Type: Improvement
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.12.0, Lustre 2.14.0
• Affects Version/s: Lustre 2.6.0, Lustre 2.7.0, Lustre 2.8.0
• Labels: dne3
• Assignee: Lai Siyao
• Reporter: Andreas Dilger

Description

The lfs migrate patch http://review.whamcloud.com/6662 will only allow migrating from a 1-stripe directory to another 1-stripe directory. That is unfortunate given that DNE2 is implementing striped directories. At a minimum this limitation needs to be documented for lfs migrate in the 2.6.0 release.

I think that implementing migration from one striped directory to another striped directory (restripe) is not very much more complex. Essentially, walk the source directory(ies), hash the name, migrate it to the correct target directory stripe. For migration, the "right" directory will never be the source directory, but for restriping a single directory to multiple, some of the entries will already be in the right place (stripe 0) and others would need to move to stripe N based on the hash. If changing from M to N stripes or changing the hash function, 1/M entries would already be on the right MDT and wouldn't need to be moved.

Issue Links

• is blocked by: LU-7357 Add layout lock for striped directories.
• is duplicated by: LU-11191 Striped DIR accross 2 MDTs very slow
• is duplicated by: LU-4029 DNE (distributed namespace) and legacy filesystems
• is related to: LU-4876 LFSCK remove entry from /REMOTE_PARENT_DIR if MDT-object name reside on the same MDT after migration
• is related to: LU-5550 LFSCK 5: Migrate the name entry from one MDT to another MDT if the name hash does not match the striped directory
• is related to: LU-11611 Incorrect return value in mdt_reint_unlink
• is related to: LU-6515 Always true check in mdd_migrate_create
• is related to: LU-11025 DNE3: directory restripe
• is related to: LU-10329 DNE3: REMOTE_PARENT_DIR scalability
• is related to: LUDOC-395 Add documentation for DNE striped directory migration
• is related to: LU-3531 DNE2: striped directory

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/41359/
Subject: LU-4684 tests: enable racer directory migration
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3070ca9b18206025d9fd55817bf4da1ec486b6be

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/41360
Subject: LU-4684 tests: enable racer directory migration
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 1c412393a42d454aa949fb0d87e788056d771c75

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/41359
Subject: LU-4684 tests: enable racer directory migration
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 51f4f706aa94538af7d1703a4e46b47579b0cad6

[Andreas Dilger]

This ticket is still listed as the reason racer.sh is skipping directory migration.

Either directory migration is working and can be enabled in racer.sh and remove LU-4684 from the comment, or a new ticket should be filed (and marked always_except) with the issue(s) and a patch submitted to change racer.sh to list that ticket as the reason it is disabled and the always_except label removed from this ticket.

[Peter Jones]

Landed for 2.12

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/32946/
Subject: LU-4684 llite: add lock for dir layout data
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: ae828cd3b092a38adbc86a4da320dd9d3a0fc80c

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33456/
Subject: LU-4684 lod: parse layout for migrating directory
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 83aa8ffcf2e8a5b7da9ead941ed0b22a435d6790

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33324/
Subject: LU-4684 migrate: replace PFID via source
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: bd596fbe449f4fbab18ed184ccce1e141928b116

[Gerrit Updater]

Lai Siyao ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33456
Subject: LU-4684 lod: parse layout for migrating directory
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 515953a6ece1deb01720e63145e07e62add5f553

[Gerrit Updater]

Lai Siyao ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33325
Subject: LU-4684 migrate: link parents lock may deadlock
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a410f20e682267e20e30eaab6aee299548322837