[LU-4684] DNE3: allow migrating DNE striped directory Created: 28/Feb/14 Updated: 16/May/22 Resolved: 01/Apr/22 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.6.0, Lustre 2.7.0, Lustre 2.8.0 |
| Fix Version/s: | Lustre 2.12.0, Lustre 2.14.0 |
| Type: | Improvement | Priority: | Major |
| Reporter: | Andreas Dilger | Assignee: | Lai Siyao |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | dne3 | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Rank (Obsolete): | 12872 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
The lfs migrate patch http://review.whamcloud.com/6662 will only allow migrating from a 1-stripe directory to another 1-stripe directory. That is unfortunate given that DNE2 is implementing striped directories. At a minimum this limitation needs to be documented for lfs migrate in the 2.6.0 release. I think that implementing migration from one striped directory to another striped directory (restripe) is not very much more complex. Essentially, walk the source directory(ies), hash the name, migrate it to the correct target directory stripe. For migration, the "right" directory will never be the source directory, but for restriping a single directory to multiple, some of the entries will already be in the right place (stripe 0) and others would need to move to stripe N based on the hash. If changing from M to N stripes or changing the hash function, 1/M entries would already be on the right MDT and wouldn't need to be moved. |
| Comments |
| Comment by Di Wang [ 14/Jul/15 ] |
|
I probably can not fix this in 2.8, so move it to 2.9? |
| Comment by Andreas Dilger [ 09/Sep/16 ] |
|
Assign to Lai for now, you can work out with Fan Yong if he has more time for this one (after the ZFS OI Scrub is done?) |
| Comment by Gerrit Updater [ 01/Apr/17 ] |
|
Lai Siyao (lai.siyao@intel.com) uploaded a new patch: https://review.whamcloud.com/26297 |
| Comment by Andreas Dilger [ 11/Apr/17 ] |
|
Directory migration should be implemented very similar to directory restriping, so that they can share the same code. Essentially, create a new layout for the directory that includes old and new shards (directory stripes), then iterate over all of the entries and move them to the shard that they belong to in the new layout. This ensures that if the (long and not atomic) operation is interrupted in the middle all of the entries can still be found in either the old or new location (which is the reason for LMV_HASH_FLAG_MIGRATION - so that the client will try all shards to find the entry) since the entries will always remain in the one directory layout even during movement. It would even allow migration to happen while the directory is being accessed, since we only need to lock one entry exclusively at a time. With directory restriping, only the existing name entries (not inodes) would be moved to the new shards, and newly-created entries would be created on the correct MDTs. When moving from a non-striped directory to a striped directory, the non-striped directory would become the master entry, and it would gain new shards to hold all of the name entries. When moving from one stripe count to another (or changing hash functions), then new shards would be added at the start of the migration operation, all entries are migrated according to the new hash function/modulus, then if the stripe count is reduced then the (verified-to-be-empty) old shard(s) would be removed. With directory migration the hash function would need to be enhanced to add new shard(s) at the end and store an "offset" or similar parameter in the LMV EA, or otherwise flag the old shards as "do not use", and restripe from the source shards to the target shards. All of these operations would essentially be handled by introducing a new hash function/flags that properly maps entries from one set of shards to another. Either redistributing entries evenly across shards in the restriping case, or leaving one or more shards empty in the migration case. We might consider optimizing the case of removing only a single shard (e.g. MDT removal) during migration, where the new shard is added at the end, all entries are moved from the old shard to the new one, and then the new shard replaces the old one in the layout. During this time, the hash function should keep names mapped to the other existing shards unchanged so that they do not all need to be migrated. Otherwise, all name entries in this directory will become remote and hurt access performance. The inodes themselves would also need to be moved during migration (not necessarily restriping, if we do this only when there are a relatively small number of entries in the directory, say < 32000), but inode migration is largely independent of how the directory layout causes name entries to be hashed. Essentially, the DNE code would be told whether only the name is being migrated, or if the whole inode needs to be migrated also. The name migration can happen in one step, then the inode migration can happen in a second step (either interleaved with name migration, or as a second step after all names have been moved. |
| Comment by Gerrit Updater [ 27/Oct/17 ] |
|
Lai Siyao (lai.siyao@intel.com) uploaded a new patch: https://review.whamcloud.com/29822 |
| Comment by Gerrit Updater [ 27/Feb/18 ] |
|
Lai Siyao (lai.siyao@intel.com) uploaded a new patch: https://review.whamcloud.com/31424 |
| Comment by Gerrit Updater [ 27/Feb/18 ] |
|
Lai Siyao (lai.siyao@intel.com) uploaded a new patch: https://review.whamcloud.com/31425 |
| Comment by Gerrit Updater [ 27/Feb/18 ] |
|
Lai Siyao (lai.siyao@intel.com) uploaded a new patch: https://review.whamcloud.com/31426 |
| Comment by Gerrit Updater [ 27/Feb/18 ] |
|
Lai Siyao (lai.siyao@intel.com) uploaded a new patch: https://review.whamcloud.com/31427 |
| Comment by Gerrit Updater [ 03/Mar/18 ] |
|
Lai Siyao (lai.siyao@intel.com) uploaded a new patch: https://review.whamcloud.com/31504 |
| Comment by Gerrit Updater [ 13/Mar/18 ] |
|
Lai Siyao (lai.siyao@intel.com) uploaded a new patch: https://review.whamcloud.com/31626 |
| Comment by Gerrit Updater [ 09/Apr/18 ] |
|
Lai Siyao (lai.siyao@intel.com) uploaded a new patch: https://review.whamcloud.com/31914 |
| Comment by Gerrit Updater [ 06/May/18 ] |
|
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/31914/ |
| Comment by Gerrit Updater [ 11/May/18 ] |
|
Lai Siyao (lai.siyao@intel.com) uploaded a new patch: https://review.whamcloud.com/32356 |
| Comment by Gerrit Updater [ 17/May/18 ] |
|
Lai Siyao (lai.siyao@intel.com) uploaded a new patch: https://review.whamcloud.com/32445 |
| Comment by Gerrit Updater [ 29/May/18 ] |
|
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/31425/ |
| Comment by Gerrit Updater [ 29/May/18 ] |
|
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/31426/ |
| Comment by Gerrit Updater [ 31/May/18 ] |
|
Lai Siyao (lai.siyao@intel.com) uploaded a new patch: https://review.whamcloud.com/32589 |
| Comment by Gerrit Updater [ 12/Jun/18 ] |
|
Lai Siyao (lai.siyao@intel.com) uploaded a new patch: https://review.whamcloud.com/32701 |
| Comment by Gerrit Updater [ 07/Aug/18 ] |
|
Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/32946 |
| Comment by Gerrit Updater [ 07/Aug/18 ] |
|
Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/32947 |
| Comment by Gerrit Updater [ 09/Aug/18 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/31424/ |
| Comment by Gerrit Updater [ 17/Sep/18 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/31427/ |
| Comment by Gerrit Updater [ 17/Sep/18 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/31626/ |
| Comment by Gerrit Updater [ 01/Oct/18 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/31504/ |
| Comment by Gerrit Updater [ 09/Oct/18 ] |
|
Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/33324 |
| Comment by Gerrit Updater [ 09/Oct/18 ] |
|
Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/33325 |
| Comment by Gerrit Updater [ 23/Oct/18 ] |
|
Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/33456 |
| Comment by Gerrit Updater [ 29/Oct/18 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/33324/ |
| Comment by Gerrit Updater [ 02/Nov/18 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/33456/ |
| Comment by Gerrit Updater [ 02/Nov/18 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/32946/ |
| Comment by Peter Jones [ 02/Nov/18 ] |
|
Landed for 2.12 |
| Comment by Andreas Dilger [ 27/Aug/19 ] |
|
This ticket is still listed as the reason racer.sh is skipping directory migration. Either directory migration is working and can be enabled in racer.sh and remove |
| Comment by Gerrit Updater [ 28/Jan/21 ] |
|
Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/41359 |
| Comment by Gerrit Updater [ 28/Jan/21 ] |
|
|
| Comment by Gerrit Updater [ 08/Jul/21 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/41359/ |