[LU-8235] Add pool awareness to lfs_migrate Created: 03/Jun/16 Updated: 22/Jun/20 Resolved: 09/Jan/18 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.7.0, Lustre 2.5.3 |
| Fix Version/s: | Lustre 2.11.0 |
| Type: | New Feature | Priority: | Minor |
| Reporter: | Eric Schnepp | Assignee: | Steve Guminski (Inactive) |
| Resolution: | Fixed | Votes: | 1 |
| Labels: | patch | ||
| Issue Links: |
|
||||||||||||||||||||||||
| Rank (Obsolete): | 9223372036854775807 | ||||||||||||||||||||||||
| Description |
|
The primary reason for this feature is to be able to allow users to specify where files are migrated to. The past focus of lfs_migrate seems to be to drain an OST and write objects to any and all other available targets. However, other use cases for lfs_migrate include rebalancing capacity, and moving data between tiers of storage with different performance characteristics. When new targets are added to a file system, it is beneficial to keep the allocation policy set such that active work will write new files that use ALL targets and retain performance; then old data can be identified with Robinhood or lfs find, and moved to the new targets with lfs_migrate. Adding pool awareness (with a "-p" command line option) would greatly facilitate this work, as all that would need to be done by the admin is to create a pool with the new targets in it. Note: additional (future) work to build on this and accomplish automatic capacity balancing would be to add another tool or two which: |
| Comments |
| Comment by Gerrit Updater [ 03/Jun/16 ] |
|
Nathan Dauchy (nathan.dauchy@nasa.gov) uploaded a new patch: http://review.whamcloud.com/20621 |
| Comment by Nathan Dauchy (Inactive) [ 03/Jun/16 ] |
|
A first draft patch is uploaded to gerrit, though it has only been tested lightly as a proof-of-concept. Probably there are things that could make it more efficient. Additional work is needed to try to remove the dependency on external tools such as egrep (though readlink may be OK since it is in the coreutils package). The additional "verbose" option from Note that NOAA is coordinating with DDN (DE28954) on this development. |
| Comment by Andreas Dilger [ 06/Jun/16 ] |
|
I'm not against this per se, but my previous thinking on improving lfs_migrate has been that we should allow passing (potentially all?) unknown options through from lfs_migrate to "lfs migrate", so that we don't need to special-case every layout option that we want lfs_migrate to support. That will be doubly true when many more options for file layouts appear in the future for Progressive File Layouts, File Level Redundancy, etc. The main differentiation is that lfs_migrate is intended to handle multiple files and checksums, and making things easier for users (e.g. 1.8/2.1 using rsync vs. 2.4+ using "lfs migrate", default values, etc.), while "lfs migrate" is for a single file. |
| Comment by Gerrit Updater [ 09/Jan/18 ] |
|
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/20621/ |
| Comment by Gerrit Updater [ 09/Jan/18 ] |
|
Minh Diep (minh.diep@intel.com) uploaded a new patch: https://review.whamcloud.com/30802 |