[LU-6081] hsm: add file migrate support

• Status: Open
• Project: Lustre
• Type: Improvement
• Priority: Minor
• Reporter: Frank Zago (Inactive)
• Assignee: Ben Evans (Inactive)
• Resolution: Unresolved
• Labels: HSM, cea, patch

Description

Currently file migration is done by "lfs migrate". This has a couple problems:

• It doesn't scale well to large files and/or large number of files to copy.
• If an external tools (such as Robinhood) wants to migrate a file, it has to spawn lfs to do the job, for every file, which has a big overhead.

A solution to this issue is to move the copy operation into the copytool. The copytool already knows how to copy files, and Lustre has control over these files. This imply to re-use and extend the infrastructure created for HSM, and would solve the two issues mentioned above:

• Migrating a file, from an application point of view, becomes more or less the same as archiving a file.
• Since the file copy is done by the copytool, this copy can be optimized in one place.

Issue Links

• is blocked by: LU-7507 the data version doesn't always chang...
• is blocked by: LU-6246 Add a JSON decoder
• is duplicated by: LU-6242 some version of gcc fails to compile ...
• is related to: LU-12513 Address YAML limitations
• is related to: LU-6234 lfs computes pool name length incorre...
• is related to: LU-7037 sanity-hsm test_57:gethostbyname("sha...
• is related to: LU-6463 Interop 2.5.3<->master ost-pools test...
• is related to: LU-10606 HSM info as part of LOV layout xattr
• is related to: LU-12890 Use HSM to mirror sync files
• is related to: LU-9897 Tracking of lustre library improvements.
• is related to: LU-11621 Add copy_file_range() API and use it ...
• is related to: LU-10968 add coordinator bypass upcalls for HS...

Comments

[Frank Zago (Inactive)]

The proposed changes to Lustre are:

Issuing the command

User API

struct hsm_request is expanded to support migration. hr_action will be a new value: HUA_MIGRATE. hr_archive_id will be unused. hr_flags will take a new flag (HSM_MIGRATION_BLOCKS, with the meaning of MIGRATION_BLOCKS in current lfs implementation). The command parameters themselves, contained in a struct llapi_stripe_param are appended at the end of the structure, and hr_data_len will be set appropriately. hr_itemcount will be 1.

The request is then sent to the kernel will llapi_hsm_request, which won't need to change.

In a future expansion, it should be possible (desired?) to allow more than one request per call (i.e. hr_itemcount > 1). This will be possible when the first parameter of llapi_hsm_request() changes to an opaque filesystem handle. The kernel code should be prepared for a value != 1, but not necessarily to handle it.

lfs

lfs will have a new command hsm_migrate, in addition to migrate. The command line parameters should be identical for both commands. hsm_migrate will use the HSM infrastructure instead of doing the copy directly.

Inside Lustre

Once the command reaches the Lustre client, some basic checking should be done: hr_itemcount==1? Checking that the file is indeed a regular should not be performed at this stage to avoid querying the server, unless the information is already known.

The request is then forwarded to the MDT.

The MDT does extra checks, such as ensuring the file is a regular file, and that the migration command makes sense. For instance no migration should happen if the migrated file would end up in the same place (success should be returned in that case).

If a migrate command is already underway for the file, or possibly queued, the new one should be rejected with an error.

The HUA_MIGRATE command will be similar to the others; it will timeout eventually, it can be in error, ...

The command is then enqueued on the HSM queue, where an agent will pull it. Same as with other commands; no change expected here.

Performing the command by the copytool

The copytool will receive the new command. When it is ready to start, it will call llapi_hsm_action_begin(). The parameters should be similar that those of a restoration since a new temporay volatile file is to be created.

The data migration itself should be done with a faily straightforward copy of the code from lfs.c' lfs_migrate(). Or ct_copy_data() could be used, with the addition of the checking of the lease if requested.

Once the migration is done, llapi_hsm_action_end() is called, returning success or an error code to the MDT through the usual channel.

The copytool will also regularly report to the MDT through the llapi_hsm_action_progress() function, the same as an archiving or restoration operation.

These modification will have to be done to the Posix copytool and the TAS Connector.

During the migration, on the MDT

The migration really happens between llapi_hsm_action_begin() / llapi_hsm_action_end(). The file should have a new flag HS_MIGRATING returned by lfs hsm_state indicating a migration is under way. It should be set by llapi_hsm_action_begin(), and reset when either the command finishes with llapi_hsm_action_end(), when the command is canceled, or when the command times out.

A user should be able to cancel a command using lfs hsm_cancel.

If a migrate command is already underway, the new request should be rejected with an error.

Reporting to the application

The usual reporting mechanism will be used:

• lfs hsm_action
• a LAYOUT changelog event
• The JSON FIFO

[Gerrit Updater]

frank zago ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13241
Subject: LU-6081 lfs: check that pool name is not too long
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e77743c49cdf428ee1d4ced75dfe0fc3a7dd8b5e

[Gerrit Updater]

frank zago ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13242
Subject: LU-6081 user: Introducing llapi_create_volatile_idx2
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a7f36b214348947cefa1a0f42cb8d6f024d023da

[Gerrit Updater]

frank zago ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13243
Subject: LU-6081 hsm: adding migration support
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5f4e8e45c0954bbb12c39eb01f9ae674fa5c57dd

[Gerrit Updater]

frank zago ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13244
Subject: LU-6081 lfs: added room for mdt-index option to lfs migrate
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2bd5256b7fcb92050433330bd7ffd2961101bbce

[Gerrit Updater]

frank zago ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13245
Subject: LU-6081 user: replace duplicated random_group_id
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 39623e302e63a95e2ece26c188074d3ac0bb8e00

[Frank Zago (Inactive)]

These patches implement a skeleton of the proposal.

The work left to do is:

• Not enough tests. There is only two tests (113a and 113b in sanity-hsm.sh). We should test various combinations of migrations (several OSTS, pools, ...). Test all the possible parameters of lfs hsm_migrate. Write a test that doesn't use lfs, but uses the API directly (see existing HSM test on gerrit: http://review.whamcloud.com/#/c/12836/).
• The HSM_MIGRATION_BLOCKS may need to be moved into hsm_migrate_param. Currently lfs can only migrate a file at a time, so that's not an issue. However we might want to have a single call to llapi_hsm_request to migrate more than one file, with some blocking and some not.
• A couple TODO left, but they don't impact the functionality.

Bugs shared with regular migrate:

• lfs migrate and hsm_migrate still copy files when all striping parameters are the same. This may however be a policy decision.
• 'lfs (hsm_)migrate file' with no arguments except the file name still causes a copy operation to occur (should be invalid to do 'migrate' with no args)
• No man page for either lfs migrate and hsm_migrate. The usage is the same.

[Andreas Dilger]

I'm quite in favour of this feature, since I've long thought that intra-filesystem migration via the policy engine in a great way to implement tiered/heterogeneous storage within Lustre. Some comments on your proposal:

Bugs shared with regular migrate:

• lfs migrate and lfs hsm_migrate still copy files when all striping parameters are the same.

That is intentional, to allow the MDS to space rebalance and defragment files as it sees fit. This uses the same stripe count as the original file, but moves it to (usually) new OSTs. Because the migration is single-threaded, this will typically result in better file allocation.

The other question is whether there should be a separate hsm_migrate sub-command, or just an option (-H, or maybe that should be implicit depending on whether HSM is active?) to the migrate sub-command to use a different migration mechanism? The less complex for users, the better.

What is still missing for tiered storage to work well is:

• being able to specify a default filesystem-wide OST pool for new files
• maybe pool quotas (similar, but different from project quotas) to restrict space usage on OSTs

[Aurelien Degremont (Inactive)]

I'm also supporting this feature. I think this will be a great improvement for Lustre. (Defaut pool will be the next one)

hr_archive_id will be unused

These modification will have to be done to the Posix copytool and the TAS Connector.

I'm not sure this should be done this way. This means all copytool should be able to handle internal migration requests. Even if this is easy for the POSIX copytool, some very HSM-specific should only care about migration with the HSM they've been designed for. I think internal migration should have a dedicated Archive ID.

lfs will have a new command hsm_migrate, in addition to migrate.

From a code point of view, this makes sense, but from a user point of view, it does not. Regular users will not really understand why, to migrate file inside Lustre they have to deal with HSM...
I support Andreas' comment about this. Have something automatic if HSM is active, or a simple flag, another keyword, but using 'hsm' here will be misleading for users.

[Frank Zago (Inactive)]

I don't think lfs can automatically select HSM to do the work; there's no way to know the existing copytool supports the migration option. So we'll add a -H option to select HSM migration.

We'll add support for the archive_id (-a), which will be valid only if -H is selected.

[Gerrit Updater]

frank zago ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13277
Subject: LU-6081 user: use random() instead of /dev/urandom
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2db3a16b3dd3c2f7499f20afd9ca394bb20313f9

[Gerrit Updater]

frank zago ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13292
Subject: LU-6081 lfs: fixed bad return value
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e7707492a4829e5c8f894bb1617e54cb8834458c

[Gerrit Updater]

frank zago ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13293
Subject: LU-6081 lfs: fixed bad return value
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e3be5d48bccb463111c056c18c0c289ab73a4195

[Gerrit Updater]

frank zago ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13317
Subject: LU-6081 lfs: split setstripe and migrate help
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8662fbe81ffa21f9453dd92d35ee829a11290c75

[Frank Zago (Inactive)]

The latest patches should have addressed all the comments.

Migration is supported with lfs migrate -H. The archive_id option was added. The help for migrate was added to the lfs man page, with all the new options. A couple more tests were added.

The remaining design question is:

• The HSM_MIGRATION_BLOCKS may need to be moved into hsm_migrate_param. Currently lfs can only migrate a file at a time, so that's not an issue. However we might want to have a single call to llapi_hsm_request to migrate more than one file, with some blocking and some not.

[Frank Zago (Inactive)]

Note that this series will need to be adjusted when LU-4840 goes upstream, and should not go in before that happens.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13241/
Subject: LU-6081 lfs: check that pool name is not too long
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 6ec0b44ae82638b1a47b8538496936b0f53b4e49

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13293/
Subject: LU-6081 lfs: fixed bad return value
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 435df8de136a679f250f2be1e12d6c08b7a0b248

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13277/
Subject: LU-6081 user: use random() instead of /dev/urandom
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 31e7f22cdc7c77ac7b48ba6a258ca9611063320a

[Frank Zago (Inactive)]

The HSM payload currently contains a binary structure to transfer the stripe information between lfs and the copytool. Is there any objection to using JSON instead? This has no impact on the kernel code because the MDT just relays whatever the payload is.

Instead of having

struct hsm_migrate_param {
       char                    lsp_pool[LOV_MAXPOOLNAME+1];
       __u64                   lsp_stripe_size;
       __s32                   mdt_index;
       __s32                   lsp_stripe_offset;
       __s32                   lsp_stripe_pattern;
       __u32                   lsp_stripe_count;
       __u32                   lsp_osts_count;
       __u32                   lsp_osts[0];
};

we would have something like

{
    "lsp_pool": "poolname",
    "lsp_stripe_size": 1024,
    "mdt_index": -1,
....
    "lsp_osts": [
        0,
        1,
        2,
        3,
        4,
        5
    ]
}

This will allow to add extra fields later if needed, or proprietary fields.

[Robert Read (Inactive)]

No objections from me, JSON seems like a good choice. Is there a maximum size for data payload?

[Gerrit Updater]

frank zago ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13654
Subject: LU-6081 lib: don't ignore the return of read()
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 988d95e991b6d66867d0e061563237a91c711d5d

[Frank Zago (Inactive)]

After testing, the payload size is 1649 bytes. That's not enough for large filesystems, with potentially thousands of OSTs.

		/* Final size will be more than double totalsize */
		if (totalsize >= MDS_MAXREQSIZE / 3)
			RETURN(-E2BIG);

[Robert Read (Inactive)]

In theory that's what pools are for, but, yeah, this is not great.

[Frank Zago (Inactive)]

With the following format:

{"lsp":{"pool":"poolname","stripe_size":1024,"stripe_offset":0,"stripe_pattern":1,"stripe_count":3,"osts":[0,1,2,3,4]},"mdt_index": -1}

it is possible to encode 400 OSTs up to 3 numbers, or 300 OSTs up to 4 numbers, or 250 OSTs up to 5 numbers.

Would that be a reasonable limitation?

With the binary format in current patch, it is possible to encode 401 OSTs (if each OST is encoded by 32 bits) or 802 OSTS (if each OST is encoded by 16 bits).

Also, there is no JSON decoder available in Lustre. I've used this implementation (http://ccodearchive.net/info/json.html) in the past, and I could integrate it into Lustre (replacing the existing partial implementation). It's license is GPL/LGPL compatible.

[Michael MacDonald (Inactive)]

FWIW, I looked at Jansson a while ago. It's MIT-licensed, seems mature, and seems to be actively maintained. I would have preferred to use it when I added the JSON event emitter to liblustreapi_hsm, but at the time I was concerned that it would increase friction and slow down acceptance of my patch.

[Frank Zago (Inactive)]

Thanks Michael. I compared several free JSON C libraries a year and a half ago, and decided against jansson. While the doc is nice and the API is complete, it was about 3 times slower in decoding a file than the CCAN implementation. I don't recall the exact numbers, but it was something like 30000/s vs 100000/s for a 200 or 300 bytes JSON string.

[Andreas Dilger]

What about the libyaml parser that was added for DLC (as an external dependency rather than a wholesale inclusion, which people didn't like). I'd rather not have multiple parsers or dependencies for Lustre+LNet.

[Frank Zago (Inactive)]

The cYAML parser is very limited. It only supports a limited subset of YAML (see for instance the hack done for LU-6099) and doesn't handle json at all. It's already 1100 lines, not counting the external dependency on libyaml, which is also not present on SLES. Adding JSON support to it would push it above the 1500 lines that the json lexer/parser takes.

[Christopher Morrone]

I don't think that Andreas was talking about cyaml. cyaml isn't a parser, it is just some kind of wrapper around libyaml. I think you are completely free to avoid cyaml and use libyaml directly.

[Frank Zago (Inactive)]

libyaml is just a tokenizer. It doesn't build a data structure for you, which is why cYAML was added (I think). I tried to expand it for a while, but I started running into trouble when adding nested arrays.

[Gerrit Updater]

frank zago ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13723
Subject: LU-6081 lib: add a full fledge JSON parser to liblustreapi
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 907e5558869296ca04e54857763b8c3142b243fa

[Gerrit Updater]

frank zago ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13724
Subject: LU-6081 lib: integrate CCAN JSON parser
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 689a1d074056ca4eddf905eec61c4b5bbf4f84cd

[Frank Zago (Inactive)]

The above 2 patches add the CCAN JSON parser. The first one is just the code copied as-is, and the second integrates it into Lustre, with some cleanups. Both could be merged if history doesn't matter.

I left out the small test suite because it uses the CCAN test package. If this JSON implementation is acceptable, I can integrate the tests with some changes.

The posix copytool has been changed to use that API.

[Christopher Morrone]

A static data structure representation is not required for something to be a parser. libyaml is an event based parser. Depending on the data that interests you, the intermediate step that cYAML introduces could be wasteful. It could be better to convert directly to the data structures that work for your application.

[Frank Zago (Inactive)]

libyaml doesn't guarantee that the input was yaml. So that is something that a user has to do. Which means build a state machine, debug it, test it, ... That's what's been done with cYAML. Using libyaml directly to parse the structures above would take way too many lines of code I think, and wouldn't be reusable.

[Andreas Dilger]

Is it possible to replace liblustreapi_json.c with the CCAN JSON code? Is it possible to replace the DLC libyaml usage with CCAN JSON? What I don't want is 3 different implementations of nearly the same functionality.

[Michael MacDonald (Inactive)]

My $0.02, for whatever it's worth: I think that both JSON and YAML could have their places in Lustre. As a data exchange format, YAML is kind of cumbersome. Not nearly as bad as XML, of course, but JSON is the de facto standard for data exchange in modern software, and many higher-level languages have good support for it right in their standard libraries (e.g. Go, Javascript, etc.), or easily-added external libraries. As a data representation format, on the other hand, YAML is quite nice in that it's easily comprehensible by humans and computers.

I do understand the desire to keep things simple and just choose one format to rule them all, but I think a well thought out set of conventions should help us arrive at nearly the same place.

My proposal:
/proc entries are YAML
configuration is YAML
data exchange is JSON (structured events, stripe info, etc.)

All that having been said, there don't appear to be any YAML 1.2-compliant C libraries. Version 1.2 of the spec purports to bring YAML into alignment as a proper superset of JSON, such that a compliant library should be able to parse both YAML and JSON.

I agree that replacing liblustreapi_json.c with CCAN JSON makes sense. I do think it would be good to retain the unit tests for that CCAN JSON and integrate them into the regular Lustre test suite, though.

I don't agree that changing DLC over to JSON necessarily makes sense, if the goal is to retain the ability for humans to easily write the configuration files. There are newer /proc entries that are YAML-formatted as well. To my mind, these seem like good places to keep YAML's human-friendly formatting. On the other hand, if there is a shift towards configuration by tools, then maybe JSON is the best format going forward.

Maybe there will eventually be a good YAML 1.2 library that can replace libyaml and CCAN JSON (having a solid set of lustre-specific tests in place to ensure it works the same would be critical), but until then it doesn't seem like a huge deal to have them both.

[Frank Zago (Inactive)]

I agree with Michael. JSON is perfect for data exchange and YAML for user config files. Both can live in parallel.

I will update the patch to replace liblustreapi_json.c and integrate the testsuite. I think the JSON API in Lustre should become private too, for it is not Lustre's job to provide a JSON API. May be it should be available as a static library for the Lustre tools that need it, and that's it.

[Andreas Dilger]

And what is the general consensus of including CCAN JSON code directly into Lustre? When we proposed to do the same thing with libyaml to avoid external dependencies, this was loudly rejected, yet I don't know what the availability of CCAN JSON as an external library dependency is.

If the decision is for CCAN JSON to be included, it should definitely be in the form of an initial import as an unmodified library, with a clear indication of the source version (e.g. git commit hash and version number) in a README.CCAN_JSON file (or similar) in the directory, and then one or more additional patches to integrate it into Lustre as needed. That would allow importing an updated version of the library in the future and re-applying the patches, or otherwise generating a list of updates from the original source library to include later updates.

We'll also need to make sure the licensing is OK.

[Andreas Dilger]

It probably makes sense to move this discussion to a separate bug, since the HSM migrate support patches are already landed to master for 2.7 and this is a feature beyond the scope of 2.7.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13654/
Subject: LU-6081 lib: don't ignore the return of read()
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 432dc1efa71e56e364b07c9cd0d0ba1feffadce1

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13317/
Subject: LU-6081 lfs: split setstripe and migrate help
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 587a54571e3ce1c39a1995a6d1ae7066e5459e24

[Nathan Rutman]

Why is this still open if landed in 2.7?

[James A Simmons]

Not all of it landed

[Nathan Rutman]

To move this forward again, can I propose:

1. liblustreapi_json.c can generate the JSON in lfs, but there's no JSON parsing for the copytool end.
2. replace liblustreapi_json.c, with CCAN json.c,  providing both an encoder and decoder without any external library dependencies
3. don't try to address the YAML issue here. If someone wants to do that later, replacing the JSON encoder/decoder should not make a difference to any JSON users.

[James A Simmons]

No it has been requested that everything be moved over to libyaml internally. See LU-9897 for details. JSON can be handled by libyaml so all JSON specific additional code shall be purged with prejudice.

[James A Simmons]

Now that luistre libraries are sane its time to come back to this work. Are people still interested in this? 

[Nathan Rutman]

Yes, I think this enhancement makes a ton of sense. What needs to be done to help this land?

[Nathan Rutman]

While I don't advocate delaying this patch for it, I will ask the question: shouldn't we use the lfs side to set up the target striping, and let the copytool read it out of the target, rather than passing the stripe info through the HSM request?

[Jean-Baptiste Riaux (Inactive)]

I see some thing moving here. Last state of the patches (today) to get a clear view:

http://review.whamcloud.com/13241	lfs: check that pool name is not too long	merged
http://review.whamcloud.com/13242	user: Introducing llapi_create_volatile_idx2	merged
http://review.whamcloud.com/13243	hsm: adding migration support
http://review.whamcloud.com/13244	lfs: added room for mdt-index option to lfs migrate	abandoned (moved)
http://review.whamcloud.com/13245	user: replace duplicated random_group_id	abandoned
http://review.whamcloud.com/13277	user: use random() instead of /dev/urandom	merged
http://review.whamcloud.com/13292	lfs: fixed bad return value	abandoned
http://review.whamcloud.com/13293	lfs: fixed bad return value	merged
http://review.whamcloud.com/13317	lfs: split setstripe and migrate help	merged
http://review.whamcloud.com/13654	lib: don't ignore the return of read()	merged
http://review.whamcloud.com/13723	lib: add a full fledge JSON parser to liblustreapi	replaced
http://review.whamcloud.com/13724	lib: integrate CCAN JSON parser	abandoned

 

So only review 13442 and 13243 remain. 13242 was refreshed last week.

 

[James A Simmons]

Patch 13243 depends on a new JSON parser. It was agreed on that the liblustre_json.c code should be rewritten to use libyaml.

[Cory Spitz]

http://review.whamcloud.com/13242 appears ready to land.  Is it?

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/13242/
Subject: LU-6081 user: adding llapi_create_volatile_param()
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 159ccd46747f91f6236c99b6d24edaf6a1351ced

[Nathan Rutman]

Marked as "merged" - is this landed? Fix version?

[James A Simmons]

No has been put on the back burner. Andreas recommend that instead of using HSM as the backend for migration that we use FLR mirroring instead. This is being discussed on the lustreclient slack channel.

[Cory Spitz]

nangelinas, do you have an opinion about how to wrap this up? We now have the patches for FLR mirroring as with LU-12890.

[Nikitas Angelinas]

@Cory Spitz, if we were to follow the suggestion by Andreas, I believe we would have to change migrate to use FLR mirrors and work on landing LU-12890. I don't have a clear understanding of what this would involve yet, but Andreas mentioned in https://review.whamcloud.com/#/c/13243/29/lustre/utils/lhsmtool_posix.c@1392 that this should be done via "mirror extend" and "mirror split -d".

@Andreas Dilger, do you think we should work on integrating migrate with FLR mirrors at this point, or could we try to land the remaining patch for this ticket (http://review.whamcloud.com/13243) and work on integrating FLR in a separate patch?

[Andreas Dilger]

I can't find the specific comment you are referring to, but I don't think we need to implement all migration via mirroring right now. I think my suggestion is that if we are implementing an interface to send "data movement" commands to HSM agent nodes, then it needs to be generic enough to allow both migrate, mirror, and resync operations.

[Nathan Rutman]

generic enough to allow both migrate, mirror, and resync operations

The mirror op is addressed in LU-12890, which in turn is blocked on this one. So iiuc we should land them both?