[LU-6081] hsm: add file migrate support

• Type: Improvement
• Resolution: Unresolved
• Priority: Minor
• Labels: HSM cea patch
• Assignee: Ben Evans (Inactive)
• Reporter: Frank Zago (Inactive)

Description

Currently file migration is done by "lfs migrate". This has a couple problems:

• It doesn't scale well to large files and/or large number of files to copy.
• If an external tools (such as Robinhood) wants to migrate a file, it has to spawn lfs to do the job, for every file, which has a big overhead.

A solution to this issue is to move the copy operation into the copytool. The copytool already knows how to copy files, and Lustre has control over these files. This imply to re-use and extend the infrastructure created for HSM, and would solve the two issues mentioned above:

• Migrating a file, from an application point of view, becomes more or less the same as archiving a file.
• Since the file copy is done by the copytool, this copy can be optimized in one place.

Issue Links

• is blocked by: LU-7507 the data version doesn't always change after a layout swap
• is blocked by: LU-6246 Add a JSON decoder
• is duplicated by: LU-6242 some version of gcc fails to compile liblustre_util.c
• is related to: LU-6234 lfs computes pool name length incorrectly
• is related to: LU-7037 sanity-hsm test_57:gethostbyname("shadow-52vm5") failed
• is related to: LU-6463 Interop 2.5.3<->master ost-pools test_1c: Pool creation of testpool12345678 failed
• is related to: LU-10606 HSM info as part of LOV layout xattr
• is related to: LU-12890 Use HSM to mirror sync files
• is related to: LU-9897 Tracking of lustre library improvements.
• is related to: LU-11621 Add copy_file_range() API and use it for lfs migrate and mirror resync
• is related to: LU-10968 add coordinator bypass upcalls for HSM archive and remove
• is related to: LU-12513 Address YAML limitations

Comments

[Nathan Rutman]

generic enough to allow both migrate, mirror, and resync operations

The mirror op is addressed in LU-12890, which in turn is blocked on this one. So iiuc we should land them both?

[Andreas Dilger]

I can't find the specific comment you are referring to, but I don't think we need to implement all migration via mirroring right now. I think my suggestion is that if we are implementing an interface to send "data movement" commands to HSM agent nodes, then it needs to be generic enough to allow both migrate, mirror, and resync operations.

[Nikitas Angelinas]

@Cory Spitz, if we were to follow the suggestion by Andreas, I believe we would have to change migrate to use FLR mirrors and work on landing LU-12890. I don't have a clear understanding of what this would involve yet, but Andreas mentioned in https://review.whamcloud.com/#/c/13243/29/lustre/utils/lhsmtool_posix.c@1392 that this should be done via "mirror extend" and "mirror split -d".

@Andreas Dilger, do you think we should work on integrating migrate with FLR mirrors at this point, or could we try to land the remaining patch for this ticket (http://review.whamcloud.com/13243) and work on integrating FLR in a separate patch?

[Cory Spitz]

nangelinas, do you have an opinion about how to wrap this up? We now have the patches for FLR mirroring as with LU-12890.

[James A Simmons]

No has been put on the back burner. Andreas recommend that instead of using HSM as the backend for migration that we use FLR mirroring instead. This is being discussed on the lustreclient slack channel.

[Nathan Rutman]

Marked as "merged" - is this landed? Fix version?