[LU-6081] hsm: add file migrate support

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Labels: HSM cea patch
• Assignee: Ben Evans
• Reporter: Frank Zago

Description

Currently file migration is done by "lfs migrate". This has a couple problems:

• It doesn't scale well to large files and/or large number of files to copy.
• If an external tools (such as Robinhood) wants to migrate a file, it has to spawn lfs to do the job, for every file, which has a big overhead.

A solution to this issue is to move the copy operation into the copytool. The copytool already knows how to copy files, and Lustre has control over these files. This imply to re-use and extend the infrastructure created for HSM, and would solve the two issues mentioned above:

• Migrating a file, from an application point of view, becomes more or less the same as archiving a file.
• Since the file copy is done by the copytool, this copy can be optimized in one place.

Issue Links

• is blocked by: LU-7507 the data version doesn't always change after a layout swap
• is blocked by: LU-6246 Add a JSON decoder
• is duplicated by: LU-6242 some version of gcc fails to compile liblustre_util.c
• is related to: LU-6234 lfs computes pool name length incorrectly
• is related to: LU-7037 sanity-hsm test_57:gethostbyname("shadow-52vm5") failed
• is related to: LU-6463 Interop 2.5.3<->master ost-pools test_1c: Pool creation of testpool12345678 failed
• is related to: LU-10606 HSM info as a layout
• is related to: LU-9897 Tracking of lustre library improvements.
• is related to: LU-11621 Add copy_file_range() API and use it for lfs migrate and mirror resync
• is related to: LU-12513 Address YAML limitations