Currently file migration is done by "lfs migrate". This has a couple problems:
- It doesn't scale well to large files and/or large number of files to copy.
- If an external tools (such as Robinhood) wants to migrate a file, it has to spawn lfs to do the job, for every file, which has a big overhead.
A solution to this issue is to move the copy operation into the copytool. The copytool already knows how to copy files, and Lustre has control over these files. This imply to re-use and extend the infrastructure created for HSM, and would solve the two issues mentioned above:
- Migrating a file, from an application point of view, becomes more or less the same as archiving a file.
- Since the file copy is done by the copytool, this copy can be optimized in one place.