Description
Currently file migration is done by "lfs migrate". This has a couple problems:
- It doesn't scale well to large files and/or large number of files to copy.
- If an external tools (such as Robinhood) wants to migrate a file, it has to spawn lfs to do the job, for every file, which has a big overhead.
A solution to this issue is to move the copy operation into the copytool. The copytool already knows how to copy files, and Lustre has control over these files. This imply to re-use and extend the infrastructure created for HSM, and would solve the two issues mentioned above:
- Migrating a file, from an application point of view, becomes more or less the same as archiving a file.
- Since the file copy is done by the copytool, this copy can be optimized in one place.
Attachments
Issue Links
- is blocked by
-
LU-7507 the data version doesn't always change after a layout swap
- Resolved
-
LU-6246 Add a JSON decoder
- Resolved
- is duplicated by
-
LU-6242 some version of gcc fails to compile liblustre_util.c
- Closed
- is related to
-
LU-6234 lfs computes pool name length incorrectly
- Resolved
-
LU-7037 sanity-hsm test_57:gethostbyname("shadow-52vm5") failed
- Resolved
-
LU-6463 Interop 2.5.3<->master ost-pools test_1c: Pool creation of testpool12345678 failed
- Closed
-
LU-10606 HSM info as part of LOV layout xattr
- Open
-
LU-12890 Use HSM to mirror sync files
- Open
-
LU-9897 Tracking of lustre library improvements.
- Open
-
LU-11621 Add copy_file_range() API and use it for lfs migrate and mirror resync
- Open
-
LU-10968 add coordinator bypass upcalls for HSM archive and remove
- Reopened
- is related to
-
LU-12513 Address YAML limitations
- Resolved