Details

    • Improvement
    • Resolution: Unresolved
    • Minor
    • None
    • None
    • 16922

    Description

      Currently file migration is done by "lfs migrate". This has a couple problems:

      • It doesn't scale well to large files and/or large number of files to copy.
      • If an external tools (such as Robinhood) wants to migrate a file, it has to spawn lfs to do the job, for every file, which has a big overhead.

      A solution to this issue is to move the copy operation into the copytool. The copytool already knows how to copy files, and Lustre has control over these files. This imply to re-use and extend the infrastructure created for HSM, and would solve the two issues mentioned above:

      • Migrating a file, from an application point of view, becomes more or less the same as archiving a file.
      • Since the file copy is done by the copytool, this copy can be optimized in one place.

      Attachments

        Issue Links

          Activity

            [LU-6081] hsm: add file migrate support

            generic enough to allow both migrate, mirror, and resync operations

            The mirror op is addressed in LU-12890, which in turn is blocked on this one. So iiuc we should land them both?

            nrutman Nathan Rutman added a comment - generic enough to allow both migrate, mirror, and resync operations The mirror op is addressed in LU-12890 , which in turn is blocked on this one. So iiuc we should land them both?

            I can't find the specific comment you are referring to, but I don't think we need to implement all migration via mirroring right now. I think my suggestion is that if we are implementing an interface to send "data movement" commands to HSM agent nodes, then it needs to be generic enough to allow both migrate, mirror, and resync operations.

            adilger Andreas Dilger added a comment - I can't find the specific comment you are referring to, but I don't think we need to implement all migration via mirroring right now. I think my suggestion is that if we are implementing an interface to send "data movement" commands to HSM agent nodes, then it needs to be generic enough to allow both migrate, mirror, and resync operations.

            @Cory Spitz, if we were to follow the suggestion by Andreas, I believe we would have to change migrate to use FLR mirrors and work on landing LU-12890. I don't have a clear understanding of what this would involve yet, but Andreas mentioned in https://review.whamcloud.com/#/c/13243/29/lustre/utils/lhsmtool_posix.c@1392 that this should be done via "mirror extend" and "mirror split -d".

            @Andreas Dilger, do you think we should work on integrating migrate with FLR mirrors at this point, or could we try to land the remaining patch for this ticket (http://review.whamcloud.com/13243) and work on integrating FLR in a separate patch?

            nangelinas Nikitas Angelinas added a comment - @Cory Spitz, if we were to follow the suggestion by Andreas, I believe we would have to change migrate to use FLR mirrors and work on landing LU-12890 . I don't have a clear understanding of what this would involve yet, but Andreas mentioned in https://review.whamcloud.com/#/c/13243/29/lustre/utils/lhsmtool_posix.c@1392 that this should be done via "mirror extend" and "mirror split -d". @Andreas Dilger, do you think we should work on integrating migrate with FLR mirrors at this point, or could we try to land the remaining patch for this ticket ( http://review.whamcloud.com/13243 ) and work on integrating FLR in a separate patch?
            spitzcor Cory Spitz added a comment -

            nangelinas, do you have an opinion about how to wrap this up? We now have the patches for FLR mirroring as with LU-12890.

            spitzcor Cory Spitz added a comment - nangelinas , do you have an opinion about how to wrap this up? We now have the patches for FLR mirroring as with LU-12890 .

            No has been put on the back burner. Andreas recommend that instead of using HSM as the backend for migration that we use FLR mirroring instead. This is being discussed on the lustreclient slack channel.

            simmonsja James A Simmons added a comment - No has been put on the back burner. Andreas recommend that instead of using HSM as the backend for migration that we use FLR mirroring instead. This is being discussed on the lustreclient slack channel.

            Marked as "merged" - is this landed? Fix version?

            nrutman Nathan Rutman added a comment - Marked as "merged" - is this landed? Fix version?

            Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/13242/
            Subject: LU-6081 user: adding llapi_create_volatile_param()
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 159ccd46747f91f6236c99b6d24edaf6a1351ced

            gerrit Gerrit Updater added a comment - Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/13242/ Subject: LU-6081 user: adding llapi_create_volatile_param() Project: fs/lustre-release Branch: master Current Patch Set: Commit: 159ccd46747f91f6236c99b6d24edaf6a1351ced
            spitzcor Cory Spitz added a comment -

            http://review.whamcloud.com/13242 appears ready to land.  Is it?

            spitzcor Cory Spitz added a comment - http://review.whamcloud.com/13242  appears ready to land.  Is it?

            Patch 13243 depends on a new JSON parser. It was agreed on that the liblustre_json.c code should be rewritten to use libyaml.

            simmonsja James A Simmons added a comment - Patch 13243 depends on a new JSON parser. It was agreed on that the liblustre_json.c code should be rewritten to use libyaml.

            I see some thing moving here. Last state of the patches (today) to get a clear view:

            http://review.whamcloud.com/13241 lfs: check that pool name is not too long merged
            http://review.whamcloud.com/13242 user: Introducing llapi_create_volatile_idx2 merged
            http://review.whamcloud.com/13243 hsm: adding migration support  
            http://review.whamcloud.com/13244 lfs: added room for mdt-index option to lfs migrate abandoned (moved)
            http://review.whamcloud.com/13245 user: replace duplicated random_group_id abandoned
            http://review.whamcloud.com/13277 user: use random() instead of /dev/urandom merged
            http://review.whamcloud.com/13292 lfs: fixed bad return value abandoned
            http://review.whamcloud.com/13293 lfs: fixed bad return value merged
            http://review.whamcloud.com/13317 lfs: split setstripe and migrate help merged
            http://review.whamcloud.com/13654 lib: don't ignore the return of read() merged
            http://review.whamcloud.com/13723 lib: add a full fledge JSON parser to liblustreapi replaced
            http://review.whamcloud.com/13724 lib: integrate CCAN JSON parser abandoned

             

            So only review 13442 and 13243 remain. 13242 was refreshed last week.

             

            riauxjb Jean-Baptiste Riaux (Inactive) added a comment - - edited I see some thing moving here. Last state of the patches (today) to get a clear view: http://review.whamcloud.com/13241 lfs: check that pool name is not too long merged http://review.whamcloud.com/13242 user: Introducing llapi_create_volatile_idx2 merged http://review.whamcloud.com/13243 hsm: adding migration support   http://review.whamcloud.com/13244 lfs: added room for mdt-index option to lfs migrate abandoned (moved) http://review.whamcloud.com/13245 user: replace duplicated random_group_id abandoned http://review.whamcloud.com/13277 user: use random() instead of /dev/urandom merged http://review.whamcloud.com/13292 lfs: fixed bad return value abandoned http://review.whamcloud.com/13293 lfs: fixed bad return value merged http://review.whamcloud.com/13317 lfs: split setstripe and migrate help merged http://review.whamcloud.com/13654 lib: don't ignore the return of read() merged http://review.whamcloud.com/13723 lib: add a full fledge JSON parser to liblustreapi replaced http://review.whamcloud.com/13724 lib: integrate CCAN JSON parser abandoned   So only review 13442 and 13243 remain. 13242 was refreshed last week.  

            People

              bevans Ben Evans (Inactive)
              fzago Frank Zago (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              29 Start watching this issue

              Dates

                Created:
                Updated: