Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-6081

hsm: add file migrate support

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
    • Rank (Obsolete):
      16922

      Description

      Currently file migration is done by "lfs migrate". This has a couple problems:

      • It doesn't scale well to large files and/or large number of files to copy.
      • If an external tools (such as Robinhood) wants to migrate a file, it has to spawn lfs to do the job, for every file, which has a big overhead.

      A solution to this issue is to move the copy operation into the copytool. The copytool already knows how to copy files, and Lustre has control over these files. This imply to re-use and extend the infrastructure created for HSM, and would solve the two issues mentioned above:

      • Migrating a file, from an application point of view, becomes more or less the same as archiving a file.
      • Since the file copy is done by the copytool, this copy can be optimized in one place.

        Attachments

          Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. LU-7507 the data version doesn't always change after a layout swap

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. LU-6246 Add a JSON decoder

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. LU-6242 some version of gcc fails to compile liblustre_util.c

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-6234 lfs computes pool name length incorrectly

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-7037 sanity-hsm test_57:gethostbyname("shadow-52vm5") failed

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-6463 Interop 2.5.3<->master ost-pools test_1c: Pool creation of testpool12345678 failed

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. LU-10606 HSM info as part of LOV layout xattr

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. LU-12890 Use HSM to mirror sync files

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Task - A task that needs to be done. LU-9897 Tracking of lustre library improvements.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-10968 add coordinator bypass upcalls for HSM archive and remove

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-11621 Add copy_file_range() API and use it for lfs migrate and mirror resync

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-12513 Address YAML limitations

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

            Activity

              People

              • Assignee:
                bevans Ben Evans (Inactive)
                Reporter:
                fzago Frank Zago
              • Votes:
                0 Vote for this issue
                Watchers:
                27 Start watching this issue

                Dates

                • Created:
                  Updated: