Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-16789

"lfs migrate" to use AIO/DIO or io_uring (kernel 5.1+)

Details

    • Improvement
    • Resolution: Unresolved
    • Minor
    • None
    • None
    • 9223372036854775807

    Description

      It would be useful to speed up "lfs migrate", "lfs mirror extend", and "lfs mirror resync" to use asynchronous Direct IO (AIO/DIO via libaio) to do the data copying

      This should use the same mechanism as lustre/tests/aiocp.c to have a producer/consumer queue and submits some number of AIO read requests, and then submit the write requests when the reads finish.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-18647 libaio read file from Lustre in DIO mode reports errno 22

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. LU-17143 Use migrate_copy_data() for all data movement

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-18455 Make 'lfs migrate' multi-threading

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-16587 Make lfs migrate faster

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-18454 'lfs migrate' command can read filenames from file/stdin

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. LU-12649 Tracker for ongoing FLR improvements

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. LU-18032 disable unaligned AIO

          • Critical - Crashes, loss of data, severe memory leak.
          • Reopened

          Activity

            [LU-16789] "lfs migrate" to use AIO/DIO or io_uring (kernel 5.1+)
            flei Feng Lei made changes -
            Link New: This issue is related to LU-18032 [ LU-18032 ]
            flei Feng Lei made changes -
            Status Original: Open [ 1 ] New: In Progress [ 3 ]
            flei Feng Lei made changes -
            Link New: This issue is related to LU-18647 [ LU-18647 ]
            flei Feng Lei made changes -
            Link New: This issue is related to LU-18455 [ LU-18455 ]
            flei Feng Lei made changes -
            Link New: This issue is related to LU-18454 [ LU-18454 ]
            adilger Andreas Dilger made changes -
            Labels Original: lug23dd lug24dd medium New: lug23dd lug24dd medium utils
            adilger Andreas Dilger made changes -
            Link New: This issue is related to LU-16587 [ LU-16587 ]
            adilger Andreas Dilger made changes -
            Link New: This issue is related to LU-12649 [ LU-12649 ]
            flei Feng Lei made changes -
            Assignee Original: WC Triage [ wc-triage ] New: Feng Lei [ flei ]
            adilger Andreas Dilger made changes -
            Labels Original: lad23dd lug23dd medium New: lug23dd lug24dd medium

            People

              flei Feng Lei
              adilger Andreas Dilger
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated: