Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-17814

"lfs find" to scan with multiple threads

Details

    • Improvement
    • Resolution: Unresolved
    • Minor
    • None
    • None
    • 3
    • 9223372036854775807

    Description

      It would be useful for "lfs find" to perform directory scans with multiple threads in parallel. It could (potentially) fork a new thread (or put a work item into a pool) for each subdirectory so that they can be scanned in parallel.

      There is "libcircle" and "libpcircle" that can perform workload sharing to speed up directory traversal. Also, the pfind code in IO500 is also doing efficient parallel directory traversal, including splitting up large directories by hash index to traverse in parallel.

      Integrating one of these algorithms into "lfs find" with pthreads would allow a many-fold improvement in directory scanning performance.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. LU-19052 add 'lfs find --skip-rebalance' parameter to return a fraction of files for OST rebalancing

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-19125 move "lfs project" over to parallel "lfs find" infrastructure

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-17699 add 'lfs find' parameter to return only a fraction of files for rebalancing

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-5170 lfs usability

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-14610 Make mpiFileUtils better support statahead

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-18586 add llapi_find_param_alloc() function

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          mentioned in

          Page Loading...

          Activity

            [LU-17814] "lfs find" to scan with multiple threads
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57295/
            Subject: LU-17814 utils: implement real pfind
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 99f47b136790fd7ef359332f620692d4e2333ba2

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57295/ Subject: LU-17814 utils: implement real pfind Project: fs/lustre-release Branch: master Current Patch Set: Commit: 99f47b136790fd7ef359332f620692d4e2333ba2
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57294/
            Subject: LU-17814 utils: Add deep copy of find_param
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 8f2fb73bc7ff7384e9cbfebe798329e7736ec4cf

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57294/ Subject: LU-17814 utils: Add deep copy of find_param Project: fs/lustre-release Branch: master Current Patch Set: Commit: 8f2fb73bc7ff7384e9cbfebe798329e7736ec4cf
            gerrit Gerrit Updater added a comment -

            "Timothy Day <timday@amazon.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/59691
            Subject: LU-17814 utils: fix compiler errors in pfind
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 5130b3d5616e8821dff0afd041daf034653f2ec9

            gerrit Gerrit Updater added a comment - "Timothy Day <timday@amazon.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/59691 Subject: LU-17814 utils: fix compiler errors in pfind Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 5130b3d5616e8821dff0afd041daf034653f2ec9
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57293/
            Subject: LU-17814 utils: Add work unit management
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: e505e7dbfb8b58a7a2ef9178b56f8b7668a18329

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57293/ Subject: LU-17814 utils: Add work unit management Project: fs/lustre-release Branch: master Current Patch Set: Commit: e505e7dbfb8b58a7a2ef9178b56f8b7668a18329
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57292/
            Subject: LU-17814 utils: implement thread pool
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 23d9d6b1fb7de300541f39718ca4e3b1f1c5db57

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57292/ Subject: LU-17814 utils: implement thread pool Project: fs/lustre-release Branch: master Current Patch Set: Commit: 23d9d6b1fb7de300541f39718ca4e3b1f1c5db57
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57221/
            Subject: LU-17814 utils: add parallel find arg
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 93c03de4bb52e9ceeb79f89dc277d2e251f27167

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57221/ Subject: LU-17814 utils: add parallel find arg Project: fs/lustre-release Branch: master Current Patch Set: Commit: 93c03de4bb52e9ceeb79f89dc277d2e251f27167

            People

              paf0186 Patrick Farrell
              adilger Andreas Dilger
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: