Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-17814

"lfs find" to scan with multiple threads

Details

    • Improvement
    • Resolution: Unresolved
    • Minor
    • None
    • None
    • 3
    • 9223372036854775807

    Description

      It would be useful for "lfs find" to perform directory scans with multiple threads in parallel. It could (potentially) fork a new thread (or put a work item into a pool) for each subdirectory so that they can be scanned in parallel.

      There is "libcircle" and "libpcircle" that can perform workload sharing to speed up directory traversal. Also, the pfind code in IO500 is also doing efficient parallel directory traversal, including splitting up large directories by hash index to traverse in parallel.

      Integrating one of these algorithms into "lfs find" with pthreads would allow a many-fold improvement in directory scanning performance.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. LU-19052 add 'lfs find --skip-rebalance' parameter to return a fraction of files for OST rebalancing

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-19125 move "lfs project" over to parallel "lfs find" infrastructure

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-17699 add 'lfs find' parameter to return only a fraction of files for rebalancing

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-5170 lfs usability

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-14610 Make mpiFileUtils better support statahead

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-18586 add llapi_find_param_alloc() function

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          mentioned in

          Page Loading...

          Activity

            [LU-17814] "lfs find" to scan with multiple threads
            gerrit Gerrit Updater added a comment -

            "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57306
            Subject: LU-17814 utils: Enable pfind by default
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: a73886d7d7790279138cce9d6090ec625897e12e

            gerrit Gerrit Updater added a comment - "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57306 Subject: LU-17814 utils: Enable pfind by default Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: a73886d7d7790279138cce9d6090ec625897e12e
            gerrit Gerrit Updater added a comment -

            "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57295
            Subject: LU-17814 utils: Messy pfind support
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: a4ce7f4a41ec6799aed2595497fc3e8b2615af10

            gerrit Gerrit Updater added a comment - "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57295 Subject: LU-17814 utils: Messy pfind support Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: a4ce7f4a41ec6799aed2595497fc3e8b2615af10
            gerrit Gerrit Updater added a comment -

            "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57294
            Subject: LU-17814 utils: Add deep copy of find_param
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: d7b40ff5da51c415a63b7c3a46c7f7299bc2af71

            gerrit Gerrit Updater added a comment - "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57294 Subject: LU-17814 utils: Add deep copy of find_param Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: d7b40ff5da51c415a63b7c3a46c7f7299bc2af71
            gerrit Gerrit Updater added a comment -

            "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57293
            Subject: LU-17814 utils: Add work unit management
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 3ebfed47ceaac032714f6a77670999f3f6f01f6b

            gerrit Gerrit Updater added a comment - "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57293 Subject: LU-17814 utils: Add work unit management Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 3ebfed47ceaac032714f6a77670999f3f6f01f6b
            gerrit Gerrit Updater added a comment -

            "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57292
            Subject: LU-17814 utils: implement thread pool
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: ec4f69c97f2e1f8867c7d1f29c104007f88b4aa5

            gerrit Gerrit Updater added a comment - "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57292 Subject: LU-17814 utils: implement thread pool Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: ec4f69c97f2e1f8867c7d1f29c104007f88b4aa5
            gerrit Gerrit Updater added a comment -

            "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57264
            Subject: LU-17814 utils: fixes - deep copy, dupe()
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 63bcdc517f7afb11bffc324fbc34353d747b8125

            gerrit Gerrit Updater added a comment - "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57264 Subject: LU-17814 utils: fixes - deep copy, dupe() Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 63bcdc517f7afb11bffc324fbc34353d747b8125
            gerrit Gerrit Updater added a comment -

            "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57248
            Subject: LU-17814 utils: enable pfind by default
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 7966d968f17d9078ce0d16a78e5a3a6ab0997948

            gerrit Gerrit Updater added a comment - "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57248 Subject: LU-17814 utils: enable pfind by default Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 7966d968f17d9078ce0d16a78e5a3a6ab0997948
            gerrit Gerrit Updater added a comment -

            "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57247
            Subject: LU-17814 utils: Add min/max depth support
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 7d28579cf7ceceb1840c4f6db7a8928258327b13

            gerrit Gerrit Updater added a comment - "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57247 Subject: LU-17814 utils: Add min/max depth support Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 7d28579cf7ceceb1840c4f6db7a8928258327b13
            gerrit Gerrit Updater added a comment -

            "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57244
            Subject: LU-17814 utils: Add parallel fini
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 8055378c33836fd411883980a540722e0d29d812

            gerrit Gerrit Updater added a comment - "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57244 Subject: LU-17814 utils: Add parallel fini Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 8055378c33836fd411883980a540722e0d29d812
            paf0186 Patrick Farrell added a comment -

            "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57243
            Subject: LU-17814 utils: Add real parallel find support
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: d73101f0257ec2ff2d0a867a53c4e33a7b8267d4

            paf0186 Patrick Farrell added a comment - "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57243 Subject: LU-17814 utils: Add real parallel find support Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: d73101f0257ec2ff2d0a867a53c4e33a7b8267d4
            gerrit Gerrit Updater added a comment -

            "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57240
            Subject: LU-17814 utils: Split llapi_semantic_traverse
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 0e3f765aff4b83890c5db3520ab37d3fb6179c26

            gerrit Gerrit Updater added a comment - "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57240 Subject: LU-17814 utils: Split llapi_semantic_traverse Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 0e3f765aff4b83890c5db3520ab37d3fb6179c26

            People

              paf0186 Patrick Farrell
              adilger Andreas Dilger
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: