Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-17814

"lfs find" to scan with multiple threads

Details

    • Improvement
    • Resolution: Unresolved
    • Minor
    • None
    • None
    • 3
    • 9223372036854775807

    Description

      It would be useful for "lfs find" to perform directory scans with multiple threads in parallel. It could (potentially) fork a new thread (or put a work item into a pool) for each subdirectory so that they can be scanned in parallel.

      There is "libcircle" and "libpcircle" that can perform workload sharing to speed up directory traversal. Also, the pfind code in IO500 is also doing efficient parallel directory traversal, including splitting up large directories by hash index to traverse in parallel.

      Integrating one of these algorithms into "lfs find" with pthreads would allow a many-fold improvement in directory scanning performance.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. LU-19052 add 'lfs find --skip-rebalance' parameter to return a fraction of files for OST rebalancing

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-17699 add 'lfs find' parameter to return only a fraction of files for rebalancing

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-5170 lfs usability

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-14610 Make mpiFileUtils better support statahead

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-18586 add llapi_find_param_alloc() function

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          mentioned in

          Page Loading...

          Activity

            [LU-17814] "lfs find" to scan with multiple threads
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57293/
            Subject: LU-17814 utils: Add work unit management
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: e505e7dbfb8b58a7a2ef9178b56f8b7668a18329

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57293/ Subject: LU-17814 utils: Add work unit management Project: fs/lustre-release Branch: master Current Patch Set: Commit: e505e7dbfb8b58a7a2ef9178b56f8b7668a18329
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57292/
            Subject: LU-17814 utils: implement thread pool
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 23d9d6b1fb7de300541f39718ca4e3b1f1c5db57

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57292/ Subject: LU-17814 utils: implement thread pool Project: fs/lustre-release Branch: master Current Patch Set: Commit: 23d9d6b1fb7de300541f39718ca4e3b1f1c5db57
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57221/
            Subject: LU-17814 utils: add parallel find arg
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 93c03de4bb52e9ceeb79f89dc277d2e251f27167

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57221/ Subject: LU-17814 utils: add parallel find arg Project: fs/lustre-release Branch: master Current Patch Set: Commit: 93c03de4bb52e9ceeb79f89dc277d2e251f27167
            qian_wc Qian Yingjin added a comment - - edited

            I would suggest this can be more general for directory traversal workload by using parallel thread pool.

            It can not only be used by "lfs find", but also should also be able to use by setting projid for files under a directory recursively or libhsm_scanner.c code with multiple threads to traverse a directory in BFS mode.

            qian_wc Qian Yingjin added a comment - - edited I would suggest this can be more general for directory traversal workload by using parallel thread pool. It can not only be used by "lfs find", but also should also be able to use by setting projid for files under a directory recursively or libhsm_scanner.c code with multiple threads to traverse a directory in BFS mode.
            gerrit Gerrit Updater added a comment -

            "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57313
            Subject: LU-17814 utils: add error handling to pfind
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 36e7441f0d8ed55c3292bab063ed59adb023422c

            gerrit Gerrit Updater added a comment - "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57313 Subject: LU-17814 utils: add error handling to pfind Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 36e7441f0d8ed55c3292bab063ed59adb023422c
            gerrit Gerrit Updater added a comment -

            "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57306
            Subject: LU-17814 utils: Enable pfind by default
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: a73886d7d7790279138cce9d6090ec625897e12e

            gerrit Gerrit Updater added a comment - "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57306 Subject: LU-17814 utils: Enable pfind by default Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: a73886d7d7790279138cce9d6090ec625897e12e
            gerrit Gerrit Updater added a comment -

            "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57295
            Subject: LU-17814 utils: Messy pfind support
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: a4ce7f4a41ec6799aed2595497fc3e8b2615af10

            gerrit Gerrit Updater added a comment - "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57295 Subject: LU-17814 utils: Messy pfind support Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: a4ce7f4a41ec6799aed2595497fc3e8b2615af10
            gerrit Gerrit Updater added a comment -

            "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57294
            Subject: LU-17814 utils: Add deep copy of find_param
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: d7b40ff5da51c415a63b7c3a46c7f7299bc2af71

            gerrit Gerrit Updater added a comment - "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57294 Subject: LU-17814 utils: Add deep copy of find_param Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: d7b40ff5da51c415a63b7c3a46c7f7299bc2af71
            gerrit Gerrit Updater added a comment -

            "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57293
            Subject: LU-17814 utils: Add work unit management
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 3ebfed47ceaac032714f6a77670999f3f6f01f6b

            gerrit Gerrit Updater added a comment - "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57293 Subject: LU-17814 utils: Add work unit management Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 3ebfed47ceaac032714f6a77670999f3f6f01f6b
            gerrit Gerrit Updater added a comment -

            "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57292
            Subject: LU-17814 utils: implement thread pool
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: ec4f69c97f2e1f8867c7d1f29c104007f88b4aa5

            gerrit Gerrit Updater added a comment - "Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57292 Subject: LU-17814 utils: implement thread pool Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: ec4f69c97f2e1f8867c7d1f29c104007f88b4aa5

            People

              paf0186 Patrick Farrell
              adilger Andreas Dilger
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: