[LU-17814] "lfs find" to scan with multiple threads

• Type: Improvement
• Resolution: Unresolved
• Priority: Minor
• Labels: medium performance utils
• Severity: 3
• Assignee: Patrick Farrell
• Reporter: Andreas Dilger

Description

It would be useful for "lfs find" to perform directory scans with multiple threads in parallel. It could (potentially) fork a new thread (or put a work item into a pool) for each subdirectory so that they can be scanned in parallel.

There is "libcircle" and "libpcircle" that can perform workload sharing to speed up directory traversal. Also, the pfind code in IO500 is also doing efficient parallel directory traversal, including splitting up large directories by hash index to traverse in parallel.

Integrating one of these algorithms into "lfs find" with pthreads would allow a many-fold improvement in directory scanning performance.

Issue Links

• is related to: LU-19052 add 'lfs find --skip-rebalance' parameter to return a fraction of files for OST rebalancing
• is related to: LU-17699 add 'lfs find' parameter to return only a fraction of files for rebalancing
• is related to: LU-5170 lfs usability
• is related to: LU-14610 Make mpiFileUtils better support statahead
• is related to: LU-18586 add llapi_find_param_alloc() function

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57295/
Subject: LU-17814 utils: implement real pfind
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 99f47b136790fd7ef359332f620692d4e2333ba2

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57294/
Subject: LU-17814 utils: Add deep copy of find_param
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 8f2fb73bc7ff7384e9cbfebe798329e7736ec4cf

[Gerrit Updater]

"Timothy Day <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/59691
Subject: LU-17814 utils: fix compiler errors in pfind
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5130b3d5616e8821dff0afd041daf034653f2ec9

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57293/
Subject: LU-17814 utils: Add work unit management
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e505e7dbfb8b58a7a2ef9178b56f8b7668a18329

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57292/
Subject: LU-17814 utils: implement thread pool
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 23d9d6b1fb7de300541f39718ca4e3b1f1c5db57

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57221/
Subject: LU-17814 utils: add parallel find arg
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 93c03de4bb52e9ceeb79f89dc277d2e251f27167

[Qian Yingjin]

I would suggest this can be more general for directory traversal workload by using parallel thread pool.

It can not only be used by "lfs find", but also should also be able to use by setting projid for files under a directory recursively or libhsm_scanner.c code with multiple threads to traverse a directory in BFS mode.

[Gerrit Updater]

"Patrick Farrell <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57313
Subject: LU-17814 utils: add error handling to pfind
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 36e7441f0d8ed55c3292bab063ed59adb023422c

[Gerrit Updater]

"Patrick Farrell <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57306
Subject: LU-17814 utils: Enable pfind by default
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a73886d7d7790279138cce9d6090ec625897e12e