[LU-16640] parallellize generic I/O Created: 13/Mar/23  Updated: 27/Apr/23

Status: Open
Project: Lustre
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Minor
Reporter: Patrick Farrell Assignee: Patrick Farrell
Resolution: Unresolved Votes: 0
Labels: None

Issue Links:
Related
is related to LU-16540 move ll_get_user_pages to ptlrpcd thr... Resolved
Rank (Obsolete): 9223372036854775807

 Description   

LU-8964 tried to improve I/O performance by splitting I/Os to multiple threads at the ll_file/cl_io level (ie, the top level).  This is a decent idea, but had some problems.

Specifically, the original LU-8964 parallel I/O work didn't improve performance much and in some cases harmed it (mostly because of limitations in the buffered I/O path, and the synchronous nature of the DIO path at the time), the new parallel DIO path should make it possible to get large performance improvements with roughly the same approach as in LU-8964.  It's mostly a matter of using this or not intelligently - We will only do parallel IO on the DIO path, since it's mostly a net loss on the buffered path.  We will also add some sized based tuning.

But the patch will be modeled on the approach taken in LU-8964.



 Comments   
Comment by Gerrit Updater [ 13/Mar/23 ]

"Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50280
Subject: LU-16640 llite: add parallel io work queue
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 356756289459826cc716ab6af0e3cea2c32d0801

Comment by Gerrit Updater [ 13/Mar/23 ]

"Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50282
Subject: LU-16640 llite: add __ll_file_io_generic
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: eb6e3817917f06a8875488af1d8d7c9597dba131

Comment by Gerrit Updater [ 14/Mar/23 ]

"Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50295
Subject: LU-16640 llite: add pio_list to ll_file_io_generic
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e6c6834bafc6ca66970c462b16b7f827a25c8cf0

Comment by Gerrit Updater [ 14/Mar/23 ]

"Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50296
Subject: LU-16640 llite: Add ci_pio and conditions
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 92e4f350590dec2bd5a303a2656b2888f7498d61

Comment by Gerrit Updater [ 14/Mar/23 ]

"Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50297
Subject: LU-16640 llite: convert to cl_rw_io
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: fe6cf640a1c8ea8b0d20f68388213ae051f414b6

Comment by Gerrit Updater [ 15/Mar/23 ]

"Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50306
Subject: LU-16640 llite: Add other pio flag checks
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0a2b272554fe1c88653f47bce944402db446968c

Comment by Gerrit Updater [ 15/Mar/23 ]

"Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50307
Subject: LU-16640 tests: add old test
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: bc08740fc1015110c2fe145517032db732b8fa81

Comment by Gerrit Updater [ 15/Mar/23 ]

"Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50308
Subject: LU-16640 clio: Add most supporting code for pio
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: bcda7c831c852b063a508ae4435b85eca1f20226

Comment by Gerrit Updater [ 22/Mar/23 ]

"Patrick Farrell <pfarrell@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50380
Subject: LU-16640 llite: remove vui_iter and vui_iocb
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b208981d20312c4fb0d27c7dcfe12b2d95fbc11c

Generated at Sat Feb 10 03:28:45 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.