[LU-13799] DIO/AIO efficiency improvements

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.15.0
• Type: Bug
• Priority: Minor
• Reporter: Patrick Farrell
• Assignee: Patrick Farrell
• Resolution: Fixed
• Severity: 3

Description

There is a large degree of inefficiency/wasted time in the DIO/AIO path.  This does not show up for DIO normally because of the waiting model. but it shows up easily in AIO.

This ticket is to cover a set of improvements to DIO/AIO performance, which will also improve DIO performance once the waiting model is adjusted.  (More on this in LU-13798.)

There is a grab bag of patches to be submitted here, and some further proposals that will probably end up in other tickets.

The essence of the improvements is that all pages in a DIO submission are the same, and therefore much of the work done on a per-page basis is irrelevant and can be skipped for DIO.  Note this statement is still compatible with unaligned DIO if it can be implemented in the future - In the case of unaligned DIO, only the first and last pages are different, and that can still be handled.

Patches and benchmarks on each patch forthcoming.

The total effect of the initial set of patches on my testbed is to raise AIO/DIO performance from around 5 GiB/s to around 9 GiB/s.  I'll get more in to what else can be done shortly. 

Issue Links

• is related to: LU-13802 New i/o path: Buffered i/o as DIO
• is related to: LU-13805 i/o path: Unaligned direct i/o
• is related to: LU-13814 DIO performance: cl_page struct remov...
• is related to: LU-15483 Minor DIO test improvements
• is related to: LU-13801 Enable io_uring interface for Lustre ...
• is related to: LU-13798 Improve direct i/o performance with m...

Comments

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39437
Subject: LU-13799 osc: Don't get time for each page
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 344b38006506252f3dfd5494b555a8891b16bf70

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39438
Subject: LU-13799 llite: Do not get/put DIO pages
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 7983c6fff30afb43259afdcf76fbda13193588df

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39439
Subject: LU-13799 clio: Implement real list splice
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 46d18b197f24cb44b9a905449906c832e028a2f3

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39440
Subject: LU-13799 osc: Simplify clipping for transient pages
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a075a2267f0ee3cbd67059d83136b019bae6e46e

[Patrick Farrell]

There are more patches coming in this series - Will update shortly.

Eventual performance result is around 10 GiB/s from one thread, but there's still headroom to go higher.

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39441
Subject: LU-13799 llite: Remove transient page counting
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4f9ffa2f7bd575aa448b254a87d02277cf5b3059

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39442
Subject: LU-13799 llite: Modify AIO/DIO reference counting
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: c967bb0af12957e533008d6ceea98f112f4d9c99

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39443
Subject: LU-13799 llite: Move free user pages
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0518cec5ee189fb3911a3b4e686fe5de37592c0d

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39445
Subject: LU-13799 lov: Cache stripe offset calculation
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2f7966118b379d670621ef35639465a6a6ab8b9e

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39448
Subject: LU-13799 clio: Skip prep for transients
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 51f72ba24d56201908282b68c47bb39ff232ac3c

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39446
Subject: LU-13799 lov: Improve DIO submit
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 83fcd181493256d325f7c9386d1638a3ca6a8dc3

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39447
Subject: LU-13799 llite: Adjust dio refcounting
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 345ddb5bed66ac0f9169c9c4663a3fb7239bd226

[Patrick Farrell]

With all of the patches here, for a 64 MiB DIO, we get:
mpirun -np 1 $IOR -w -r -t 64M -b 64G -o ./iorfile --posix.odirect
write    8155 MiB/s

read     8566 MiB/s

For a very large DIO, such as 1 GiB, we get:
mpirun -np 1 $IOR -w -r -t 1GM -b 64G -o ./iorfile --posix.odirect

write    9033 MiB/s

read    9817 MiB/s

[Patrick Farrell]

This is getting near the limit of my testbed hardware, at least with one interface.  It's possible to use two if you configure with just one CPT though, so...

At this point, there is still actually a huge amount of overhead in the DIO/AIO path.

If you look at perf with all of these patches in place, cl_page allocation is now over 50% of CPU time.  Converting this from single calls to batch calls and removing a few per-page actions that can be done once per DIO, it's possible to get that down significantly.  This leads to a DIO performance of something like 15 GiB/s.

I don't have a patch ready for that - I'm going to open a separate ticket to discuss/describe what else I think can be done to improve this further. 

[Andreas Dilger]

Patrick, let me state that you rock. That is all.

[Andreas Dilger]

Patrick, I've also filed LU-13801 for implementing the io_uring interface, which is the successor to libaio, which is a bit sad since we just got libaio working well.

[Patrick Farrell]

(Moved from LU-13798)
Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39482
Subject: LU-13799 osc: Improve osc_queue_sync_pages
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2671920f69c1ff9c89cc7d390e9692acc809e5dd

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/43835
Subject: LU-13799 osc: Update shrink interval less often
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 644ab49701d6305fb79c41e1a5eed9406b3d30a9

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/43838
Subject: LU-13799 osc: Update shrink interval per i/o
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: d5560fc6b5f518992aadf5cf3e8861cbadcc6d86

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/39437/
Subject: LU-13799 osc: Don't get time for each page
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 485976ab451dd6708d4d46bce3bbed9991f5d356

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/39439/
Subject: LU-13799 clio: Implement real list splice
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: dfe2d225b86d4215cbd09e863e8de7547f0d4dae

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/39440/
Subject: LU-13799 osc: Simplify clipping for transient pages
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b64b9646f17b771c415e4b39cb8babcdc7541b30

[Patrick Farrell]

Current status on these patches, now that we've started landing some of this.

The following patches (in order) are ready, modulo completion of testing (errors appear unrelated) and review:
https://review.whamcloud.com/39442
https://review.whamcloud.com/39441
https://review.whamcloud.com/39446
https://review.whamcloud.com/39447
https://review.whamcloud.com/39448
https://review.whamcloud.com/39482

Several of those could use a second review; they are almost all pretty simple.

These three patches are not complete:
https://review.whamcloud.com/39445 (bugs; still trying to work it out...)

https://review.whamcloud.com/39443 (reworking implementation)
https://review.whamcloud.com/39438 (Depends on 39443)

[Patrick Farrell]

Update on status:

I've rebased this series on https://review.whamcloud.com/#/c/44131/ (LU-14805), which was causing what I thought were bugs in https://review.whamcloud.com/39445

It appears, though, that https://review.whamcloud.com/39445 is OK and should be ready for review.

So, updated status:

The following patches (in order) are ready, modulo completion of testing (errors appear unrelated) and review:
https://review.whamcloud.com/39442
https://review.whamcloud.com/39441
https://review.whamcloud.com/39446
https://review.whamcloud.com/39447
https://review.whamcloud.com/39448
[https://review.whamcloud.com/39482
] https://review.whamcloud.com/39445

These patches are not complete:

https://review.whamcloud.com/39443 (reworking implementation)
https://review.whamcloud.com/39438 (Depends on 39443)

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44153
Subject: LU-13799 osc: Always set aio in anchor
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e1c9ec5fef42f45748174177be8b7f285103ffba

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44154
Subject: LU-13799 llite: Simplify cda_no_aio_complete use
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a485a314dc9f8740ee886c64952ccb422feeac99

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44201
Subject: LU-13799 llite: Simplify sync_io references
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ad546d08251c9fc6afe66916154644d19c57315e

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44209
Subject: LU-13799 llite: Implement lower/upper aio
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 27fba688585261d4002b170060ac462eceac0fd8

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44268
Subject: LU-13799 llite: Remove ldp_file_offset
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 36f0b45793b3ae37c4facb46163062a6636ef023

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44293
Subject: LU-13799 llite: Remove unnecessary page get/put
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9b4deec3c7e79688f261f668680cf91135681c4b

[Patrick Farrell]

I wanted to update the on the overall status of this series and related patches.

The patches associated with this ticket should be ready for landing, but they're often failing testing due to bugs which have patches in flight.

So here's a summary of how I'm imagining getting things landed.

First, several bug fixes (in this order):
LU-14805 llite: No locked parallel DIO
[https://review.whamcloud.com/44131/
]
Supporting patches for the next fix:
LU-14838 osc: Remove lockless truncate
https://review.whamcloud.com/44204/
LU-14838 osc: Remove client contention support
[https://review.whamcloud.com/44205/
]The actual fix:

LU-14814 osc: osc: Do not flush on lockless cancel
https://review.whamcloud.com/44152/

Strictly speaking, LU-14838 osc: Remove lockless truncate is not required.  It's just that the required patch - LU-14838 osc: Remove client contention support - has a context dependency on it.  I could resolve that but I think we're going to land the lockless truncate patch anyway, so I didn't bother.  (If anyone feels strongly, I'm happy to rearrange.)

That fixes all the known bugs, and frees us up to consider the patches under this ticket, LU-13799.

That should make it much easier to get those patches through testing.

[Gerrit Updater]

Patrick Farrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44321
Subject: LU-13799 tests: Improve test 398b
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: d3c6b0b8d1c3c2032164f63989b18a8f0933e8ed

[Patrick Farrell]

Current status...

There's a series of patches ready for Oleg & landing, starting here:
https://review.whamcloud.com/44131/
(The second patch is this one: https://review.whamcloud.com/39442/ and I trust Oleg to just follow the sequence from there... heh)

These patches are waiting for review:
https://review.whamcloud.com/39445/
https://review.whamcloud.com/44154/
https://review.whamcloud.com/44153/

The following patches also need review & landing (not attached to this ticket, but fix related bugs):
https://review.whamcloud.com/44204/
https://review.whamcloud.com/44205
https://review.whamcloud.com/44152

There are also a bunch of patches which are reviewed and waiting for Oleg to land.

Once these are in, there is a further set, starting here:
https://review.whamcloud.com/44209/

But I would like to get the other patches landed and that patch & the ones after it rebased, etc, rather than try to include them in one giant drop.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/39442/
Subject: LU-13799 llite: Modify AIO/DIO reference counting
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b3de247b76b410101e166b024d65e2bf23d401ba

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/39441/
Subject: LU-13799 llite: Remove transient page counting
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 587e5aa8342980f761930235714add1cca80686b

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/39446/
Subject: LU-13799 lov: Improve DIO submit
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: d31647c017a390c9553a38d82c02fe7001a33a05

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/39447/
Subject: LU-13799 llite: Adjust dio refcounting
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1e4d10af3909452b0eee1f99010d80aeb01d42a7

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/39448/
Subject: LU-13799 clio: Skip prep for transients
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b8553978789ad3dd0776c0543dea4641804d0ac5

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/39482/
Subject: LU-13799 osc: Improve osc_queue_sync_pages
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 87c4535f7a5d239aad4e936545a72d0199ccd9ba

[Patrick Farrell]

Oleg,

Thanks for getting those merged in.

The latest drop to master covers all of the fixes associated with other tickets that I think are required, so this series should be good to go now.

Here's the current status of the patches here...  I am still hoping to get everything here in to 2.15.  There should not be any more patches added to this ticket at this point.

This patch needs review again, but hasn't much changed:
https://review.whamcloud.com/39445/

After that, patches get a little more complicated.  I've rebased all the remaining patches on to tip of master + the test 398b improvement, to get them some more testing.  (398b improvement: https://review.whamcloud.com/44321/)

Here's the set:
https://review.whamcloud.com/44154/
https://review.whamcloud.com/44153/
https://review.whamcloud.com/44209/

https://review.whamcloud.com/39443/

https://review.whamcloud.com/39438/

https://review.whamcloud.com/44268/

https://review.whamcloud.com/44293/

 

[Patrick Farrell]

A comment on performance.  I have not retested this recently, so definitely take this with some salt.

The currently landed set of patches should put us at around 7 GiB/s.  The lov caching patch should take that to around 8 GiB/s.

The remaining set of patches should push the rest of the way to around 10 GiB/s.

Then that's where this ticket stops and the work is picked up in other tickets.

[Shuichi Ihara]

https://jira.whamcloud.com/secure/attachment/40203/LU-13799.xlsx
This is the performance comparison of master vs master + patch LU-13799 series (https://review.whamcloud.com/#/c/44445/) on the real storage system.
1.6GB/sec vs 13GB/sec Write, 1.5GB/sec vs 14GB/sec Read at xfersize=64m in Lustre stripesize=1m.

[Patrick Farrell]

sihara:
https://review.whamcloud.com/#/c/44445/ is one of the EXA6 versions of these patches.  Did you mean to link a different one?

[Patrick Farrell]

sihara how many OSTs did you have in your system and how many stripes did you use?

lctl set_param osc..max_pages_per_rpc=16M osc..checksums=0 osc.*.max_rpcs_in_flight=16

 

I'm just curious because it's possible some of these were limited by max_rpcs_in_flight.

Still, that performance is excellent - significantly better than I expected.

[Patrick Farrell]

OK, I see in the spreadsheet you used:
https://review.whamcloud.com/#/c/44293/

That is missing:
https://review.whamcloud.com/#/c/39445/

Because I did not put it in the series.  If possible, you might want to retest with that patch added.

[Shuichi Ihara]

I'm just curious because it's possible some of these were limited by max_rpcs_in_flight.

Yeah, i also thought that helps, but didn't help very much. because it still doesn't reach to max_rpcs_in_flight limit somehow. see example.

# lctl set_param osc.*.rpc_stats=clear
# lfs setstripe -c 8 -S 1M /ai400x2/ior.out/
# mpirun -np 1 ior -w -r -t 256m -b 8G -o /ai400x2/ior.out/file --posix.odirect -e
# lctl get_param osc.*.rpc_stats
			read			write
pages per rpc         rpcs   % cum % |       rpcs   % cum %
1:		         0   0   0   |          0   0   0
2:		         0   0   0   |          0   0   0
4:		         0   0   0   |          0   0   0
8:		         0   0   0   |          0   0   0
16:		         0   0   0   |          0   0   0
32:		         0   0   0   |          0   0   0
64:		         0   0   0   |          0   0   0
128:		         0   0   0   |          0   0   0
256:		      1024 100 100   |       1024 100 100

			read			write
rpcs in flight        rpcs   % cum % |       rpcs   % cum %
1:		       142  13  13   |        856  83  83
2:		       873  85  99   |        152  14  98
3:		         5   0  99   |          9   0  99
4:		         2   0  99   |          6   0  99
5:		         1   0  99   |          1   0 100
6:		         1   0 100   |          0   0 100

98-99% of rpcs is still 2 rpcs in flight here.

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/39445/
Subject: LU-13799 lov: Cache stripe offset calculation
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 14db1faa0fbe813fed616435303753d390f45827

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/44154/
Subject: LU-13799 llite: Simplify cda_no_aio_complete use
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b60bd21ec5d5f34ed79c63158860b6f5e948dba2

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/44153/
Subject: LU-13799 osc: Always set aio in anchor
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: eadccb33ac4bbe54a01da5168f6170702f9b2629

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/44209/
Subject: LU-13799 llite: Implement lower/upper aio
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 46ff76137160b66f1d4437b3443859027faae9c4

[Gerrit Updater]

"Patrick Farrell <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/46135
Subject: LU-13799 tests: Reduce I/O sizes
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0bd3f71bdced8c80513199a6ffebed96a4d0fb54

[Gerrit Updater]

"Patrick Farrell <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/46139
Subject: LU-13799 tests: Use fallocate to fill OSTs
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 53aa72a20e7af4cdbfd825a8f35100d901266903

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/39443/
Subject: LU-13799 llite: Move free user pages
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 7f9b8465bc1125e51aa29cdc27db9a9d6fdc0b89

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/39438/
Subject: LU-13799 llite: Do not get/put DIO pages
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 881b4c722296ff7ac22c6fd7988363f2cdad9f1e

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/44293/
Subject: LU-13799 llite: Remove unnecessary page get/put
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c2e94f08cf3ff000b350faf61b6d25ebbad7970e

[Patrick Farrell]

All of the patches have landed to master for this.  Moved the last few test changes to LU-15843.

[Cory Spitz]

> Moved the last few test changes to LU-15843
Typo?
https://jira.whamcloud.com/browse/LU-15843 gives "404 - Not found" and https://review.whamcloud.com/#/q/LU-15843 doesn't turn up anything either.

[Peter Jones]

I think that it's LU-15483