[LU-8964] use parallel I/O to improve performance on machines with slow single thread performance

• Type: New Feature
• Resolution: Duplicate
• Priority: Major
• Assignee: James A Simmons
• Reporter: Dmitry Eremin (Inactive)

Description

On machines with slow single thread performance like KNL the bottleneck of I/O performance moved into code which just copy memory from one buffer to other (from user space to kernel or vice versa). In current Lustre implementation all I/O performs in single thread and this is become an issue for KNL. Significantly improve performance can be with solution which do parallel memory transfer of large buffers.

 

Issue Links

• is duplicated by: LU-1056 Single-client, single-thread and single-file is limited at 1.5GB/s
• is related to: LU-11069 ifort lseek returns wrong position on lustre 2.10.3
• is related to: LU-10367 FIO Fails to run with libaio
• is related to: LU-9618 Connect readahead to prep_partial_page to improve small (< 1 page) write performance
• is related to: LU-11825 Remove LU-8964/pio feature & supporting framework
• is related to: LU-12043 improve Lustre single thread read performances
• is related to: LU-6658 single stream write performance improvement with worker threads in llite
• is related to: LU-8709 parallel asynchronous readahead

Comments

[Cory Spitz]

simmonsja, for the record, you mean LU-12043. LU-12403 is "add e2fsprog support for RHEL-8".

[James A Simmons]

LU-12403 will do this work correctly.

[James A Simmons]

Thanks Patrick for the heads up on ktask. I will be watching it closely and give it a spin under this ticket.

[Dmitry Eremin (Inactive)]

Thanks for slides. I will loop at them carefully. But for now I disagree that padata API have a big overhead. It's mostly negligible comparing with other overhead to pass work into different thread. But having many threads will leads a sheduler delay to switch under heavy loads. So, I think padata will work more stable and predictable in this case.

[Patrick Farrell (Inactive)]

Also, apologies for not posting these last year.

[Patrick Farrell (Inactive)]

https://www.eofs.eu/_media/events/devsummit17/patrick_farrell_laddevsummit_pio.pdf

This is old and out of date, but I wanted to make sure these slides were seen.  I think the performance of the readahead code would probably be helped a lot by changes to the parallelization framework (as would the performance of pio itself).

So slides 8, 9, and 10 would probably be of particular interest here.  There are significant performance improvements available for PIO just by going from padata to something simpler.  Also, the CPU binding behavior of padata is pretty bad - Binding explicitly to one CPU is problematic.  Padata seems to assume the whole machine is dedicated, which is not a friendly assumption.  (I discovered its CPU binding behavior because I saw performance problems - A particular CPU would be busy and the work assigned to that CPU would be delayed, which delays the completion of the whole i/o.  At this time, other CPUs were idle, and not binding to a specific CPU would have allowed one of them to be used.)

[Patrick Farrell (Inactive)]

Great

[Dmitry Eremin (Inactive)]

The last version of patch don't have an issue with RPC splitting. For reading in my VM machine I have the following:

with PIO disabled:

                        read                    write
pages per rpc         rpcs   % cum % |       rpcs   % cum %
1:                       3   4   4   |          0   0   0
2:                       0   0   4   |          0   0   0
4:                       0   0   4   |          0   0   0
8:                       0   0   4   |          0   0   0
16:                      0   0   4   |          0   0   0
32:                      0   0   4   |          0   0   0
64:                      0   0   4   |          0   0   0
128:                     0   0   4   |          0   0   0
256:                     0   0   4   |          0   0   0
512:                     1   1   6   |          0   0   0
1024:                   62  93 100   |          0   0   0

with PIO enabled:

                        read                    write
pages per rpc         rpcs   % cum % |       rpcs   % cum %
1:                       2   2   2   |          0   0   0
2:                       0   0   2   |          0   0   0
4:                       0   0   2   |          0   0   0
8:                       0   0   2   |          0   0   0
16:                      0   0   2   |          0   0   0
32:                      0   0   2   |          0   0   0
64:                      0   0   2   |          0   0   0
128:                     0   0   2   |          0   0   0
256:                     1   1   4   |          0   0   0
512:                     4   5  10   |          0   0   0
1024:                   61  89 100   |          0   0   0