[LU-4198] Improve IO performance when using DIRECT IO using libaio

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.14.0
• Affects Version/s: Lustre 2.4.1
• Labels: clio
• Environment: Seen in two environments. AWS cloud (Robert R.) and a dual-OSS setup (3 SSD per OST) over 2x10 GbE.
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Brett Lee (Inactive)

Description

Attached to this Jira are some numbers from the direct IO tests. Write operations only.

It was noticed that setting RPCs in flight to 256 in these tests gives poorer performance. max rpc here is set to 32.

• A sample FIO output:

  fio.4k.write.1.23499: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
  fio-2.1.2
  Starting 1 process
  fio.4k.write.1.23499: Laying out IO file(s) (1 file(s) / 10MB)
  
  fio.4k.write.1.23499: (groupid=0, jobs=1): err= 0: pid=10709: Fri Nov  1 11:47:29 2013
    write: io=10240KB, bw=2619.7KB/s, iops=654, runt=  3909msec
      clat (usec): min=579, max=5283, avg=1520.43, stdev=1216.20
       lat (usec): min=580, max=5299, avg=1521.37, stdev=1216.22
      clat percentiles (usec):
       |  1.00th=[  604],  5.00th=[  652], 10.00th=[  668], 20.00th=[  708],
       | 30.00th=[  732], 40.00th=[  756], 50.00th=[  796], 60.00th=[  844],
       | 70.00th=[ 1320], 80.00th=[ 3440], 90.00th=[ 3568], 95.00th=[ 3632],
       | 99.00th=[ 3824], 99.50th=[ 5024], 99.90th=[ 5216], 99.95th=[ 5280],
       | 99.99th=[ 5280]
      bw (KB  /s): min= 1224, max= 4366, per=97.64%, avg=2557.14, stdev=1375.64
      lat (usec) : 750=37.50%, 1000=30.12%
      lat (msec) : 2=5.00%, 4=26.76%, 10=0.62%
    cpu          : usr=0.92%, sys=8.70%, ctx=2562, majf=0, minf=25
    IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
       complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
       issued    : total=r=0/w=2560/d=0, short=r=0/w=0/d=0
  
  Run status group 0 (all jobs):
    WRITE: io=10240KB, aggrb=2619KB/s, minb=2619KB/s, maxb=2619KB/s, mint=3909msec, maxt=3909msec
  

Issue Links

• is duplicated by: LU-13786 Take server-side locks for direct i/o
• is related to: LU-13900 don't call aio_complete() in lustre upon errors
• is related to: LU-247 Lustre client slow performance on BG/P IONs: unaligned DIRECT_IO
• is related to: LU-13697 short io for AIO
• is related to: LU-9409 Lustre small IO write performance improvement
• is related to: LU-10278 lfs migrate: make use of direct i/o optional
• is related to: LU-13801 Enable io_uring interface for Lustre client
• is related to: LU-12687 Fast ENOSPC on direct I/O
• is related to: LU-13798 Improve direct i/o performance with multiple stripes: Submit all stripes of a DIO and then wait

Comments

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39733
Subject: LU-4198 clio: turn on lockless for some kind of IO
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 2acd73b09c86ca7ee436152274f9d1beab1ad571

[Wang Shilong (Inactive)]

nrutman

I think LU-13798 will parallel DIO with stripes enabled, and we see big improvements with that, so i guess you could take a look there.

[Andreas Dilger]

Nathan, I think regardless of how this ticket started, it ended up being used to land the AIO/DIO support for 2.14. If there are still issues that need to be addressed, they should be done in the context of a new ticket.

[Nathan Rutman]

Can someone please summarize what the state of this ticket is? The subject seems to have wandered from "Additional stripes on a file does not increase IO performance when using DIRECT IO" to lockless DIO to AIO. Johann's and Jinshan's comments seem to be at odds as to whether DIO stripes are parallelized or not.

Rough testing (DIO, not AIO) seems to indicate they are not.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/37824/
Subject: LU-4198 clio: Remove pl_owner
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e880a6fccbc57ae335949c1dd20335359b1cb220

[Gerrit Updater]

Wang Shilong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37621
Subject: LU-4198 clio: return error for short direct IO
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9913a5411bb305d617ed3bba6bd5d000ffc11121

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/32416/
Subject: LU-4198 clio: AIO support for direct IO
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: d1dded6e28473d889a9b24b47cbc804f90dd2956

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/8201/
Subject: LU-4198 clio: turn on lockless for some kind of IO
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 6bce536725efd166d2772f13fe954f271f9c53b8

[Gerrit Updater]

Wang Shilong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34774
Subject: LU-4198 llite: transient page simplification
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 704390928ce55173d2b2fca0e0fe244907d750b2

[Shuichi Ihara]

Here is test resutls of patch https://review.whamcloud.com/32416
LU-4198.png

Client
2 x E5-2650v4@2.20GHz, 128GB memory, 1 x EDR
OSS/MDS
DDN AI200(2xOSS/MDS, 20 x NVMe, 1 x EDR, master branch)

Without patch, we only get 80K IOPS at 4k random read with DIO even increased number of threads. Here is fio parameters.

[randread]
ioengine=sync
;ioengine=libaio
rw=randread
blocksize=4096
iodepth=32
direct=1
size=1g
runtime=120
numjobs=128
group_reporting
directory=/cache0/fio.out
filename_format=f.$jobnum.$filenum

With aio patch https://review.whamcloud.com/32416, it could readch more than 600K IOPS per an client.
patch helps not only supporting libaio on Lustre, but also benchmark (e.g. we only very small number of clients to satulate storage IOPS) and libaio supoprted applications. (e.g. database, virtual machine enviorment).
One thing I need to inform here, but patch worked well only at less number of max_rpcs_in_flight. For instance, max_rpcs_in_flight=1 was scaling very well so far, but max_rpcs_in_flight=256 was problematic and didn't scale anything.