Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-4198

Improve IO performance when using DIRECT IO using libaio

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: Lustre 2.4.1
    • Fix Version/s: Lustre 2.14.0
    • Labels:
    • Environment:
      Seen in two environments. AWS cloud (Robert R.) and a dual-OSS setup (3 SSD per OST) over 2x10 GbE.
    • Rank (Obsolete):
      11385

      Description

      Attached to this Jira are some numbers from the direct IO tests. Write operations only.

      It was noticed that setting RPCs in flight to 256 in these tests gives poorer performance. max rpc here is set to 32.

      • A sample FIO output:
        fio.4k.write.1.23499: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
        fio-2.1.2
        Starting 1 process
        fio.4k.write.1.23499: Laying out IO file(s) (1 file(s) / 10MB)
        
        fio.4k.write.1.23499: (groupid=0, jobs=1): err= 0: pid=10709: Fri Nov  1 11:47:29 2013
          write: io=10240KB, bw=2619.7KB/s, iops=654, runt=  3909msec
            clat (usec): min=579, max=5283, avg=1520.43, stdev=1216.20
             lat (usec): min=580, max=5299, avg=1521.37, stdev=1216.22
            clat percentiles (usec):
             |  1.00th=[  604],  5.00th=[  652], 10.00th=[  668], 20.00th=[  708],
             | 30.00th=[  732], 40.00th=[  756], 50.00th=[  796], 60.00th=[  844],
             | 70.00th=[ 1320], 80.00th=[ 3440], 90.00th=[ 3568], 95.00th=[ 3632],
             | 99.00th=[ 3824], 99.50th=[ 5024], 99.90th=[ 5216], 99.95th=[ 5280],
             | 99.99th=[ 5280]
            bw (KB  /s): min= 1224, max= 4366, per=97.64%, avg=2557.14, stdev=1375.64
            lat (usec) : 750=37.50%, 1000=30.12%
            lat (msec) : 2=5.00%, 4=26.76%, 10=0.62%
          cpu          : usr=0.92%, sys=8.70%, ctx=2562, majf=0, minf=25
          IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
             submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
             complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
             issued    : total=r=0/w=2560/d=0, short=r=0/w=0/d=0
        
        Run status group 0 (all jobs):
          WRITE: io=10240KB, aggrb=2619KB/s, minb=2619KB/s, maxb=2619KB/s, mint=3909msec, maxt=3909msec
        

        Attachments

        1. fio.direct.xls
          13 kB
        2. JinshanPatchesTesting.xlsx
          141 kB
        3. LU-4198.png
          LU-4198.png
          129 kB
        4. vvp_io.c.dio_i_size.patch
          2 kB

          Issue Links

            Activity

              People

              • Assignee:
                bobijam Zhenyu Xu
                Reporter:
                brett Brett Lee (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                23 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: