Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-4198

Improve IO performance when using DIRECT IO using libaio

Details

    • Improvement
    • Resolution: Fixed
    • Minor
    • Lustre 2.14.0
    • Lustre 2.4.1
    • Seen in two environments. AWS cloud (Robert R.) and a dual-OSS setup (3 SSD per OST) over 2x10 GbE.
    • 3
    • 11385

    Description

      Attached to this Jira are some numbers from the direct IO tests. Write operations only.

      It was noticed that setting RPCs in flight to 256 in these tests gives poorer performance. max rpc here is set to 32.

      • A sample FIO output:
        fio.4k.write.1.23499: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
        fio-2.1.2
        Starting 1 process
        fio.4k.write.1.23499: Laying out IO file(s) (1 file(s) / 10MB)
        
        fio.4k.write.1.23499: (groupid=0, jobs=1): err= 0: pid=10709: Fri Nov  1 11:47:29 2013
          write: io=10240KB, bw=2619.7KB/s, iops=654, runt=  3909msec
            clat (usec): min=579, max=5283, avg=1520.43, stdev=1216.20
             lat (usec): min=580, max=5299, avg=1521.37, stdev=1216.22
            clat percentiles (usec):
             |  1.00th=[  604],  5.00th=[  652], 10.00th=[  668], 20.00th=[  708],
             | 30.00th=[  732], 40.00th=[  756], 50.00th=[  796], 60.00th=[  844],
             | 70.00th=[ 1320], 80.00th=[ 3440], 90.00th=[ 3568], 95.00th=[ 3632],
             | 99.00th=[ 3824], 99.50th=[ 5024], 99.90th=[ 5216], 99.95th=[ 5280],
             | 99.99th=[ 5280]
            bw (KB  /s): min= 1224, max= 4366, per=97.64%, avg=2557.14, stdev=1375.64
            lat (usec) : 750=37.50%, 1000=30.12%
            lat (msec) : 2=5.00%, 4=26.76%, 10=0.62%
          cpu          : usr=0.92%, sys=8.70%, ctx=2562, majf=0, minf=25
          IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
             submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
             complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
             issued    : total=r=0/w=2560/d=0, short=r=0/w=0/d=0
        
        Run status group 0 (all jobs):
          WRITE: io=10240KB, aggrb=2619KB/s, minb=2619KB/s, maxb=2619KB/s, mint=3909msec, maxt=3909msec
        

      Attachments

        1. fio.direct.xls
          13 kB
          Brett Lee
        2. JinshanPatchesTesting.xlsx
          141 kB
          Brett Lee
        3. LU-4198.png
          129 kB
          Shuichi Ihara
        4. vvp_io.c.dio_i_size.patch
          2 kB
          Patrick Farrell

        Issue Links

          Activity

            [LU-4198] Improve IO performance when using DIRECT IO using libaio
            adilger Andreas Dilger made changes -
            Link New: This issue is related to LU-13798 [ LU-13798 ]
            wshilong Wang Shilong (Inactive) made changes -
            Link New: This issue is related to LU-13900 [ LU-13900 ]
            adilger Andreas Dilger made changes -
            Link New: This issue is related to LU-13801 [ LU-13801 ]
            zam Alexander Zarochentsev made changes -
            Link New: This issue is duplicated by LU-13786 [ LU-13786 ]
            wshilong Wang Shilong (Inactive) made changes -
            Link New: This issue is related to LU-13697 [ LU-13697 ]
            adilger Andreas Dilger made changes -
            Summary Original: Additional stripes on a file does not increase IO performance when using DIRECT IO New: Improve IO performance when using DIRECT IO using libaio
            adilger Andreas Dilger made changes -
            Issue Type Original: Bug [ 1 ] New: Improvement [ 4 ]
            adilger Andreas Dilger made changes -
            Fix Version/s New: Lustre 2.14.0 [ 14490 ]
            Resolution New: Fixed [ 1 ]
            Status Original: Reopened [ 4 ] New: Resolved [ 5 ]
            adilger Andreas Dilger made changes -
            Link New: This issue is related to LU-12687 [ LU-12687 ]
            pjones Peter Jones made changes -
            Link New: This issue is related to EX-722 [ EX-722 ]

            People

              bobijam Zhenyu Xu
              brett Brett Lee (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: