Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-14055

Write performance regression caused by an commit from LU-13344

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • Lustre 2.15.0
    • Lustre 2.14.0
    • None
    • master (commit: 56526a90ae)
    • 3
    • 9223372036854775807

    Description

      commit 76626d6c52 "LU-13344 all: Separate debugfs and procfs handling" caused write performance regression. Here is a reproducer and tested workload.

      Single Client(Ubuntu 18.04, 5.4.0-47-generic), 16MB O_DIRECT, FPP (128 processes)

      # mpirun --allow-run-as-root -np 128 --oversubscribe --mca btl_openib_warn_default_gid_prefix 0 --bind-to none ior -u -w -r
       -k -e -F -t 16384k -b 16384k -s 1000 -u -o /mnt/ai400x/ior.out/file --posix.odirect 
      

      "git bisect" indentified an commit where regression started.

      Here is test results.
      76626d6c52 LU-13344 all: Separate debugfs and procfs handling

      access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
      ------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
      write     21861      1366.33    60.78       16384      16384      0.091573   93.68      40.38      93.68      0   
      read      38547      2409.18    46.14       16384      16384      0.005706   53.13      8.26       53.13      0   
      

      5bc1fe092c LU-13196 llite: Remove mutex on dio read

      access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
      ------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
      write     32678      2042.40    58.96       16384      16384      0.105843   62.67      4.98       62.67      0   
      read      38588      2411.78    45.89       16384      16384      0.004074   53.07      8.11       53.07      0   
      

      master (commit 56526a90ae)

      access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
      ------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
      write     17046      1065.37    119.02      16384      16384      0.084449   120.15     67.76      120.15     0   
      read      38512      2407.00    45.04       16384      16384      0.006462   53.18      9.07       53.18      0   
      

      master still has this regression and when commit 76626d6c52 reverts from master, the performrance is back.

      master (commit 56526a90ae)+ revert commit 76626d6c52

      access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
      ------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
      write     32425      2026.59    59.88       16384      16384      0.095842   63.16      4.79       63.16      0   
      read      39601      2475.09    47.22       16384      16384      0.003637   51.72      5.73       51.72      0  
      

      Attachments

        1. master.svg
          245 kB
        2. master-revert.svg
          414 kB

        Issue Links

          Activity

            People

              paf0186 Patrick Farrell (Inactive)
              sihara Shuichi Ihara
              Votes:
              0 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: