Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-8775

ZFS OST has an extra 128 byte for each IO

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Lustre 2.9.0
    • Labels:
      None
    • Environment:
      ZFS-Lustre: lustre 2.8.57_62_g919224d, zfs-0.7.0
      ldiskfs-lustre: lustre 2.7.16.7
    • Rank (Obsolete):
      9223372036854775807

      Description

      It seems that, when ZFS is used as the backend file system for Lustre OST, there is an additional 128 byte for each IO.

      For example, write 1GB write to a ZFS based file system.

      # dd if=/dev/zero of=File_1GB count=1024 bs=1M
      1024+0 records in
      1024+0 records out
      1073741824 bytes (1.1 GB) copied, 0.869192 s, 1.2 GB/s
      
      [root@kapollo01 ~]# lctl get_param obdfilter.edafs-OST0001.brw_stats
      obdfilter.edafs-OST0001.brw_stats=
      snapshot_time:         1477281605.706719 (secs.usecs)
      
                                 read      |     write
      pages per bulk r/w     rpcs  % cum % |  rpcs        % cum %
      1:                       0   0   0   | 1024  50  50
      2:                       0   0   0   |    0   0  50
      4:                       0   0   0   |    0   0  50
      8:                       0   0   0   |    0   0  50
      16:                      0   0   0   |    0   0  50
      32:                      0   0   0   |    0   0  50
      64:                      0   0   0   |    0   0  50
      128:                     0   0   0   |    0   0  50
      256:                     0   0   0   | 1024  50 100
      
                                 read      |     write
      discontiguous pages    rpcs  % cum % |  rpcs        % cum %
      0:                       0   0   0   | 1024  50  50
      1:                       0   0   0   |    0   0  50
      2:                       0   0   0   |    0   0  50
      3:                       0   0   0   |    0   0  50
      4:                       0   0   0   |    0   0  50
      5:                       0   0   0   |    0   0  50
      6:                       0   0   0   |    0   0  50
      7:                       0   0   0   |    0   0  50
      8:                       0   0   0   |    0   0  50
      9:                       0   0   0   |    0   0  50
      10:                      0   0   0   |    0   0  50
      11:                      0   0   0   |    0   0  50
      12:                      0   0   0   |    0   0  50
      13:                      0   0   0   |    0   0  50
      14:                      0   0   0   |    0   0  50
      15:                      0   0   0   |    0   0  50
      16:                      0   0   0   |    0   0  50
      17:                      0   0   0   |    0   0  50
      18:                      0   0   0   |    0   0  50
      19:                      0   0   0   |    0   0  50
      20:                      0   0   0   |    0   0  50
      21:                      0   0   0   |    0   0  50
      22:                      0   0   0   |    0   0  50
      23:                      0   0   0   |    0   0  50
      24:                      0   0   0   |    0   0  50
      25:                      0   0   0   |    0   0  50
      26:                      0   0   0   |    0   0  50
      27:                      0   0   0   |    0   0  50
      28:                      0   0   0   |    0   0  50
      29:                      0   0   0   |    0   0  50
      30:                      0   0   0   |    0   0  50
      31:                      0   0   0   | 1024  50 100
      
                                 read      |     write
      disk I/Os in flight    ios   % cum % |  ios         % cum %
      1:                       0   0   0   | 2039  99  99
      2:                       0   0   0   |    9   0 100
      
                                 read      |     write
      I/O time (1/1000s)     ios   % cum % |  ios         % cum %
      
                                 read      |     write
      disk I/O size          ios   % cum % |  ios         % cum %
      128:                     0   0   0   | 1024  50  50
      256:                     0   0   0   |    0   0  50
      512:                     0   0   0   |    0   0  50
      1K:                      0   0   0   |    0   0  50
      2K:                      0   0   0   |    0   0  50
      4K:                      0   0   0   |    0   0  50
      8K:                      0   0   0   |    0   0  50
      16K:                     0   0   0   |    0   0  50
      32K:                     0   0   0   |    0   0  50
      64K:                     0   0   0   |    0   0  50
      128K:                    0   0   0   |    0   0  50
      256K:                    0   0   0   |    0   0  50
      512K:                    0   0   0   |    0   0  50
      1M:                      0   0   0   | 1024  50 100
      

      On the other side, such 128 bytes additional IO does not exist.

      [root@kcli01 ost01]# dd if=/dev/zero of=File_1GB count=1024 bs=1M
      1024+0 records in
      1024+0 records out
      1073741824 bytes (1.1 GB) copied, 1.79642 s, 598 MB/s
      
      [root@koss02 ~]# lctl get_param obdfilter.krakenfs-OST0001.brw_stats
      obdfilter.krakenfs-OST0001.brw_stats=
      snapshot_time:         1477297530.705071 (secs.usecs)
      
                                 read      |     write
      pages per bulk r/w     rpcs  % cum % |  rpcs        % cum %
      1K:                      0   0   0   |  256 100 100
      
                                 read      |     write
      discontiguous pages    rpcs  % cum % |  rpcs        % cum %
      0:                       0   0   0   |  256 100 100
      
                                 read      |     write
      discontiguous blocks   rpcs  % cum % |  rpcs        % cum %
      0:                       0   0   0   |  256 100 100
      
                                 read      |     write
      disk fragmented I/Os   ios   % cum % |  ios         % cum %
      4:                       0   0   0   |  255  99  99
      5:                       0   0   0   |    0   0  99
      6:                       0   0   0   |    1   0 100
      
                                 read      |     write
      disk I/Os in flight    ios   % cum % |  ios         % cum %
      1:                       0   0   0   |    4   0   0
      2:                       0   0   0   |    5   0   0
      3:                       0   0   0   |    5   0   1
      4:                       0   0   0   |    5   0   1
      5:                       0   0   0   |    2   0   2
      6:                       0   0   0   |    2   0   2
      7:                       0   0   0   |    1   0   2
      8:                       0   0   0   |    1   0   2
      9:                       0   0   0   |    1   0   2
      10:                      0   0   0   |    1   0   2
      11:                      0   0   0   |    1   0   2
      12:                      0   0   0   |    1   0   2
      13:                      0   0   0   |    3   0   3
      14:                      0   0   0   |   11   1   4
      15:                      0   0   0   |   23   2   6
      16:                      0   0   0   |   44   4  10
      17:                      0   0   0   |   81   7  18
      18:                      0   0   0   |  118  11  30
      19:                      0   0   0   |  144  14  44
      20:                      0   0   0   |  133  12  57
      21:                      0   0   0   |  104  10  67
      22:                      0   0   0   |   72   7  74
      23:                      0   0   0   |   45   4  78
      24:                      0   0   0   |   40   3  82
      25:                      0   0   0   |   42   4  86
      26:                      0   0   0   |   42   4  90
      27:                      0   0   0   |   37   3  94
      28:                      0   0   0   |   29   2  97
      29:                      0   0   0   |   16   1  98
      30:                      0   0   0   |    8   0  99
      31:                      0   0   0   |    5   0 100
      
                                 read      |     write
      I/O time (1/1000s)     ios   % cum % |  ios         % cum %
      4:                       0   0   0   |    3   1   1
      8:                       0   0   0   |    0   0   1
      16:                      0   0   0   |   13   5   6
      32:                      0   0   0   |  172  67  73
      64:                      0   0   0   |   54  21  94
      128:                     0   0   0   |   14   5 100
      
                                 read      |     write
      disk I/O size          ios   % cum % |  ios         % cum %
      512K:                    0   0   0   |    2   0   0
      1M:                      0   0   0   | 1024  99 100
       
      

        Attachments

          Activity

            People

            • Assignee:
              bzzz Alex Zhuravlev
              Reporter:
              zhiqi Zhiqi Tao (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: