Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-8964

use parallel I/O to improve performance on machines with slow single thread performance

Details

    • New Feature
    • Resolution: Duplicate
    • Major
    • None
    • None
    • None
    • 9223372036854775807

    Description

      On machines with slow single thread performance like KNL the bottleneck of I/O performance moved into code which just copy memory from one buffer to other (from user space to kernel or vice versa). In current Lustre implementation all I/O performs in single thread and this is become an issue for KNL. Significantly improve performance can be with solution which do parallel memory transfer of large buffers.

       

      Attachments

        1. 40ost_rpc_stats.txt
          91 kB
          Shuichi Ihara
        2. read_readahead_test.c
          3 kB
          Patrick Farrell

        Issue Links

          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. LU-1056 Single-client, single-thread and single-file is limited at 1.5GB/s

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-11069 ifort lseek returns wrong position on lustre 2.10.3

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-10367 FIO Fails to run with libaio

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-9618 Connect readahead to prep_partial_page to improve small (< 1 page) write performance

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-11825 Remove LU-8964/pio feature & supporting framework

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-12043 improve Lustre single thread read performances

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          New Feature - A new feature of the product, which has yet to be developed. LU-6658 single stream write performance improvement with worker threads in llite

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-8709 parallel asynchronous readahead

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            [LU-8964] use parallel I/O to improve performance on machines with slow single thread performance
            spitzcor Cory Spitz added a comment -

            simmonsja, for the record, you mean LU-12043. LU-12403 is "add e2fsprog support for RHEL-8".

            spitzcor Cory Spitz added a comment - simmonsja , for the record, you mean LU-12043 . LU-12403 is "add e2fsprog support for RHEL-8".
            simmonsja James A Simmons made changes -
            Resolution New: Duplicate [ 3 ]
            Status Original: Open [ 1 ] New: Resolved [ 5 ]
            simmonsja James A Simmons added a comment -

            LU-12403 will do this work correctly.

            simmonsja James A Simmons added a comment - LU-12403 will do this work correctly.
            simmonsja James A Simmons made changes -
            Link New: This issue is related to LU-12043 [ LU-12043 ]
            simmonsja James A Simmons made changes -
            Status Original: In Progress [ 3 ] New: Open [ 1 ]
            simmonsja James A Simmons made changes -
            Link New: This issue is related to LU-11825 [ LU-11825 ]
            simmonsja James A Simmons added a comment -

            Thanks Patrick for the heads up on ktask. I will be watching it closely and give it a spin under this ticket.

            simmonsja James A Simmons added a comment - Thanks Patrick for the heads up on ktask. I will be watching it closely and give it a spin under this ticket.
            simmonsja James A Simmons made changes -
            Assignee Original: Dmitry Eremin [ dmiter ] New: James A Simmons [ simmonsja ]
            adilger Andreas Dilger made changes -
            Link New: This issue is related to LU-10367 [ LU-10367 ]
            jhammond John Hammond made changes -
            Link New: This issue is related to LU-11069 [ LU-11069 ]

            People

              simmonsja James A Simmons
              dmiter Dmitry Eremin (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              26 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: