Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-10026

Client-side data compression

    XMLWordPrintable

Details

    • New Feature
    • Resolution: Unresolved
    • Minor
    • Lustre 2.17.0
    • None
    • 9223372036854775807

    Description

      Due to the increasing gap between computational speed, network speed and storage capacity, it has become necessary to investigate data reduction techniques. Storage systems have become a significant part of the total cost of ownership due to the increased amount of storage devices, their associated acquisition cost and energy consumption.

      Ultimately, we are aiming for compression support in Lustre at multiple levels:

      • Client-side compression allows using the available network and storage capacity more efficiently,
      • Client hints empower applications to provide information useful for compression and
      • Adaptive compression makes it possible to choose appropriate settings depending on performance metrics and projected benefits.

      Compression will be completely transparent to the applications because it will be performed by the client and/or server on their behalf. However, it will be possible for users to tune Lustre's behavior to obtain the best performance/compression/etc. When using client-side compression, the single stream performance bottleneck will directly benefit from the compression. Initial studies have shown that a compression ratio of 1.5 can be achieved for scientific data using lz4.

      Attachments

        1. BGI-Intel-QAT-Performance-2016-09.pdf
          488 kB
          Andreas Dilger
        2. compression-lz4-chunksize.png
          90 kB
          Andreas Dilger
        3. intel_hub_isc17.pdf
          271 kB
          Sebastien Buisson
        4. ipcc_12_17.pdf
          424 kB
          Sebastien Buisson
        5. LAD2016-compression_kuhn.pdf
          1.34 MB
          Andreas Dilger
        6. Lustre-client-compression-poster_ISC17.pdf
          425 kB
          Andreas Dilger

        Issue Links

          Activity

            People

              ablagodarenko Artem Blagodarenko
              adilger Andreas Dilger
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated: