[LU-10026] Client-side data compression

• Type: New Feature
• Resolution: Unresolved
• Priority: Minor
• Fix Version/s: Lustre 2.17.0
• Labels: patch
• Assignee: Artem Blagodarenko
• Reporter: Andreas Dilger

Description

Due to the increasing gap between computational speed, network speed and storage capacity, it has become necessary to investigate data reduction techniques. Storage systems have become a significant part of the total cost of ownership due to the increased amount of storage devices, their associated acquisition cost and energy consumption.

Ultimately, we are aiming for compression support in Lustre at multiple levels:

• Client-side compression allows using the available network and storage capacity more efficiently,
• Client hints empower applications to provide information useful for compression and
• Adaptive compression makes it possible to choose appropriate settings depending on performance metrics and projected benefits.

Compression will be completely transparent to the applications because it will be performed by the client and/or server on their behalf. However, it will be possible for users to tune Lustre's behavior to obtain the best performance/compression/etc. When using client-side compression, the single stream performance bottleneck will directly benefit from the compression. Initial studies have shown that a compression ratio of 1.5 can be achieved for scientific data using lz4.

Issue Links

• is related to: LU-16085 Ubuntu 22.04 sanityn test_106c: supported stx_attributes: got '2030', expected '30'
• is related to: LU-18586 add llapi_find_param_alloc() function
• is related to: LU-16837 interop: client skip unknown component mirror

Comments

[Gerrit Updater]

"Artem Blagodarenko <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/60073
Subject: LU-10026 osc: improve can_merge_pages()
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ca217a3fdb4f605309226329c1ec6aa8fb4f5f2e

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/59877/
Subject: LU-10026 utils: reserve fields for compress find
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 98e2d3230ef02ea41f7999cfe395bb68e69976f1

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/59877
Subject: LU-10026 utils: reserve fields for compress find
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 055e7f126aa0afb97457f9fcb2f9763add343ff5

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57844/
Subject: LU-10026 ptlrpc: verify large allocations are aligned
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4c972aa6696a50ddfbc0dcb3e8db36f4b435fa4e

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57845/
Subject: LU-10026 sptlrpc: don't crash for too-large chunk size
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 110fcc188c60fe60215c53d6a21de0a56bcb1d69

[Gerrit Updater]

"Artem Blagodarenko <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57857
Subject: LU-10026 csdc: fix compr header for recent kernels
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 20c2bb554a7c11f75166eed57736792a62ebacc5

[Gerrit Updater]

"Artem Blagodarenko <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57847
Subject: LU-10026 sec: PTLRPC_MAX_BRW_BITS is maximum pool size
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: dafc36deba7f01b9972f3dc00ded9a6851f583c5

[Gerrit Updater]

"Artem Blagodarenko <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57846
Subject: LU-10026 sec: additional memory page for an object
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e497b3e80a1dda362cfaebd25a0480507bb1a1b4

[Gerrit Updater]

"Artem Blagodarenko <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57845
Subject: LU-10026 sptlrpc: don't crash for too-large chunk size
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2b7ee177786a29f8974afe3849e9cbe3ce1022b1