https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10026 Lustre <p>Due to the increasing gap between computational speed, network speed and storage capacity, it has become necessary to investigate data reduction techniques. Storage systems have become a significant part of the total cost of ownership due to the increased amount of storage devices, their associated acquisition cost and energy consumption.</p> <p>Ultimately, we are aiming for compression support in Lustre at multiple levels:</p> <ul class="alternate" type="square"> <li>Client-side compression allows using the available network and storage capacity more efficiently,</li> <li>Client hints empower applications to provide information useful for compression and</li> <li>Adaptive compression makes it possible to choose appropriate settings depending on performance metrics and projected benefits.</li> </ul> <p>Compression will be completely transparent to the applications because it will be performed by the client and/or server on their behalf. However, it will be possible for users to tune Lustre's behavior to obtain the best performance/compression/etc. When using client-side compression, the single stream performance bottleneck will directly benefit from the compression. Initial studies have shown that a compression ratio of 1.5 can be achieved for scientific data using lz4.</p> LU-10026

Client-side data compression

New Feature Minor Open Unresolved Artem Blagodarenko Andreas Dilger patch Sat, 23 Sep 2017 08:01:41 +0000 Wed, 25 Oct 2023 18:08:49 +0000 Lustre 2.17.0 0 12 <p>Anna Fuchs (anna.fuchs@informatik.uni-hamburg.de) uploaded a new patch: <a href="https://review.whamcloud.com/30393" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30393</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10026" title="Client-side data compression" class="issue-link" data-issue-key="LU-10026">LU-10026</a>: WIP: client-side compression (write path)<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bc288aebb2d07df8d931a7a0a30e188659380473</p> <p>The patch contains a first version of application-transparent client-side compression (write path).</p> <p>The data is compressed on the client, transferred compressed over<br/> the network and decompressed on the object server side. It reaches the<br/> ZFS-backend unmodified and is stored like it was never touched.</p> <p>The current version works with clients and object servers built<br/> with configure option --enable-compression. It supports only new file<br/> writes, not modifying of existing ones.</p> <p>The size of compression pool is currently static. For test purposes, please, change cmp_pool_init(number_of_buffers, size_of_buffer_in_kb) to an appropriate size in osc/osc_request.c:osc_init() and target/tgt_main.c:tgt_mod_init(). Also, set ppchunk (pages per chunk) in osc_request.c:calc_chunks() to a desired value for testing. See TODO 20.</p> <p><b>Approach:</b></p> <p>Compression is introduced on the client side just before the data transfer happens. <br/> The potential RPC is split into chunks. The chunk size is at least 1 page and at most <br/> as large as the preset ZFS record size. In general, chunking is based on stripes.</p> <p>Every chunk is compressed separately and currently serially. Parallel compression within<br/> one RPC is generally possible, but is not prioritized due to already parallel processing <br/> of RPCs. All compressed chunks are concatenated without any gaps in between and sent as 1<br/> RPC over the network. Every chunk is forced to be a separate network I/O buffer. If a chunk<br/> could not be compressed for any uncritical reasons (random data), it stays uncompressed at <br/> the beginning, end or between other potentially compressed chunks in a stripe.</p> <p>In this version, the data is supposed to be decompressed on the server. In future version it <br/> could be passed through the server layers and be persisted compressed on the ZFS-backend. <br/> However, to decompress the data and deliver it to the application, one needs to know at least<br/> which algorithm has been used and, in the most cases, how large the compressed data is and how<br/> large it was. We differ here between physical size (actual byte exact size in memory, compressed) <br/> and the logical size (application view, uncompressed). This information must not be lost after <br/> compression and is currently present in two places. Also see TODOs for storing compressed data.</p> <p>On the one side, it is the chunk descriptor. There exists 1 chunk descriptor per chunk. This <br/> descriptor is to be sent as part of the RPC (together with the body, ioobject, niobuf) before<br/> the transfer. For comfort reasons, it currently contains more information than necessary (see <br/> comments in code), but could be shrunk to fewer fields. Based on that information, every chunk <br/> can be decompressed on the server-side. Very important to notice is that some algorithms like <br/> lz4 require the byte-exact size of compressed data for decompression.</p> <p>If the data is supposed to be stored compressed, we still need that information to be present. <br/> ZFS, which supports its own compression, can already remember the rounded physical size and its <br/> own compression algorithms. This list is to be extended by a value, which indicates external <br/> compression ("data has been compressed by Lustre, don't touch it"). The actually used compression <br/> algorithm and the byte-exact sizes have to be stored as a data header before the actual data. <br/> There exists a header for every compressed chunk. Uncompressed or incompressible chunks do not <br/> have any header and are marked as not externally compressed in ZFS. The size of this header <br/> defines the file layout for compressed data and should not be ever changed for compatibility <br/> reasons in the future. The code for this already exists (and is included in the patch) but is<br/> currently not necessary since data is decompressed by the server.</p> <p>The read path in not affected in this version and the data is read like in an unpatched build.</p> <p>It is not planned to compress any data transfer between clients and MDSs. <br/> The MDS will be involved only for configuration and further settings.</p> <p>This version supports only lz4.</p> <p><b>Technical details:</b></p> <p>Compression forces the use of at least one additional memory buffer for de-/compressed data. <br/> Algorithms like lz4 can handle only contiguous memory buffers for in- and output. The data <br/> in Lustre is accessible page-wise; the pages are commonly not contiguous. Therefore, the original data <br/> has to be first copied into temporary contiguous buffers. This fact increases the memory usage<br/> and forces additional copies. To keep the memory usage minimal, this version uses one temporary <br/> buffer for one chunk (src) and a second large buffer for the output. The output (dst) buffer is <br/> as large as the original data size (page_count). Also see TODOs for page-based compression.</p> <p>Compression process (client-side). Mainly in osc_request.c:compressed_osc_brw_prep_request:</p> <p>1. Input data array (pga), page_count many pages. <br/> 2. Calculate chunks, from 1 page to zfs_record_size large<br/> 3. Prepare 1 src-buffer (sizeof chunk) and many dst-buffesr (sizeof logical_page_count in total)<br/> 4. Copy chunk into src-buffer<br/> 5. Compress src-buffer to dst<span class="error">&#91;chunk&#93;</span> buffer. <br/> 6. Continue composing output data array (cpga) from dst (no additional copy!). <br/> Skip gaps in dst, compressed chunks are consecutively. If chunk was compressible, write a header before compressed data. If chunk was incompressible, use pga pages, write no header and release dst<span class="error">&#91;uncompressible_chunk&#93;</span> immediately<br/> 7. Repeat 4 + 5 for all chunks<br/> 8. Release src-buffer.<br/> 9. Build request with cpga. <br/> 10. Transfer cpga.<br/> 11. After request successfully finished, release compressed pages, try to release uncompressed pages (see TODOs).</p> <p>Transfer. Main changes in tgt_handler.c:tgt_brw_write_compressed:</p> <p>1. Receive RPC. <br/> 2. Prepare remote network buffers for compressed data, recognize physical sizes<br/> 3. Prepare local network buffers for decompressed data, recognize logical sizes<br/> 4. Build bulk descriptor with local network buffers, but physical sizes<br/> 5. Receive compressed (physical size) data into local network buffers<br/> 6. Decompress data from and to local network buffers (see next section)<br/> 7. Commit decompressed data to the backend</p> <p>Decompression process (server-side). Main changes in tgt_handler.c:tgt_brw_write_compressed:</p> <p>1. Input data in lnb<span class="error">&#91;logical_page_count&#93;</span> array. <br/> 2. Prepare 1 src-buffer (sizeof chunk) and many dst-buffers (sizeof logical_page_count in total)<br/> 3. If input chunk is not contiguous, copy chunk from lnb into src-buffer<br/> 4. Decompress src-buffer into dst<span class="error">&#91;chunk&#93;</span> buffer. <br/> 5. When the chunk was not compressible, still need to copy src to dst. <br/> 6. Repeat 3-5 for every chunk. <br/> 7. Copy complete dst to local buffers. <br/> 8. Release src and dst-buffers.</p> <p>Since memory allocation (especially contiguous) in the kernel is not only expensive, but also not<br/> always possible, we have introduced an exclusive memory pool for compression. This module is <br/> initialized at system start, when the osc and the target modules are inserted. At that time most of <br/> the kernel memory is hopefully not in use and we should have the best chances to get big contiguous <br/> memory blocks.</p> <p>The buffers can be loaned from pool only in fixed size. Once you request a smaller buffer, you would <br/> get the bigger fixed size buffer. When you try to request a larger buffer, the request will fail.</p> <p>The current state is work in progress. There is a long list of things to be done. Some of them are <br/> already in progress, some need to be discussed until we can proceed with the development.</p> <p><b>TODOs:</b></p> <p>1. Style check (checkpatch.pl)</p> <p>2. The compression pool already provides the ability to loan several buffers atomically to prevent deadlocks<br/> for large parallel requests. It needs some few fixes for releasing those buffers.</p> <p>3. Discuss whether the chunk descriptor can be present even if the build was configured without compression. Otherwise, isolate everything that affects compression.</p> <p>4. Add/improve error handling. Check return values on failure and prepare fallbacks. Add debugging messages.</p> <p>5. The header size within the data should be discussed.</p> <p>6. Refactoring and merging duplicated functions into original ones as far as possible (compressed_osc_brw_prep_request, tgt_brw_write_compressed, osd_bufs_get_compressed_write, etc.)</p> <p>7. Shrink chunk descriptor to really necessary values, since it is sent over the network</p> <p>8. Add read path.</p> <p>9. Currently there are no compatibility checks for the compression feature. The current version is based on the <br/> 2.10+ state. Clients and object servers have to be configured with enabled compression. The MDS is in this version<br/> not affected by compression and might be just compatible with unpatched 2.10 version. For the future, is also has to<br/> support compression.</p> <p>10. Mixing of different algos of the same type should be possible chunk-wise to enable more flexibility for the future features.</p> <p>11. Currently, compression is only controlled via the configuration setting --enable-compression when building.<br/> We plan to enable/disable it additionally and dynamically (lctl, lfs, /proc).</p> <p>12. Expand compression feature to store the data compressed.</p> <p>13. Release original pga-pages on the client. See comments in osc_request.c:osc_release_cppga</p> <p>14. Memory summary on the server might not recognize all the used memory. See comments in ofd_io.c:ofd_preprw_write_compressed</p> <p>15. Minor TODOs in the code comments</p> <p>16. Write compression tests (build and functionality) for Gerrit</p> <p>17. Support read-modify-write.</p> <p>18. In the near future we will support an algorithm than can handle page arrays as in- and output (pga-able) instead of contiguous buffers (cbuf-able). <br/> We will not need to allocate so many buffers and make additional copies.</p> <p>19. We should restrict mixing of cbuf-able and pga-able algorithms. In theory, mixing all algorithm for every chunk is possible, in practice it results <br/> in complex code and potentially not very smart use of resources. Maybe restrict to one type (cbuf-able of page-able) per file.</p> <p>20. Get/set correct values for <br/> – compression threshold: max(short IO, x)<br/> – chunk size: get from ZFS<br/> – compression pool size: allocate MAX(32, number_of_cpus)*default_RPC_size large pool with record-size sized contiguous chunks<br/> – for future: algorithm (certain one, quick one, best compression ratio, etc.), policy (force, dynamic, etc.) via ladvise</p> <p>21. Dynamic decision making for compression usage, algorithms, strategy, etc.</p> <p>Further, see the project description and updated information at <a href="http://wiki.lustre.org/Enhanced_Adaptive_Compression_in_Lustre" class="external-link" target="_blank" rel="nofollow noopener">http://wiki.lustre.org/Enhanced_Adaptive_Compression_in_Lustre</a></p> <p>Anna Fuchs (anna.fuchs@informatik.uni-hamburg.de) uploaded a new patch: <a href="https://review.whamcloud.com/34603" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34603</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10026" title="Client-side data compression" class="issue-link" data-issue-key="LU-10026">LU-10026</a>: Configuration for compression support and lz4 module backport<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3599c495bb2a430d780ecdd0459270a1c9463820</p> <p>We have taken care of some of the todos and split up the feature into multiple patches.</p> <p>The first of the series adds compression support to the build system and a backport for the lz4 module needed for kernel version &lt;3.11.</p> <p>The remaining patches will follow soon and add support for the page buffer pool, basic data structures, client-side compression and server-side handling.</p> <p>Anna Fuchs (anna.fuchs@informatik.uni-hamburg.de) uploaded a new patch: <a href="https://review.whamcloud.com/34649" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34649</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10026" title="Client-side data compression" class="issue-link" data-issue-key="LU-10026">LU-10026</a>: Page buffer pool for compression<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2446a55b697d744d7f503ce6c1ae9c17ca6d0bd2</p> <p><del>Anna Fuchs (anna.fuchs@informatik.uni-hamburg.de) uploaded a new patch:</del> <a href="https://review.whamcloud.com/34650" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34650</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-10026" title="Client-side data compression" class="issue-link" data-issue-key="LU-10026">LU-10026</a>: Prepare structures for compression</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: 9e39536b65f078d5f2dc3d2455b020451c35ea33</del><br/> (duplicate with 34651)</p> <p>Anna Fuchs (anna.fuchs@informatik.uni-hamburg.de) uploaded a new patch: <a href="https://review.whamcloud.com/34651" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34651</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10026" title="Client-side data compression" class="issue-link" data-issue-key="LU-10026">LU-10026</a>: Prepare structures for compression<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8170c9d07f38e0ead8dfd0db99e380d9691aaf03</p> <p>Anna Fuchs (anna.fuchs@informatik.uni-hamburg.de) uploaded a new patch: <a href="https://review.whamcloud.com/34810" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34810</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10026" title="Client-side data compression" class="issue-link" data-issue-key="LU-10026">LU-10026</a>: Compress data on client with lz4, send compressed over<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c216f2445d829e776ed8db1fe25eb71ee22091bb</p> <p>Anna Fuchs (anna.fuchs@informatik.uni-hamburg.de) uploaded a new patch: <a href="https://review.whamcloud.com/34831" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34831</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10026" title="Client-side data compression" class="issue-link" data-issue-key="LU-10026">LU-10026</a>: Decompress data by server (previously compressed by client) and<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 59b59c6ff201119bc6eb43af5ae280b30a6a4869</p> <p><span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/44027/44027_intel_hub_isc17.pdf" title="intel_hub_isc17.pdf attached to LU-10026">intel_hub_isc17.pdf^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> is the initial description of work to be done.<br/> <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/44028/44028_ipcc_12_17.pdf" title="ipcc_12_17.pdf attached to LU-10026">ipcc_12_17.pdf^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> is the status as of December 2017.</p> <p>Hi Anna, Michael, Matt,<br/> I'm wondering if any of you could share some "real" data input files for testing compression effectiveness, rather than just using e.g. the Linux or Lustre source tree for input? It doesn't need to be huge, tens of MB to single GB is enough, but small parts of it would be included directly into the Lustre source tree for regression testing, so it should be freely available/redistributable. I'm sure there are e.g. weather or CFD or other datasets that can be downloaded, I just don't know where to look for them.</p> <p>Links from Anna with publicly available data:</p> <p><a href="https://www.earthdata.nasa.gov/eosdis/daacs" class="external-link" target="_blank" rel="nofollow noopener">https://www.earthdata.nasa.gov/eosdis/daacs</a><br/> <a href="https://github.com/pangeo-data/WeatherBench" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/pangeo-data/WeatherBench</a></p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49170/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49170/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10026" title="Client-side data compression" class="issue-link" data-issue-key="LU-10026">LU-10026</a> csdc: reserve layout bits for compress component<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 570d76dc3fd677357d76187ba718e339dd695788</p> <p>"Zhenyu Xu &lt;bobijam@hotmail.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51978" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51978</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10026" title="Client-side data compression" class="issue-link" data-issue-key="LU-10026">LU-10026</a> csdc: DoM pattern could be a combined value<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 12f9b0d34cd172374133b159236b8d2c8bc14652</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51978/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51978/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10026" title="Client-side data compression" class="issue-link" data-issue-key="LU-10026">LU-10026</a> csdc: DoM pattern could be a combined value<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: bb0cc84fbed51e006bfac230dada426bfac4f500</p> <p>"Sarah Liu &lt;sarah@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52572" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52572</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10026" title="Client-side data compression" class="issue-link" data-issue-key="LU-10026">LU-10026</a> tests: Fix sanity test_56ab for CSDC<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1ce661dd56fb4b6ecc9e909805c6101bbd9c3161</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52572/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52572/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10026" title="Client-side data compression" class="issue-link" data-issue-key="LU-10026">LU-10026</a> tests: Fix sanity test_56ab for CSDC<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4fc8ca3325d2eb49d8342eb943e3a17932f24517</p> Related LU-16085 LU-16837 Development Rank 1|hzzkpj: Rank (Obsolete) 9223372036854775807