https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-682 Lustre <p>Kit Westneat commented:</p> <p>"Older versions of tar have to read in the entire file to figure out<br/> what parts are sparse. Newer versions should skip that if the # of<br/> blocks are 0, but I'm not sure if that made it into lustre-tar yet.</p> <p>Here's the patch:<br/> <a href="http://lists.gnu.org/archive/html/bug-tar/2010-08/msg00043.html" class="external-link" target="_blank" rel="nofollow noopener">http://lists.gnu.org/archive/html/bug-tar/2010-08/msg00043.html</a><br/> "</p> <p>This hasn't made it into lustre-tar yet, and may be worth looking into.</p> LU-682

optimization for Lustre-tar on completely sparse files.

Improvement Minor Closed Won't Fix Richard Henwood Richard Henwood Mon, 12 Sep 2011 15:49:49 +0000 Thu, 3 Jan 2013 16:21:40 +0000 Thu, 3 Jan 2013 16:21:37 +0000 0 1 <p>RHEL5 tar is now being built with this patch.</p> <p>A patch for RHEL6 tar will ideally be received upstream.</p> <p>The patch looks something like this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">--- tar-1.19/orig/src/sparse.c +++ tar-1.19/src/sparse.c @@ -216,15 +216,17 @@ struct tar_stat_info *st = file-&gt;stat_info; <span class="code-object">int</span> fd = file-&gt;fd; <span class="code-object">char</span> buffer[BLOCKSIZE]; - size_t count; + size_t count = 0; off_t offset = 0; struct sp_array sp = {0, 0}; - <span class="code-keyword">if</span> (!lseek_or_error (file, 0)) - <span class="code-keyword">return</span> <span class="code-keyword">false</span>; - st-&gt;archive_file_size = 0; + <span class="code-keyword">if</span> (ST_NBLOCKS (st-&gt;stat) == 0) + offset = st-&gt;stat.st_size; + <span class="code-keyword">else</span> + { + <span class="code-keyword">if</span> (!tar_sparse_scan (file, scan_begin, NULL)) <span class="code-keyword">return</span> <span class="code-keyword">false</span>; @@ -254,6 +256,7 @@ offset += count; } + } <span class="code-keyword">if</span> (sp.numbytes == 0) sp.offset = offset; </pre> </div></div> <h3><a name="UseCase"></a>Use Case</h3> <p>A administrator wishes to perform a file-level back up of a MDT.</p> <p>A MDT on a production file system may have many millions of files. Each of these file will be completely sparse (ST_NBLOCKS is zero). Tar without this patch will scan large completely sparse files even though the blocks are zero. Scanning large, completely sparse files is time-consuming.</p> <p>Andreas adds: "This is useful for 1.8.x MDTs right now, and once Fan Yong has implemented OI Scrub it will also be useful for 2.x MDTs."</p> <p>It seems that this patch is up-stream in Gnu tar, starting at version: 1.24</p> <p>I've been told that RHEL 6.3 will include the completely sparse file optimization patch. </p> <p>Users on 6.0, 6.1 and 6.2 will be able to use tar from 6.3 when it is available.</p> <p>This is available in the RHEL5 patched lustre-tar, and will be available in RHEL6.3 as well.</p> <p>Doe this mean WC's tar should be replaced by mainstream tar 1.24?</p> <p>These patches are in RHEL 6.3 beta tar 1.23-7</p> <ol> <li>optimization for -c --sparse for completely sparse files (#760665)<br/> Patch12: tar-1.23-optimize-packing-entirely-sparse-files.patch</li> <li>fix for filename corruption when --sparse and --posix options are used. (#656834)<br/> Patch9: tar-1.23-long-name-corruption.patch</li> </ol> <p>I haven't seen the beta, but I have had this confirmed by Red Hat - tar will be 1.23+patches for sparse files (among other I assume).</p> <p>Once 6.3 is available, then you will have more choices for RHEL6 users: use vanilla Red Hat tar from 6.3, use WC tar, or roll your own gnu tar &gt;1.24.</p> <p>WC tar was created specifically to target RHEL5 with the requirement: we want to build on the same platform we run on.</p> <p>WC tar achieves this. If I remember correctly: The problem with bumping the gun tar version is that more recent (&gt;1.23) versions, that have sparse and other patches included, require a version of autoconf (&gt;2.60) that is not readily available on RHEL5.</p> <p>No longer relevant: tar with sparse is used to file-level backup MDT. Restoring a MDT from a file-level backup is only supported on 2.3 and beyond. 2.3 only supports rhel6. rhel6 tar distribution includes the sparse patch.</p> Related LU-417 Development Rank 1|hzvy7z: Rank (Obsolete) 9728