[LU-682] optimization for Lustre-tar on completely sparse files.

• Type: Improvement
• Resolution: Won't Fix
• Priority: Minor
• Assignee: Richard Henwood (Inactive)
• Reporter: Richard Henwood (Inactive)

Description

Kit Westneat commented:

"Older versions of tar have to read in the entire file to figure out
what parts are sparse. Newer versions should skip that if the # of
blocks are 0, but I'm not sure if that made it into lustre-tar yet.

Here's the patch:
http://lists.gnu.org/archive/html/bug-tar/2010-08/msg00043.html
"

This hasn't made it into lustre-tar yet, and may be worth looking into.

Issue Links

• is related to: LU-417 block usage is reported as zero by stat call for tens of seconds after creating a file
• Trackbacks: Lustre Tools In addition to the Lustre code, a number of tools exist to help maintain a Lustre filesystem at

Comments

[Richard Henwood (Inactive)]

I haven't seen the beta, but I have had this confirmed by Red Hat - tar will be 1.23+patches for sparse files (among other I assume).

Once 6.3 is available, then you will have more choices for RHEL6 users: use vanilla Red Hat tar from 6.3, use WC tar, or roll your own gnu tar >1.24.

WC tar was created specifically to target RHEL5 with the requirement: we want to build on the same platform we run on.

WC tar achieves this. If I remember correctly: The problem with bumping the gun tar version is that more recent (>1.23) versions, that have sparse and other patches included, require a version of autoconf (>2.60) that is not readily available on RHEL5.

[Nathan Rutman]

These patches are in RHEL 6.3 beta tar 1.23-7

1. optimization for -c --sparse for completely sparse files (#760665)
   Patch12: tar-1.23-optimize-packing-entirely-sparse-files.patch
2. fix for filename corruption when --sparse and --posix options are used. (#656834)
   Patch9: tar-1.23-long-name-corruption.patch

[Nathan Rutman]

Doe this mean WC's tar should be replaced by mainstream tar 1.24?

[Andreas Dilger]

This is available in the RHEL5 patched lustre-tar, and will be available in RHEL6.3 as well.

[Richard Henwood (Inactive)]

I've been told that RHEL 6.3 will include the completely sparse file optimization patch.

Users on 6.0, 6.1 and 6.2 will be able to use tar from 6.3 when it is available.

[Richard Henwood (Inactive)]

It seems that this patch is up-stream in Gnu tar, starting at version: 1.24

[Richard Henwood (Inactive)]

Andreas adds: "This is useful for 1.8.x MDTs right now, and once Fan Yong has implemented OI Scrub it will also be useful for 2.x MDTs."

[Richard Henwood (Inactive)]

Use Case

A administrator wishes to perform a file-level back up of a MDT.

A MDT on a production file system may have many millions of files. Each of these file will be completely sparse (ST_NBLOCKS is zero). Tar without this patch will scan large completely sparse files even though the blocks are zero. Scanning large, completely sparse files is time-consuming.

[Richard Henwood (Inactive)]

The patch looks something like this:

--- tar-1.19/orig/src/sparse.c
+++ tar-1.19/src/sparse.c
@@ -216,15 +216,17 @@
   struct tar_stat_info *st = file->stat_info;
   int fd = file->fd;
   char buffer[BLOCKSIZE];
-  size_t count;
+  size_t count = 0;
   off_t offset = 0;
   struct sp_array sp = {0, 0};
 
-  if (!lseek_or_error (file, 0))
-    return false;
-
   st->archive_file_size = 0;
   
+  if (ST_NBLOCKS (st->stat) == 0)
+    offset = st->stat.st_size;
+  else
+    {
+
   if (!tar_sparse_scan (file, scan_begin, NULL))
     return false;
 
@@ -254,6 +256,7 @@
 
       offset += count;
     }
+  }
 
   if (sp.numbytes == 0)
     sp.offset = offset;

[Richard Henwood (Inactive)]

RHEL5 tar is now being built with this patch.

A patch for RHEL6 tar will ideally be received upstream.