[LU-682] optimization for Lustre-tar on completely sparse files.

• Type: Improvement
• Resolution: Won't Fix
• Priority: Minor
• Assignee: Richard Henwood (Inactive)
• Reporter: Richard Henwood (Inactive)

Description

Kit Westneat commented:

"Older versions of tar have to read in the entire file to figure out
what parts are sparse. Newer versions should skip that if the # of
blocks are 0, but I'm not sure if that made it into lustre-tar yet.

Here's the patch:
http://lists.gnu.org/archive/html/bug-tar/2010-08/msg00043.html
"

This hasn't made it into lustre-tar yet, and may be worth looking into.

Issue Links

• is related to: LU-417 block usage is reported as zero by stat call for tens of seconds after creating a file
• Trackbacks: Lustre Tools In addition to the Lustre code, a number of tools exist to help maintain a Lustre filesystem at

Comments

[Nathan Rutman]

Doe this mean WC's tar should be replaced by mainstream tar 1.24?

[Andreas Dilger]

This is available in the RHEL5 patched lustre-tar, and will be available in RHEL6.3 as well.

[Richard Henwood (Inactive)]

I've been told that RHEL 6.3 will include the completely sparse file optimization patch.

Users on 6.0, 6.1 and 6.2 will be able to use tar from 6.3 when it is available.

[Richard Henwood (Inactive)]

It seems that this patch is up-stream in Gnu tar, starting at version: 1.24

[Richard Henwood (Inactive)]

Andreas adds: "This is useful for 1.8.x MDTs right now, and once Fan Yong has implemented OI Scrub it will also be useful for 2.x MDTs."

[Richard Henwood (Inactive)]

Use Case

A administrator wishes to perform a file-level back up of a MDT.

A MDT on a production file system may have many millions of files. Each of these file will be completely sparse (ST_NBLOCKS is zero). Tar without this patch will scan large completely sparse files even though the blocks are zero. Scanning large, completely sparse files is time-consuming.

[Richard Henwood (Inactive)]

The patch looks something like this:

--- tar-1.19/orig/src/sparse.c
+++ tar-1.19/src/sparse.c
@@ -216,15 +216,17 @@
   struct tar_stat_info *st = file->stat_info;
   int fd = file->fd;
   char buffer[BLOCKSIZE];
-  size_t count;
+  size_t count = 0;
   off_t offset = 0;
   struct sp_array sp = {0, 0};
 
-  if (!lseek_or_error (file, 0))
-    return false;
-
   st->archive_file_size = 0;
   
+  if (ST_NBLOCKS (st->stat) == 0)
+    offset = st->stat.st_size;
+  else
+    {
+
   if (!tar_sparse_scan (file, scan_begin, NULL))
     return false;
 
@@ -254,6 +256,7 @@
 
       offset += count;
     }
+  }
 
   if (sp.numbytes == 0)
     sp.offset = offset;

[Richard Henwood (Inactive)]

RHEL5 tar is now being built with this patch.

A patch for RHEL6 tar will ideally be received upstream.