[LU-3864] stat() on HSM released file returns st_blocks = 0

• Type: Technical task
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.5.0
• Affects Version/s: Lustre 2.5.0
• Labels: HSM
• Assignee: Bruno Faccini (Inactive)
• Reporter: John Hammond

Description

HSM release in not preserving block count as reported by stat()

# cd /mnt/lustre
# dd if=/dev/zero of=Antoshka bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0740321 s, 142 MB/s
# stat Antoshka 
  File: `Antoshka'
  Size: 10485760  	Blocks: 20480      IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 144115205255725060  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2013-08-30 10:13:48.000000000 -0500
Modify: 2013-08-30 10:13:48.000000000 -0500
Change: 2013-08-30 10:13:48.000000000 -0500
# lfs hsm_archive Antoshka 
# lfs hsm_release Antoshka
# stat Antoshka
  File: `Antoshka'
  Size: 10485760  	Blocks: 0          IO Block: 4194304 regular file
Device: 2c54f966h/743766374d	Inode: 144115205255725060  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2013-08-30 10:13:48.000000000 -0500
Modify: 2013-08-30 10:13:48.000000000 -0500
Change: 2013-08-30 10:13:48.000000000 -0500

I had intended to fix this with LU-3811 but it will require some work in the MD* attr_set patch.

If you're thinking (philosophically) hmm, well maybe it should report a block count of 0 here then you're just wrong.

Comments

[Bruno Faccini (Inactive)]

Hello Aurelien,
Seems everybody "agree" with st_blocks to be Null for released files. This is the reason ticket resolution has been set as "won't fix".

Concerning change in FIEMAP, this can be tracked as a new ticket (or still this one), but with lower priority.

BTW, do we know what answer to FIEMAP do provide DMF and/or GHI ??

Seems to me that it is not a frozen interface, because when googling I found that a FIEMAP_EXTENT_SECONDARY flag exists (to indicate that extent data are on HSM) in some implementations that may fit our needs here, but then is it used/tested by the coreutils and other tools ?

But anyway, I wrote a 1st patch attempt that returns a single extent with (FIEMAP_EXTENT_DELALLOC | FIEMAP_EXTENT_UNKNOWN | FIEMAP_EXTENT_LAST). It is http://review.whamcloud.com/7584.

[Aurelien Degremont (Inactive)]

OK, I've checked with GHI (GPFS/HPSS Interface), which is the only concurrent product of Lustre/HSM.
GHI also returns st_blocks == 0 when files are RELEASED.

I propose that FIEMAP is modified for RELEASED files, and st_block still return 0. We will see if lot of people complains about that.

[Aurelien Degremont (Inactive)]

Why this bug as been put as wontfix?

We must, at least, fix FIEMAP.

[Aurelien Degremont (Inactive)]

John,

I agree with you that checking each possible tool is not the right way.

• Users will be confused at first by this, but we should teach them. You cannot really consider this will be really fully transparent for them. I'm pretty sure that users will ask admins why their file accesses are random, because they do not realize that files could restored when accessed and this could requires mounting tapes which could be really long.

• DMF, which is the major HSM in HPC environment, and probably the HSM that will be used the most with Lustre/HSM, is exactly doing this. DMF, on their XFS/CXFS frontend, reports st_block == 0 for released(offline) files. DMF is probably doing this for 20 years now. This is the main solution right now on the market and DMF is very interested by Lustre as a frontend because CXFS does not scale (and because Lustre is very well deployed).

• Admins like they can see this difference between theoretical space and real disk usage through stat/du.

I think this is very simple and that we're being a bit lazy here.

What will be your proposal?

I still consider this behavior could be changed, but this is not as simple as: this is totally wrong to reports 0 for st_blocks. I think reporting st_block as 0 is a feature, not a bug. Only the potential data corruption for some "too-smart" tools is a concern to me.

[Jinshan Xiong (Inactive)]

Hi John, by design, st_blocks is the actual disk blocks consumed by the file data. So it should be reasonable to report 0 for st_blocks for released files, this matches the . But I agree that reporting st_blocks to be zero is confusing here because this may cause applications to think this is a totally sparse file so fiemap won't be consulted at all.

Any decisions for this problem are reasonable to me

[John Hammond]

I think this is very simple and that we're being a bit lazy here.

It is a bug for release to change st_blocks. Remember that HSM is supposed to be transparent to the user. We shouldn't be trying to guess if some application will depend on st_blocks. Instead we should assume that some application depends on st_blocks and that there will be data loss because of this bug. It can be a blocker or not, I don't care so much. But it's a bug.

The fact that we're focused on cp and tar is troubling. What about bbcp, rsync, zip, and all of those terrible data movers that the open science sites use? Have we checked them too?

Moreover, it's very easy to think of situations where regular users will be confused by this. For example, when they run 'du -hs' on their homedir and then tell admin that the 8TB of data they just created must be lost. The admin who uses HSM on the other hand can be told and should know that 'dh -hs' will not always reflect the frontend FS usage.

[Jinshan Xiong (Inactive)]

Hi Aurelien and Bruno,

If we can fix this issue by fiemap that'll be great. I was afraid that we would fix tar issue but import another one by reporting incorrect block number.

Jinshan

[Aurelien Degremont (Inactive)]

> Did I understand and is this what you mean ?

This is perfectly correct!

> Then, only difference I see already is that we will not use the same cp logic for "real" non-sparse (plain or dense) but released files, if any specifics are made in cp mainly for performance but this could be just nothing regarding time for restore.

I do not understand what you mean

What I see is that cp will read sequentially the just-previously-released-but-now-restored file, instead of doing something more efficient because the file could really be "a little bit" sparse.
So before releasing the file, cp would have copied it efficiently, creating a sparse file.
After restore, it will copy it fully, and the file copy will be no more sparse.
I think this is acceptable.

[Bruno Faccini (Inactive)]

Aurelien,
Just to be sure that I fully understand what you mean in your last comment :

_ for a released file, we can return st_blocks = 0 with correct st_size as already.
_ this allows cp (and tar?) to presume it is a sparse-file and go thru related special logic.
_ but then we need to change Lustre FIEMAP to return something like FIEMAP_EXTENT_DELALLOC or FIEMAP_EXTENT_UNKNOWN for released files to make cp happy.

Did I understand and is this what you mean ?

Then, only difference I see already is that we will not use the same cp logic for "real" non-sparse (plain or dense) but released files, if any specifics are made in cp mainly for performance but this could be just nothing regarding time for restore.

[Aurelien Degremont (Inactive)]

Jinshan, even I thought that reporting a 0 value for st_block was a good idea first, we cannot afford to keep doing this due to cp/tar issue.

This is unacceptable that users doing "cp --sparse" or "tar --sparse" copy WRONG data if this is applied to a released file.
After looking at coreutils code and LU-2580 I think the bug here is that FIEMAP output for released file is wrong.
coreutils is just using st_blocks to detect if the file looks sparse.

It seems we can return a special FIEMAP for released file, with one region, with FIEMAP_EXTENT_DELALLOC or FIEMAP_EXTENT_UNKNOWN set.

I've not checked tar.

Andreas, do you think this solution is reasonable?

[Bruno Faccini (Inactive)]

I am late in this thread (even if ticket is assigned to me !), but even if I understand the possible issue when using "smart" tools, I wonder what will happen when using "simple" tools like "du" if st_blocks no longer reflects the current number of consumed blocks ?

Do we know how behave other HSMs, like DataMigration and others ? I seem to remember DM did not do anything tricky with st_blocks, but may be it changed since.