[LU-6214] HSM released files archive with 'tar --xattr -c' cannot be extracted to lustre

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.7.0
• Fix Version/s: Lustre 2.8.0
• Type: Bug
• Priority: Major
• Reporter: John Hammond
• Assignee: Bruno Faccini (Inactive)
• Resolution: Fixed
• Labels: hsm, lov, patch, tar
• Severity: 3

Description

# echo XXX > f0
# lfs hsm_archive f0
# lfs hsm_state f0
f0: (0x00000009) exists archived, archive_id:1
# lfs hsm_release f0
# lfs hsm_state f0
f0: (0x0000000d) released exists archived, archive_id:1
# tar --xattr -cf /tmp/f0.tar f0
tar: f0: file changed as we read it
# rm f0
# tar --xattr -xf /tmp/f0.tar f0
tar: f0: Cannot write: No data available
tar: Exiting with failure status due to previous errors
# echo $?
2
# ls
f0
# cat f0
cat: f0: No data available

It does not matter if --xattr is used during extraction.

# rm f0
# tar -xf /tmp/f0.tar f0
tar: f0: Cannot write: No data available
tar: Exiting with failure status due to previous errors
# ls
f0
# cat f0
cat: f0: No data available

Extraction is creating the file and giving it a trusted.lov xattr which has a released pattern. From strace:

mknod("f0", 0600)                       = 0
setxattr("f0", "trusted.hsm", "\x00\x00\x00\x00\x0d\x00\x00\x00\x01\x00\x00\x00\x00\x00\x0
0\x00\x15\x00\x00\x00\x01\x00\x00\x00", 24, 0) = 0
setxattr("f0", "trusted.link", "\xdf\xf1\xea\x11\x01\x00\x00\x00,\x00\x00\x00\x00\x00\x00\
x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x14\x00\x00\x00\x02\x00\x00\x00\x07\x00\x00\x00\x0
1\x00\x00\x00\x00f0", 44, 0) = 0
setxattr("f0", "trusted.lov", "\xd0\x0b\xd1\x0b\x01\x00\x00\x80\x0f\x00\x00\x00\x00\x00\x0
0\x00\x00\x04\x00\x00\x02\x00\x00\x00\x00\x00\x10\x00\x00\x00\x00\x00", 32, 0) = 0
setxattr("f0", "trusted.lma", "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x02\x00\x0
0\x00\x0f\x00\x00\x00\x00\x00\x00\x00", 24, 0) = 0
open("f0", O_WRONLY|O_CREAT, 0600)      = 4
write(4, "XXX\n", 4)                    = -1 ENODATA (No data available)

# lfs getstripe f0
f0
lmm_stripe_count:   1
lmm_stripe_size:    1048576
lmm_pattern:        80000001
lmm_layout_gen:     0
lmm_stripe_offset:  0
# getfattr -d -m- f0
# file: f0
trusted.link=0s3/HqEQEAAAAsAAAAAAAAAAAAAAAAAAAAABQAAAACAAAABwAAAAEAAAAAZjA=
trusted.lma=0sAAAAAAAAAAAABAAAAgAAABcAAAAAAAAA
trusted.lov=0s0AvRCwEAAIAXAAAAAAAAAAAEAAACAAAAAAAQAAAAAAA=

The same archive works fine when extracting to an ext4 filesystem.

# cd /tmp
# tar -xf /tmp/f0.tar f0
# cat f0
XXX

Comments

[Andreas Dilger]

It probably doesn't make sense to expose the RELEASED flag to userspace in this way. I'd think that we want the userspace visible attributes of the file to remain unchanged regardless of whether the file is archived or not.

Should we just mask off the RELEASED flag before returning lustre.lov to userspace? Should that layout also inherit the default striping from the parent (LU-6211)?

[Aditya Pandit]

I did the following analysis:
1. Execute the lfs hsm_archive filename. This executes ioctl.
2. The ioctl triggers the copytool an up call driver
3. Went through the copytool in lustre/utils/lhsmtool_posix.c
4. ct_save_stripe and other fuction uses fgetxattr function to retrive the lov_user_md buffer. This buffer contains magic number, pattern, object id, size of stripe, number of stripes offset, layout generation number and unsized array of ost data object.
5. When hsm_release is called it release the space and only keep pointers.
6. The file is archived using normal tar with --xattr it calls fgetxattr which save the xattrs. but note that it would not store the stripe information of data objects.
7. Now in the test case the file is deleted using the lmv_unlink -> ll_unlink -> md_unlink -> mdo_unlink call.
8. Now we have tar file with corrupted xattrs data which we would like to restore using tar --xattrs -xf file.tar file. This tries to apply the layout map and fails.The file is present in the archive.
9. If you perform same operation without --xattr it works perfectly fine.
10. If file is archived, released, tar, restore, untar it does not work because same problem comes while restoring the attributes.
11. If file is tarred before the release operation it works.
12. Might be tar is not enabled to read the striping information from file that is already released.

[Gerrit Updater]

Aditya Pandit ([e-mail]) uploaded a new patch: http://review.whamcloud.com/14086
Subject: LU-6214: ignore hsm special xattr for now
Project: fs/lustre-release
Branch: b2_5
Current Patch Set: 1
Commit: 290f638056fc17613b2028b521933eb1f5c86259

[Aditya Pandit]

I looked more closely at the bug and the flow. flow goes like this
tar -> listxattr -> getxattr. Then while restoring using tar, it has all the xattrs with incorrect parameters where it calls create -> setxattr. Here it tries to set the stripe information. I think ideally tar should not be allowed to manipulate the internal structures of lustre because it is not fully aware of lustre.
lustre manipulates the stripe information in ll_setxattr and also exposes attributes in listxattr, getxattr and setxattr.

How about we separate extended attributes manipulated by lustre and those that use getxattr/setxattr? Put the implementation of manipulation of setxattr/getxattr as ioctls. lhsmtool_posix would call the ioctl for manipulating the attributes.

We can completely filter out the lustre specific attributes in getxattr and setxattr, so that applications like tar would not manipulate lustre specific attributes at all. It would be a normal read,write,getxattr and setxattr operation for tar or any other application that manipulate the xattr. Such applications would only handle the attributes which they have set and not the xattrs of lustre.

[Andreas Dilger]

Being able to backip and restore the stripe parameters with tar is a feature that users asked for. I think the right solution here is to just mask out the "RELEASED" flag from the "lov" xattr when it is set. That avoids the current problem of the new file being marked released when it is being restored, but keeps the ability to backup and restore the file layout. It also makes sense to ignore the HSM xattr as the existing patch 14086 does, but it needs updating as mentioned in the comments there. Also, patches should be submitted to master first.

[Gerrit Updater]

Aditya Pandit ([e-mail]) uploaded a new patch: http://review.whamcloud.com/16060
Subject: LU-6214 llite: HSM released files archive.
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a2af454aeb1d501647be4de127ebaf945de7e721

[Bruno Faccini (Inactive)]

Don't forget to abandon #14086.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/16060/
Subject: LU-6214 llite: tar restore fails for HSM released files.
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b567f5066b7ca2d15d4f492ec205651c9a63ef98

[Peter Jones]

Landed for 2.8