[LU-13397] lfs migrate/mirror extend/resync does not preserve sparse file

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.15.0
• Labels: HPv3 HPv4p1
• Severity: 3
• Epic Link: Hot Pools - v4 phase 1
• Assignee: Mikhail Pershin
• Reporter: Andreas Dilger

Description

While testing "lfs migrate", "lfs mirror extend", and "lfs mirror resync", I was doing high-offset writes to initialize later PFL components of the file layout (e.g. at 1GB offset and 32GB offset). When trying to mirror/resync those files to another mirror copy, it resulted in the commands failing due to ENOSPC because there was not enough free space in the test filesystem to write out the data.

These commands should be updated to share a common "data copy" routine (if they don't already) to reduce code duplication when fixing these issues. Then, the code needs to handle sparse input files properly, first by checking for sparse files (e.g. blocks << size) to enable checking the source file, and not copying holes in the file. There should probably be options added for each command like "--sparse=<auto,no,yes>" (default = auto) to force a specific behavior.

Unfortunately, there is no optimal way to handle reading of sparse files in Lustre today. In all cases, it makes little sense to be doing these operations on in-use files, so there are already checks if the file is modified during migrate/mirror/resync.

• For 1-stripe files, the ioctl(FIEMAP) will return a current map of data for the file (it flushes data if FIEMAP_FLAG_SYNC is used, but doesn't prevent further modification). Multi-stripe and PFL files return multiple maps in per-object offset order, and that is not useful if the files are using different layouts (likely a common case). Also, ZFS does not yet support FIEMAP despite some efforts in that direction.
• Using SEEK_HOLE and SEEK_DATA would be the preferred solution, but this needs a Lustre-level update to pass these through from the client to the OST (and MDT for DoM). This is described in LU-10801, and may be able to leverage some infrastructure from the patch https://review.whamcloud.com/9275 "LU-3606 fallocate: Implement fallocate preallocate operation". While SEEK_HOLE and SEEK_DATA "work" for Lustre by kernel emulation, they just assume that every block is "data" and the first hole is the end of the file.
• The simplest (though least efficient) option would be to do zero-block detection during the copy phase. This has quite high CPU and IO overhead, because it requires reading the whole file and checking every byte. It would only be done if the file appears to be very sparse, and the layout is complex (i.e. not single stripe where FIEMAP is useful).

Issue Links

• is duplicated by: LU-10810 SEEK_HOLE and SEEK_DATA support for lseek
• is related to: LU-3833 HSM POSIX copytool sparse file handling
• is related to: LU-14176 lfs mirror resync may truncate file to wrong size
• is related to: LU-14160 Implement fallocate FALLOCATE_FL_PUNCH_HOLE support

Comments

[Gerrit Updater]

"Etienne AUJAMES <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50329
Subject: LU-13397 lfs: mirror extend/copy keeps sparseness
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: d36e3f1a5599eb557e07550651d855c19d18ca36

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/44721/
Subject: LU-13397 llite: support fallocate() on selected mirror
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 89736d502cc99f095237dde7520fc4ca86191882

[Gerrit Updater]

"Mike Pershin <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/44721
Subject: LU-13397 llite: support fallocate() on selected mirror
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 84ab0da804fda75262dd0022528d0043cb3e4558

[Mikhail Pershin]

I've found the problem with mirror resync and sparse file, it is not working as intended, so I am keeping ticket opened.

The problem is in fallocate punch operation during resync. If it is done prior write operation then write itself fails due to changed layout version.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/40773/
Subject: LU-13397 lfs: mirror resync to keep sparseness
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4126fbb30c125050ea2e1fdf3d446201b826ce29

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/40772/
Subject: LU-13397 lfs: mirror extend/copy keeps sparseness
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 0561c144cc1bb623e05d08b5055009e8d86047f4

[Li Xi]

"LU-14174 lfs: llapi_mirror_find() return code check" is also in https://review.whamcloud.com/#/c/40773/9

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/40773
Subject: LU-13397 lfs: mirror resync to keep sparseness
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e916affaabd65e55222431eebfaa92aa6d54989b

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/40772
Subject: LU-13397 lfs: mirror extend/copy keeps sparseness
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5b38421a548116edb5e45d3cdbd0fd6d2f09e314

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/40530
Subject: LU-13397 hsm: lhsmtool to handle sparse files
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ae180a1080dc1cb3990d8f53caee95e11a160248