[LU-16374] Implement backup/restore of encrypted files Created: 08/Dec/22 Updated: 14/Nov/23 |
|
| Status: | Open |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.16.0 |
| Fix Version/s: | None |
| Type: | New Feature | Priority: | Minor |
| Reporter: | Sebastien Buisson | Assignee: | Sebastien Buisson |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Issue Links: |
|
||||||||||||
| Rank (Obsolete): | 9223372036854775807 | ||||||||||||
| Epic Link: | Client side Encrypted backup/restore | ||||||||||||
| Description |
|
Implement backup/restore of encrypted files, according the the HLD as available at https://datadirectnetworks-my.sharepoint.com/:w:/g/personal/sbuisson_ddn_com/EeWD3Q7Ku69Anntda03QPDUBs6oxRCxtlxELM7xxy-S1qQ This effort can be divided into the following steps:
|
| Comments |
| Comment by Gerrit Updater [ 14/Dec/22 ] |
|
"Sebastien Buisson <sbuisson@ddn.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49410 |
| Comment by Gerrit Updater [ 20/Dec/22 ] |
|
"Sebastien Buisson <sbuisson@ddn.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49456 |
| Comment by Gerrit Updater [ 09/Jan/23 ] |
|
"Sebastien Buisson <sbuisson@ddn.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49581 |
| Comment by Gerrit Updater [ 31/Jan/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49581/ |
| Comment by Andreas Dilger [ 08/Feb/23 ] |
|
Sebastien, can you please add a test case for the bas64 vs. base64url compatibility. Best would be to update test32_newtarball() to add a 2.14.0 directory with data-only encryption and a 2.15.0 directory with filename encryption (with base64 name encoding) enabled. The test_32 script should contain an "ls -l" of the encrypted directories to verify they are all unchanged. This listing would fail for the 2.15.0 filename encrypted directory with a master client due to the base64url encoding, unless llite.*.filename_enc_use_old_base64=1 is set. |
| Comment by Gerrit Updater [ 08/Feb/23 ] |
|
"Sebastien Buisson <sbuisson@ddn.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49945 |
| Comment by Gerrit Updater [ 21/Mar/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49410/ |
| Comment by Gerrit Updater [ 06/Apr/23 ] |
|
"Sebastien Buisson <sbuisson@ddn.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50561 |
| Comment by Gerrit Updater [ 11/Apr/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49945/ |
| Comment by Andreas Dilger [ 16/Apr/23 ] |
|
I noticed while running strace tar that it is always opening files with the equivalent of O_FILE_ENC: #define O_FILE_ENC (O_NOCTTY | O_NDELAY) openat(AT_FDCWD, "/mnt/testfs/sparse", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC) = 3 because of include/uapi/asm-generic/fcntl.h: #ifndef O_NDELAY #define O_NDELAY O_NONBLOCK #endif I think this is mostly saved by the additional requirement for O_DIRECT (which is not part of O_FILE_ENC, but should be). We should consider changing this to use a different flag combination for O_FILE_ENC to avoid the risk of accidental issues if tar ever starts reading with O_DIRECT before we can get the xattr handling implemented. One possibility includes O_DSYNC, which doesn't make much sense for O_RDONLY files, but would force writes on encrypted restore to be synchronous. With O_DIRECT and large enough writes (32MB?) that might be OK, but not ideal for small files. Another possibility is O_APPEND, which would prevent multiple writers to an encrypted file during restore, but would otherwise be OK. I haven't looked at any of the options in detail yet, just wanted to point out this issue. |
| Comment by Gerrit Updater [ 12/Jul/23 ] |
|
"Sebastien Buisson <sbuisson@ddn.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51640 |
| Comment by Andreas Dilger [ 02/Aug/23 ] |
|
For reference, there was a discussion about fscrypt file backup and restore on the linux-ext4, linux-fscrypt, and linux-fsdevel mailing lists in "Backup/restore of fscrypt files and directories" and this also referenced other discussions "backup/restore of fscrypt files" and "fs: interface for directly reading/writing compressed data". |
| Comment by Gerrit Updater [ 19/Aug/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51640/ |
| Comment by Gerrit Updater [ 31/Aug/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49456/ |