[LU-12275] Client-side file data encryption Created: 09/May/19 Updated: 28/Mar/23 Resolved: 19/Sep/20 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.14.0 |
| Fix Version/s: | Lustre 2.14.0 |
| Type: | New Feature | Priority: | Critical |
| Reporter: | Sebastien Buisson | Assignee: | Sebastien Buisson |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | encryption, sec | ||
| Attachments: |
|
||||||||||||||||||||||||||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||
| Rank (Obsolete): | 9223372036854775807 | ||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
This ticket is a place-holder to describe work to be done for client-side encryption. The high-level requirements are the following:
We are proposing to address these requirements by:
So the workflow would be the following:
Further details will be added as the feature design makes progress. |
| Comments |
| Comment by Sebastien Buisson [ 28/May/19 ] |
|
lustre_encryption_threat_model.txt is the description of the threat model for Lustre client-side encryption. |
| Comment by Sebastien Buisson [ 28/May/19 ] |
|
lustre_encryption_key_hierarchy.txt is the description of the key hierarchy for Lustre client-side encryption. |
| Comment by Sebastien Buisson [ 28/May/19 ] |
|
lustre_encryption_modes_usage.txt is the description of the encryption modes and usage for Lustre client-side encryption. |
| Comment by Sebastien Buisson [ 28/May/19 ] |
|
lustre_encryption_access_semantics.txt is the description of the access semantics for Lustre client-side encryption. |
| Comment by Andreas Dilger [ 04/Jun/19 ] |
It would be better to not tie the encryption directly to the filesystem blocksize, if possible, as this would break the network transparency of the filesystem compared to the backend storage. Also, for ZFS, the blocksize may be up to 1MB and possibly variable per file and per stripe of the file, while the ext4 blocksize is only 4KB. Instead, if we can pick a fixed blocksize for the crypto like 32KB, similar to how the data compression is done. Instead of using the block number as the IV, it should use the byte offset as the IV, since this is agnostic of the blocksize. The drawback is that we may have to write/allocate a larger portion of the files data (rounded up to 32KB or similar), but this would already be true if we are using a feature like bigalloc on ext4, or recordsize for ZFS. |
| Comment by Andreas Dilger [ 04/Jun/19 ] |
I thought this was one of the features we were interested in having? |
| Comment by Sebastien Buisson [ 05/Jun/19 ] |
|
Being able to encrypt/decrypt a directory with multiple different keys is definitely something we would like to propose. But if we want to have it directly at the Lustre level, it means we cannot rely on fscrypt. However, basing the Lustre client encryption feature on fscrypt kernel API means we would be able to leverage the fscrypt userspace tool https://github.com/google/fscrypt. This tool enables to manipulate encryption policies on directories, and makes use of key wrapping. One of the benefits of this technique is to support using multiple protectors for a policy: |
| Comment by Gerrit Updater [ 10/Sep/19 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/36143 |
| Comment by Gerrit Updater [ 10/Sep/19 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/36144 |
| Comment by Gerrit Updater [ 10/Sep/19 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/36145 |
| Comment by Gerrit Updater [ 10/Sep/19 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/36146 |
| Comment by Patrick Farrell (Inactive) [ 10/Sep/19 ] |
|
sebastien, what's your plan for tests for these? I note that none of the patches seem to have tests today. Are you planning to add a bunch of tests in a separate patch, or tests to each patch, etc? They're also helpful for review because they show the reviewer the interface(s). |
| Comment by Sebastien Buisson [ 10/Sep/19 ] |
|
Given that all 4 patches are just here to serve one purpose, which is doing IOs (write and read) on encrypted files, I was planning to add a separate patch after them. At the moment, there is no specific interface to exercise this code. You have to use the dummy encryption mode, as described in the mount.lustre man page. A dummy key can be added to the keyring thanks to the following script, after that you just need to mount the client with the 'test_dummy_encryption' option, and all files created under a subdirectory (not the root of the Lustre fs) will be encrypted. #!/bin/bash
mode='\x00\x00\x00\x00'
raw="$(printf ""\\\\x%02x"" $(seq 0 63))"
if lscpu | grep "Byte Order" | grep -q Little ; then
size='\x40\x00\x00\x00'
else
size='\x00\x00\x00\x40'
fi
key="${mode}${raw}${size}"
echo -n -e "${key}" | keyctl padd logon fscrypt:4242424242424242 @s
|
| Comment by Patrick Farrell (Inactive) [ 10/Sep/19 ] |
|
So it's not possible to apply encryption to the root? |
| Comment by Patrick Farrell (Inactive) [ 10/Sep/19 ] |
|
Also, that makes sense - thanks. |
| Comment by Sebastien Buisson [ 11/Sep/19 ] |
|
So far I have decided to apply the same restriction as ext4, which forbids encrypting the root of the file system. With ext4, having the root encrypted is a problem for file system check (lost+found cannot be encrypted), but I am not sure it would be a problem for Lustre. However, setting a policy on a directory imposes that this directory is empty. And this is an issue with Lustre's root directory, because it always contains .lustre directory. Maybe we could think about how to release this restriction in the future, but I am not sure this is very limiting. And we still have the possibility to use subdirectory mount if we want clients to see only encrypted stuff. |
| Comment by Andreas Dilger [ 11/Sep/19 ] |
|
Since the Lustre-visible root directory is already the "ROOT/" subdirectory, it would be possible to set the encryption before the .lustre/ subdirectory is created. However, I think this would also cause problems. LFSCK would need to have access to the keyring on the client in order to create files during LFSCK, since it would be against the fscrypt policy to have an unencrypted Lustre lost+found/ subdirectory where it could generate unencrypted filenames. It probably makes the most sense to keep the root directory unencrypted, and only encrypt subdirectories. If users don't want to expose information about what those subdirectories are, they can use generic names like "dir1, dir2, dir3" (maybe matching the UID, since that is visible information anyway). |
| Comment by Andreas Dilger [ 11/Sep/19 ] |
|
from https://review.whamcloud.com/36146
Is it possible that the Lustre client/OSS always does the same thing as ext4, reading/writing the full PAGE_SIZE of data, but storing the actual file size on the object(s)? That means some data would be stored explicitly beyond EOF, but would avoid the complexity of storing a separate trusted.enc xattr, which could become out-of-sync with the actual file size in a number of ways. Replying to my own comment:
One possible option would be to do the encryption in chunks of the minimum PAGE_SIZE=4KiB, and use the IV for each chunk based on the 4KB index and not the actual PAGE_SIZE if it is different. This means that for the common PAGE_SIZE=4KiB the encryption would work exactly the same (i.e. compatible with files written by x86 clients), but for PAGE_SIZE=64KiB there would be 16 4KiB chunks within the page that are encrypted/decrypted separately. |
| Comment by Gerrit Updater [ 16/Sep/19 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/36191 |
| Comment by Gerrit Updater [ 19/Sep/19 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/36238 |
| Comment by Gerrit Updater [ 03/Oct/19 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/36360 |
| Comment by Gerrit Updater [ 03/Oct/19 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/36361 |
| Comment by Gerrit Updater [ 11/Oct/19 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/36433 |
| Comment by Gerrit Updater [ 22/Oct/19 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36360/ |
| Comment by Gerrit Updater [ 06/Dec/19 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36238/ |
| Comment by Patrick Farrell [ 17/Dec/19 ] |
|
Question for Sebastien and/or Andreas: Would this be incompatible with allowing unaligned direct i/o? It seems clear the answer is yes - Today, buffered i/o can easily be forced to work in full pages (which this code does) and direct i/o must work in full pages due to alignment requirements. I had some interest a while back in lifting the alignment requirement for DIO, which could improve performance for some scenarios of interest in certain cases. So it seems it would be necessary to prevent unaligned DIO on encrypted files, which would be relatively straightforward, if potentially a little confusing. Looking at Actually, it occurs to me that we could fall back to buffered i/o for DIO on encrypted files, if we decide that's important. It's unusual, but it's something that GPFS already does, so it's not unprecedented. Any strong reactions to this or aspects I've missed in the context of this work? |
| Comment by Sebastien Buisson [ 17/Dec/19 ] |
|
Hi Patrick, As explained in the attachment https://jira.whamcloud.com/secure/attachment/32658/lustre_encryption_access_semantics.txt, "direct I/O is not supported on encrypted files. Attempts to use direct I/O on such files will fall back to buffered I/O." So yes, just like GPFS |
| Comment by Patrick Farrell [ 17/Dec/19 ] |
|
Ah, OK! Well that's easy, then. Thanks. |
| Comment by Gerrit Updater [ 21/Feb/20 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/37672 |
| Comment by Gerrit Updater [ 21/Feb/20 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/37673 |
| Comment by Gerrit Updater [ 04/Mar/20 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/37794 |
| Comment by Gerrit Updater [ 02/Apr/20 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/38127 |
| Comment by Gerrit Updater [ 30/Apr/20 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/38430 |
| Comment by Gerrit Updater [ 01/May/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/38127/ |
| Comment by Gerrit Updater [ 22/May/20 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/38702 |
| Comment by Gerrit Updater [ 29/May/20 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/38759 |
| Comment by Gerrit Updater [ 06/Jun/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/38759/ |
| Comment by Gerrit Updater [ 06/Jun/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36143/ |
| Comment by Gerrit Updater [ 09/Jun/20 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/38881 |
| Comment by Gerrit Updater [ 09/Jun/20 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/38882 |
| Comment by Gerrit Updater [ 10/Jun/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36433/ |
| Comment by Gerrit Updater [ 12/Jun/20 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/38918 |
| Comment by Gerrit Updater [ 16/Jun/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36144/ |
| Comment by Gerrit Updater [ 16/Jun/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36145/ |
| Comment by Gerrit Updater [ 16/Jun/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36146/ |
| Comment by Gerrit Updater [ 16/Jun/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/37794/ |
| Comment by Gerrit Updater [ 17/Jun/20 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/38967 |
| Comment by Gerrit Updater [ 19/Jun/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36191/ |
| Comment by Gerrit Updater [ 19/Jun/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/37673/ |
| Comment by Gerrit Updater [ 30/Jun/20 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/39220 |
| Comment by Gerrit Updater [ 08/Jul/20 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/39315 |
| Comment by Gerrit Updater [ 17/Jul/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/38881/ |
| Comment by Gerrit Updater [ 20/Jul/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/38882/ |
| Comment by Gerrit Updater [ 20/Jul/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/38430/ |
| Comment by Gerrit Updater [ 20/Jul/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/38702/ |
| Comment by Gerrit Updater [ 20/Jul/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/38918/ |
| Comment by Gerrit Updater [ 21/Jul/20 ] |
|
Neil Brown (neilb@suse.de) uploaded a new patch: https://review.whamcloud.com/39459 |
| Comment by Gerrit Updater [ 31/Jul/20 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/39558 |
| Comment by Gerrit Updater [ 11/Aug/20 ] |
|
Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: https://review.whamcloud.com/39617 |
| Comment by Gerrit Updater [ 13/Aug/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/39459/ |
| Comment by Gerrit Updater [ 08/Sep/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/38967/ |
| Comment by Gerrit Updater [ 08/Sep/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/39220/ |
| Comment by Gerrit Updater [ 08/Sep/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/39558/ |
| Comment by Gerrit Updater [ 12/Sep/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/39315/ |
| Comment by Gerrit Updater [ 19/Sep/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/39617/ |
| Comment by Peter Jones [ 19/Sep/20 ] |
|
I believe that everything planned for 2.14 has now landed |
| Comment by Gerrit Updater [ 30/Jan/23 ] |
|
"Andreas Dilger <adilger@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49828 |
| Comment by Gerrit Updater [ 08/Feb/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/49828/ |
| Comment by Gerrit Updater [ 16/Feb/23 ] |
|
"Andreas Dilger <adilger@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50023 |
| Comment by Gerrit Updater [ 24/Feb/23 ] |
|
"Andreas Dilger <adilger@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50140 |
| Comment by Gerrit Updater [ 01/Mar/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50023/ |
| Comment by Gerrit Updater [ 28/Mar/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50140/ |