Details
-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
-
Lustre 2.15.0
-
3
-
9223372036854775807
Description
On Client 1:
# fscrypt setup Defaulting to policy_version 1 because kernel doesn't support v2. Customizing passphrase hashing difficulty for this system... Created global config file at "/etc/fscrypt.conf". Metadata directories created at "/.fscrypt". [root@trevis-90vm4 ~]# fscrypt setup /mnt/lustre Metadata directories created at "/mnt/lustre/.fscrypt". Change in /etc/fscrypt.conf policy_version field from 1 to 2
On Client 2:
# fscrypt setup Defaulting to policy_version 1 because kernel doesn't support v2. Customizing passphrase hashing difficulty for this system... Created global config file at "/etc/fscrypt.conf". Metadata directories created at "/.fscrypt". Change in /etc/fscrypt.conf policy_version field from 1 to 2
On Client 1:
# mkdir /mnt/lustre/private # ls -la /mnt/lustre/ total 16 drwxr-xr-x 5 root root 4096 Apr 20 12:37 . drwxr-xr-x. 3 root root 4096 Mar 1 02:01 .. drwxr-xr-x 4 root root 4096 Apr 20 08:41 .fscrypt drwxr-xr-x 2 root root 4096 Apr 20 12:37 private
On Client 2:
[root@trevis-90vm5 ~]# ls -la /mnt/lustre/
total 16
drwxr-xr-x 5 root root 4096 Apr 20 12:37 .
drwxr-xr-x. 3 root root 4096 Mar 1 02:01 ..
drwxr-xr-x 4 root root 4096 Apr 20 08:41 .fscrypt
drwxr-xr-x 2 root root 4096 Apr 20 12:37 private
On Client 1:
# fscrypt encrypt /mnt/lustre/private The following protector sources are available: 1 - Your login passphrase (pam_passphrase) 2 - A custom passphrase (custom_passphrase) 3 - A raw 256-bit key (raw_key) Enter the source number for the new protector [2 - custom_passphrase]: 2 Enter a name for the new protector: shield Enter custom passphrase for protector "shield": Confirm passphrase: "/mnt/lustre/private" is now encrypted, unlocked, and ready for use. # echo "encrypt data" > /mnt/lustre/private/encrypt_file1 # echo "encrypt data" > /mnt/lustre/private/encrypt_file2 # ls -la /mnt/lustre/private total 8 drwx------ 2 root root 4096 Apr 20 12:42 . drwxr-xr-x 5 root root 4096 Apr 20 12:37 .. -rw-r--r-- 1 root root 0 Apr 20 12:42 encrypt_file1 -rw-r--r-- 1 root root 0 Apr 20 12:42 encrypt_file2 # cat /mnt/lustre/private/encrypt_file1 encrypt data # cat /mnt/lustre/private/encrypt_file2 encrypt data
On Client 2:
# ls -la /mnt/lustre/private/ total 8 drwx------ 2 root root 4096 Apr 20 12:42 . drwxr-xr-x 5 root root 4096 Apr 20 12:37 .. -rw-r--r-- 1 root root 0 Apr 20 12:42 1SSaHn+b7U0XyzL82SNzjWDOS6PGKx3pAjn7WD8I1vC -rw-r--r-- 1 root root 0 Apr 20 12:42 piJmwoKUzymPjQ+53PFWANKGtXX82h0+HFPC7gFLR1B # cat /mnt/lustre/private/1SSaHn+b7U0XyzL82SNzjWDOS6PGKx3pAjn7WD8I1vC cat: /mnt/lustre/private/1SSaHn+b7U0XyzL82SNzjWDOS6PGKx3pAjn7WD8I1vC: Required key not available # cat /mnt/lustre/private/piJmwoKUzymPjQ+53PFWANKGtXX82h0+HFPC7gFLR1B cat: /mnt/lustre/private/piJmwoKUzymPjQ+53PFWANKGtXX82h0+HFPC7gFLR1B: Required key not available # echo "new data" > /mnt/lustre/private/1SSaHn+b7U0XyzL82SNzjWDOS6PGKx3pAjn7WD8I1vC -bash: /mnt/lustre/private/1SSaHn+b7U0XyzL82SNzjWDOS6PGKx3pAjn7WD8I1vC: Required key not available # echo "new data" > /mnt/lustre/private/piJmwoKUzymPjQ+53PFWANKGtXX82h0+HFPC7gFLR1B -bash: /mnt/lustre/private/piJmwoKUzymPjQ+53PFWANKGtXX82h0+HFPC7gFLR1B: Required key not available
On Client 2:
# fscrypt unlock /mnt/lustre/private Enter custom passphrase for protector "shield": "/mnt/lustre/private" is now unlocked and ready for use.
After that the file size is 0
On Client 1:
# ls -la /mnt/lustre/private/ total 8 drwx------ 2 root root 4096 Apr 20 12:42 . drwxr-xr-x 6 root root 4096 Apr 20 13:00 .. -rw-r--r-- 1 root root 0 Apr 20 12:42 encrypt_file1 -rw-r--r-- 1 root root 0 Apr 20 12:42 encrypt_file2 # cat /mnt/lustre/private/encrypt_file1 # cat /mnt/lustre/private/encrypt_file2
On Client 2:
# ls -la /mnt/lustre/private/ total 8 drwx------ 2 root root 4096 Apr 20 12:42 . drwxr-xr-x 6 root root 4096 Apr 20 13:00 .. -rw-r--r-- 1 root root 0 Apr 20 12:42 encrypt_file1 -rw-r--r-- 1 root root 0 Apr 20 12:42 encrypt_file2 # cat /mnt/lustre/private/encrypt_file1 # cat /mnt/lustre/private/encrypt_file2
This is the first encrypted directory on this file system.
With all the following encrypted directories, the bug could not be repeated.
Attachments
Issue Links
Activity
Is it possible that this problem is the same as LU-15787, where a client w/o encryption support is able to access the file? This would be the case if you tested on a RHEL7.9 client, or a client built without encryption support.
Oleg, if you are able to reproduce this, could you please apply the patch https://review.whamcloud.com/47156 "LU--15787 sec: block enc unaware clients on enc files" on the server to see if that fixes it.
- fscrypt --version to get version of fscrypt command line utility being used;
fscrypt --version fscrypt version v0.2.9
- distribution and kernel version on server side;
4.18.0-348.2.1.el8_lustre.x86_64 CentOS Linux release 8.5.2111
- distribution and kernel version on client side;
4.18.0-348.2.1.el8_lustre.x86_64 CentOS Linux release 8.5.2111
- what is the exact Lustre version being used? As 2.15.0 is not released, which tag/commit have you taken from Git?
lustre-release 03e4b451b0dc95fc4af2f1f48ba88b8b57ad366c Build #4283
I tried to reproduce with servers and clients installed with CentOS 8.3 and kernel 4.18.0-240.10.1.el8_3.x86_64, and current tip of Lustre master branch (commit "c6cf295d8f LU-15719 doc: Update changelog to properly reflect supported kernels").
I did not manage to hit the issue reported here. Even right after creation of files encrypt_file1 and encrypt_file2, their size is reported correctly on client1 and client2.
This step, carried out on client1, is already a problem:
# echo "encrypt data" > /mnt/lustre/private/encrypt_file1 # echo "encrypt data" > /mnt/lustre/private/encrypt_file2 # ls -la /mnt/lustre/private total 8 drwx------ 2 root root 4096 Apr 20 12:42 . drwxr-xr-x 5 root root 4096 Apr 20 12:37 .. -rw-r--r-- 1 root root 0 Apr 20 12:42 encrypt_file1 -rw-r--r-- 1 root root 0 Apr 20 12:42 encrypt_file2
Right after you created files encrypt_file1 and encrypt_file2, they appear with a size of 0, locally on the node where you created them. This should certainly not happen. Can you reproduce the same behavior on a regular, non-encrypted directory, as you explain making sure this is the first directory on this file system?
Could you please specify the following:
- fscrypt --version to get version of fscrypt command line utility being used;
- distribution and kernel version on server side;
- distribution and kernel version on client side;
- what is the exact Lustre version being used? As 2.15.0 is not released, which tag/commit have you taken from Git?
This is weird, as truncate with or without the encryption key is exercised in sanity-sec test_48a.
There are some operations that can be done without the encryption key:
Without the key --------------- Some filesystem operations may be performed on encrypted regular files, directories, and symlinks even before their encryption key has been added, or after their encryption key has been removed: : - Files may be deleted. That is, nondirectory files may be deleted with unlink() as usual, and empty directories may be deleted with rmdir() as usual. Therefore, ``rm`` and ``rm -r`` will work as expected. : Without the key, regular files cannot be opened or truncated. Attempts to do so will fail with ENOKEY. This implies that any regular file operations that require a file descriptor, such as read(), write(), mmap(), fallocate(), and ioctl(), are also forbidden.
So it looks like truncating the file shouldn't be allowed without a key, as that might e.g. corrupt the last block, or (potentially) remove important records from the end of a file or similar.
Could not be reproduced, perhaps addressed by
LU-15787