Details
-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
-
Lustre 2.15.0
-
3
-
9223372036854775807
Description
On Client 1:
# fscrypt setup Defaulting to policy_version 1 because kernel doesn't support v2. Customizing passphrase hashing difficulty for this system... Created global config file at "/etc/fscrypt.conf". Metadata directories created at "/.fscrypt". [root@trevis-90vm4 ~]# fscrypt setup /mnt/lustre Metadata directories created at "/mnt/lustre/.fscrypt". Change in /etc/fscrypt.conf policy_version field from 1 to 2
On Client 2:
# fscrypt setup Defaulting to policy_version 1 because kernel doesn't support v2. Customizing passphrase hashing difficulty for this system... Created global config file at "/etc/fscrypt.conf". Metadata directories created at "/.fscrypt". Change in /etc/fscrypt.conf policy_version field from 1 to 2
On Client 1:
# mkdir /mnt/lustre/private # ls -la /mnt/lustre/ total 16 drwxr-xr-x 5 root root 4096 Apr 20 12:37 . drwxr-xr-x. 3 root root 4096 Mar 1 02:01 .. drwxr-xr-x 4 root root 4096 Apr 20 08:41 .fscrypt drwxr-xr-x 2 root root 4096 Apr 20 12:37 private
On Client 2:
[root@trevis-90vm5 ~]# ls -la /mnt/lustre/
total 16
drwxr-xr-x 5 root root 4096 Apr 20 12:37 .
drwxr-xr-x. 3 root root 4096 Mar 1 02:01 ..
drwxr-xr-x 4 root root 4096 Apr 20 08:41 .fscrypt
drwxr-xr-x 2 root root 4096 Apr 20 12:37 private
On Client 1:
# fscrypt encrypt /mnt/lustre/private The following protector sources are available: 1 - Your login passphrase (pam_passphrase) 2 - A custom passphrase (custom_passphrase) 3 - A raw 256-bit key (raw_key) Enter the source number for the new protector [2 - custom_passphrase]: 2 Enter a name for the new protector: shield Enter custom passphrase for protector "shield": Confirm passphrase: "/mnt/lustre/private" is now encrypted, unlocked, and ready for use. # echo "encrypt data" > /mnt/lustre/private/encrypt_file1 # echo "encrypt data" > /mnt/lustre/private/encrypt_file2 # ls -la /mnt/lustre/private total 8 drwx------ 2 root root 4096 Apr 20 12:42 . drwxr-xr-x 5 root root 4096 Apr 20 12:37 .. -rw-r--r-- 1 root root 0 Apr 20 12:42 encrypt_file1 -rw-r--r-- 1 root root 0 Apr 20 12:42 encrypt_file2 # cat /mnt/lustre/private/encrypt_file1 encrypt data # cat /mnt/lustre/private/encrypt_file2 encrypt data
On Client 2:
# ls -la /mnt/lustre/private/ total 8 drwx------ 2 root root 4096 Apr 20 12:42 . drwxr-xr-x 5 root root 4096 Apr 20 12:37 .. -rw-r--r-- 1 root root 0 Apr 20 12:42 1SSaHn+b7U0XyzL82SNzjWDOS6PGKx3pAjn7WD8I1vC -rw-r--r-- 1 root root 0 Apr 20 12:42 piJmwoKUzymPjQ+53PFWANKGtXX82h0+HFPC7gFLR1B # cat /mnt/lustre/private/1SSaHn+b7U0XyzL82SNzjWDOS6PGKx3pAjn7WD8I1vC cat: /mnt/lustre/private/1SSaHn+b7U0XyzL82SNzjWDOS6PGKx3pAjn7WD8I1vC: Required key not available # cat /mnt/lustre/private/piJmwoKUzymPjQ+53PFWANKGtXX82h0+HFPC7gFLR1B cat: /mnt/lustre/private/piJmwoKUzymPjQ+53PFWANKGtXX82h0+HFPC7gFLR1B: Required key not available # echo "new data" > /mnt/lustre/private/1SSaHn+b7U0XyzL82SNzjWDOS6PGKx3pAjn7WD8I1vC -bash: /mnt/lustre/private/1SSaHn+b7U0XyzL82SNzjWDOS6PGKx3pAjn7WD8I1vC: Required key not available # echo "new data" > /mnt/lustre/private/piJmwoKUzymPjQ+53PFWANKGtXX82h0+HFPC7gFLR1B -bash: /mnt/lustre/private/piJmwoKUzymPjQ+53PFWANKGtXX82h0+HFPC7gFLR1B: Required key not available
On Client 2:
# fscrypt unlock /mnt/lustre/private Enter custom passphrase for protector "shield": "/mnt/lustre/private" is now unlocked and ready for use.
After that the file size is 0
On Client 1:
# ls -la /mnt/lustre/private/ total 8 drwx------ 2 root root 4096 Apr 20 12:42 . drwxr-xr-x 6 root root 4096 Apr 20 13:00 .. -rw-r--r-- 1 root root 0 Apr 20 12:42 encrypt_file1 -rw-r--r-- 1 root root 0 Apr 20 12:42 encrypt_file2 # cat /mnt/lustre/private/encrypt_file1 # cat /mnt/lustre/private/encrypt_file2
On Client 2:
# ls -la /mnt/lustre/private/ total 8 drwx------ 2 root root 4096 Apr 20 12:42 . drwxr-xr-x 6 root root 4096 Apr 20 13:00 .. -rw-r--r-- 1 root root 0 Apr 20 12:42 encrypt_file1 -rw-r--r-- 1 root root 0 Apr 20 12:42 encrypt_file2 # cat /mnt/lustre/private/encrypt_file1 # cat /mnt/lustre/private/encrypt_file2
This is the first encrypted directory on this file system.
With all the following encrypted directories, the bug could not be repeated.
Attachments
Issue Links
![[Whamcloud Community Wiki] Page](/images/icons/generic_link_16.png "[Whamcloud Community Wiki] Page"){kind=icon}
Activity
Is it possible that this problem is the same as LU-15787, where a client w/o encryption support is able to access the file? This would be the case if you tested on a RHEL7.9 client, or a client built without encryption support.
Oleg, if you are able to reproduce this, could you please apply the patch https://review.whamcloud.com/47156 "LU--15787 sec: block enc unaware clients on enc files" on the server to see if that fixes it.
- fscrypt --version to get version of fscrypt command line utility being used;
fscrypt --version fscrypt version v0.2.9
- distribution and kernel version on server side;
4.18.0-348.2.1.el8_lustre.x86_64 CentOS Linux release 8.5.2111
- distribution and kernel version on client side;
4.18.0-348.2.1.el8_lustre.x86_64 CentOS Linux release 8.5.2111
- what is the exact Lustre version being used? As 2.15.0 is not released, which tag/commit have you taken from Git?
lustre-release 03e4b451b0dc95fc4af2f1f48ba88b8b57ad366c Build #4283
I tried to reproduce with servers and clients installed with CentOS 8.3 and kernel 4.18.0-240.10.1.el8_3.x86_64, and current tip of Lustre master branch (commit "c6cf295d8f LU-15719 doc: Update changelog to properly reflect supported kernels").
I did not manage to hit the issue reported here. Even right after creation of files encrypt_file1 and encrypt_file2, their size is reported correctly on client1 and client2.
This step, carried out on client1, is already a problem:
# echo "encrypt data" > /mnt/lustre/private/encrypt_file1 # echo "encrypt data" > /mnt/lustre/private/encrypt_file2 # ls -la /mnt/lustre/private total 8 drwx------ 2 root root 4096 Apr 20 12:42 . drwxr-xr-x 5 root root 4096 Apr 20 12:37 .. -rw-r--r-- 1 root root 0 Apr 20 12:42 encrypt_file1 -rw-r--r-- 1 root root 0 Apr 20 12:42 encrypt_file2
Right after you created files encrypt_file1 and encrypt_file2, they appear with a size of 0, locally on the node where you created them. This should certainly not happen. Can you reproduce the same behavior on a regular, non-encrypted directory, as you explain making sure this is the first directory on this file system?
Could you please specify the following:
- fscrypt --version to get version of fscrypt command line utility being used;
- distribution and kernel version on server side;
- distribution and kernel version on client side;
- what is the exact Lustre version being used? As 2.15.0 is not released, which tag/commit have you taken from Git?
This is weird, as truncate with or without the encryption key is exercised in sanity-sec test_48a.
There are some operations that can be done without the encryption key:
Without the key --------------- Some filesystem operations may be performed on encrypted regular files, directories, and symlinks even before their encryption key has been added, or after their encryption key has been removed: : - Files may be deleted. That is, nondirectory files may be deleted with unlink() as usual, and empty directories may be deleted with rmdir() as usual. Therefore, ``rm`` and ``rm -r`` will work as expected. : Without the key, regular files cannot be opened or truncated. Attempts to do so will fail with ENOKEY. This implies that any regular file operations that require a file descriptor, such as read(), write(), mmap(), fallocate(), and ioctl(), are also forbidden.
So it looks like truncating the file shouldn't be allowed without a key, as that might e.g. corrupt the last block, or (potentially) remove important records from the end of a file or similar.
adilger thank you very much.
I'll try to reproduce it