[LU-15767] If the user regain access to the encrypted directory from another client, the contents of the files will be deleted

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.15.0
• Type: Bug
• Priority: Major
• Reporter: Oleg Kulachenko (Inactive)
• Assignee: Sebastien Buisson
• Resolution: Cannot Reproduce
• Labels: encryption, sec, security
• Severity: 3

Description

On Client 1:

 

# fscrypt setup
Defaulting to policy_version 1 because kernel doesn't support v2.
Customizing passphrase hashing difficulty for this system...
Created global config file at "/etc/fscrypt.conf".
Metadata directories created at "/.fscrypt".
[root@trevis-90vm4 ~]# fscrypt setup /mnt/lustre
Metadata directories created at "/mnt/lustre/.fscrypt".
Change in /etc/fscrypt.conf policy_version field from 1 to 2 

On Client 2:

 

# fscrypt setup
Defaulting to policy_version 1 because kernel doesn't support v2.
Customizing passphrase hashing difficulty for this system...
Created global config file at "/etc/fscrypt.conf".
Metadata directories created at "/.fscrypt".
Change in /etc/fscrypt.conf policy_version field from 1 to 2 

 

 

On Client 1:

 

# mkdir /mnt/lustre/private
# ls -la  /mnt/lustre/
total 16
drwxr-xr-x  5 root root 4096 Apr 20 12:37 .
drwxr-xr-x. 3 root root 4096 Mar  1 02:01 ..
drwxr-xr-x  4 root root 4096 Apr 20 08:41 .fscrypt
drwxr-xr-x  2 root root 4096 Apr 20 12:37 private 

 

 

On Client 2:

 

[root@trevis-90vm5 ~]# ls -la  /mnt/lustre/
total 16
drwxr-xr-x  5 root root 4096 Apr 20 12:37 .
drwxr-xr-x. 3 root root 4096 Mar  1 02:01 ..
drwxr-xr-x  4 root root 4096 Apr 20 08:41 .fscrypt
drwxr-xr-x  2 root root 4096 Apr 20 12:37 private 

 

 

On Client 1:

 

# fscrypt encrypt /mnt/lustre/private
The following protector sources are available:
1 - Your login passphrase (pam_passphrase)
2 - A custom passphrase (custom_passphrase)
3 - A raw 256-bit key (raw_key)
Enter the source number for the new protector [2 - custom_passphrase]: 2
Enter a name for the new protector: shield
Enter custom passphrase for protector "shield":
Confirm passphrase:
"/mnt/lustre/private" is now encrypted, unlocked, and ready for use.

# echo "encrypt data" > /mnt/lustre/private/encrypt_file1
# echo "encrypt data" > /mnt/lustre/private/encrypt_file2

# ls -la /mnt/lustre/private
total 8
drwx------ 2 root root 4096 Apr 20 12:42 .
drwxr-xr-x 5 root root 4096 Apr 20 12:37 ..
-rw-r--r-- 1 root root    0 Apr 20 12:42 encrypt_file1
-rw-r--r-- 1 root root    0 Apr 20 12:42 encrypt_file2

# cat /mnt/lustre/private/encrypt_file1
encrypt data
# cat /mnt/lustre/private/encrypt_file2
encrypt data 

 

 

On Client 2:

# ls -la /mnt/lustre/private/
total 8
drwx------ 2 root root 4096 Apr 20 12:42 .
drwxr-xr-x 5 root root 4096 Apr 20 12:37 ..
-rw-r--r-- 1 root root    0 Apr 20 12:42 1SSaHn+b7U0XyzL82SNzjWDOS6PGKx3pAjn7WD8I1vC
-rw-r--r-- 1 root root    0 Apr 20 12:42 piJmwoKUzymPjQ+53PFWANKGtXX82h0+HFPC7gFLR1B
  

# cat /mnt/lustre/private/1SSaHn+b7U0XyzL82SNzjWDOS6PGKx3pAjn7WD8I1vC
cat: /mnt/lustre/private/1SSaHn+b7U0XyzL82SNzjWDOS6PGKx3pAjn7WD8I1vC: Required key not available
# cat /mnt/lustre/private/piJmwoKUzymPjQ+53PFWANKGtXX82h0+HFPC7gFLR1B
cat: /mnt/lustre/private/piJmwoKUzymPjQ+53PFWANKGtXX82h0+HFPC7gFLR1B: Required key not available


# echo "new data" > /mnt/lustre/private/1SSaHn+b7U0XyzL82SNzjWDOS6PGKx3pAjn7WD8I1vC
-bash: /mnt/lustre/private/1SSaHn+b7U0XyzL82SNzjWDOS6PGKx3pAjn7WD8I1vC: Required key not available
# echo "new data" > /mnt/lustre/private/piJmwoKUzymPjQ+53PFWANKGtXX82h0+HFPC7gFLR1B
-bash: /mnt/lustre/private/piJmwoKUzymPjQ+53PFWANKGtXX82h0+HFPC7gFLR1B: Required key not available

 

On Client 2:

# fscrypt unlock /mnt/lustre/private
Enter custom passphrase for protector "shield":
"/mnt/lustre/private" is now unlocked and ready for use.

 

After that the file size is 0

On Client 1:

# ls -la /mnt/lustre/private/
total 8
drwx------ 2 root root 4096 Apr 20 12:42 .
drwxr-xr-x 6 root root 4096 Apr 20 13:00 ..
-rw-r--r-- 1 root root    0 Apr 20 12:42 encrypt_file1
-rw-r--r-- 1 root root    0 Apr 20 12:42 encrypt_file2

# cat /mnt/lustre/private/encrypt_file1
# cat /mnt/lustre/private/encrypt_file2

 

On Client 2:

# ls -la /mnt/lustre/private/
total 8
drwx------ 2 root root 4096 Apr 20 12:42 .
drwxr-xr-x 6 root root 4096 Apr 20 13:00 ..
-rw-r--r-- 1 root root    0 Apr 20 12:42 encrypt_file1
-rw-r--r-- 1 root root    0 Apr 20 12:42 encrypt_file2

# cat /mnt/lustre/private/encrypt_file1 
# cat /mnt/lustre/private/encrypt_file2

 

 

 

This is the first encrypted directory on this file system.

With all the following encrypted directories, the bug could not be repeated.

 

 

 

 

Issue Links

• is related to: LU-15787 clients built without encryption supp...

Comments

[Andreas Dilger]

There are some operations that can be done without the encryption key:

Without the key
---------------
Some filesystem operations may be performed on encrypted regular
files, directories, and symlinks even before their encryption key has
been added, or after their encryption key has been removed:
:
- Files may be deleted.  That is, nondirectory files may be deleted
  with unlink() as usual, and empty directories may be deleted with
  rmdir() as usual.  Therefore, ``rm`` and ``rm -r`` will work as
  expected.
:
Without the key, regular files cannot be opened or truncated.
Attempts to do so will fail with ENOKEY.  This implies that any
regular file operations that require a file descriptor, such as
read(), write(), mmap(), fallocate(), and ioctl(), are also forbidden.

So it looks like truncating the file shouldn't be allowed without a key, as that might e.g. corrupt the last block, or (potentially) remove important records from the end of a file or similar.

[Sebastien Buisson]

This is weird, as truncate with or without the encryption key is exercised in sanity-sec test_48a.

[Sebastien Buisson]

Could you please specify the following:

• fscrypt --version to get version of fscrypt command line utility being used;
• distribution and kernel version on server side;
• distribution and kernel version on client side;
• what is the exact Lustre version being used? As 2.15.0 is not released, which tag/commit have you taken from Git?

[Sebastien Buisson]

This step, carried out on client1, is already a problem:

# echo "encrypt data" > /mnt/lustre/private/encrypt_file1
# echo "encrypt data" > /mnt/lustre/private/encrypt_file2

# ls -la /mnt/lustre/private
total 8
drwx------ 2 root root 4096 Apr 20 12:42 .
drwxr-xr-x 5 root root 4096 Apr 20 12:37 ..
-rw-r--r-- 1 root root    0 Apr 20 12:42 encrypt_file1
-rw-r--r-- 1 root root    0 Apr 20 12:42 encrypt_file2

Right after you created files encrypt_file1 and encrypt_file2, they appear with a size of 0, locally on the node where you created them. This should certainly not happen. Can you reproduce the same behavior on a regular, non-encrypted directory, as you explain making sure this is the first directory on this file system?

[Sebastien Buisson]

I tried to reproduce with servers and clients installed with CentOS 8.3 and kernel 4.18.0-240.10.1.el8_3.x86_64, and current tip of Lustre master branch (commit "c6cf295d8f LU-15719 doc: Update changelog to properly reflect supported kernels").
I did not manage to hit the issue reported here. Even right after creation of files encrypt_file1 and encrypt_file2, their size is reported correctly on client1 and client2.

[Oleg Kulachenko (Inactive)]

sebastien 

• fscrypt --version to get version of fscrypt command line utility being used;

fscrypt --version
fscrypt version v0.2.9

• distribution and kernel version on server side;

4.18.0-348.2.1.el8_lustre.x86_64
CentOS Linux release 8.5.2111 

• distribution and kernel version on client side;

4.18.0-348.2.1.el8_lustre.x86_64
CentOS Linux release 8.5.2111

• what is the exact Lustre version being used? As 2.15.0 is not released, which tag/commit have you taken from Git?

lustre-release 03e4b451b0dc95fc4af2f1f48ba88b8b57ad366c Build #4283  

[Andreas Dilger]

Is it possible that this problem is the same as LU-15787, where a client w/o encryption support is able to access the file? This would be the case if you tested on a RHEL7.9 client, or a client built without encryption support.

Oleg, if you are able to reproduce this, could you please apply the patch https://review.whamcloud.com/47156 "LU--15787 sec: block enc unaware clients on enc files" on the server to see if that fixes it.

[Oleg Kulachenko (Inactive)]

adilger thank you very much.

I'll try to reproduce it

[Peter Jones]

Could not be reproduced, perhaps addressed by LU-15787