[LU-14228] mv file operation completed with different results to encrypted directory and to unencrypted

• Status: Closed
• Project: Lustre
• Type: Bug
• Priority: Minor
• Reporter: Vikentsi Lapa
• Assignee: Sebastien Buisson
• Resolution: Fixed

Environment

Client: 2.13.57_12_g3c75d25
Server: 2.13.57-1

• Severity: 3

Description

Create big file, more than half of file system size. File should fit to file system.
Check file system size.

[root@trevis-59vm10 ~]# dd if=/dev/vda of=/mnt/lustre/sda bs=1M count=500
[root@trevis-59vm10 ~]# du -sh /mnt/lustre/sda
501M    /mnt/lustre/sdc

[root@trevis-59vm10 ~]# lfs df -h
UUID                       bytes        Used   Available Use% Mounted on
lustre-MDT0000_UUID       122.1M        2.6M      108.6M   3% /mnt/lustre[MDT:0]
lustre-OST0000_UUID       305.8M      254.3M       24.9M  92% /mnt/lustre[OST:0]
lustre-OST0001_UUID       305.8M      270.2M        9.0M  97% /mnt/lustre[OST:1]

filesystem_summary:       611.5M      524.6M       33.9M  94% /mnt/lustre

Move file to unencrypted directory. Result was successful. File renamed.

[root@trevis-59vm10 ~]# mv /mnt/lustre/sda /mnt/lustre/sdb
[root@trevis-59vm10 ~]# du -sh /mnt/lustre/sdb
501M    /mnt/lustre/sdb

Move file to encrypted directory. Result was No space left on device

[root@trevis-59vm10 ~]# mv /mnt/lustre/sdb /mnt/lustre/dir1/
mv: error writing '/mnt/lustre/dir1/sdb': No space left on device

Below are details about test environment.

Clients was installed with commands:

loadjenkinsbuild -j lustre-master -b 0 -n trevis-59vm10 -d el8.1 -a x86_64 -t client -p test -rv

loadjenkinsbuild -j lustre-master -b 0 -n trevis-59vm12 -d ubuntu2004 -a x86_64 -t client -p test -rv

Lustre client version Ubuntu 20.04.

[18829.299323] Lustre: Lustre: Build Version: 2.13.57_12_g3c75d25
[18829.344486] LNet: Added LNI 10.9.6.30@tcp [8/256/0/180]
[18829.345220] LNet: Accept secure, port 988
[18830.416612] Lustre: Mounted lustre-client
 dpkg -l | grep lustre
ii  lustre-client-modules-5.4.0-40-generic 2.13.57-12-g3c75d25-1             amd64        Lustre Linux kernel module (kernel 5.4.0-40-generic)
ii  lustre-client-utils                    2.13.57-12-g3c75d25-1             amd64        Userspace utilities for the Lustre filesystem (client)
ii  lustre-dev                             2.13.57-12-g3c75d25-1             amd64        Development files for the Lustre filesystem
ii  lustre-iokit                           2.13.57-12-g3c75d25-1             amd64        Collection of benchmark tools for the Lustre filesystem
ii  lustre-tests                           2.13.57-12-g3c75d25-1             amd64        Test suite for the Lustre filesystem
ii  posix                                  1.2                               amd64        This is POSIX compliance source to test POSIX on lustre.

[24669.478523] Lustre: lustre-OST0000-osc-ffff89bba284d800: disconnect after 24s idle
[24669.479479] Lustre: Skipped 1 previous similar message
[26524.543507] Lustre: 9090:0:(policy.c:449:llcrypt_ioctl_get_policy_ex()) inode 144115205272502273: returning policy filenames_en
cryption_mode as 4, but is in fact null
[26766.726913] Lustre: 9128:0:(policy.c:449:llcrypt_ioctl_get_policy_ex()) inode 144115205272502273: returning policy filenames_en
cryption_mode as 4, but is in fact null
[28575.857999] Lustre: lustre-OST0001-osc-ffff89bba284d800: disconnect after 23s idle
[30695.439703] Lustre: lustre-OST0000-osc-ffff89bba284d800: disconnect after 24s idle
[119819.762873] Lustre: 7988:0:(client.c:2282:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 16
08140338/real 1608140338]  req@000000005606ed07 x1686133235745664/t0(0) o104->lustre-MDT0000@10.9.6.30@tcp:15/16 lens 328/224 e 0
to 1 dl 1608140345 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:''

Lustre server configuration and versions

[ 1427.387306] Lustre: Lustre: Build Version: 2.13.57
[ 1427.430301] LNet: Added LNI 10.9.6.27@tcp [8/256/0/180]
[ 1427.431051] LNet: Accept secure, port 988

[root@trevis-59vm9 ~]# rpm -qa | grep lustre
kernel-modules-4.18.0-147.8.1.el8_lustre.x86_64
kernel-core-4.18.0-147.8.1.el8_lustre.x86_64
lustre-2.13.57-1.el8.x86_64
kmod-lustre-osd-ldiskfs-2.13.57-1.el8.x86_64
lustre-osd-ldiskfs-mount-2.13.57-1.el8.x86_64
lustre-tests-2.13.57-1.el8.x86_64
lustre-iokit-2.13.57-1.el8.x86_64
kernel-4.18.0-147.8.1.el8_lustre.x86_64

Comments

[Andreas Dilger]

Moving a file to an encrypted directory will cause it to be copied and encrypted. I think this behavior is expected, but possibly needs to be documented in the feature notes.

[Sebastien Buisson]

Hi,

In section "30.5.1. Client-side encryption access semantics" of the Lustre doc updated with content related to the client encryption feature, that you can preview via https://build.whamcloud.com/job/lustre-manual-reviews/1109/artifact/lustre_manual.xhtml#managingSecurity.clientencryption.semantics , there is a special note for this use case:

Note

"moving" an unencrypted file into an encrypted directory, e.g. with the mv program, is
implemented in userspace by a copy followed by a delete. Be aware that the original
unencrypted data may remain recoverable from free space on the disk; prefer to keep all
files encrypted from the very beginning.

So I think it covers your concern. Do you agree to close this ticket?
Thanks,
Sebastien.

[Vikentsi Lapa]

Yes it can be closed because it is covered by documentation. User informed about this case and can plan move/rename operation accordingly to free disk space