[LU-12475] Lustre file system is causing kernel panic issue when using with fips enabled kernel

Related — Mon, 15 Jun 2020 20:04:24 +0000

While using lustre filesystem with FIPS enabled kernel on rhel7.6 I am facing kernel panic.

System Information & Panic string.

crash> sys|grep -e NODENAME -e RELEASE -e PANIC
NODENAME: ip-172-31-14-243.ec2.internal
RELEASE: 3.10.0-957.21.3.el7.x86_64
PANIC: "Kernel panic - not syncing: Module libcfs signature verification failed in FIPS mode"

#cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.21.3.el7.x86_64 root=UUID=50a9826b-3a50-44d0-ad12-28f2056e9927 ro console=ttyS0,115200n8 console=tty0 net.ifnames=0 crashkernel=auto fips=1

Backtrace:-

crash> bt
PID: 11597 TASK: ffff8f4a9de88000 CPU: 1 COMMAND: "modprobe"
#0 [ffff8f493b3d7af0] machine_kexec at ffffffffbbc63934
#1 [ffff8f493b3d7b50] __crash_kexec at ffffffffbbd1d162
#2 [ffff8f493b3d7c20] panic at ffffffffbc35c81b
#3 [ffff8f493b3d7ca0] crypto_check_alg at ffffffffbbf2286a
#4 [ffff8f493b3d7cc8] crypto_register_alg at ffffffffbbf23144
#5 [ffff8f493b3d7ce8] crypto_register_shash at ffffffffbbf2961f
#6 [ffff8f493b3d7d00] cfs_crypto_adler32_register at ffffffffc049f225 [libcfs]
#7 [ffff8f493b3d7d10] cfs_crypto_register at ffffffffc049ea11 [libcfs]
#8 [ffff8f493b3d7d20] init_module at ffffffffc04391a9 [libcfs]
#9 [ffff8f493b3d7d38] do_one_initcall at ffffffffbbc0210a
#10 [ffff8f493b3d7d68] load_module at ffffffffbbd192dc
#11 [ffff8f493b3d7eb8] sys_finit_module at ffffffffbbd19956
#12 [ffff8f493b3d7f50] system_call_fastpath at ffffffffbc375ddb
RIP: 00007f378a796349 RSP: 00007ffd08ad98a8 RFLAGS: 00010202
RAX: 0000000000000139 RBX: 000000000101b850 RCX: 00007f378a78d0f0
RDX: 0000000000000000 RSI: 000000000041a94e RDI: 0000000000000000
RBP: 000000000041a94e R8: 0000000000000000 R9: 00000000010192d0
R10: 0000000000000000 R11: 0000000000000206 R12: 0000000000000000
R13: 000000000101b7f0 R14: 0000000000040000 R15: 0000000000000000
ORIG_RAX: 0000000000000139 CS: 0033 SS: 002b

Fuction which is responsible for Kernel Panic here.

>>kernel-3.10.0-957.el7/crypto/algapi.c

static inline void crypto_check_module_sig(struct module *mod)

{ #ifdef CONFIG_CRYPTO_FIPS if (fips_enabled && mod && !mod->sig_ok) panic("Module %s signature verification failed in FIPS mode\n", mod->name); #endif return; }

As the function "crypto_check_alg" is calling "crypto_check_module_sig" to check the signature of module here.

static int crypto_check_alg(struct crypto_alg *alg)

{ crypto_check_module_sig(alg->cra_module); if (alg->cra_alignmask & (alg->cra_alignmask + 1)) return -EINVAL; if (alg->cra_blocksize > PAGE_SIZE / 8) return -EINVAL; if (alg->cra_priority < 0) return -EINVAL; atomic_set(&alg->cra_refcnt, 1); return crypto_set_driver_name(alg); }

Further checking module libcsf doesn't have the signature field here as we can see in case of nvme_core modules we are getting sig_ok field as true but in case of libcfs sig_ok is false.

crash> struct module ffffffffc00b1480|grep -e name -e sig_ok
name = "nvme_core\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000",
name = 0xffff8f48b4c63280 "nvme_core",
sig_ok = true,

crash> struct module ffffffffc04c5120|grep -e name -e sig_ok
name = "libcfs\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000",
name = 0xffff8f48b40b77b8 "libcfs",
sig_ok = false,

Sequence of events :-

1. Mounting lustre file system leads to libcfs.ko being loaded
2. libcfs calls a kernel function: crypto_register_shash() used for defining new hash type
3. That function calls crypto_register_alg which is calling crypto_check_alg() which further uses or calls
crypto_check_module_sig()
4. A check is performed: are we in FIPS mode and does libcfs have a valid signature? ... If not, panic.

In Summary "libcfs calls a kernel function(kernel crypto function) whose use is forbidden (by non-signed modules) in FIPS mode" .

modinfo libcfs
filename: /lib/modules/3.10.0-957.21.2.el7.x86_64/weak-updates/lustre-client/net/libcfs.ko
license: GPL
version: 0.5.0
description: Lustre helper library
author: OpenSFS, Inc. <http://www.lustre.org/>
retpoline: Y
rhelversion: 7.6
srcversion: B62994CF6EF3B80D9BF4F03
depends:
vermagic: 3.10.0-957.el7.x86_64 SMP mod_unload modversions
parm: libcfs_subsystem_debug:Lustre kernel debug subsystem mask (int)
parm: libcfs_debug:Lustre kernel debug mask (int)
parm: libcfs_debug_mb:Total debug buffer size. (uint)
parm: libcfs_printk:Lustre kernel debug console mask (uint)
parm: libcfs_console_ratelimit:Lustre kernel debug console ratelimit (0 to disable) (uint)
parm: libcfs_console_max_delay:Lustre kernel debug console max delay (jiffies) (uint)
parm: libcfs_console_min_delay:Lustre kernel debug console min delay (jiffies) (uint)
parm: libcfs_console_backoff:Lustre kernel debug console backoff factor (uint)
parm: libcfs_panic_on_lbug:Lustre kernel panic on LBUG (uint)
parm: libcfs_debug_file_path:Path for dumping debug logs, set 'NONE' to prevent log dumping (charp)
parm: cpu_npartitions:# of CPU partitions (int)
parm: cpu_pattern:CPU partitions pattern (charp)

Can we signing the modules so they will work in a FIPS enabled kernel

Whamcloud Community JIRA

[LU-12475] Lustre file system is causing kernel panic issue when using with fips enabled kernel