[LU-14673] panic: crc32-table: crc32 alg self test failed in fips mode!

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.12.7, Lustre 2.15.0
• Affects Version/s: Lustre 2.12.6
• Labels: llnl
• Environment: lustre-2.12.6_4.llnl-2.t4.x86_64 ; 4.18.0-240.22.1.1toss.t4.x86_64 ; fips=1
• Severity: 3
• Assignee: Sebastien Buisson
• Reporter: Olaf Faaland

Description

upon loading LNet, node panics with this:

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
LNet: HW NUMA nodes: 2, HW CPU cores: 64, npartitions: 2
alg: No test for adler32 (adler32-zlib)
alg: hash: digest failed on test 1 for crc32-table: ret=126
Kernel panic - not syncing: crc32-table: crc32 alg self test failed in fips mode!

CPU: 11 PID: 70553 Comm: cryptomgr_test Tainted: G           OE    --------- -  - 4.18.0-240.22.1.1toss.t4.x86_64 #1
Hardware name: HPE ProLiant DL385 Gen10 Plus/ProLiant DL385 Gen10 Plus, BIOS A42 10/30/2020
Call Trace:
 dump_stack+0x5c/0x80
 panic+0xe7/0x2a9
 ? __alg_test_hash+0x55/0x80
 alg_test.cold.21+0x13/0x44
 ? __switch_to_asm+0x41/0x70
 ? __switch_to_asm+0x35/0x70
 ? __switch_to_asm+0x41/0x70
 ? __switch_to+0x7a/0x400
 ? __schedule+0x2cf/0x720
 ? crypto_acomp_scomp_free_ctx+0x30/0x30
 cryptomgr_test+0x27/0x50
 kthread+0x11d/0x140
 ? kthread_flush_work_fn+0x10/0x10
 ret_from_fork+0x22/0x40
Kernel Offset: 0x18c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

Loading 2.14.0 LNet under fips=1 does not cause the panic

Issue Links

• is related to: LU-13355 adler32 wrapper in libcfs

Comments

[James A Simmons]

The special crc32 handling is left overs from the RHEL6 days which is why it was removed in newer lustre versions. All the special crc32 handling Lustre did is now apart of the supported kernels.

[Sebastien Buisson]

Hi Olaf,

Sorry for the confusion, I meant:

• it fails on RHEL8.3, no matter FIPS is enabled or not, but only panics when FIPS is enabled;
• it does not fail on RHEL7.

[Olaf Faaland]

Hi Sebastian,

When you say "it [the digest test] always fails", do you mean it always fails under the RHEL 8.3 kernel, but succeeds under the RHEL 7 kernel?

thanks

[Sebastien Buisson]

The fact that it does not crash with 2.14 is due to patch https://review.whamcloud.com/35342, only landed to master in early January 2020. This patch is a (very) big one, whose objective was to simplify the Lustre code by removing obsolete config checks. Among those was cfs_crypto_crc32_register() and all the Lustre specific crc32 implementation done in libcfs/libcfs/linux/linux-crypto-crc32.c.

No crc32, no crash

I can try to see why the digest test is failing (the fact that it fails is not due to FIPS, it always fails, but with FIPS enabled it triggers a panic). But maybe the most obvious move would be to remove the call to cfs_crypto_crc32_register(). Any suggestion adilger?

[Andreas Dilger]

Olaf, since it looks like you are building your own patched "1toss" client kernel, you could potentially disable this check until the problem is understood and fixed.

[Andreas Dilger]

This looks related to LU-13355, but according to patch https://review.whamcloud.com/38205 "LU-13355 crypto: crypto engine wrappers in libcfs" the crc32 crypto wrapper should be fixed since 2.12.5.

Possibly something has changed in how FIPS is being checked in the 4.18 kernel?

[Olaf Faaland]

For my records, my internal ticket is TOSS5190