[LU-14673] panic: crc32-table: crc32 alg self test failed in fips mode!

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.12.7, Lustre 2.15.0
• Affects Version/s: Lustre 2.12.6
• Labels: llnl
• Environment: lustre-2.12.6_4.llnl-2.t4.x86_64 ; 4.18.0-240.22.1.1toss.t4.x86_64 ; fips=1
• Severity: 3
• Assignee: Sebastien Buisson
• Reporter: Olaf Faaland

Description

upon loading LNet, node panics with this:

libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
LNet: HW NUMA nodes: 2, HW CPU cores: 64, npartitions: 2
alg: No test for adler32 (adler32-zlib)
alg: hash: digest failed on test 1 for crc32-table: ret=126
Kernel panic - not syncing: crc32-table: crc32 alg self test failed in fips mode!

CPU: 11 PID: 70553 Comm: cryptomgr_test Tainted: G           OE    --------- -  - 4.18.0-240.22.1.1toss.t4.x86_64 #1
Hardware name: HPE ProLiant DL385 Gen10 Plus/ProLiant DL385 Gen10 Plus, BIOS A42 10/30/2020
Call Trace:
 dump_stack+0x5c/0x80
 panic+0xe7/0x2a9
 ? __alg_test_hash+0x55/0x80
 alg_test.cold.21+0x13/0x44
 ? __switch_to_asm+0x41/0x70
 ? __switch_to_asm+0x35/0x70
 ? __switch_to_asm+0x41/0x70
 ? __switch_to+0x7a/0x400
 ? __schedule+0x2cf/0x720
 ? crypto_acomp_scomp_free_ctx+0x30/0x30
 cryptomgr_test+0x27/0x50
 kthread+0x11d/0x140
 ? kthread_flush_work_fn+0x10/0x10
 ret_from_fork+0x22/0x40
Kernel Offset: 0x18c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

Loading 2.14.0 LNet under fips=1 does not cause the panic

Issue Links

• is related to: LU-13355 adler32 wrapper in libcfs

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/43653/
Subject: LU-14673 sec: annotate algorithms taking optional key
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 0aefae53fb03eaca7229d9d4b3e48c4a8c33de1a

[Olaf Faaland]

Keeping the topllnl label until the patch lands to b2_12

[Peter Jones]

Landed for 2.15

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/43656/
Subject: LU-14673 sec: annotate algorithms taking optional key
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b161e7b777e63bb4328aeab9e50560f919fedc31

[Gerrit Updater]

Sebastien Buisson ([e-mail]) uploaded a new patch: https://review.whamcloud.com/43656
Subject: LU-14673 sec: annotate algorithms taking optional key
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9b32694f26424030024a16c91a7e4575d5b281c2

[Gerrit Updater]

Sebastien Buisson ([e-mail]) uploaded a new patch: https://review.whamcloud.com/43653
Subject: LU-14673 sec: annotate algorithms taking optional key
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 2
Commit: cc066c79a26d927e11dcfdef96eb5ab77dc7025a

[Sebastien Buisson]

Patch https://review.whamcloud.com/35342 has been abandoned as we cannot drop RHEL6 support in 2.12.

An even cleaner approach is to fix the root cause of the error:

alg: hash: digest failed on test 1 for crc32-table: ret=126

Errno 126 is:

#define ENOKEY          126     /* Required key not available */

And it appears that crc32 needs to set the CRYPTO_ALG_OPTIONAL_KEY flag to work properly. I will push a patch to fix this.

[Sebastien Buisson]

Maybe the most straightforward way to get rid of this problem is to not call LIBCFS_HAVE_CRC32 config check in libcfs/autoconf/lustre-libcfs.m4, resulting in built-in crc32 not being used. I tested this quick solution, it works.

A cleaner approach is to backport patch https://review.whamcloud.com/35342 to b2_12, I just pushed a patch for this:
https://review.whamcloud.com/43623

[James A Simmons]

The special crc32 handling is left overs from the RHEL6 days which is why it was removed in newer lustre versions. All the special crc32 handling Lustre did is now apart of the supported kernels.

[Sebastien Buisson]

Hi Olaf,

Sorry for the confusion, I meant:

• it fails on RHEL8.3, no matter FIPS is enabled or not, but only panics when FIPS is enabled;
• it does not fail on RHEL7.