https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14673 Lustre <p>upon loading LNet, node panics with this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>libcfs: loading out-of-tree module taints kernel. libcfs: module verification failed: signature and/or required key missing - tainting kernel LNet: HW NUMA nodes: 2, HW CPU cores: 64, npartitions: 2 alg: No test for adler32 (adler32-zlib) alg: hash: digest failed on test 1 for crc32-table: ret=126 Kernel panic - not syncing: crc32-table: crc32 alg self test failed in fips mode! CPU: 11 PID: 70553 Comm: cryptomgr_test Tainted: G OE --------- - - 4.18.0-240.22.1.1toss.t4.x86_64 #1 Hardware name: HPE ProLiant DL385 Gen10 Plus/ProLiant DL385 Gen10 Plus, BIOS A42 10/30/2020 Call Trace: dump_stack+0x5c/0x80 panic+0xe7/0x2a9 ? __alg_test_hash+0x55/0x80 alg_test.cold.21+0x13/0x44 ? __switch_to_asm+0x41/0x70 ? __switch_to_asm+0x35/0x70 ? __switch_to_asm+0x41/0x70 ? __switch_to+0x7a/0x400 ? __schedule+0x2cf/0x720 ? crypto_acomp_scomp_free_ctx+0x30/0x30 cryptomgr_test+0x27/0x50 kthread+0x11d/0x140 ? kthread_flush_work_fn+0x10/0x10 ret_from_fork+0x22/0x40 Kernel Offset: 0x18c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) </pre> </div></div> <p>Loading 2.14.0 LNet under fips=1 does not cause the panic</p> lustre-2.12.6_4.llnl-2.t4.x86_64 <br/> 4.18.0-240.22.1.1toss.t4.x86_64 <br/> fips=1 <br/> LU-14673

panic: crc32-table: crc32 alg self test failed in fips mode!

Bug Major Resolved Fixed Sebastien Buisson Olaf Faaland llnl Thu, 6 May 2021 01:11:40 +0000 Tue, 13 Jul 2021 08:24:25 +0000 Thu, 27 May 2021 19:08:35 +0000 Lustre 2.12.6 Lustre 2.12.7 Lustre 2.15.0 0 7 <p>For my records, my internal ticket is TOSS5190</p> <p>This looks related to <a href="https://jira.whamcloud.com/browse/LU-13355" title="adler32 wrapper in libcfs" class="issue-link" data-issue-key="LU-13355"><del>LU-13355</del></a>, but according to patch <a href="https://review.whamcloud.com/38205" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38205</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-13355" title="adler32 wrapper in libcfs" class="issue-link" data-issue-key="LU-13355"><del>LU-13355</del></a> crypto: crypto engine wrappers in libcfs</tt>" the crc32 crypto wrapper should be fixed since 2.12.5.</p> <p>Possibly something has changed in how FIPS is being checked in the 4.18 kernel? </p> <p>Olaf, since it looks like you are building your own patched "<tt>1toss</tt>" client kernel, you could potentially disable this check until the problem is understood and fixed.</p> <p>The fact that it does not crash with 2.14 is due to patch <a href="https://review.whamcloud.com/35342" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35342</a>, only landed to master in early January 2020. This patch is a (very) big one, whose objective was to simplify the Lustre code by removing obsolete config checks. Among those was cfs_crypto_crc32_register() and all the Lustre specific crc32 implementation done in libcfs/libcfs/linux/linux-crypto-crc32.c.</p> <p>No crc32, no crash <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I can try to see why the digest test is failing (the fact that it fails is not due to FIPS, it always fails, but with FIPS enabled it triggers a panic). But maybe the most obvious move would be to remove the call to <tt>cfs_crypto_crc32_register()</tt>. Any suggestion <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a>?</p> <p>Hi Sebastian,</p> <p>When you say "it <span class="error">&#91;the digest test&#93;</span> always fails", do you mean it always fails under the RHEL 8.3 kernel, but succeeds under the RHEL 7 kernel?</p> <p>thanks</p> <p>Hi Olaf,</p> <p>Sorry for the confusion, I meant:</p> <ul class="alternate" type="square"> <li>it fails on RHEL8.3, no matter FIPS is enabled or not, but only panics when FIPS is enabled;</li> <li>it does not fail on RHEL7.</li> </ul> <p>The special crc32 handling is left overs from the RHEL6 days which is why it was removed in newer lustre versions. All the special crc32 handling Lustre did is now apart of the supported kernels.</p> <p>Maybe the most straightforward way to get rid of this problem is to <b>not</b> call <tt>LIBCFS_HAVE_CRC32</tt> config check in libcfs/autoconf/lustre-libcfs.m4, resulting in built-in crc32 not being used. I tested this quick solution, it works.</p> <p>A cleaner approach is to backport patch <a href="https://review.whamcloud.com/35342" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35342</a> to b2_12, I just pushed a patch for this:<br/> <a href="https://review.whamcloud.com/43623" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43623</a></p> <p>Patch <a href="https://review.whamcloud.com/35342" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35342</a> has been abandoned as we cannot drop RHEL6 support in 2.12.</p> <p>An even cleaner approach is to fix the root cause of the error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>alg: hash: digest failed on test 1 for crc32-table: ret=126 </pre> </div></div> <p>Errno 126 is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#define ENOKEY 126 /* Required key not available */ </pre> </div></div> <p>And it appears that <tt>crc32</tt> needs to set the <tt>CRYPTO_ALG_OPTIONAL_KEY</tt> flag to work properly. I will push a patch to fix this.</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/43653" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43653</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14673" title="panic: crc32-table: crc32 alg self test failed in fips mode!" class="issue-link" data-issue-key="LU-14673"><del>LU-14673</del></a> sec: annotate algorithms taking optional key<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 2<br/> Commit: cc066c79a26d927e11dcfdef96eb5ab77dc7025a</p> <p>Sebastien Buisson (sbuisson@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/43656" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43656</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14673" title="panic: crc32-table: crc32 alg self test failed in fips mode!" class="issue-link" data-issue-key="LU-14673"><del>LU-14673</del></a> sec: annotate algorithms taking optional key<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9b32694f26424030024a16c91a7e4575d5b281c2</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/43656/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43656/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14673" title="panic: crc32-table: crc32 alg self test failed in fips mode!" class="issue-link" data-issue-key="LU-14673"><del>LU-14673</del></a> sec: annotate algorithms taking optional key<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b161e7b777e63bb4328aeab9e50560f919fedc31</p> <p>Landed for 2.15</p> <p>Keeping the topllnl label until the patch lands to b2_12</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/43653/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43653/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14673" title="panic: crc32-table: crc32 alg self test failed in fips mode!" class="issue-link" data-issue-key="LU-14673"><del>LU-14673</del></a> sec: annotate algorithms taking optional key<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 0aefae53fb03eaca7229d9d4b3e48c4a8c33de1a</p> Related LU-13355 Development Rank 1|i01tu7: Rank (Obsolete) 9223372036854775807 Severity