[LU-17794] UBSAN: array-index-out-of-bounds errors

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0
• Affects Version/s: Lustre 2.16.0
• Labels: janitor9x
• Severity: 3
• Assignee: Jian Yu
• Reporter: Jian Yu

Description

While testing master branch client build on Ubuntu 24.04 with kernel 6.8.0-31-generic, lots of runtime UBSAN array-index-out-of-bounds errors occurred in dmesg/syslog:

------------[ cut here ]------------
UBSAN: array-index-out-of-bounds in /root/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1168:3
index 0 is out of range for type 'void *[*]'
CPU: 0 PID: 111835 Comm: modprobe Tainted: G           OE      6.8.0-31-generic #31-Ubuntu
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Call Trace:
 <TASK>
 dump_stack_lvl+0x48/0x70
 dump_stack+0x10/0x20
 __ubsan_handle_out_of_bounds+0xc6/0x110
 cfs_percpt_alloc+0x138/0x480 [lnet]
 cfs_percpt_lock_create+0xe8/0x2d0 [lnet]
 lnet_lib_init+0xe7/0x400 [lnet]
 lnet_init+0xac/0xff0 [lnet]
 ? __pfx_lnet_init+0x10/0x10 [lnet]
 do_one_initcall+0x5e/0x340
 do_init_module+0xc0/0x2c0
 load_module+0xba1/0xcf0
 init_module_from_file+0x96/0x100
 ? init_module_from_file+0x96/0x100
 idempotent_init_module+0x11c/0x2b0
 __x64_sys_finit_module+0x64/0xd0
 x64_sys_call+0x1d6e/0x25c0
 do_syscall_64+0x7f/0x180
 ? do_syscall_64+0x8c/0x180
 ? syscall_exit_to_user_mode+0x86/0x260
 ? do_syscall_64+0x8c/0x180
 ? syscall_exit_to_user_mode+0x86/0x260
 ? do_syscall_64+0x8c/0x180
 ? exc_page_fault+0x94/0x1b0
 entry_SYSCALL_64_after_hwframe+0x73/0x7b
RIP: 0033:0x7ce001b2725d
Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 8b bb 0d 00 f7 d8 64 89 01 48
RSP: 002b:00007ffde0e96948 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
RAX: ffffffffffffffda RBX: 00005a202b655da0 RCX: 00007ce001b2725d
RDX: 0000000000000000 RSI: 00005a202a23be52 RDI: 0000000000000004
RBP: 00007ffde0e96a00 R08: 0000000000000040 R09: 00007ffde0e96990
R10: 00007ce001c03b20 R11: 0000000000000246 R12: 00005a202a23be52
R13: 0000000000040000 R14: 00005a202b655ce0 R15: 0000000000000000
 </TASK>
---[ end trace ]---

A full list after running runtests is in:
https://testing.whamcloud.com/test_logs/6fbc4c52-3c60-4441-8370-43386a3673c4/show_text

Issue Links

• is related to: LU-18221 improve check_dmesg_for_errors() to check UBSAN and KASAN errors
• is related to: LU-16807 Resolve newer debug kernel warnings

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/56105/
Subject: LU-17794 crypto: replace 1-element array in llcrypt_symlink_data
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 5e32fe84c1776857931497ba261100dc3b0e8e24

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/56077/
Subject: LU-17794 lustre: replace 0-length arrays with flexible arrays (3/3)
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 37acaa912f34d68f37f788d5478e04838b040fc9

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/55940/
Subject: LU-17794 lustre: replace 0-length arrays with flexible arrays (2/3)
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4920a934baf06a81157df90abb08225517352479

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/55874/
Subject: LU-17794 lustre: replace 0-length arrays with flexible arrays (1/3)
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 51a281cdb405fec311f280cfabc156c8746e2412

[Gerrit Updater]

"Jian Yu <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/56105
Subject: LU-17794 crypto: replace 1-element array in llcrypt_symlink_data
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8353fb2bcbfefeb58936789fe20159ff8c3607ce

[Gerrit Updater]

"Jian Yu <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/56077
Subject: LU-17794 lustre: replace 0-length arrays with flexible arrays (3/3)
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9c447054811a79191f7e8bbc0999b6142edd119b

[Gerrit Updater]

"Jian Yu <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/55940
Subject: LU-17794 lustre: replace 0-length arrays with flexible arrays (2/3)
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b2a01826ebb7b4f0ce5b3fb43446c8bd4c956d82

[Gerrit Updater]

"Jian Yu <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/55874
Subject: LU-17794 lustre: replace 0-length arrays with flexible arrays
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6761f8f42aa8458b0757ed53fa59e352e76c3b8f

[Jian Yu]

According to https://people.kernel.org/kees/bounded-flexible-arrays-in-c, I made changes to replace 0-length arrays with flexible arrays.
I'm testing the patch locally and will push it to Gerrit.

[Jian Yu]

Thank you, sihara. Let me work on this ticket.

[Shuichi Ihara]

here is all "UBSAN: array-index-out-of-bounds" errors when lustre modules are loaded

# dmesg | grep -B1 "out of range"
[ 1004.087907] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1168:3
[ 1004.090141] index 0 is out of range for type 'void *[*]'
--
[ 1004.091413] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1168:3
[ 1004.093513] index 0 is out of range for type 'void *[*]'
--
[ 1004.094677] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1169:20
[ 1004.096786] index 0 is out of range for type 'void *[*]'
--
[ 1005.380898] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1130:19
[ 1005.383060] index 0 is out of range for type 'void *[*]'
--
[ 1005.384761] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1131:4
[ 1005.386899] index 0 is out of range for type 'void *[*]'
--
[ 1005.388516] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1131:4
[ 1005.390622] index 0 is out of range for type 'void *[*]'
--
[ 1057.948558] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1168:3
[ 1057.950703] index 0 is out of range for type 'void *[*]'
--
[ 1057.951936] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1168:3
[ 1057.954044] index 0 is out of range for type 'void *[*]'
--
[ 1057.955216] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1169:20
[ 1057.957341] index 0 is out of range for type 'void *[*]'
--
[ 1059.178717] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/klnds/in-kernel-o2iblnd/o2iblnd.c:1431:15
[ 1059.181134] index 0 is out of range for type 'page *[*]'
--
[ 1059.182833] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/klnds/in-kernel-o2iblnd/o2iblnd.c:1433:33
[ 1059.185767] index 0 is out of range for type 'page *[*]'
--
[ 1059.190183] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/klnds/in-kernel-o2iblnd/o2iblnd.c:1584:26
[ 1059.192577] index 0 is out of range for type 'page *[*]'
--
[ 1059.234282] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-ptl.c:74:18
[ 1059.236456] index 0 is out of range for type 'int [*]'
--
[ 1059.238575] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lustre/ptlrpc/ptlrpcd.c:613:28
[ 1059.240929] index 1 is out of range for type 'ptlrpcd_ctl [*]'
--
[ 1059.242472] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lustre/ptlrpc/ptlrpcd.c:930:36
[ 1059.244617] index 1 is out of range for type 'ptlrpcd_ctl [*]'
--
[ 1059.246105] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lustre/ptlrpc/ptlrpcd.c:596:22
[ 1059.248252] index 1 is out of range for type 'ptlrpcd_ctl [*]'
--
[ 1059.249993] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lustre/ptlrpc/ptlrpcd.c:956:38
[ 1059.252677] index 1 is out of range for type 'ptlrpcd_ctl [*]'