Details
-
Bug
-
Resolution: Fixed
-
Minor
-
Lustre 2.16.0
-
3
-
9223372036854775807
Description
While testing master branch client build on Ubuntu 24.04 with kernel 6.8.0-31-generic, lots of runtime UBSAN array-index-out-of-bounds errors occurred in dmesg/syslog:
------------[ cut here ]------------ UBSAN: array-index-out-of-bounds in /root/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1168:3 index 0 is out of range for type 'void *[*]' CPU: 0 PID: 111835 Comm: modprobe Tainted: G OE 6.8.0-31-generic #31-Ubuntu Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 Call Trace: <TASK> dump_stack_lvl+0x48/0x70 dump_stack+0x10/0x20 __ubsan_handle_out_of_bounds+0xc6/0x110 cfs_percpt_alloc+0x138/0x480 [lnet] cfs_percpt_lock_create+0xe8/0x2d0 [lnet] lnet_lib_init+0xe7/0x400 [lnet] lnet_init+0xac/0xff0 [lnet] ? __pfx_lnet_init+0x10/0x10 [lnet] do_one_initcall+0x5e/0x340 do_init_module+0xc0/0x2c0 load_module+0xba1/0xcf0 init_module_from_file+0x96/0x100 ? init_module_from_file+0x96/0x100 idempotent_init_module+0x11c/0x2b0 __x64_sys_finit_module+0x64/0xd0 x64_sys_call+0x1d6e/0x25c0 do_syscall_64+0x7f/0x180 ? do_syscall_64+0x8c/0x180 ? syscall_exit_to_user_mode+0x86/0x260 ? do_syscall_64+0x8c/0x180 ? syscall_exit_to_user_mode+0x86/0x260 ? do_syscall_64+0x8c/0x180 ? exc_page_fault+0x94/0x1b0 entry_SYSCALL_64_after_hwframe+0x73/0x7b RIP: 0033:0x7ce001b2725d Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 8b bb 0d 00 f7 d8 64 89 01 48 RSP: 002b:00007ffde0e96948 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 RAX: ffffffffffffffda RBX: 00005a202b655da0 RCX: 00007ce001b2725d RDX: 0000000000000000 RSI: 00005a202a23be52 RDI: 0000000000000004 RBP: 00007ffde0e96a00 R08: 0000000000000040 R09: 00007ffde0e96990 R10: 00007ce001c03b20 R11: 0000000000000246 R12: 00005a202a23be52 R13: 0000000000040000 R14: 00005a202b655ce0 R15: 0000000000000000 </TASK> ---[ end trace ]---
A full list after running runtests is in:
https://testing.whamcloud.com/test_logs/6fbc4c52-3c60-4441-8370-43386a3673c4/show_text
Attachments
Issue Links
Activity
"Jian Yu <yujian@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/55940
Subject: LU-17794 lustre: replace 0-length arrays with flexible arrays (2/3)
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b2a01826ebb7b4f0ce5b3fb43446c8bd4c956d82
"Jian Yu <yujian@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/55874
Subject: LU-17794 lustre: replace 0-length arrays with flexible arrays
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6761f8f42aa8458b0757ed53fa59e352e76c3b8f
According to https://people.kernel.org/kees/bounded-flexible-arrays-in-c, I made changes to replace 0-length arrays with flexible arrays.
I'm testing the patch locally and will push it to Gerrit.
here is all "UBSAN: array-index-out-of-bounds" errors when lustre modules are loaded
# dmesg | grep -B1 "out of range" [ 1004.087907] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1168:3 [ 1004.090141] index 0 is out of range for type 'void *[*]' -- [ 1004.091413] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1168:3 [ 1004.093513] index 0 is out of range for type 'void *[*]' -- [ 1004.094677] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1169:20 [ 1004.096786] index 0 is out of range for type 'void *[*]' -- [ 1005.380898] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1130:19 [ 1005.383060] index 0 is out of range for type 'void *[*]' -- [ 1005.384761] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1131:4 [ 1005.386899] index 0 is out of range for type 'void *[*]' -- [ 1005.388516] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1131:4 [ 1005.390622] index 0 is out of range for type 'void *[*]' -- [ 1057.948558] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1168:3 [ 1057.950703] index 0 is out of range for type 'void *[*]' -- [ 1057.951936] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1168:3 [ 1057.954044] index 0 is out of range for type 'void *[*]' -- [ 1057.955216] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-cpt.c:1169:20 [ 1057.957341] index 0 is out of range for type 'void *[*]' -- [ 1059.178717] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/klnds/in-kernel-o2iblnd/o2iblnd.c:1431:15 [ 1059.181134] index 0 is out of range for type 'page *[*]' -- [ 1059.182833] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/klnds/in-kernel-o2iblnd/o2iblnd.c:1433:33 [ 1059.185767] index 0 is out of range for type 'page *[*]' -- [ 1059.190183] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/klnds/in-kernel-o2iblnd/o2iblnd.c:1584:26 [ 1059.192577] index 0 is out of range for type 'page *[*]' -- [ 1059.234282] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lnet/lnet/lib-ptl.c:74:18 [ 1059.236456] index 0 is out of range for type 'int [*]' -- [ 1059.238575] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lustre/ptlrpc/ptlrpcd.c:613:28 [ 1059.240929] index 1 is out of range for type 'ptlrpcd_ctl [*]' -- [ 1059.242472] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lustre/ptlrpc/ptlrpcd.c:930:36 [ 1059.244617] index 1 is out of range for type 'ptlrpcd_ctl [*]' -- [ 1059.246105] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lustre/ptlrpc/ptlrpcd.c:596:22 [ 1059.248252] index 1 is out of range for type 'ptlrpcd_ctl [*]' -- [ 1059.249993] UBSAN: array-index-out-of-bounds in /work/ihara/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lustre/ptlrpc/ptlrpcd.c:956:38 [ 1059.252677] index 1 is out of range for type 'ptlrpcd_ctl [*]'
Oh, i see.. patch https://review.whamcloud.com/55418 was not merged, but abandoned?
Hi yujian
I've built master 4ad0861d90 (with LU-18011, LU-16350 and LU-18010) for my ubuntu 24.04 server.
But, I still saw same warnings below. I thought this was resolved by https://review.whamcloud.com/55418 and patch should be included. Anything am I still missing here?
[32409.549000] ------------[ cut here ]------------ [32409.549003] UBSAN: array-index-out-of-bounds in /home/ubuntu/lustre-release/debian/tmp/modules-deb/usr_src/modules/lustre/lustre/ptlrpc/ptlrpcd.c:956:38 [32409.551024] index 1 is out of range for type 'ptlrpcd_ctl [*]' [32409.551711] CPU: 10 PID: 442784 Comm: modprobe Tainted: G OE 6.8.0-38-generic #38-Ubuntu [32409.551714] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009)/Incus, BIOS 2024.02-2 03/11/2024 [32409.551715] Call Trace: [32409.551718] <TASK> [32409.551720] dump_stack_lvl+0x76/0xa0 [32409.551724] dump_stack+0x10/0x20 [32409.551726] __ubsan_handle_out_of_bounds+0xc6/0x110 [32409.551729] ptlrpcd_addref+0x143e/0x1460 [ptlrpc] [32409.551827] ? percpu_ref_init+0x99/0x180 [32409.551830] ptlrpc_init_portals+0x68/0x2c0 [ptlrpc] [32409.551924] ptlrpc_init+0x241/0xff0 [ptlrpc] [32409.552019] ? __pfx_ptlrpc_init+0x10/0x10 [ptlrpc] [32409.552112] do_one_initcall+0x5b/0x340 [32409.552117] do_init_module+0xc0/0x2c0 [32409.552119] load_module+0xba1/0xcf0 [32409.552122] init_module_from_file+0x96/0x100 [32409.552124] ? init_module_from_file+0x96/0x100 [32409.552127] idempotent_init_module+0x11c/0x2b0 [32409.552129] __x64_sys_finit_module+0x64/0xd0 [32409.552132] x64_sys_call+0x1d6e/0x25c0 [32409.552134] do_syscall_64+0x7f/0x180 [32409.552136] ? ext4_llseek+0xc3/0x130 [32409.552138] ? ksys_lseek+0x7d/0xd0 [32409.552141] ? syscall_exit_to_user_mode+0x89/0x260 [32409.552144] ? do_syscall_64+0x8c/0x180 [32409.552147] ? putname+0x5b/0x80 [32409.552149] ? do_sys_openat2+0x9f/0xe0 [32409.552151] ? __x64_sys_openat+0x55/0xa0 [32409.552153] ? syscall_exit_to_user_mode+0x89/0x260 [32409.552156] ? do_syscall_64+0x8c/0x180 [32409.552157] ? putname+0x5b/0x80 [32409.552159] ? do_sys_openat2+0x9f/0xe0 [32409.552161] ? __x64_sys_openat+0x55/0xa0 [32409.552163] ? syscall_exit_to_user_mode+0x89/0x260 [32409.552165] ? do_syscall_64+0x8c/0x180 [32409.552167] ? do_syscall_64+0x8c/0x180 [32409.552169] ? clear_bhb_loop+0x15/0x70 [32409.552172] ? clear_bhb_loop+0x15/0x70 [32409.552175] ? clear_bhb_loop+0x15/0x70 [32409.552177] entry_SYSCALL_64_after_hwframe+0x78/0x80 [32409.552180] RIP: 0033:0x7db400f2725d [32409.552192] Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 8b bb 0d 00 f7 d8 64 89 01 48 [32409.552193] RSP: 002b:00007ffcfbdfbd78 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [32409.552195] RAX: ffffffffffffffda RBX: 000055e148b04880 RCX: 00007db400f2725d [32409.552197] RDX: 0000000000000000 RSI: 000055e148883e52 RDI: 0000000000000007 [32409.552198] RBP: 00007ffcfbdfbe30 R08: 0000000000000040 R09: 00007ffcfbdfbdc0 [32409.552199] R10: 00007db401003b20 R11: 0000000000000246 R12: 000055e148883e52 [32409.552200] R13: 0000000000040000 R14: 000055e148b04830 R15: 0000000000000000 [32409.552202] </TASK> [32409.552203] ---[ end trace ]---
After https://review.whamcloud.com/55418 ("LU-17793 build: disable objtool and UBSAN warnings") lands to master branch, this ticket will not be blocking Lustre 2.16.0 release.
Comments from Andreas:
Since UBSAN warnings are generated at runtime when code is executed, it depends on actually exercising code paths in order to generate warnings. It probably makes sense to start with running a patch that re-enables UBSAN and adds a search for UBSAN errors during the test, something like the following in run_one():
do_nodes $all_nodes dmesg | tac | sed "/${TESTNAME/_ / }/,$ d" |
����������������grep "UBSAN" && error "UBSAN issue detected"
and runs with fortestonly on only a very simple test like "runtests" to collect the warnings, fix them, then move on to increasingly complex tests like sanity, sanityn, replay-, recovery-, conf-sanity, racer, etc. and fix them incrementally. Patches should be pushed directly to master/b_es6_0 without the UBSAN-enable patch
After the majority of warnings are fixed during normal operations, the UBSAN-enable patch can land and we will just get bug reports as they are hit in the field.
"Jian Yu <yujian@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/56077
Subject:
LU-17794lustre: replace 0-length arrays with flexible arrays (3/3)Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9c447054811a79191f7e8bbc0999b6142edd119b