[LU-5568] kernel crash when when network initialization failed

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.7.0
• Affects Version/s: Lustre 2.7.0
• Labels: MB patch

Epic/Theme

lnet

• Severity: 3
• Assignee: Amir Shehata (Inactive)
• Reporter: Wang Shilong (Inactive)

Description

105.976884] Lustre: Lustre: Build Version: v2_6_51_0-g16c7568-CHANGED-3.10.1.el7_lustre
[ 105.990490] LNetError: 2145:0:(socklnd.c:2660:ksocknal_enumerate_interfaces()) Can't find any usable interfaces
[ 106.990120] LNetError: 105-4: Error -100 starting up LNI tcp
[ 106.992703] LNetError: 2145:0:(api-ni.c:823:lnet_unprepare()) ASSERTION( list_empty(&the_lnet.ln_nis) ) failed:
[ 106.994560] LNetError: 2145:0:(api-ni.c:823:lnet_unprepare()) LBUG
[ 106.994561] Pid: 2145, comm: modprobe
[ 106.994561] \x0aCall Trace:
[ 106.994574] [<ffffffffa044f853>] libcfs_debug_dumpstack+0x53/0x80 [libcfs]
[ 106.994578] [<ffffffffa044fdf5>] lbug_with_loc+0x45/0xc0 [libcfs]
[ 106.994585] [<ffffffffa04f3267>] lnet_unprepare+0x297/0x340 [lnet]
[ 106.994587] [<ffffffffa04f3b5c>] LNetNIInit+0x25c/0x3e0 [lnet]
[ 106.994592] [<ffffffff81061bc6>] ? put_online_cpus+0x56/0x80
[ 106.994631] [<ffffffffa0983000>] ? init_module+0x0/0x1000 [ptlrpc]
[ 106.994658] [<ffffffffa081310c>] ptlrpc_ni_init+0x2c/0x1a0 [ptlrpc]
[ 106.994679] [<ffffffffa0983000>] ? init_module+0x0/0x1000 [ptlrpc]
[ 106.994703] [<ffffffffa0813291>] ptlrpc_init_portals+0x11/0xf0 [ptlrpc]
[ 106.994722] [<ffffffffa0983000>] ? init_module+0x0/0x1000 [ptlrpc]
[ 106.994739] [<ffffffffa09831c4>] init_module+0x1c4/0x1000 [ptlrpc]
[ 106.994742] [<ffffffff810020e2>] do_one_initcall+0xe2/0x190
[ 106.994744] [<ffffffff810ca7fb>] load_module+0x129b/0x1a90
[ 106.994745] [<ffffffff812da590>] ? ddebug_dyndbg_module_param_cb+0x0/0x60
[ 106.994747] [<ffffffff810c7133>] ? copy_module_from_fd.isra.43+0x53/0x150
[ 106.994748] [<ffffffff810cb1a6>] SyS_finit_module+0xa6/0xd0
[ 106.994750] [<ffffffff815f2119>] system_call_fastpath+0x16/0x1b
[ 106.994750]
[ 106.995032] Kernel panic - not syncing: LBUG
[ 106.995034] CPU: 3 PID: 2145 Comm: modprobe Tainted: GF O-------------- 3.10.1.el7_lustre #1
[ 106.995034] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
[ 106.995036] ffffffffa0474d0d 00000000d711e588 ffff880036601bf0 ffffffff815e19ba
[ 106.995037] ffff880036601c70 ffffffff815db549 ffffffff00000008 ffff880036601c80
[ 106.995037] ffff880036601c20 00000000d711e588 ffffffffa051574f 0000000000000000
[ 106.995038] Call Trace:
[ 106.995052] [<ffffffff815e19ba>] dump_stack+0x19/0x1b
[ 106.995055] [<ffffffff815db549>] panic+0xd8/0x1e7
[ 106.995062] [<ffffffffa044fe5b>] lbug_with_loc+0xab/0xc0 [libcfs]
[ 106.995067] [<ffffffffa04f3267>] lnet_unprepare+0x297/0x340 [lnet]
[ 106.995070] [<ffffffffa04f3b5c>] LNetNIInit+0x25c/0x3e0 [lnet]
[ 106.995072] [<ffffffff81061bc6>] ? put_online_cpus+0x56/0x80
[ 106.995088] [<ffffffffa0983000>] ? 0xffffffffa0982fff
[ 106.995113] [<ffffffffa081310c>] ptlrpc_ni_init+0x2c/0x1a0 [ptlrpc]
[ 106.995117] [<ffffffffa0983000>] ? 0xffffffffa0982fff
[ 106.995138] [<ffffffffa0813291>] ptlrpc_init_portals+0x11/0xf0 [ptlrpc]
[ 106.995141] [<ffffffffa0983000>] ? 0xffffffffa0982fff
[ 106.995179] [<ffffffffa09831c4>] init_module+0x1c4/0x1000 [ptlrpc]
[ 106.995181] [<ffffffff810020e2>] do_one_initcall+0xe2/0x190
[ 106.995182] [<ffffffff810ca7fb>] load_module+0x129b/0x1a90
[ 106.995185] [<ffffffff812da590>] ? ddebug_proc_write+0xf0/0xf0
[ 106.995186] [<ffffffff810c7133>] ? copy_module_from_fd.isra.43+0x53/0x150
[ 106.995187] [<ffffffff810cb1a6>] SyS_finit_module+0xa6/0xd0
[ 106.995189] [<ffffffff815f2119>] system_call_fastpath+0x16/0x1b

Issue Links

• duplicates: LU-5664 assertion in failure handling of LNetNIInit
• is duplicated by: LU-5884 bad lnet conf causes LBUG
• is related to: LU-5734 LNet dynamic control: lnet_dyn_add_ni() can't clean up failed NI in some cases
• is related to: LU-2456 Dynamic LNet Config Main Development Work