-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
Problem
lnetctl fault delay del --all causes a kernel oops after insmod of lnet.ko and before the first lnetctl lnet configure. The oops is a NULL pointer dereference in lnet_delay_rule_del(). The oopsed thread holds delay_dd.dd_mutex and LNET_LOCK_EX. After the oops, lnetctl lnet unconfigure stays in D state, LNet cannot unload, and the node needs a reboot.
Environment
- master at 4a7bddaf15, plus the unlanded LU-15909 patches. These patches do not change the fault code.
- Ubuntu 22.04, kernel 5.15.0-47-generic, x86_64, MLNX_OFED 23.10. Modules loaded with insmod from the build tree.
Reproduction
This sequence happened during a lab test. We did not run it again as a standalone reproducer.
1. Load libcfs.ko and lnet.ko with insmod.
2. Do not run lnetctl lnet configure.
3. Run lnetctl fault delay del --all.
Analysis
The console log:
BUG: kernel NULL pointer dereference, address: 0000000000000000 Oops: 0000 [#1] SMP NOPTI CPU: 6 PID: 1335053 Comm: lnetctl Tainted: G OE 5.15.0-47-generic #51-Ubuntu RIP: 0010:lnet_delay_rule_del+0x92/0x610 [lnet] RSI: 00000000ffffffff RBX: 0000000000000000 Call Trace: lnet_fault_cmd+0x25a/0xc50 [lnet] genl_family_rcv_msg_doit+0xe7/0x150 genl_rcv_msg+0xe2/0x1f0 netlink_rcv_skb+0x55/0x100 genl_rcv+0x29/0x40
lnet_delay_rule_del() takes delay_dd.dd_mutex and lnet_net_lock(LNET_LOCK_EX) (RSI = -1). It then walks the_lnet.ln_delay_rules. Only lnet_prepare() initializes that list head, in LNetNIInit(). Before the first lnet configure, the list head is zero, so the walk reads address 0 (RBX = 0).
The ioctl path does not have this problem. lnet_ioctl() calls LNetNIInit() before LNetCtl(), so lctl net_delay_del initializes LNet first. The netlink handler lnet_fault_cmd() calls the rule functions directly, and it does not check the_lnet.ln_refcount. Other netlink handlers in api-ni.c check ln_refcount.
This paragraph is an inference from the source. We did not test it. The other fault operations have the same problem. The functions lnet_drop_rule_del(), lnet_drop_rule_reset() and lnet_delay_rule_reset() walk an uninitialized list. The functions lnet_drop_rule_add() and lnet_delay_rule_add() add a rule to an uninitialized list head.
The netlink path came from commit f1c66236c6 ("LU-10391 lnet: Fault injection add/del ioctls to netlink").
Proposed fix
In lnet_fault_cmd(), take the_lnet.ln_api_mutex and return -ENETDOWN when the_lnet.ln_refcount is 0, as the other netlink handlers do. Add a sanity-lnet test that loads the lnet module, runs each fault command before lnetctl lnet configure, and checks for an error.