-
Bug
-
Resolution: Fixed
-
Medium
-
Lustre 2.18.0
-
None
-
2.17.54_183_gc088134
-
3
-
9223372036854775807
Repeatedly deleting and re-adding one NI of a two-rail LNet network on a Lustre server
panics the kernel. Reproduced 4 times on master (`2.17.54_183_gc088134`), always the
same assertion and call stack, in the `lnet_discovery` kernel thread.
[ 1680.840533] LNetError: 18633:0:(peer.c:278:lnet_destroy_peer_locked()) ASSERTION( list_empty(&lp->lp_peer_nets) ) failed: [ 1680.844355] LNetError: 18633:0:(peer.c:278:lnet_destroy_peer_locked()) LBUG [ 1680.846782] CPU: 1 PID: 18633 Comm: lnet_discovery Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.123.1.el8_lustre.ddn17.x86_64 #1 [ 1680.851151] Hardware name: DDN SFA400X3ME, BIOS 1.17.0-20260414_171146-ponderosa 04/01/2014 [ 1680.853954] Call Trace: [ 1680.854946] dump_stack+0x41/0x60 [ 1680.856202] lbug_with_loc.cold.7+0x5/0x43 [libcfs] [ 1680.857949] lnet_destroy_peer_locked+0x293/0x300 [lnet] [ 1680.859879] lnet_peer_attach_peer_ni+0x544/0x600 [lnet] [ 1680.861804] ? lnet_peer_add_nid+0x131/0x510 [lnet] [ 1680.863597] lnet_peer_merge_data+0x756/0x1250 [lnet] [ 1680.865414] lnet_peer_discovery+0x155f/0x1b50 [lnet] [ 1680.867230] ? lnet_peer_merge_data+0x1250/0x1250 [lnet] [ 1680.869121] kthread+0x134/0x150 [ 1680.870349] ? set_kthread_struct+0x50/0x50 [ 1680.871875] ret_from_fork+0x1f/0x40 [ 1680.873250] Kernel panic - not syncing: LBUG
Environment
- Lustre master, commit `c088134a8e` (`2.17.54_183_gc088134`), server build, EL8
- Kernel `4.18.0-553.123.1.el8_lustre.ddn17.x86_64`, Rocky Linux 8.10
- Server has 4 multi-rail LNet networks configured simultaneously (`tcp10`, `tcp11`, `o2ib12`, `o2ib13`), each with 2 NIs on VLAN sub-interfaces (`mlxen0.N` + `mlxen1.N`, same `/24` subnet per net) — only `o2ib12` was churned to trigger the crash
- `o2ib12`: 2 NIs on the server, `mlxen0.12` + `mlxen1.12`
- At least one client mounted over `o2ib12` with ordinary I/O in progress
Steps to Reproduce
lnetctl net del --net o2ib12 lnetctl net add --net o2ib12 --if mlxen0.12 # Reduce to a single NI sleep 3 lnetctl net add --net o2ib12 --if mlxen1.12 # Restore two NIs sleep 3
On the clients:
Verify that all peers are discovered correctly.
Repeat the above sequence multiple times.
The server occasionally hits an LBUG in lnet_destroy_peer_locked() during peer discovery. The issue occurs intermittently after repeating the above sequence several times.