-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
kiblnd_startup() calls lnet_get_link_status() from inside the
write_lock_irqsave(&kiblnd_data.kib_global_lock) section:
write_lock_irqsave(&kiblnd_data.kib_global_lock, flags);
ibdev->ibd_nnets++;
list_add_tail(&net->ibn_list, &ibdev->ibd_nets);
if (ibdev->ibd_hdev->ibh_state == IBLND_DEV_PORT_DOWN)
kiblnd_set_ni_fatal_on(ibdev->ibd_hdev, 1);
netdev = dev_get_by_name(ni->ni_net_ns, net->ibn_dev->ibd_ifname);
if (netdev &&
((netdev->reg_state == NETREG_UNREGISTERING) ||
(netdev->operstate != IF_OPER_UP) ||
(lnet_get_link_status(netdev) == 0))) {
kiblnd_set_ni_fatal_on(ibdev->ibd_hdev, 1);
}
dev_put(netdev);
write_unlock_irqrestore(&kiblnd_data.kib_global_lock, flags);
lnet_get_link_status() takes rtnl_lock() and netdev_lock_ops(). rtnl_lock()
is mutex_lock(), which can sleep. We call it with preemption disabled and
interrupts off.
LU-20102 removed the rcu_read_lock() around this code but left the call
inside the rwlock section, so the sleep in atomic context stays.
The failure is silent while rtnl_mutex is free, because the mutex fast path
does not schedule, and might_sleep() reports nothing unless
CONFIG_DEBUG_ATOMIC_SLEEP is set. When rtnl_mutex is held by another task,
__mutex_lock_slowpath() calls schedule_preempt_disabled() and the kernel
reports:
BUG: scheduling while atomic: lnetctl/8668/0x00000002
preempt_count 0x2 is 1 from write_lock_irqsave() and 1 from
schedule_preempt_disabled().
This was seen on an aarch64 client (kernel 7.0.0-1013-nvidia-64k, 64K pages,
NVIDIA GB300 NVL) during boot, when systemd-networkd and udev hold
rtnl_mutex while LNet is configured. The node does not boot.
After the report the node panics. schedule_debug() calls __schedule_bug()
and then preempt_count_set(PREEMPT_DISABLED), so the later unlocks no longer
balance. The task returns to user space with a non-zero preempt count, and
the next ordinary user page fault takes the faulthandler_disabled() path:
Unable to handle kernel paging request at virtual address 0000fffff7cc59f4 ESR = 0x0000000082000007 EC = 0x20: IABT (lower EL), IL = 32 bits pc : 0000fffff7cc59f4 lr : 0000fffff7fb4910 ... Kernel panic - not syncing: Aiee, killing interrupt handler!
The address is a user address (x16 is the aarch64 PLT register and holds the
same value), so the Oops carries no kernel call trace.
The fix is to read the net device state before the lock is taken, and to
take kib_global_lock only to publish the result.