Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20580

o2iblnd: sleeping in atomic in kiblnd_startup() via rtnl_lock()

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      kiblnd_startup() calls lnet_get_link_status() from inside the
      write_lock_irqsave(&kiblnd_data.kib_global_lock) section:

        write_lock_irqsave(&kiblnd_data.kib_global_lock, flags);
        ibdev->ibd_nnets++;
        list_add_tail(&net->ibn_list, &ibdev->ibd_nets);
        if (ibdev->ibd_hdev->ibh_state == IBLND_DEV_PORT_DOWN)
                kiblnd_set_ni_fatal_on(ibdev->ibd_hdev, 1);
      
        netdev = dev_get_by_name(ni->ni_net_ns, net->ibn_dev->ibd_ifname);
      
        if (netdev &&
            ((netdev->reg_state == NETREG_UNREGISTERING) ||
             (netdev->operstate != IF_OPER_UP) ||
            (lnet_get_link_status(netdev) == 0))) {
                kiblnd_set_ni_fatal_on(ibdev->ibd_hdev, 1);
        }
        dev_put(netdev);
        write_unlock_irqrestore(&kiblnd_data.kib_global_lock, flags);
      

      lnet_get_link_status() takes rtnl_lock() and netdev_lock_ops(). rtnl_lock()
      is mutex_lock(), which can sleep. We call it with preemption disabled and
      interrupts off.

      LU-20102 removed the rcu_read_lock() around this code but left the call
      inside the rwlock section, so the sleep in atomic context stays.

      The failure is silent while rtnl_mutex is free, because the mutex fast path
      does not schedule, and might_sleep() reports nothing unless
      CONFIG_DEBUG_ATOMIC_SLEEP is set. When rtnl_mutex is held by another task,
      __mutex_lock_slowpath() calls schedule_preempt_disabled() and the kernel
      reports:

        BUG: scheduling while atomic: lnetctl/8668/0x00000002
      

      preempt_count 0x2 is 1 from write_lock_irqsave() and 1 from
      schedule_preempt_disabled().

      This was seen on an aarch64 client (kernel 7.0.0-1013-nvidia-64k, 64K pages,
      NVIDIA GB300 NVL) during boot, when systemd-networkd and udev hold
      rtnl_mutex while LNet is configured. The node does not boot.

      After the report the node panics. schedule_debug() calls __schedule_bug()
      and then preempt_count_set(PREEMPT_DISABLED), so the later unlocks no longer
      balance. The task returns to user space with a non-zero preempt count, and
      the next ordinary user page fault takes the faulthandler_disabled() path:

        Unable to handle kernel paging request at virtual address 0000fffff7cc59f4
        ESR = 0x0000000082000007
        EC = 0x20: IABT (lower EL), IL = 32 bits
        pc : 0000fffff7cc59f4
        lr : 0000fffff7fb4910
        ...
        Kernel panic - not syncing: Aiee, killing interrupt handler!
      

      The address is a user address (x16 is the aarch64 PLT register and holds the
      same value), so the Oops carries no kernel call trace.

      The fix is to read the net device state before the lock is taken, and to
      take kib_global_lock only to publish the result.

            hornc Chris Horn
            hornc Chris Horn
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: