Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20950

lnet: netlink fault command oopses before LNet is configured

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      Problem

      lnetctl fault delay del --all causes a kernel oops after insmod of lnet.ko and before the first lnetctl lnet configure. The oops is a NULL pointer dereference in lnet_delay_rule_del(). The oopsed thread holds delay_dd.dd_mutex and LNET_LOCK_EX. After the oops, lnetctl lnet unconfigure stays in D state, LNet cannot unload, and the node needs a reboot.

      Environment

      • master at 4a7bddaf15, plus the unlanded LU-15909 patches. These patches do not change the fault code.
      • Ubuntu 22.04, kernel 5.15.0-47-generic, x86_64, MLNX_OFED 23.10. Modules loaded with insmod from the build tree.

      Reproduction

      This sequence happened during a lab test. We did not run it again as a standalone reproducer.

      1. Load libcfs.ko and lnet.ko with insmod.
      2. Do not run lnetctl lnet configure.
      3. Run lnetctl fault delay del --all.

      Analysis

      The console log:

      BUG: kernel NULL pointer dereference, address: 0000000000000000
      Oops: 0000 [#1] SMP NOPTI
      CPU: 6 PID: 1335053 Comm: lnetctl Tainted: G           OE     5.15.0-47-generic #51-Ubuntu
      RIP: 0010:lnet_delay_rule_del+0x92/0x610 [lnet]
      RSI: 00000000ffffffff RBX: 0000000000000000
      Call Trace:
       lnet_fault_cmd+0x25a/0xc50 [lnet]
       genl_family_rcv_msg_doit+0xe7/0x150
       genl_rcv_msg+0xe2/0x1f0
       netlink_rcv_skb+0x55/0x100
       genl_rcv+0x29/0x40
      

      lnet_delay_rule_del() takes delay_dd.dd_mutex and lnet_net_lock(LNET_LOCK_EX) (RSI = -1). It then walks the_lnet.ln_delay_rules. Only lnet_prepare() initializes that list head, in LNetNIInit(). Before the first lnet configure, the list head is zero, so the walk reads address 0 (RBX = 0).

      The ioctl path does not have this problem. lnet_ioctl() calls LNetNIInit() before LNetCtl(), so lctl net_delay_del initializes LNet first. The netlink handler lnet_fault_cmd() calls the rule functions directly, and it does not check the_lnet.ln_refcount. Other netlink handlers in api-ni.c check ln_refcount.

      This paragraph is an inference from the source. We did not test it. The other fault operations have the same problem. The functions lnet_drop_rule_del(), lnet_drop_rule_reset() and lnet_delay_rule_reset() walk an uninitialized list. The functions lnet_drop_rule_add() and lnet_delay_rule_add() add a rule to an uninitialized list head.

      The netlink path came from commit f1c66236c6 ("LU-10391 lnet: Fault injection add/del ioctls to netlink").

      Proposed fix

      In lnet_fault_cmd(), take the_lnet.ln_api_mutex and return -ENETDOWN when the_lnet.ln_refcount is 0, as the other netlink handlers do. Add a sanity-lnet test that loads the lnet module, runs each fault command before lnetctl lnet configure, and checks for an error.

            hornc Chris Horn
            hornc Chris Horn
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: