Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20871

MDS NULL pointer dereference in lod_it_init() when the lower layer's do_index_ops is not yet visible

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • MDS on aarch64, Lustre 2.15.8
    • 3
    • 9223372036854775807

      An MDS servicing MDS_READPAGE (readdir) takes a NULL pointer dereference at
      offset 0x28 and panics:

        Unable to handle kernel NULL pointer dereference at virtual address 0000000000000028
        Comm: mdt_rdpg03_060
        pc : lod_it_init+0x4c/0xdc [lod]
        lr : lod_it_init+0x30/0xdc [lod]
        Code: aa0003f3 2a1603e2 aa1503e0 aa1403e1 (f9401463)
        Call trace:
         lod_it_init+0x4c/0xdc [lod]
         dt_index_walk+0xb0/0x4e8 [obdclass]
         mdd_readpage+0xc0/0x5bc [mdd]
         mdt_readpage+0x788/0xd00 [mdt]
         tgt_handle_request0+0x2bc/0x660 [ptlrpc]
         tgt_request_handle+0x268/0xaac [ptlrpc]
         ptlrpc_server_handle_request+0x460/0xf20 [ptlrpc]
         ptlrpc_main+0xd18/0x15b8 [ptlrpc]
        

      The faulting instruction is ldr x3, x3, #0x28 with x3 == 0. x3 was
      loaded at lod_it_init+52 from dt_object_child(dt)->do_index_ops, and 0x28 is the offset of dio_it.init within
      struct dt_index_operations. The NULL is therefore the lower layer's
      ->do_index_ops.

      Root cause: a missing read-side acquire

      lod_index_try() sets the lower layer's ->do_index_ops (via
      ->do_index_try()) and then its own. A second thread never runs that
      function: dt_try_as_dir() returns early when the upper layer's
      ->do_index_ops is already set, so it neither re-runs >do_index_try()
      nor establishes any ordering. That thread performs two independent loads:

      1. the upper layer's ->do_index_ops, in dt_try_as_dir() / dt_index_walk()
      2. the lower layer's, in lod_it_init()

      with nothing ordering them. On a weakly ordered architecture the second load may
      be satisfied before the first store is observed, returning NULL. dt_index_walk()

      asserts only the upper layer's ops, so it does not catch this.

      A vmcore confirms the mechanism: the field holds a valid osd_dir_ops in the
      dump while the faulting thread had loaded NULL, so another CPU's store became
      visible in the interval. Nothing in the tree ever assigns do_index_ops = NULL,
      which rules out a teardown or clear race and leaves ordering as the only
      explanation.

      This is also why it has not been seen on x86: x86-TSO forbids load-load
      reordering, so the window is unreachable there. It is reachable on weakly
      ordered architectures.

      A second, related exposure on the lookup path

      lod_lookup() and lod_striped_lookup() call dt_lookup() (since
      eb5e3cc6, LU-13139), and dt_lookup() begins with
      LASSERT(dt->do_index_ops); on the object passed in, which is the lower
      layer. The same race therefore produces an LBUG rather than a NULL
      dereference on that path. Same defect, different symptom.

      Reproducer

      Deterministic and architecture-independent, via fault injection. Under a new
      fail_loc, have lod_index_try() publish its own ->do_index_ops while
      leaving the lower layer's unset – the state weak ordering produces – then
      return:

        #define OBD_FAIL_LOD_INDEX_OPS_RACE   0x1902
        
              if (CFS_FAIL_CHECK(OBD_FAIL_LOD_INDEX_OPS_RACE)) {
                      dt->do_index_ops = &lod_index_ops;
                      RETURN(0);
              }       
        
        mkdir $DIR/d && createmany -o $DIR/d/f 20
        fail mds1                        # cold the MDT's object cache; dt_try_as_dir()
                                         # fast-paths once ->do_index_ops is set, so the
                                         # object must be instantiated fresh
        lctl set_param fail_loc=0x1902   # on the MDS
        ls -U $DIR/d                     # unpatched: MDS panics as above
        

      A two-thread race on a single directory is not reachable, because concurrent
      readdirs serialise on the directory's LDLM lock, so the single-threaded form
      above is what reproduces it.

      Impact

      Any client able to issue readdir can crash an MDS, with no special privileges
      and no unusual arguments. The window is narrow, so it presents as a rare
      unexplained MDS panic rather than a reproducible failure, and the panic recurs
      for as long as the workload keeps instantiating directory objects on a weakly
      ordered MDS.

      Proposed fix

      1. Add an acquire-side accessor exported from obdclass/dt_object.c alongside dt_try_as_dir() (per LU-13139's direction): dt_index_ops(dt) returning smp_load_acquire(&dt->do_index_ops).
      2. Publish with release in lod_index_try(): smp_store_release(&dt->do_index_ops, &lod_index_ops) and the striped variant.
      3. A repairing accessor for the lower layer, lod_child_index_ops(), used by lod_it_init() and lod_striped_it_init(). On NULL it calls dt_try_as_dir(env, next, true) to establish the ops (idempotent for directories) and returns -ENOTDIR only if the object genuinely cannot be an index. Self-healing is required, not merely defensive: once the upper layer's ops are published, every subsequent caller takes the dt_try_as_dir() fast path, so lod_index_try() never runs again for that object and nothing would otherwise ever set the lower layer's ops. A guard that only returned an error leaves the directory unreadable until it is evicted from the object cache; this was observed during testing.
      4. Soften dt_lookup()'s LASSERT(dt->do_index_ops) to an error return for the lookup path.
      5. Add sanity.sh test_132 driving the fail_loc above, asserting that readdir succeeds, that the repair path actually ran (so the test cannot pass vacuously), and that the directory is still readable after the fault is cleared.

            wc-triage WC Triage
            wamax Max Wang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: