Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20842

LNet: Peer discovery ping/push concurrency ref leak

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Medium Medium
    • Lustre 2.18.0
    • None
    • None
    • 3
    • 9223372036854775807

      Description

      A peer with both `lp_ping_mdh` and `lp_push_mdh` outstanding gets both unlinked by `lnet_peer_cancel_discovery()` during queue cleanup 1. `LNetMDUnlink()` only delivers the unlink event inline when `md_refcount == 0`; a busy MD defers it to the LND completion path. The inline event moves the peer off `ln_dc_expired`, so cleanup 2 stops waiting and cleanup 3 dequeues it via `lnet_peer_discovery_complete(lp, -ESHUTDOWN)` — which sets `LNET_PEER_REDISCOVER`, so `lnet_peer_is_uptodate()` stays false. When the deferred event finally arrives, `lnet_peer_queue_for_discovery()` finds `lp_dc_list` empty, takes a fresh ref, and puts the peer back on `ln_dc_request`, ultimately causing a reference leak. 

      Trigger conditions & timing

      1. A peer has both `lp_ping_mdh` and `lp_push_mdh` outstanding;
      2. `lnet_peer_cancel_discovery()` triggers unlink of both MDs during queue cleanup 1;
      3. The unlink includes one inline if idle and one deferred event if busy;
      4. The inline unlink completes cleanup 2 and cleanup 3, and drains the queue;
      5. If the deferred unlink arrives afterward, it re-inserts the peer into `ln_dc_request`, and this reference will never be released, causing a leak.

      Impact & errors

      1. May trigger `LASSERT(list_empty(&the_lnet.ln_dc_request))`;
      2. May report "Memory leaks detected" at module unload;
      3. May trigger `LASSERT(md->md_handler != handler)` in `lnet_assert_handler_unused()`.

      Once the erroneous timing window is hit, a reference leak occurs; the final error point depends on the concurrency timing of the peer re-inserting and the detection point.

            xqu Xinxia Qu
            xqu Xinxia Qu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: