Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20947

o2iblnd: PUT_DONE on a finalised conn loses tx_conn with a send posted

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      Problem

      o2iblnd can call kiblnd_tx_complete() on a PUT_DONE tx with a NULL tx_conn. The node then crashes with a NULL pointer dereference. The crash needs a conn that finalises while the scheduler handles a PUT_ACK.

      Environment

      • master at 4a7bddaf15, and each branch that has a155c3fca38d ("LU-11756 o2iblnd: kib_conn leak").
      • o2ib with any HCA. The window is wider when a device removal or a reset closes conns under traffic.

      Reproduction

      We have no reproducer. We found the path in the source during a review of the LU-6132 series.

      Analysis

      1. o2iblnd posts a PUT_REQ tx.
      2. The PUT_ACK arrives before the PUT_REQ send completes. kiblnd_handle_rx() removes the tx from ibc_active_txs. tx_sending is still 1.
      3. connd finalises the conn and sets IBLND_CONN_DISCONNECTED. kiblnd_abort_txs() does not see the tx, because the tx is on no list.
      4. kiblnd_handle_rx() calls kiblnd_queue_tx_locked() for the PUT_DONE. The DISCONNECTED branch puts the tx on ibc_zombie_txs, clears tx_conn, and drops the conn ref.
      5. The HCA flushes the PUT_REQ send. kiblnd_tx_complete() reads tx->tx_conn, which is NULL.

      If the HCA does not flush, no code completes the posted send. kiblnd_destroy_conn() then hits LASSERT(conn->ibc_nsends_posted == 0) when the last conn ref goes.

      Proposed fix

      In the DISCONNECTED branch of kiblnd_queue_tx_locked(), keep a PUT_DONE with tx_sending != 0 on ibc_active_txs with its conn ref. The send completion then releases the tx through the normal idle path.

            hornc Chris Horn
            hornc Chris Horn
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: