Problem

The lnet_discovery thread can spin for up to 150 s. The watchdog reports a soft lockup in lnet_discovery. Messages to the affected peer stop until the spin ends. With softlockup_panic=1, the node panics.

The stack is the same as the stack in LU-19309:

lnet_peer_discovery_complete
lnet_send
lnet_select_pathway
lnet_initiate_peer_discovery

Environment

• master (2.17.60), b2_16, b_es6_0 and b_es7_0. The defect started with 72726a311814 ("LU-14810 lnet: Do not issue multiple PUSHes", 2.15.65).
• Rocky 9 aarch64 VMs, 4KB pages, tcp and o2ib (Soft-RoCE).

Reproduction

1. Configure LNet on 2 nodes, L and R. Let L discover R.
2. On L, drop each push ACK from R.
3. On L, run lnetctl discover --force for R 2 times. The second command sets LNET_PEER_FORCE_PUSH while LNET_PEER_PUSH_SENT is set.
4. Make R push to L.
5. Send an lst RPC from L to R.

Result: 5 of 5 runs spin. The discovery thread did 10M loops in 15 s.

A device removal can also start the spin. The removal loses a push ACK. At the re-add, lnet_set_link_fatal_state(ni, 0) sets LNET_PEER_FORCE_PUSH.

Analysis

Two functions disagree about LNET_PEER_FORCE_PUSH while LNET_PEER_PUSH_SENT is set:

• lnet_peer_needs_push() returns false while PUSH_SENT is set. So the discovery thread calls lnet_peer_discovered(). Then lnet_peer_discovery_complete() resends the messages on lp_dc_pendq.
• lnet_peer_is_uptodate_locked() returns false when FORCE_PUSH is set, whatever PUSH_SENT is. So each resend calls lnet_initiate_peer_discovery(). That call puts the message back on lp_dc_pendq and the peer back on ln_dc_request.

The inner loop of lnet_peer_discovery() never empties ln_dc_request, so the thread never sleeps. The spin stops when the push ACK clears PUSH_SENT. If the ACK is lost, the spin stops when the response timeout (transaction_timeout, 150 s) unlinks the push MD.

Evidence:

• A debug log at each 10M loops showed state 0x4511 (FORCE_PUSH, PUSH_SENT, DISCOVERED, MULTI_RAIL) and 1 message on lp_dc_pendq.
• In a vmcore, lnet_discovery was running in lnet_peer_discovery_complete(). The peer had lp_state 0x4511, and its push MD was still live.
• Without a panic, the watchdog reported "stuck for 183s". Then lnet_finalize_expired_responses() unlinked the push, and the spin stopped.

The LU-19309 Maloo vmcore shows a different state: a gateway peer with MARK_DELETED and no PUSH_SENT. This ticket does not cover that state.

Proposed fix

In lnet_peer_is_uptodate_locked(), do not check LNET_PEER_FORCE_PUSH. lnet_peer_needs_push() already checks it and obeys PUSH_SENT. The forced push still goes out after the ACK.

With the fix, the reproducer did not spin in 15 runs (0 of 15).