-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
Problem
A router that marked a peer NI down drops every message routed to that
peer but never degrades its health. The peer NI stays at
LNET_MAX_HEALTH_VALUE, so it is never queued for recovery, no recovery
ping is sent, and the router black-holes the peer until LNet restarts.
Seen in LUS-13527: 5 of 13 routers black-holed an MDS peer for ~30
minutes, evicting clients. Fingerprint on an affected router:
state: down
health value: 1000
Root cause
On a router the forward runs inside lnet_finalize(): it health checks
the receive leg, then lnet_complete_msg_locked() calls lnet_send() to
forward (lib-msg.c:411-433). The forward fails in
lnet_post_send_locked(), which skips lnet_finalize() because do_send is
false, and the error returns to "if (rc != 0) goto again;" and is
discarded. The health check already ran, for the receive.
Two effects hide it: msg_ev.status is still 0 while msg_health_status is
not, which lnet_is_health_check() rejects as inconsistent; and
lnet_send() overwrites the precise status with REMOTE_ERROR.
The same gap applies to the ACK send, not just routed forwards.