A multi-rail server suffered an IB port failure that triggered an extended outage with all routes periodically flapping.

Server is Multi-Rail on o2ib100 (10.80 + 10.81). When the 10.81 port went down, LNet kept attempting connections to the routers over that dead port. Each failed attempt (rdma_resolve_addr → ADDR ERROR -110) caused LNet, because the peers are routers and discovery is disabled, to mark the route through that router down, even though the router was fully reachable over the healthy 10.80 port. Router pings then restored the routes, the next failed connection knocked them down again, and the routes flapped continuously -> no route to <client> on the healthy rail -> I/O stalls -> client evictions. In short: a dead local port took down routes to routers that were reachable on the surviving port, because LNet attributed the local-port connection failure to the router.

In lnet_notify(), this logic assumes that if discovery is disabled then the peers are not multi-rail. However, the site was statically defining MR peers with DD disabled, so the logic is flawed:

        if (lp->lp_rtr_refcount) {
                if (reset)
                        /* reset flag indicates gateway peer went up or down */
                        lp->lp_alive = alive;
                /* If discovery is disabled, locally or on the gateway, then
                 * any routes using lpni as next-hop need to be updated
                 */
                if (lnet_is_discovery_disabled(lp)) {
                        list_for_each_entry(route, &lp->lp_routes, lr_gwlist) {
                                if (nid_same(&route->lr_nid, &lpni->lpni_nid))
                                        lnet_set_route_aliveness(route, alive);