Problem

The route round robin in lnet_find_route_locked() is not wrap-safe. After 2^31 routed sends to one remote net, lr_seq wraps. From then on, one route takes all the traffic to that net, and the round robin does not recover.

Since LU-18444, router_sensitivity_percentage has no effect. Three comments and the lnetctl help text still describe the old effect.

Environment

• master at 4a7bddaf15. The LU-18444 fix (70c5f58403) is in this tree.
• Any node that sends to one remote net through 2 or more routes with equal priority, hops and gateway health.

Reproduction

1. Configure 2 routes to one remote net with equal priority and hops.
2. Send 2^31 or more messages through the 2 routes. To get there faster, use a debugger to set lr_seq of both routes near INT_MAX.
3. Look at the gateway of each send in the D_NET log ("Selected next-hop").

Result: all sends use one route.

Analysis

lr_seq in struct lnet_route is an int. After each routed send, lnet_handle_find_routed_path() sets best_route->lr_seq = last_route->lr_seq + 1. Thus the largest lr_seq on the net grows by 1 for each send.

lnet_find_route_locked() picks two routes:

• last_route is the route with the largest lr_seq (last_route->lr_seq < route->lr_seq).
• best_route is the route with the smallest lr_seq among equal routes (route->lr_seq > best_route->lr_seq skips a route).

Example with routes A and B. A has lr_seq INT_MAX. The next send uses B and sets B to INT_MAX + 1, which wraps to INT_MIN. Now B has the smallest value, and A has the largest. Each send then picks B and sets B to INT_MAX + 1 again. A gets no more traffic.

Before LU-18444, the last_route compare used a difference (last_route->lr_seq - route->lr_seq < 0). With that compare, the skew stopped after about 2^32 sends. The LU-18444 refactor changed it to a plain compare, so the skew is now permanent.

The stale text:

• lnet/lnet/router.c: the comment above router_sensitivity_percentage describes the old health threshold.
• lnet/utils/lnetctl.c: the lnetctl set router_sensitivity help says that 100 needs fully healthy router interfaces.
• lnet/utils/lnetconfig/liblnetconfig.h: the lustre_lnet_config_rtr_sensitivity comment says that LNet treats an interface below the percentage as failed.
• lnet/lnet/lib-msg.c: a comment says that routers "rely on health feature to manage route and peer aliveness". Route aliveness now uses only the NI status. Peer NI health only ranks gateways and MR forwarding paths.

Proposed fix

• Make lr_seq an unsigned int.
• Compare with a signed difference, as time_after() does: (int)(route->lr_seq - last_route->lr_seq) > 0 and (int)(route->lr_seq - best_route->lr_seq) > 0.
• In the router.c comment, the lnetctl help and liblnetconfig.h, say that router_sensitivity_percentage is deprecated and has no effect.
• Change the lib-msg.c comment: gateway selection and MR forwarding use peer NI health.

The finding is source-only. We did not run the wrap on a VM.