[LU-11143] Multi-Rail/Dynamic Discovery break LNet router checker and asymmetric route failure detection

Related — Fri, 27 Jan 2023 17:04:53 +0000

The LNet router checker needs to ping the interface defined in the route table, but MR can choose a different interface for those pings.
As a result the router can end up marking some interfaces down because they aren't seeing any traffic in the check_interval + router ping timeout window.
This causes routes to be considered down because of asymmetric route failure detection.

Reproduced on a three node VM.
Router:

sles15build01:/tmp # lctl list_nids
192.168.2.20@tcp99
192.168.2.20@tcp1
192.168.2.20@tcp
sles15build01:~ # lnetctl route show -v
sles15build01:~ # lnetctl peer show -v
peer:
    - primary nid: 192.168.2.22@tcp99
      Multi-Rail: True
      peer ni:
        - nid: 192.168.2.22@tcp1
          state: up
          max_ni_tx_credits: 8
          available_tx_credits: 8
          min_tx_credits: 7
          tx_q_num_of_buf: 0
          available_rtr_credits: 8
          min_rtr_credits: 7
          refcount: 1
          statistics:
              send_count: 22
              recv_count: 22
              drop_count: 0
        - nid: 192.168.2.22@tcp99
          state: up
          max_ni_tx_credits: 8
          available_tx_credits: 8
          min_tx_credits: 7
          tx_q_num_of_buf: 0
          available_rtr_credits: 8
          min_rtr_credits: 7
          refcount: 1
          statistics:
              send_count: 18
              recv_count: 18
              drop_count: 0
    - primary nid: 192.168.2.21@tcp
      Multi-Rail: True
      peer ni:
        - nid: 192.168.2.21@tcp
          state: up
          max_ni_tx_credits: 8
          available_tx_credits: 8
          min_tx_credits: 7
          tx_q_num_of_buf: 0
          available_rtr_credits: 8
          min_rtr_credits: 7
          refcount: 1
          statistics:
              send_count: 41
              recv_count: 41
              drop_count: 0
sles15build01:~ #

Client:

sles15c01:/tmp # lctl list_nids
192.168.2.22@tcp99
192.168.2.22@tcp1
sles15c01:/tmp # lctl show_route
net                tcp hops 4294967295 gw                192.168.2.20@tcp1 up pri 0
sles15c01:/tmp #
sles15c01:~ # lnetctl route show -v
route:
    - net: tcp
      gateway: 192.168.2.20@tcp1
      hop: -1
      priority: 0
      state: up
sles15c01:~ # lnetctl peer show -v
peer:
    - primary nid: 192.168.2.20@tcp99
      Multi-Rail: True
      peer ni:
        - nid: 192.168.2.20@tcp1
          state: up
          max_ni_tx_credits: 8
          available_tx_credits: 8
          min_tx_credits: 7
          tx_q_num_of_buf: 0
          available_rtr_credits: 8
          min_rtr_credits: 8
          refcount: 4
          statistics:
              send_count: 23
              recv_count: 23
              drop_count: 0
        - nid: 192.168.2.20@tcp99
          state: NA
          max_ni_tx_credits: 8
          available_tx_credits: 8
          min_tx_credits: 7
          tx_q_num_of_buf: 0
          available_rtr_credits: 8
          min_rtr_credits: 8
          refcount: 1
          statistics:
              send_count: 18
              recv_count: 18
              drop_count: 0
        - nid: 192.168.2.20@tcp
          state: NA
          max_ni_tx_credits: 0
          available_tx_credits: 0
          min_tx_credits: 0
          tx_q_num_of_buf: 0
          available_rtr_credits: 0
          min_rtr_credits: 0
          refcount: 2
          statistics:
              send_count: 0
              recv_count: 0
              drop_count: 0
    - primary nid: 192.168.2.21@tcp
      Multi-Rail: True
      peer ni:
        - nid: 192.168.2.21@tcp
          state: NA
          max_ni_tx_credits: 0
          available_tx_credits: 0
          min_tx_credits: 0
          tx_q_num_of_buf: 0
          available_rtr_credits: 0
          min_rtr_credits: 0
          refcount: 2
          statistics:
              send_count: 0
              recv_count: 0
              drop_count: 0
sles15c01:~ #

Server:

sles15s01:/tmp # lctl list_nids
192.168.2.21@tcp
sles15s01:/tmp # lctl show_route
net               tcp1 hops 4294967295 gw                 192.168.2.20@tcp up pri 0
net              tcp99 hops 4294967295 gw                 192.168.2.20@tcp down pri 0
sles15s01:~ # lnetctl route show -v
route:
    - net: tcp1
      gateway: 192.168.2.20@tcp
      hop: -1
      priority: 0
      state: up
    - net: tcp99
      gateway: 192.168.2.20@tcp
      hop: -1
      priority: 0
      state: up
sles15s01:~ # lnetctl peer show -v
peer:
    - primary nid: 192.168.2.20@tcp99
      Multi-Rail: True
      peer ni:
        - nid: 192.168.2.20@tcp
          state: up
          max_ni_tx_credits: 8
          available_tx_credits: 8
          min_tx_credits: 7
          tx_q_num_of_buf: 0
          available_rtr_credits: 8
          min_rtr_credits: 8
          refcount: 5
          statistics:
              send_count: 42
              recv_count: 42
              drop_count: 0
        - nid: 192.168.2.20@tcp99
          state: NA
          max_ni_tx_credits: 0
          available_tx_credits: 0
          min_tx_credits: 0
          tx_q_num_of_buf: 0
          available_rtr_credits: 0
          min_rtr_credits: 0
          refcount: 2
          statistics:
              send_count: 0
              recv_count: 0
              drop_count: 0
        - nid: 192.168.2.20@tcp1
          state: NA
          max_ni_tx_credits: 0
          available_tx_credits: 0
          min_tx_credits: 0
          tx_q_num_of_buf: 0
          available_rtr_credits: 0
          min_rtr_credits: 0
          refcount: 2
          statistics:
              send_count: 0
              recv_count: 0
              drop_count: 0
    - primary nid: 192.168.2.22@tcp99
      Multi-Rail: True
      peer ni:
        - nid: 192.168.2.22@tcp99
          state: NA
          max_ni_tx_credits: 0
          available_tx_credits: 0
          min_tx_credits: 0
          tx_q_num_of_buf: 0
          available_rtr_credits: 0
          min_rtr_credits: 0
          refcount: 2
          statistics:
              send_count: 0
              recv_count: 0
              drop_count: 0
        - nid: 192.168.2.22@tcp1
          state: NA
          max_ni_tx_credits: 0
          available_tx_credits: 0
          min_tx_credits: 0
          tx_q_num_of_buf: 0
          available_rtr_credits: 0
          min_rtr_credits: 0
          refcount: 2
          statistics:
              send_count: 0
              recv_count: 0
              drop_count: 0
sles15s01:~ #

Here we see the router checker thread on the client needs to ping the @tcp1 nid but lnet_select_pathway() chooses a different nid for the router.

00000400:00000200:2.0:1531326577.044094:0:29632:0:(router.c:1099:lnet_ping_router_locked()) Check: 12345-192.168.2.20@tcp1
00000400:00000200:2.0:1531326577.044100:0:29632:0:(lib-move.c:3251:LNetGet()) LNetGet -> 12345-192.168.2.20@tcp1
00000400:00000200:2.0:1531326577.044213:0:29632:0:(lib-move.c:2172:lnet_select_pathway()) TRACE: 192.168.2.22@tcp99(192.168.2.22@tcp99:<?>) -> 192.168.2.20@tcp99(192.168.2.20@tcp1:192.168.2.20@tcp99) : GET

Whamcloud Community JIRA

[LU-11143] Multi-Rail/Dynamic Discovery break LNet router checker and asymmetric route failure detection