Details
-
Bug
-
Resolution: Fixed
-
Major
-
None
-
Lustre 2.11.0
-
None
-
3
-
9223372036854775807
Description
The LNet router checker needs to ping the interface defined in the route table, but MR can choose a different interface for those pings.
As a result the router can end up marking some interfaces down because they aren't seeing any traffic in the check_interval + router ping timeout window.
This causes routes to be considered down because of asymmetric route failure detection.
Reproduced on a three node VM.
Router:
sles15build01:/tmp # lctl list_nids
192.168.2.20@tcp99
192.168.2.20@tcp1
192.168.2.20@tcp
sles15build01:~ # lnetctl route show -v
sles15build01:~ # lnetctl peer show -v
peer:
- primary nid: 192.168.2.22@tcp99
Multi-Rail: True
peer ni:
- nid: 192.168.2.22@tcp1
state: up
max_ni_tx_credits: 8
available_tx_credits: 8
min_tx_credits: 7
tx_q_num_of_buf: 0
available_rtr_credits: 8
min_rtr_credits: 7
refcount: 1
statistics:
send_count: 22
recv_count: 22
drop_count: 0
- nid: 192.168.2.22@tcp99
state: up
max_ni_tx_credits: 8
available_tx_credits: 8
min_tx_credits: 7
tx_q_num_of_buf: 0
available_rtr_credits: 8
min_rtr_credits: 7
refcount: 1
statistics:
send_count: 18
recv_count: 18
drop_count: 0
- primary nid: 192.168.2.21@tcp
Multi-Rail: True
peer ni:
- nid: 192.168.2.21@tcp
state: up
max_ni_tx_credits: 8
available_tx_credits: 8
min_tx_credits: 7
tx_q_num_of_buf: 0
available_rtr_credits: 8
min_rtr_credits: 7
refcount: 1
statistics:
send_count: 41
recv_count: 41
drop_count: 0
sles15build01:~ #
Client:
sles15c01:/tmp # lctl list_nids
192.168.2.22@tcp99
192.168.2.22@tcp1
sles15c01:/tmp # lctl show_route
net tcp hops 4294967295 gw 192.168.2.20@tcp1 up pri 0
sles15c01:/tmp #
sles15c01:~ # lnetctl route show -v
route:
- net: tcp
gateway: 192.168.2.20@tcp1
hop: -1
priority: 0
state: up
sles15c01:~ # lnetctl peer show -v
peer:
- primary nid: 192.168.2.20@tcp99
Multi-Rail: True
peer ni:
- nid: 192.168.2.20@tcp1
state: up
max_ni_tx_credits: 8
available_tx_credits: 8
min_tx_credits: 7
tx_q_num_of_buf: 0
available_rtr_credits: 8
min_rtr_credits: 8
refcount: 4
statistics:
send_count: 23
recv_count: 23
drop_count: 0
- nid: 192.168.2.20@tcp99
state: NA
max_ni_tx_credits: 8
available_tx_credits: 8
min_tx_credits: 7
tx_q_num_of_buf: 0
available_rtr_credits: 8
min_rtr_credits: 8
refcount: 1
statistics:
send_count: 18
recv_count: 18
drop_count: 0
- nid: 192.168.2.20@tcp
state: NA
max_ni_tx_credits: 0
available_tx_credits: 0
min_tx_credits: 0
tx_q_num_of_buf: 0
available_rtr_credits: 0
min_rtr_credits: 0
refcount: 2
statistics:
send_count: 0
recv_count: 0
drop_count: 0
- primary nid: 192.168.2.21@tcp
Multi-Rail: True
peer ni:
- nid: 192.168.2.21@tcp
state: NA
max_ni_tx_credits: 0
available_tx_credits: 0
min_tx_credits: 0
tx_q_num_of_buf: 0
available_rtr_credits: 0
min_rtr_credits: 0
refcount: 2
statistics:
send_count: 0
recv_count: 0
drop_count: 0
sles15c01:~ #
Server:
sles15s01:/tmp # lctl list_nids
192.168.2.21@tcp
sles15s01:/tmp # lctl show_route
net tcp1 hops 4294967295 gw 192.168.2.20@tcp up pri 0
net tcp99 hops 4294967295 gw 192.168.2.20@tcp down pri 0
sles15s01:~ # lnetctl route show -v
route:
- net: tcp1
gateway: 192.168.2.20@tcp
hop: -1
priority: 0
state: up
- net: tcp99
gateway: 192.168.2.20@tcp
hop: -1
priority: 0
state: up
sles15s01:~ # lnetctl peer show -v
peer:
- primary nid: 192.168.2.20@tcp99
Multi-Rail: True
peer ni:
- nid: 192.168.2.20@tcp
state: up
max_ni_tx_credits: 8
available_tx_credits: 8
min_tx_credits: 7
tx_q_num_of_buf: 0
available_rtr_credits: 8
min_rtr_credits: 8
refcount: 5
statistics:
send_count: 42
recv_count: 42
drop_count: 0
- nid: 192.168.2.20@tcp99
state: NA
max_ni_tx_credits: 0
available_tx_credits: 0
min_tx_credits: 0
tx_q_num_of_buf: 0
available_rtr_credits: 0
min_rtr_credits: 0
refcount: 2
statistics:
send_count: 0
recv_count: 0
drop_count: 0
- nid: 192.168.2.20@tcp1
state: NA
max_ni_tx_credits: 0
available_tx_credits: 0
min_tx_credits: 0
tx_q_num_of_buf: 0
available_rtr_credits: 0
min_rtr_credits: 0
refcount: 2
statistics:
send_count: 0
recv_count: 0
drop_count: 0
- primary nid: 192.168.2.22@tcp99
Multi-Rail: True
peer ni:
- nid: 192.168.2.22@tcp99
state: NA
max_ni_tx_credits: 0
available_tx_credits: 0
min_tx_credits: 0
tx_q_num_of_buf: 0
available_rtr_credits: 0
min_rtr_credits: 0
refcount: 2
statistics:
send_count: 0
recv_count: 0
drop_count: 0
- nid: 192.168.2.22@tcp1
state: NA
max_ni_tx_credits: 0
available_tx_credits: 0
min_tx_credits: 0
tx_q_num_of_buf: 0
available_rtr_credits: 0
min_rtr_credits: 0
refcount: 2
statistics:
send_count: 0
recv_count: 0
drop_count: 0
sles15s01:~ #
Here we see the router checker thread on the client needs to ping the @tcp1 nid but lnet_select_pathway() chooses a different nid for the router.
00000400:00000200:2.0:1531326577.044094:0:29632:0:(router.c:1099:lnet_ping_router_locked()) Check: 12345-192.168.2.20@tcp1 00000400:00000200:2.0:1531326577.044100:0:29632:0:(lib-move.c:3251:LNetGet()) LNetGet -> 12345-192.168.2.20@tcp1 00000400:00000200:2.0:1531326577.044213:0:29632:0:(lib-move.c:2172:lnet_select_pathway()) TRACE: 192.168.2.22@tcp99(192.168.2.22@tcp99:<?>) -> 192.168.2.20@tcp99(192.168.2.20@tcp1:192.168.2.20@tcp99) : GET
