[LU-11413] Large performance degradation in routing environment.

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.13.0
• Environment: any LST server/client and two routers
• Severity: 3
• Assignee: Alexey Lyashkov
• Reporter: Alexey Lyashkov

Description

Large performance degradation found with MD testing in routed env. While simplify a test case, I found an async routes was is root cause of this. Replication is quite simple,
you need LST server and client in different logical networks and ask server to be route traffic via 'roter1' while client have route a traffic via route2.
Example of results is
server have route via 73@o2ib and lst results in case different router used:
[LNet Rates of 172.18.1.4@o2ib1]
[R] Avg: 8615 RPC/s Min: 8615 RPC/s Max: 8615 RPC/s
[W] Avg: 8615 RPC/s Min: 8615 RPC/s Max: 8615 RPC/s
but once routing changed - results is much better.
[root@c-lmo069 ~]# lnetctl route del --net o2ib1 --gateway 172.18.2.76@o2ib10
[root@c-lmo069 ~]# lnetctl route add --net o2ib1 --gateway 172.18.2.73@o2ib10

[root@c-lmo069 ~]# lctl dk > log; bash /root/lnet.sh write 4k 1; lctl dk > log-r4
Performing write
SESSION: read/write FEATURES: 1 TIMEOUT: 300 FORCE: No
172.18.1.4@o2ib1 are added to session
172.18.2.69@o2ib10 are added to session
Test was added successfully
bulk_rw is running now
[LNet Rates of 172.18.1.4@o2ib1]
[R] Avg: 11349 RPC/s Min: 11349 RPC/s Max: 11349 RPC/s
[W] Avg: 11349 RPC/s Min: 11349 RPC/s Max: 11349 RPC/s

lnet.sh is a simple script to send a 4k rpc, with 1 send in parallel (concurrent sends == 1).

This issue can replicated with socklnd also.

socklnd problem is - reply is scheduled to the different thread and can be send quickly as possible.

o2ib problem - it looks need additional network messaged to distribute a credits between nodes, as o2ib protocol have send an additional credits which sending a request which assume a reply.

First part of these issues is looks easy and caused an incomplete implementation with with lnet_send() function call, it's never use a right source NID to make it preferable.

               /* NB: we probably want to use NID of msg::msg_from as 3rd
                 * parameter (router NID) if it's routed message */
                rc = lnet_send(msg->msg_ev.target.nid, msg, LNET_NID_ANY);

while second part of problem, more complex.
two problems in this area.
1) server can set just a single router and we need to route all traffic to this to avoid performance penalty.
It can be done with adding a incoming message counter as similar as lpni_seq counted as outgunning events.

2) server router is outside of router's list. did we need to add this router to make ability to choose it in lnet_find_route_locked() ?