|
This is what I think is going wrong, do you agree?
As noted before, without LU-17062, the problem goes away.
For the code lines below, I'm referring to where we added LU-17062 in our 2.15.4_1-llnl branch https://github.com/LLNL/lustre/commit/2e2719341590edef547e16b4051684781cf1b795
The debug data is in mutt-garter-debug-2fcd4d27f.tar.gz in the dk.TOSS-6214.garter3.lustre-2.15.4_1.llnl_3_g2fcd4d2-1.t4.x86_64.1706816872 file. Lines 737 to 771
The debug patch version of lustre that was running on garter3 is at https://github.com/LLNL/lustre/tree/debug_print-lnet_handle_find_routed_path
The setup is
server garter3: 172.19.1.135@o2ib100
routers mutt[1-4]: 172.19.1.[105-108]@o2ib100 and 192.168.128.[1-4]@o2ib44
client mutt7: 192.168.128.7@o2ib44
garter3 <—> o2ib100 <—> mutt[1-4] <—> o2ib44 <—> mutt7
garter3 has routes to o2ib44 via mutt[1-3], mutt7 has routes to o2ib100 via mutt[1-4], so the is an asymmetry regarding mutt4
mutt7 pings garter3 via mutt4
Looking at function lnet_handle_find_routed_path():
at line 2182, there is a call to lnet_peer_ni_find_locked()
the call stack is lnet_handle_find_routed_path() -> lnet_peer_ni_find_locked() -> lnet_get_peer_ni_locked() -> lnet_peer_ni_addref_locked()
The return value is gwni. If gwni is not NULL, it is the gateway NI. In the debug logs for garter3, I see the nid for mutt4 when gwni is returned.
The refcount for mutt4 is incremented due to lnet_peer_ni_addref_locked() being in the call stack above.
before LU-17062, if gwni is not NULL, the refcount for mutt4 is decremented almost immediately at line 2184
after LU-17062, the decrement happens at the end of the function.
at line 2291, net_find_route_locked() is called. In the debug logs I can see the value of gwni->lpni_nid change after this call.
It changes from the nid of mutt4 to the nid of another mutt router that garter3 does have a route through.
before the LU-17062, there were no other calls to lnet_peer_ni_decref_locked()
after LU-17062 there is a call to lnet_peer_ni_decref_locked() if gwni was not NULL when it was modified at line 2182
and hasn't subsequently become NULL. It shouldn't have become NULL if there's some NI that garter3 can use to get to o2ib44, which
there is in this case.
after LU-17062 at line 2365 lnet_peer_ni_decref_locked() is called on the a router with a refcount of 1 (assuming the gwni changed values due to net_find_route_locked()). The refcount goes to 0 (or would go to 0) so lnet_destroy_peer_ni_locked() is called. lnet_destroy_peer_ni_locked() has asserts to make sure something like this doesn't happen, so the asserts fail.
| 
asp-orelic-console.tar.gz