[LU-12222] LNet should select loopback NI when possible

• Type: Improvement
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.14.0, Lustre 2.12.6
• Affects Version/s: Lustre 2.12.0, Lustre 2.13.0
• Labels: HPv3
• Assignee: Alexey Lyashkov
• Reporter: Chris Horn

Description

I happened to notice that when mounting an MGT and MDT co-located on the same node that we end up sending RPCs to ourselves:

00000100:00000040:11.0:1556165595.786175:0:30688:0:(niobuf.c:905:ptl_send_rpc()) @@@ send flg=0  req@ffff881f275f8000 x1631757890748560/t0(0) o502->MGC10.12.0.50@o2ib40@10.12.0.50@o2ib40:26/25 lens 272/8472 e 0 to 0 dl 1556165690 ref 3 fl Rpc:/0/ffffffff rc 0/-1
00000100:00000001:11.0:1556165595.786177:0:30688:0:(niobuf.c:54:ptl_send_buf()) Process entered
00000100:00000040:11.0:1556165595.786177:0:30688:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.12.0.50@o2ib40
00000400:00000010:11.0:1556165595.786178:0:30688:0:(lib-lnet.h:221:lnet_md_alloc()) slab-alloced 'md' of size 136 at ffff881f279d26e8.
00000100:00000200:11.0:1556165595.786178:0:30688:0:(niobuf.c:85:ptl_send_buf()) Sending 272 bytes to portal 26, xid 1631757890748560, offset 0
00000400:00000010:11.0:1556165595.786179:0:30688:0:(lib-lnet.h:472:lnet_msg_alloc()) alloc '(msg)': 440 at ffff881f279c3200 (tot 331800384).
00000400:00000200:11.0:1556165595.786181:0:30688:0:(lib-move.c:4637:LNetPut()) LNetPut -> 12345-10.12.0.50@o2ib40
00000400:00000200:11.0:1556165595.786183:0:30688:0:(lib-move.c:2545:lnet_handle_send_case_locked()) Source ANY to MR:  10.12.0.50@o2ib40 local destination
00000400:00000200:11.0:1556165595.786184:0:30688:0:(lib-move.c:1619:lnet_get_best_ni()) compare ni 0@lo [c:0, d:10, s:0] with best_ni not seleced [c:-2147483648, d:-1, s:0]
00000400:00000200:11.0:1556165595.786185:0:30688:0:(lib-move.c:1662:lnet_get_best_ni()) selected best_ni 0@lo
00000400:00000200:11.0:1556165595.786186:0:30688:0:(lib-move.c:1619:lnet_get_best_ni()) compare ni 10.12.0.50@o2ib40 [c:2045, d:10, s:26] with best_ni 0@lo [c:0, d:10, s:0]
00000400:00000200:11.0:1556165595.786188:0:30688:0:(lib-move.c:1662:lnet_get_best_ni()) selected best_ni 10.12.0.50@o2ib40
00000400:00000200:11.0:1556165595.786188:0:30688:0:(lib-move.c:1367:lnet_select_peer_ni()) 10.12.0.50@o2ib40 ni_is_pref = 0
00000400:00000200:11.0:1556165595.786189:0:30688:0:(lib-move.c:1428:lnet_select_peer_ni()) sd_best_lpni = 10.12.0.50@o2ib40
00000400:00000200:11.0:1556165595.786192:0:30688:0:(lib-move.c:1837:lnet_handle_send()) TRACE: 10.12.0.50@o2ib40(10.12.0.50@o2ib40:<?>) -> 10.12.0.50@o2ib40(10.12.0.50@o2ib40:10.12.0.50@o2ib40) : PUT try# 0

In the above excerpt we can see that LNet has enough information to determine that we're sending this to ourselves (the nid of best_ni == the nid of best_lpni). Ideally we'd just use the loopback NI whenever possible.

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/38865/
Subject: LU-12222 ptlrpc: Check if NID is local, not just lolnd NID
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 7635e6326de88bfd035c82ac9261e90e4f9b4a1b

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/38864/
Subject: LU-12222 lnet: Primary NID of lolnd NID is the lolnd NID
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 4494f55aa2fe7d2ae274405fc0f2314193db1c0e

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/38863/
Subject: LU-12222 lnet: Introduce constant for the lolnd NID
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: b5fb6f1fe6dedca0ba6e405e90d7ec72d9b97e83

[Gerrit Updater]

Chris Horn ([e-mail]) uploaded a new patch: https://review.whamcloud.com/38865
Subject: LU-12222 ptlrpc: Check if NID is local, not just lolnd NID
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 35c6cfeec9d5b9a354bc49319694d2360c48c68c

[Gerrit Updater]

Chris Horn ([e-mail]) uploaded a new patch: https://review.whamcloud.com/38864
Subject: LU-12222 lnet: Primary NID of lolnd NID is the lolnd NID
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 7e924bc9a53ad82a0c4e6da4084e4de8536e4ada

[Gerrit Updater]

Chris Horn ([e-mail]) uploaded a new patch: https://review.whamcloud.com/38863
Subject: LU-12222 lnet: Introduce constant for the lolnd NID
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: a0bb5d5f52414dcfc95bfa699c5d800c1d7daa7a

[Gerrit Updater]

[Deleted]

[Gerrit Updater]

[Deleted]

[Gerrit Updater]

[Deleted]

[Peter Jones]

Landed for 2.14