Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-12222

LNet should select loopback NI when possible

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Lustre 2.12.0, Lustre 2.13.0
    • Fix Version/s: Lustre 2.14.0
    • Labels:
    • Rank (Obsolete):
      9223372036854775807

      Description

      I happened to notice that when mounting an MGT and MDT co-located on the same node that we end up sending RPCs to ourselves:

      00000100:00000040:11.0:1556165595.786175:0:30688:0:(niobuf.c:905:ptl_send_rpc()) @@@ send flg=0  req@ffff881f275f8000 x1631757890748560/t0(0) o502->MGC10.12.0.50@o2ib40@10.12.0.50@o2ib40:26/25 lens 272/8472 e 0 to 0 dl 1556165690 ref 3 fl Rpc:/0/ffffffff rc 0/-1
      00000100:00000001:11.0:1556165595.786177:0:30688:0:(niobuf.c:54:ptl_send_buf()) Process entered
      00000100:00000040:11.0:1556165595.786177:0:30688:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.12.0.50@o2ib40
      00000400:00000010:11.0:1556165595.786178:0:30688:0:(lib-lnet.h:221:lnet_md_alloc()) slab-alloced 'md' of size 136 at ffff881f279d26e8.
      00000100:00000200:11.0:1556165595.786178:0:30688:0:(niobuf.c:85:ptl_send_buf()) Sending 272 bytes to portal 26, xid 1631757890748560, offset 0
      00000400:00000010:11.0:1556165595.786179:0:30688:0:(lib-lnet.h:472:lnet_msg_alloc()) alloc '(msg)': 440 at ffff881f279c3200 (tot 331800384).
      00000400:00000200:11.0:1556165595.786181:0:30688:0:(lib-move.c:4637:LNetPut()) LNetPut -> 12345-10.12.0.50@o2ib40
      00000400:00000200:11.0:1556165595.786183:0:30688:0:(lib-move.c:2545:lnet_handle_send_case_locked()) Source ANY to MR:  10.12.0.50@o2ib40 local destination
      00000400:00000200:11.0:1556165595.786184:0:30688:0:(lib-move.c:1619:lnet_get_best_ni()) compare ni 0@lo [c:0, d:10, s:0] with best_ni not seleced [c:-2147483648, d:-1, s:0]
      00000400:00000200:11.0:1556165595.786185:0:30688:0:(lib-move.c:1662:lnet_get_best_ni()) selected best_ni 0@lo
      00000400:00000200:11.0:1556165595.786186:0:30688:0:(lib-move.c:1619:lnet_get_best_ni()) compare ni 10.12.0.50@o2ib40 [c:2045, d:10, s:26] with best_ni 0@lo [c:0, d:10, s:0]
      00000400:00000200:11.0:1556165595.786188:0:30688:0:(lib-move.c:1662:lnet_get_best_ni()) selected best_ni 10.12.0.50@o2ib40
      00000400:00000200:11.0:1556165595.786188:0:30688:0:(lib-move.c:1367:lnet_select_peer_ni()) 10.12.0.50@o2ib40 ni_is_pref = 0
      00000400:00000200:11.0:1556165595.786189:0:30688:0:(lib-move.c:1428:lnet_select_peer_ni()) sd_best_lpni = 10.12.0.50@o2ib40
      00000400:00000200:11.0:1556165595.786192:0:30688:0:(lib-move.c:1837:lnet_handle_send()) TRACE: 10.12.0.50@o2ib40(10.12.0.50@o2ib40:<?>) -> 10.12.0.50@o2ib40(10.12.0.50@o2ib40:10.12.0.50@o2ib40) : PUT try# 0

      In the above excerpt we can see that LNet has enough information to determine that we're sending this to ourselves (the nid of best_ni == the nid of best_lpni). Ideally we'd just use the loopback NI whenever possible.

        Attachments

          Activity

            People

            • Assignee:
              shadow Alexey Lyashkov
              Reporter:
              hornc Chris Horn
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: