Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-12222

LNet should select loopback NI when possible

Details

    • Improvement
    • Resolution: Fixed
    • Major
    • Lustre 2.14.0, Lustre 2.12.6
    • Lustre 2.12.0, Lustre 2.13.0
    • 9223372036854775807

    Description

      I happened to notice that when mounting an MGT and MDT co-located on the same node that we end up sending RPCs to ourselves:

      00000100:00000040:11.0:1556165595.786175:0:30688:0:(niobuf.c:905:ptl_send_rpc()) @@@ send flg=0  req@ffff881f275f8000 x1631757890748560/t0(0) o502->MGC10.12.0.50@o2ib40@10.12.0.50@o2ib40:26/25 lens 272/8472 e 0 to 0 dl 1556165690 ref 3 fl Rpc:/0/ffffffff rc 0/-1
      00000100:00000001:11.0:1556165595.786177:0:30688:0:(niobuf.c:54:ptl_send_buf()) Process entered
      00000100:00000040:11.0:1556165595.786177:0:30688:0:(niobuf.c:57:ptl_send_buf()) peer_id 12345-10.12.0.50@o2ib40
      00000400:00000010:11.0:1556165595.786178:0:30688:0:(lib-lnet.h:221:lnet_md_alloc()) slab-alloced 'md' of size 136 at ffff881f279d26e8.
      00000100:00000200:11.0:1556165595.786178:0:30688:0:(niobuf.c:85:ptl_send_buf()) Sending 272 bytes to portal 26, xid 1631757890748560, offset 0
      00000400:00000010:11.0:1556165595.786179:0:30688:0:(lib-lnet.h:472:lnet_msg_alloc()) alloc '(msg)': 440 at ffff881f279c3200 (tot 331800384).
      00000400:00000200:11.0:1556165595.786181:0:30688:0:(lib-move.c:4637:LNetPut()) LNetPut -> 12345-10.12.0.50@o2ib40
      00000400:00000200:11.0:1556165595.786183:0:30688:0:(lib-move.c:2545:lnet_handle_send_case_locked()) Source ANY to MR:  10.12.0.50@o2ib40 local destination
      00000400:00000200:11.0:1556165595.786184:0:30688:0:(lib-move.c:1619:lnet_get_best_ni()) compare ni 0@lo [c:0, d:10, s:0] with best_ni not seleced [c:-2147483648, d:-1, s:0]
      00000400:00000200:11.0:1556165595.786185:0:30688:0:(lib-move.c:1662:lnet_get_best_ni()) selected best_ni 0@lo
      00000400:00000200:11.0:1556165595.786186:0:30688:0:(lib-move.c:1619:lnet_get_best_ni()) compare ni 10.12.0.50@o2ib40 [c:2045, d:10, s:26] with best_ni 0@lo [c:0, d:10, s:0]
      00000400:00000200:11.0:1556165595.786188:0:30688:0:(lib-move.c:1662:lnet_get_best_ni()) selected best_ni 10.12.0.50@o2ib40
      00000400:00000200:11.0:1556165595.786188:0:30688:0:(lib-move.c:1367:lnet_select_peer_ni()) 10.12.0.50@o2ib40 ni_is_pref = 0
      00000400:00000200:11.0:1556165595.786189:0:30688:0:(lib-move.c:1428:lnet_select_peer_ni()) sd_best_lpni = 10.12.0.50@o2ib40
      00000400:00000200:11.0:1556165595.786192:0:30688:0:(lib-move.c:1837:lnet_handle_send()) TRACE: 10.12.0.50@o2ib40(10.12.0.50@o2ib40:<?>) -> 10.12.0.50@o2ib40(10.12.0.50@o2ib40:10.12.0.50@o2ib40) : PUT try# 0

      In the above excerpt we can see that LNet has enough information to determine that we're sending this to ourselves (the nid of best_ni == the nid of best_lpni). Ideally we'd just use the loopback NI whenever possible.

      Attachments

        Activity

          [LU-12222] LNet should select loopback NI when possible

          Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/38865/
          Subject: LU-12222 ptlrpc: Check if NID is local, not just lolnd NID
          Project: fs/lustre-release
          Branch: b2_12
          Current Patch Set:
          Commit: 7635e6326de88bfd035c82ac9261e90e4f9b4a1b

          gerrit Gerrit Updater added a comment - Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/38865/ Subject: LU-12222 ptlrpc: Check if NID is local, not just lolnd NID Project: fs/lustre-release Branch: b2_12 Current Patch Set: Commit: 7635e6326de88bfd035c82ac9261e90e4f9b4a1b

          Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/38864/
          Subject: LU-12222 lnet: Primary NID of lolnd NID is the lolnd NID
          Project: fs/lustre-release
          Branch: b2_12
          Current Patch Set:
          Commit: 4494f55aa2fe7d2ae274405fc0f2314193db1c0e

          gerrit Gerrit Updater added a comment - Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/38864/ Subject: LU-12222 lnet: Primary NID of lolnd NID is the lolnd NID Project: fs/lustre-release Branch: b2_12 Current Patch Set: Commit: 4494f55aa2fe7d2ae274405fc0f2314193db1c0e

          Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/38863/
          Subject: LU-12222 lnet: Introduce constant for the lolnd NID
          Project: fs/lustre-release
          Branch: b2_12
          Current Patch Set:
          Commit: b5fb6f1fe6dedca0ba6e405e90d7ec72d9b97e83

          gerrit Gerrit Updater added a comment - Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/38863/ Subject: LU-12222 lnet: Introduce constant for the lolnd NID Project: fs/lustre-release Branch: b2_12 Current Patch Set: Commit: b5fb6f1fe6dedca0ba6e405e90d7ec72d9b97e83

          Chris Horn (chris.horn@hpe.com) uploaded a new patch: https://review.whamcloud.com/38865
          Subject: LU-12222 ptlrpc: Check if NID is local, not just lolnd NID
          Project: fs/lustre-release
          Branch: b2_12
          Current Patch Set: 1
          Commit: 35c6cfeec9d5b9a354bc49319694d2360c48c68c

          gerrit Gerrit Updater added a comment - Chris Horn (chris.horn@hpe.com) uploaded a new patch: https://review.whamcloud.com/38865 Subject: LU-12222 ptlrpc: Check if NID is local, not just lolnd NID Project: fs/lustre-release Branch: b2_12 Current Patch Set: 1 Commit: 35c6cfeec9d5b9a354bc49319694d2360c48c68c

          Chris Horn (chris.horn@hpe.com) uploaded a new patch: https://review.whamcloud.com/38864
          Subject: LU-12222 lnet: Primary NID of lolnd NID is the lolnd NID
          Project: fs/lustre-release
          Branch: b2_12
          Current Patch Set: 1
          Commit: 7e924bc9a53ad82a0c4e6da4084e4de8536e4ada

          gerrit Gerrit Updater added a comment - Chris Horn (chris.horn@hpe.com) uploaded a new patch: https://review.whamcloud.com/38864 Subject: LU-12222 lnet: Primary NID of lolnd NID is the lolnd NID Project: fs/lustre-release Branch: b2_12 Current Patch Set: 1 Commit: 7e924bc9a53ad82a0c4e6da4084e4de8536e4ada

          Chris Horn (chris.horn@hpe.com) uploaded a new patch: https://review.whamcloud.com/38863
          Subject: LU-12222 lnet: Introduce constant for the lolnd NID
          Project: fs/lustre-release
          Branch: b2_12
          Current Patch Set: 1
          Commit: a0bb5d5f52414dcfc95bfa699c5d800c1d7daa7a

          gerrit Gerrit Updater added a comment - Chris Horn (chris.horn@hpe.com) uploaded a new patch: https://review.whamcloud.com/38863 Subject: LU-12222 lnet: Introduce constant for the lolnd NID Project: fs/lustre-release Branch: b2_12 Current Patch Set: 1 Commit: a0bb5d5f52414dcfc95bfa699c5d800c1d7daa7a
          gerrit Gerrit Updater added a comment - - edited

          [Deleted]

          gerrit Gerrit Updater added a comment - - edited [Deleted]
          gerrit Gerrit Updater added a comment - - edited

          [Deleted]

          gerrit Gerrit Updater added a comment - - edited [Deleted]
          gerrit Gerrit Updater added a comment - - edited

          [Deleted]

          gerrit Gerrit Updater added a comment - - edited [Deleted]
          pjones Peter Jones added a comment -

          Landed for 2.14

          pjones Peter Jones added a comment - Landed for 2.14

          People

            shadow Alexey Lyashkov
            hornc Chris Horn
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: