• Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
    • Severity:
    • Rank (Obsolete):


      While testing the fix for LU-13708, we found that communication was still severely disrupted.

      I ran some ping tests between a client and a server. From the logs, we could see the router was attempting to forward a message using an interface that had been disabled:

      00000400:00000200:0.0:1593005930.168691:0:7991:0:(lib-move.c:4323:lnet_parse()) TRACE: <- 610@gni : GET - routed
      00000800:00000200:0.0:1593005930.168698:0:7991:0:(gnilnd_cb.c:2450:kgnilnd_recv()) $$ conn ffff88082b37c800, rxmsg ffffc900201321c8, lntmsg ffff880716ef6040 niov=0 kiov=          (null) iov=          (null) offset=0 mlen=0 rlen=0 from 610@gni  msg@0xffffc900201321c8 m/v/ty/ck/pck/pl b00fbabe/8/2/0/0/0 x2156869:GNILND_MSG_IMMEDIATE
      00000800:00000200:0.0:1593005930.168707:0:7991:0:(gnilnd_cb.c:2092:kgnilnd_consume_rx()) $$ rx ffff880829491d80 processed from 610@gni  msg@0xffffc900201321c8 m/v/ty/ck/pck/pl b00fbabe/8/2/0/0/0 x2156869:GNILND_MSG_IMMEDIATE
      00000800:00000200:0.0:1593005930.168711:0:7991:0:(gnilnd_cb.c:2058:kgnilnd_release_msg()) consuming ffff88082b37c800
      00000400:00000200:0.0:1593005930.168718:0:7991:0:(lib-msg.c:996:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0
      00000400:00000200:0.0:1593005930.168722:0:7991:0:(lib-msg.c:825:lnet_health_check()) health check: 605@gni->610@gni: GET: OK
      00000400:00000200:0.0:1593005930.168727:0:7991:0:(lib-move.c:2624:lnet_handle_send_case_locked()) Source ANY to NMR: local destination
      00000400:00000200:0.0:1593005930.168737:0:7991:0:(lib-move.c:1853:lnet_handle_send()) TRACE: 610@gni(<?>) -> <?> : GET try# 0 <<<< is disabled interface

      There's a flaw in the logic used to forward the message. The path selection code treats this like a local send to a non-multi-rail peer. The reason for this is that we don't want the router to modify the destination interface. However, this code path sets a "preferred NI" that gets used for future sends. In this case, the first time the router forwarded a message to, it set as the "preferred NI" to be used when communicating with this node. Now, even though that interface is down, it is still selecting it to forward messages because of this preferred status.




            • Assignee:
              hornc Chris Horn
              hornc Chris Horn
            • Votes:
              0 Vote for this issue
              2 Start watching this issue


              • Created: