Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-21057

LNet: discovery Push can use a source NID and a destination NID from different replies

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      sanity-lnet test_237 failed with "Failed to ping 10.240.26.110@tcp1" (-113) on b_es7_0:
      https://testing.whamcloud.com/test_sets/eb812b2b-d1a4-4982-8eeb-3e8ffd79e973

      The route to tcp1 went down because a discovery Push from the client to the router failed. The Push used a source NID and a destination NID on different local networks:

      lnet_handle_send_case_locked()) Source Specified: 10.240.26.107@tcp2 to MR:  10.240.26.109@tcp routed destination
      lnet_handle_find_routed_path()) Unable to send message from 10.240.26.107@tcp2 to 10.240.26.109@tcp - Route table may be misconfigured
      LNetPut()) Error sending PUT to 12345-10.240.26.109@tcp: -113
      lnet_router_discovery_complete()) 10.240.26.109@tcp: Router discovery failed -113
      lnet_set_route_aliveness()) route to tcp1 through 10.240.26.109@tcp has gone from up to down
      

      The client debug log shows this sequence:

      1. The discovery thread sends a ping to 10.240.26.109@tcp. LNet sends it from 10.240.26.107@tcp2 to 10.240.26.109@tcp2.
      2. Before the reply arrives, a Push from the router arrives. The discovery thread merges the Push data and calls lnet_peer_send_push().
      3. lnet_peer_send_push() reads lp_disc_dst_nid. The value is LNET_ANY_NID, so the destination is the primary NID 10.240.26.109@tcp.
      4. The ping reply arrives. lnet_discovery_event_reply() sets lp_disc_src_nid to 10.240.26.107@tcp2 and lp_disc_dst_nid to 10.240.26.109@tcp2.
      5. LNetPut() reads lp->lp_disc_src_nid through a pointer and gets 10.240.26.107@tcp2.

      lnet_peer_send_push() drops lp_lock before it reads lp_disc_dst_nid and lp_disc_src_nid. lnet_discovery_event_reply() writes both fields under lp_lock. The two reads can see different replies. The same code is on master.

            hornc Chris Horn
            hornc Chris Horn
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: