1. 1. Problem
      lnet_peer_set_primary_nid() writes a new lp_primary_nid before lnet_peer_add_nid() runs. If the add fails, the function writes the old value back. A reader with only the net lock can see this transient NID. lnet_parse() can then stamp msg_initiator with that NID, and LNetPrimaryNID() can return it.

1. 1. Environment

• master at 550451c0e4 and earlier.
• Any node with lock_prim_nid=0, or a peer without LNET_PEER_LOCK_PRIMARY.

1. 1. Reproduction
      Source analysis only. We did not reproduce the failure.
      1. Configure a peer with lnetctl, so that it has LNET_PEER_CONFIGURED and no LNET_PEER_LOCK_PRIMARY.
      2. Let a ping reply from that peer report primary NID B, where B is another NID of the same peer.
      3. lnet_peer_data_present() calls lnet_peer_set_primary_nid(lp, B, flags). flags has no LNET_PEER_CONFIGURED.
      4. The function writes B under lp_lock. lnet_peer_add_nid() returns -EPERM, and the function writes the old NID back.
      5. A lookup between steps 3 and 4 returns B.

1. 1. Analysis
      The writers of lp_primary_nid hold lp_lock. Readers such as lnet_peer_primary_nid_locked() hold the net lock only. The window lasts from the first write to the restore. It recurs on each discovery round of such a peer.

For LU-20403, ptlrpc_refresh_import_primary() can move an import to the transient NID. The reply ME then keys on that NID, but lnet_parse() stamps the reply with the restored primary. The reply does not match, and the RPC waits until it times out. The next send moves the import back. LU-20403 rejects a transient NID that no peer owns, but not one that the same peer owns.

1. 1. Proposed fix
      In lnet_peer_set_primary_nid(), test the -EPERM condition of lnet_peer_add_nid() before the write: a peer with LNET_PEER_CONFIGURED and no LNET_PEER_CONFIGURED in flags. Skip the write in that case. A move of the write after lnet_peer_add_nid() is not a simple change, because lnet_peer_add_nid() and lnet_peer_attach_peer_ni() read lp_primary_nid to place the primary NI.