[LU-12197] incorrect peer nids with discovery enabled

• Type: Bug
• Resolution: Duplicate
• Priority: Major
• Affects Version/s: Lustre 2.12.0
• Labels: arm
• Environment: ARM clients: kernel 4.14.0-115.2.2.el7a.aarch64 MLNX_OFED_LINUX-4.5-1.0.1.0 (OFED-4.5-1.0.1) ; x86 servers: rhel 7.6, same mofed ; 2.12.0 no patches
• Severity: 3
• Assignee: Amir Shehata (Inactive)
• Reporter: Ruth Klundt (Inactive)

Description

The client side symptom is alternating success/failure of lnet ping to an oss. On the oss we see:

# lnetctl peer show --nid n1-ib0@o2ib
peer:
    - primary nid: xxx.xxx.xxx.17@o2ib
      Multi-Rail: True
      peer ni:
        - nid: xxx.xxx.xxx.17@o2ib
          state: NA
        - nid: xxx.xxx.xxx.182@o2ib
          state: NA
# lnetctl peer show --nid n2-ib0@o2ib
peer:
    - primary nid: xxx.xxx.xxx.17@o2ib
      Multi-Rail: True
      peer ni:
        - nid: xxx.xxx.xxx.17@o2ib
          state: NA
        - nid: xxx.xxx.xxx.182@o2ib
          state: NA

where n1 has ipaddr ending in 182, and n2 has ipaddr ending 17.

The results in logs is lots of timeouts, put NAKs, mount failures and general chaos including plenty of the following message:

kernel: LustreError: 21309:0:(events.c:450:server_bulk_callback()) event type 3, status -61, desc ffff9c46e0303200 

The logs lead us to believe there were IB problems, but the fabric was found to be clean and responsive between the affected client nodes and oss servers.

Planning to turn off discovery going forward. I'll leave a few clients drained for awhile in case there is info you might need.

fyi, rebooting the client does not change the behavior, rebooting the server clears it. Also manually deleting the incorrect peer nid on the server and re-adding the correct info for the missing peer also clears it.

Also, clients are running socket direct, but only one IPoIB interface is configured and in use by lnet.

Issue Links

• duplicates: LU-11478 LNet: discovery sequence numbers could be misleading