Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-12197

incorrect peer nids with discovery enabled

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: Lustre 2.12.0
    • Fix Version/s: None
    • Labels:
    • Environment:
      ARM clients: kernel 4.14.0-115.2.2.el7a.aarch64 MLNX_OFED_LINUX-4.5-1.0.1.0 (OFED-4.5-1.0.1)
      x86 servers: rhel 7.6, same mofed
      2.12.0 no patches
    • Severity:
      3
    • Rank (Obsolete):
      9223372036854775807

      Description

      The client side symptom is alternating success/failure of lnet ping to an oss. On the oss we see:

      # lnetctl peer show --nid n1-ib0@o2ib
      peer:
          - primary nid: xxx.xxx.xxx.17@o2ib
            Multi-Rail: True
            peer ni:
              - nid: xxx.xxx.xxx.17@o2ib
                state: NA
              - nid: xxx.xxx.xxx.182@o2ib
                state: NA
      # lnetctl peer show --nid n2-ib0@o2ib
      peer:
          - primary nid: xxx.xxx.xxx.17@o2ib
            Multi-Rail: True
            peer ni:
              - nid: xxx.xxx.xxx.17@o2ib
                state: NA
              - nid: xxx.xxx.xxx.182@o2ib
                state: NA

      where n1 has ipaddr ending in 182, and n2 has ipaddr ending 17.

      The results in logs is lots of timeouts, put NAKs, mount failures and general chaos including plenty of the following message:

      kernel: LustreError: 21309:0:(events.c:450:server_bulk_callback()) event type 3, status -61, desc ffff9c46e0303200 

      The logs lead us to believe there were IB problems, but the fabric was found to be clean and responsive between the affected client nodes and oss servers.

      Planning to turn off discovery going forward. I'll leave a few clients drained for awhile in case there is info you might need.

      fyi, rebooting the client does not change the behavior, rebooting the server clears it. Also manually deleting the incorrect peer nid on the server and re-adding the correct info for the missing peer also clears it.

      Also, clients are running socket direct, but only one IPoIB interface is configured and in use by lnet.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ashehata Amir Shehata
                Reporter:
                ruth.klundt@gmail.com Ruth Klundt
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: