Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-21247

Report NI address removal on the console and clarify the o2ib "consumer defined fatal error" reject

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      When the IP address of an NI is removed or changed while LNet is running (e.g. by NetworkManager), ksocklnd and ko2iblnd mark the NI down but only log it under D_NET. Nothing appears on the console of the node where the address changed.

      With o2ib, peers that keep connecting to the old address are rejected by the remote RDMA CM, which finds no listener bound to it. The reject carries an all-zero payload, and kiblnd_rejected() logs:

      LNetError: ... 172.16.10.101@o2ib rejected: consumer defined fatal error
      

      This message is printed on the peers, not on the node where the address changed, and it looks like a hardware or fabric fault. Such cases end up being investigated as network problems, while the cause is an address changed without stopping Lustre/LNet.

      Proposal:

      • In the inetaddr notifiers of ksocklnd and ko2iblnd, when the event matches the NI address, print on the console: <nid>: address removed from <ifname>, NI unusable until it is restored. If the address was changed, stop Lustre and reconfigure LNet and, when the address comes back: "<nid>: address restored on <ifname>".
      • In kiblnd_rejected(), when the reject payload is all zeros, print: "<nid> rejected: remote host has no LNet NI on this address". Other non-o2iblnd payloads keep the current message.

       

            cbordage Cyril Bordage
            cbordage Cyril Bordage
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: