Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20944

LNet: errors detected while forwarding a message escape the health check

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      Problem
      A router that marked a peer NI down drops every message routed to that
      peer but never degrades its health. The peer NI stays at
      LNET_MAX_HEALTH_VALUE, so it is never queued for recovery, no recovery
      ping is sent, and the router black-holes the peer until LNet restarts.

      Seen in LUS-13527: 5 of 13 routers black-holed an MDS peer for ~30
      minutes, evicting clients. Fingerprint on an affected router:

      state: down
      health value: 1000
      Root cause
      On a router the forward runs inside lnet_finalize(): it health checks
      the receive leg, then lnet_complete_msg_locked() calls lnet_send() to
      forward (lib-msg.c:411-433). The forward fails in
      lnet_post_send_locked(), which skips lnet_finalize() because do_send is
      false, and the error returns to "if (rc != 0) goto again;" and is
      discarded. The health check already ran, for the receive.

      Two effects hide it: msg_ev.status is still 0 while msg_health_status is
      not, which lnet_is_health_check() rejects as inconsistent; and
      lnet_send() overwrites the precise status with REMOTE_ERROR.

      The same gap applies to the ACK send, not just routed forwards.

            shadow Alexey Lyashkov
            shadow Alexey Lyashkov
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: