Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20907

client evicted by "Invalid replay without recovery" when final ping races end of recovery

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      During server failover testing (2.17.59 servers, 2.17.58 clients, a small cluster with imperative recovery), clients were evicted just after a target completed recovery successfully. A few seconds before each client eviction the server logs:

      LustreError: (service.c:1411:ptlrpc_check_req()) @@@ Invalid replay without recovery  req@ffff89898f314000 x1878493176588800/t0(0) o400->6575c3d5-a001-4ccc-b292-c6208abc155e@192.168.100.204@tcp:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1
      

      Request flags 0x2c0 are MSG_REQ_REPLAY_DONE | MSG_LOCK_REPLAY_DONE | MSG_PACK_UID_GID: the client's final recovery ping from signal_completed_replay(), sent for the first time (no MSG_RESENT).

      The sequence:

      1. The client finishes replay and sends its final ping. The target queues it until every client is done.
      2. That ping times out or fails on the client, and the client reconnects. The target is still recovering, so the connect reply carries MSG_CONNECT_RECOVERING | MSG_CONNECT_RECONNECT, the import goes back to REPLAY_LOCKS -> REPLAY_WAIT, and a new final ping is sent.
      3. The other clients finish and recovery completes in between. This client counts as recovered, because its first final ping was queued.
      4. The second final ping reaches ptlrpc_check_req() after OBDF_RECOVERING has cleared. MSG_REQ_REPLAY_DONE without recovery calls class_fail_export(), evicting a client that recovered successfully.

      target_queue_recovery_request() already handles a final ping that arrives after recovery, by processing it as an ordinary ping; only the earlier ptlrpc_check_req() check fails the export. The race is long-standing and is not related to LU-18154 / LU-20376 (it reproduces with both reverted).

      Reproduced on master (4a7bdda) using replay-single test_93a's setup (OBD_FAIL_TGT_REPLAY_RECONNECT holding the final stage on the OST), with the second final ping delayed past the end of recovery: the OST logs "Recovery over ... 0 were evicted", then "Invalid replay without recovery", and the client is evicted.

      Fix: in ptlrpc_check_req(), let an OBD_PING carrying MSG_LOCK_REPLAY_DONE through when the export has already queued its final ping in this recovery (exp_replay_done). New test replay-single test_93c uses a new client fail_loc, OBD_FAIL_PTLRPC_DELAY_FINAL_PING (0x538), to make the race deterministic.

            paf0186 Patrick Farrell
            paf0186 Patrick Farrell
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: