-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
During server failover testing (2.17.59 servers, 2.17.58 clients, a small cluster with imperative recovery), clients were evicted just after a target completed recovery successfully. A few seconds before each client eviction the server logs:
LustreError: (service.c:1411:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff89898f314000 x1878493176588800/t0(0) o400->6575c3d5-a001-4ccc-b292-c6208abc155e@192.168.100.204@tcp:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1
Request flags 0x2c0 are MSG_REQ_REPLAY_DONE | MSG_LOCK_REPLAY_DONE | MSG_PACK_UID_GID: the client's final recovery ping from signal_completed_replay(), sent for the first time (no MSG_RESENT).
The sequence:
- The client finishes replay and sends its final ping. The target queues it until every client is done.
- That ping times out or fails on the client, and the client reconnects. The target is still recovering, so the connect reply carries MSG_CONNECT_RECOVERING | MSG_CONNECT_RECONNECT, the import goes back to REPLAY_LOCKS -> REPLAY_WAIT, and a new final ping is sent.
- The other clients finish and recovery completes in between. This client counts as recovered, because its first final ping was queued.
- The second final ping reaches ptlrpc_check_req() after OBDF_RECOVERING has cleared. MSG_REQ_REPLAY_DONE without recovery calls class_fail_export(), evicting a client that recovered successfully.
target_queue_recovery_request() already handles a final ping that arrives after recovery, by processing it as an ordinary ping; only the earlier ptlrpc_check_req() check fails the export. The race is long-standing and is not related to LU-18154 / LU-20376 (it reproduces with both reverted).
Reproduced on master (4a7bdda) using replay-single test_93a's setup (OBD_FAIL_TGT_REPLAY_RECONNECT holding the final stage on the OST), with the second final ping delayed past the end of recovery: the OST logs "Recovery over ... 0 were evicted", then "Invalid replay without recovery", and the client is evicted.
Fix: in ptlrpc_check_req(), let an OBD_PING carrying MSG_LOCK_REPLAY_DONE through when the export has already queued its final ping in this recovery (exp_replay_done). New test replay-single test_93c uses a new client fail_loc, OBD_FAIL_PTLRPC_DELAY_FINAL_PING (0x538), to make the race deterministic.