-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
2.15.5 servers with some patches
2.15.x client with many patches back ported from 2.17
-
3
-
9223372036854775807
During an OST HA failover, service NID 10.131.100.78@tcp moved to another OSS. Client 10.131.0.131@tcp (node002) received and logged a TCP reset from that NID after imperative-recovery (IR) processing had issued an LNet discovery GET.
The reset is reported to LNet via lnet_notify(..., alive = false), but the outstanding discovery GET is not cancelled then. Its peer remains in a state containing LNET_PEER_PING_SENT until the response MD expires. In this trace, the GET waited 215.359 seconds from send, or 209.664 seconds after the TCP reset, before Ping Unlink ran.
The OSS used an effective IR soft recovery window of 150 seconds and timed out recovery after 175 seconds. It evicted the client export before the client could rediscover the peer and issue a valid recovery connection. The eventual connect was handled without recovery/reconnect flags, so the client became EVICTED.
This is not a request to extend recovery_time_soft: that timeout is the victim. Once a known TCP reset has been reported to LNet, discovery should not remain PING_SENT beyond the server recovery window. Also, another workaround is to reduce the lnet transaction_timeout so that clients timeout the transaction early and get to recovery within the server's recovery window.
Reproduction outline:
- Run I/O from the client to lustrefs-OST000c.
- Fail over the OST, moving 10.131.100.78@tcp from the old OSS to oss23.
- The new OSS mounts the target and starts imperative recovery.
- MGC sends the lustrefs-cliir log; the client creates an LNet peer and sends a discovery GET.
- SockLND receives a TCP reset for the service NID.
- The discovery GET remains outstanding until response expiry; the OSS evicts the stale export before the later retry can complete recovery.
| UTC, client debug | Elapsed | Evidence and state |
|---|---|---|
| 2026-08-20 19:18:01.328139 | T0 | ost_read to 10.131.100.78 fails with -107 (ENOTCONN). |
| 19:18:01.480912 | +0.153 s | PtlRPC declares abrupt disconnect; import changes FULL -> DISCONN -> CONNECTING. |
| 19:18:01.481650 | +0.154 s | First recovery connect finishes -19 (ENODEV), expected while target is unavailable. |
| 19:18:38.689228 | +37.361 s | MGC starts processing lustrefs-cliir. |
| 19:18:38.690575-.690620 | +37.362 s | LNet sends discovery GET; state is 0x100241, including PING_SENT. |
| 19:18:44.385197-.385208 | +43.057 s | SockLND logs TCP reset, then lnet_notify(... down) for the exact service NID. No Ping Unlink follows. |
| 19:19:14.529365 | +73.201 s | Recovery RPC ends -110; pinger begins another recovery attempt, but peer is discovered: 0, ready=0, and import remains CONNECTING. |
| 19:19:19-19:22:10 | +78-249 s | Repeated not pinging (in recovery ... CONNECTING) for OST000c. |
| 19:22:14.049281 | +252.721 s | Ping Unlink and Response timeout finally occur. This is 215.358661 s after GET send and 209.664073 s after reset. |
| 19:22:14.049322 | +252.721 s | Discovery fails: state 0x102041 rc -110. |
| 19:22:14.049477 | +252.721 s | Pinger starts recovery again; import changes DISCONN -> CONNECTING. |
| 19:22:14.050342 | +252.722 s | New socket attempt sees -99 (EADDRNOTAVAIL) on source port 1023. This is after the long discovery stall and is not asserted as its cause. |
| 19:22:14.245375 | +252.917 s | Server reply lacks reconnect/recover flags; import changes CONNECTING -> EVICTED. |
Client debug excerpts
# MGC IR handling creates peer and sends discovery GET
1787253518.689228 (mgc_request.c:606:do_requeue()) updating log lustrefs-cliir
1787253518.690575 (lib-move.c:5498:LNetGet()) LNetGet -> 12345-10.131.100.78@tcp
1787253518.690601 ... GET try# 0
1787253518.690619 (peer.c:3800:lnet_peer_send_ping()) peer 10.131.100.78@tcp
1787253518.690620 (peer.c:4132:lnet_peer_discovery()) ... state 0x100241 rc 0
1787253518.690670 (peer.c:3037:lnet_discovery_event_send()) Ping Send to 10.131.100.78@tcp: 0
# 5.695 seconds later, SockLND reports reset and peer down
1787253524.385197 (socklnd_cb.c:2387:ksocknal_find_timed_out_conn())
A connection with 12345-10.131.100.78@tcp (10.131.100.78:988) was reset
1787253524.385208 (router.c:1666:lnet_notify())
10.131.0.131@tcp notifying 10.131.100.78@tcp: down
There is no Ping Unlink at the peer-down notification. It occurs only at response expiry:
1787253734.049281 (peer.c:3058:lnet_discovery_event_unlink())
Ping Unlink for message to peer 10.131.100.78@tcp
1787253734.049289 (lib-move.c:3367:lnet_finalize_expired_responses())
Response timeout: md = 00000000bbb3dfd2: nid = 10.131.100.78@tcp
1787253734.049320 (peer.c:3758:lnet_peer_ping_failed())
peer 10.131.100.78@tcp:-110
1787253734.049322 (peer.c:4132:lnet_peer_discovery())
peer 10.131.100.78@tcp(...) state 0x102041 rc -110
1787253734.049324 (peer.c:2426:lnet_peer_discovery_complete())
Discovery complete. Dequeue peer 10.131.100.78@tcp
The later server response is interpreted as eviction because it has no recovery/reconnect semantics:
1787253734.245370 (import.c:1321:ptlrpc_connect_interpret()) lustrefs-OST000c-osc-...: evicting (reconnect/recover flags not set: 4) 1787253734.245375 (import.c:83:import_set_state_nolock()) lustrefs-OST000c_UUID: changing import state from CONNECTING to EVICTED 1787253734.245376 (import.c:1638:ptlrpc_import_recovery_state_machine()) This client was evicted by lustrefs-OST000c; in progress operations using this service will fail.
OSS and client-dmesg evidence
The OSS dmesg confirms mount, effective IR window, and stale-export eviction:
[Thu Aug 20 19:18:03 2026] LNet: Added LNI 10.131.100.78@tcp ...
[Thu Aug 20 19:18:10 2026] LDISKFS-fs ... mounted filesystem ...
[Thu Aug 20 19:18:10 2026] Lustre: lustrefs-OST000c: Imperative Recovery enabled,
recovery window shrunk from 300-900 down to 150-900
[Thu Aug 20 19:18:13 2026] Lustre: lustrefs-OST000c: Will be in recovery for at
least 2:30, or until 77 clients reconnect
...
[Thu Aug 20 19:21:08 2026] Lustre: lustrefs-OST000c: recovery is timed out,
evict stale exports
[Thu Aug 20 19:21:08 2026] Lustre: lustrefs-OST000c: disconnecting 54 stale clients
[Thu Aug 20 19:21:08 2026] Lustre: lustrefs-OST000c: Recovery over after 2:55,
of 77 clients 23 recovered and 54 were evicted.
The OSS debug trace names the same client UUID as stale before node002's eventual connection:
1787253685.735853 (genops.c:1570:class_disconnect_stale_exports()) lustrefs-OST000c: disconnect stale client a7f883aa-4b65-4c3f-9892-1f297a723c85@<unknown> 1787253734.050920 (ldlm_lib.c:1371:target_handle_connect())
lustrefs-OST000c: connection from a7f883aa-4b65-4c3f-9892-1f297a723c85@10.131.0.131@tcp
Client dmesg records the application-visible failure:
[Thu Aug 20 19:17:34 2026] LustreError: lustrefs-OST000c-osc-...:
operation ost_read to node 10.131.100.78@tcp failed: rc = -107
...
[Thu Aug 20 19:21:47 2026] LustreError: lustrefs-OST000c-osc-...:
This client was evicted by lustrefs-OST000c; in progress operations using
this service will fail.
Why this appears to be an LNet discovery state-machine gap
In the client code:
- lnet_peer_send_ping() sends discovery GET and records an outstanding PING_SENT operation.
- SockLND detects the reset and calls lnet_notify() with alive = false. The trace proves this notification reached router.c:lnet_notify().
- The normal peer-down path updates peer status and may call the LND peer-down callback, but does not cancel the outstanding discovery MD.
- lnet_discovery_event_unlink() is the existing path that clears PING_SENT, sets PING_FAILED, and records ETIMEDOUT. Here it runs only from lnet_finalize_expired_responses() at response-MD expiry.
Thus the transport has conclusively notified LNet that the connection is down, but discovery treats the previous GET as in flight for another 209.664 seconds.
Expected behavior and requested guidance
For an LND-originated peer-down notification, if the matching peer has an outstanding discovery ping MD, LNet should unlink that MD immediately. The existing unlink handler can then perform:
PING_SENT -> PING_FAILED (ETIMEDOUT) -> rediscovery/recovery can proceed
This should be limited to LND peer-down handling; it should not change normal I/O paths, MGC behavior, PtlRPC expiry values, or server recovery-timer policy.
Some Question:
- Could cancelling an outstanding discovery MD from the LND peer-down path the intended fix for this state transition?
- Is there an existing upstream issue or fix for a discovery GET that remains PING_SENT after a SockLND reset notification?
- If not, what implementation and test coverage would be preferred upstream?
Local proof-of-concept validation
After this reproduction, a local diagnostic patch was tested. It retains the exact peer during lnet_notify(..., alive=false), drops lnet_net_lock, calls the existing lnet_peer_cancel_discovery(), then releases the retained peer. It does not change PtlRPC timeout behavior.
In a later 64-client test, five clients hit PING_SENT -> TCP reset -> lnet_notify(down). All five logged Ping Unlink immediately after the down notification. The OSS completed recovery in 45 seconds:
Recovery over after 0:45, of 77 clients 77 recovered and 0 were evicted.
This supports immediate discovery cancellation as the direction, but the local patch is a proof of concept rather than an asserted upstream-ready change.