-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
Problem
On o2ib, sanity-lnet test_233 fails intermittently with "ping failed with rc = 255". The failed lnetctl ping takes about 60 s and returns -EIO. The failure occurs at a step that changes from run to run: local_dropped, local_timeout or local_no_route. At the end, the peer NI is in the recovery queue.
On tcp, the test passed 134 of 134 times in 30 days. On o2ib, it failed 5 of about 14 times in review sessions of https://review.whamcloud.com/69652 to https://review.whamcloud.com/69657. Those changes do not touch the code in the analysis below.
Environment
- Branch: master, at v2_17_57 and later.
- Maloo: lustre-reviews custom sessions with nettype=o2ib testlist=sanity-lnet.
- Example test sets: https://testing.whamcloud.com/test_sets/7fad76f9-03b6-487a-bcda-e4f74b544338 and https://testing.whamcloud.com/test_sets/2161bb2b-2daf-4577-bce0-2ac80009eb6f
Reproduction
- Run sanity-lnet test_233 on o2ib with 2 nodes.
- Repeat it. The failure rate on Maloo o2ib was about 1 in 3.
The race window is short. We did not reproduce the failure on Soft-RoCE VMs in 36 runs.
Analysis
The failure has two causes. The first cause loses the REPLY. The second cause makes the loss take 60 s to show.
1. The peer aborts the new conn. The initiator debug log shows this sequence (test set 7fad76f9):
- The drop rule simulates LOCAL_DROPPED. kiblnd_post_tx_locked() closes conn A with -22.
- LNet resends the GET. o2iblnd connects conn B 4 ms later, and the GET send completes on conn B.
- The peer closes conn B: Closing conn ... to 172.16.10.1@o2ib: error -103
- No REPLY arrives.
The peer debug log has no net mask, so it does not show the close of conn A. We infer the mechanism from the code. On the peer, the DISCONNECTED event of conn A runs kiblnd_close_conn_locked(A, 0). Conn B is still in the accept phase, so ibp_conns is empty. The function then unlinks the peer NI, although ibp_accepting is 1. When the peer gets the ESTABLISHED event of conn B, kiblnd_connreq_done() finds !kiblnd_peer_active(peer_ni) and closes conn B with -ECONNABORTED. The peer then drops the GET that arrived early on conn B.
kiblnd_peer_connect_failed() does not unlink a peer NI while kiblnd_peer_connecting() is true. kiblnd_close_conn_locked() does not make this check.
Test set 2161bb2b shows the same sequence.
2. The response timeout ignores a lower transaction_timeout. The step sets transaction_timeout to 10 s. The REPLY timeout came 62 s after the send. lnet_monitor_thread() checks for expired responses at rsp_timeout. It sets rsp_timeout = now + lnet_transaction_timeout / 2 after each check. LNet started with the default of 150 s, so the next check was 75 s after the start. The new value does not move that check. The lnetctl ping wait of 60 s ended first, so the ping returned -EIO, not -ETIMEDOUT.
Proposed fix
- o2iblnd: in kiblnd_close_conn_locked(), do not unlink the peer NI while kiblnd_peer_connecting(peer_ni) is true. The pending connect then either adds its conn, or kiblnd_peer_connect_failed() unlinks the peer NI.
- lnet: in lnet_monitor_thread(), run the response check when rsp_timeout is later than now + lnet_transaction_timeout / 2.
Fix 2 alone does not stop the test failure, because LNet does not resend a GET after a REPLY timeout. Fix 2 makes the failure fast and clear.
Test results
We tested both fixes on 2 Soft-RoCE VMs. For each fix, we ran the base build first as a negative control.
| Test | Base | With the fix |
|---|---|---|
| Issue 1, race widened by injected fail_loc | Peer closes conn B with -103 in 5 of 5 runs | No -103 in 10 runs, 10 of 10 pings pass |
| Issue 2, new sanity-lnet test_243 (lost REPLY after transaction_timeout 10) | Ping fails after 72 s | Ping fails after 13 to 14 s, 3 of 3 runs |
| sanity-lnet test_233, o2ib | - | 5 of 5 pass |
| sanity-lnet tests 200 to 260, o2ib | - | 23 pass, 0 fail |
The fail_loc for issue 1 is synthetic and is not in the patch. Issue 1 has no test that we can push.