-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
Problem
When a PtlRPC request expires, the request can still wait in LNet for a peer tx credit. PtlRPC cannot finish the request until LNet releases the request MD. The import thus stays busy, and it cannot try the next NID. When bulk RPCs hold all peer credits to a dead server, a client can take minutes to reconnect, and the restarted target can evict it.
Environment
- master, at commit 40b80bc048 ("
LU-20867gss: re-resolve lsvcgssd ioctl device on failure"), on rocky9 VMs. - tcp, with ksocklnd peer_credits=8, sock_timeout=50, credits=256, and lnet discovery=0.
- One client, an MDS, and an OSS failover pair with 2 OSTs.
Reproduction
- On the client, set osc.*.max_rpcs_in_flight=32.
- Start 32 dd writers with direct I/O and 1 MiB blocks, across both OSTs.
- Confirm with lnetctl peer show -v that the client has no tx credits for the OSS.
- On the client, add a static ARP entry for the OSS IP to a MAC address that does not exist. This step makes the dead NID give no reply, as on some cloud networks.
- Kill the OSS VM. The partner mounts the OSTs.
- Time the interval until both OSC imports are FULL. Check for eviction.
On master, step 6 took 159.7-172.3 s, and the target evicted the client in 3 of 3 runs. While the import waited, it stayed in CONNECTING on the dead NID for 24-49 s at a time.
Analysis
- ptlrpc_unregister_reply() calls LNetMDUnlink() on rq_reply_md_h only. It then waits for rq_req_unlinked. PtlRPC never unlinks the request MD.
LU-4006makes lnet_post_send_locked() drop a message on an aborted MD. The drop occurs only when the message gets a credit and reaches the head of lpni_txq. A queued message thus waits for the credit holders to complete.- With a silent dead peer, socklnd holds each credit holder until its conn fails, about sock_timeout. Each connect attempt waits behind them.
Earlier attempts:
LU-13368let PtlRPC discard the request-out callback, so it stopped its wait for rq_req_unlinked.LU-17379fixed a check in that code, because a CONNECT waited on lp_dc_pendq past its timeout.LU-17704reverted theLU-17379fix (LU-17759errors), andLU-14499revertedLU-13368(o2iblnd shutdown hang, memory leak).- These attempts let PtlRPC stop its wait while the LND could still own the message. The fix below keeps that wait.
LU-21058 (change 70136) solves the connect case differently: a connect message skips the LNet credits. The two fixes complement each other. The fix below helps every RPC type, but it frees the import only at the RPC timeout. 70136 sends the connect at once, so it can also reach a server that restarted on the same NID.
Proposed fix
- PtlRPC: in ptlrpc_unregister_reply(), also call LNetMDUnlink(rq_req_md_h) in three cases: the request timed out (rq_timedout), failed (rq_err) or waits for a resend (rq_resend). Skip the call when LNet already unlinked the request MD. Read these flags under rq_lock. Use no discard flag. Keep the wait for rq_req_unlinked, so PtlRPC never frees a buffer that the LND holds. Do not unlink the request MD after a normal reply. LNet reports a send that completes on an unlinked MD as failed. A good request then ends as a network error, which is the
LU-17759failure. PtlRPC uses no result of the old send for a resend or for an aborted request (rq_err). But for these requests, the cancelled send would set rq_net_err. Then ptlrpc_check_set() would call ptlrpc_expire_one_request(), which logs a false network error and can set a FULL import to DISCONN. Thus request_out_callback() does not set rq_net_err for a request with rq_resend or rq_err. ptl_send_rpc() clears both flags before the next send. Side effect: a real LND failure of the old send no longer expires the request. The resend or the abort replaces the old send, so I think this is correct. Reviewers must agree. - LNet: each MD keeps a list of its attached messages. In LNetMDUnlink(), for a busy MD, take each message that LNet still queues off lpni_txq, tq_delayed or lp_dc_pendq. Return its credits, and finalize it with -ECANCELED after LNetMDUnlink() drops the locks. A message that the LND already holds stays as it is. The lock order is lnet_res_lock (MD CPT), then lnet_net_lock (message CPT), then lpni_lock or lp_lock. The unlink path does not take LNET_LOCK_EX.
A prototype of both parts gave these results with the reproduction above. Each row has 2-5 runs.
| Build | Kill only | Kill and static ARP | Evicted |
|---|---|---|---|
| master | 42.9-66.2 s | 159.7-172.3 s | 3 of 3, with static ARP |
| Part 1 only | 42.0-57.1 s | 96.3-106.4 s | 0 of 4 |
| Part 2 only, first prototype | 48.6-59.3 s | 160.5-168.2 s | 2 of 2, with static ARP |
| Parts 1 and 2 | 38.3-40.0 s | 50.2-52.2 s | 0 of 5 |
| |
37.0-41.1 s | 49.2-49.9 s | 0 of 4 |
The client debug log shows the cancel in every run, with 16-26 cancelled messages for each run. In one run, these were 14 OST_WRITE, 7 OBD_PING and 4 CONNECT requests. No node had an LBUG, an assertion or a slub_debug report. No node logged "already unlinked".
A CONNECT does not wait on lp_dc_pendq on master. Since LU-17906, PtlRPC does not send a connect over a peer that is not up to date. Other LNet users can still wait there. In an lnet_selftest test to a dead NID that LNet did not discover, master released the MD after 51.2 s, at the end of discovery. The series released it after 30.2-30.6 s, when lnet_selftest stopped the RPC.
The series adds sanity-lnet test_261. A delay rule holds all 8 peer tx credits, and an lnetctl ping with a 5 s timeout then waits for a credit. The test needs proof that the ping fails in less than 20 s, and that all credits come back. On master the test failed in 4 of 4 runs: the ping returned after 60-64 s. On the series the test passed in 7 of 7 runs: the ping failed after 5 s.
Other results on the series, with the same ping test:
| Case | master | Series |
|---|---|---|
| Message waits for an NI credit (tq_delayed) | 58.6-63.8 s | 5.0-5.1 s, 3 of 3 |
| Routed path (tcp1, router, tcp2) | 61-63 s | 5.0-5.1 s, 4 of 4 |
The credits came back in every run.
Full suites on 4-node clusters, with slub_debug=FZPU:
| Suite | master | Series |
|---|---|---|
| sanity-lnet | 162 pass, 1 fail (test_261) | 163 pass, 0 fail |
| recovery-small | 132 pass, 0 fail | 132 pass, 0 fail |
Both builds skipped the same tests. All router tests ran. lustre_rmmod passed on all nodes after each suite. No node had an LBUG, an assertion, a slub_debug report or an "already unlinked" message.
lnet_selftest brw throughput was not lower on the series. The VM network limited both builds to about 35 MiB/s. struct lnet_msg grows by 24 bytes, and struct lnet_libmd grows by 16 bytes.
The series also adds recovery-small test_165. The client delays requests to the OST by 10 s, sends 16 ladvise requests and then deactivates the OSC. The test fails if the import goes to DISCONN or if the import timeout count changes. Without the rq_err check, the test failed in 3 of 3 runs: each run logged 8 false network errors and set the import to DISCONN. With the check, the test passed in 4 of 4 runs. On master it passed in 3 of 3 runs. With the check, recovery-small gave 133 pass and 0 fail.
recovery-small test_166 covers an aborted request with a send state other than FULL. A delay rule holds an OSC CONNECT, and the test then deactivates the OSC. Without the rq_err check, ptlrpc_expire_one_request() changed the status from -EIO to -ETIMEDOUT and added 1 to the import timeout count. The test failed in 3 of 3 runs. With the check, and on master, the interpret callback got -EIO, and the test passed in 3 of 3 runs. The console output and the import state were the same on all builds.
Master has the same false network error for bulk RPCs. The function client_bulk_callback() sets rq_net_err when an abort unlinks the bulk MD. That function does not check rq_err. This series does not change that path.
Work before the fix can land:
- A test of o2iblnd. The
LU-14499hang was on o2iblnd. - A lockdep run.
- A runtime test of the resend unlink. ptlrpc_abort_set() has no runtime trigger, so no test covers it. This test needs a reconnect to a second server NID while the credits to the old NID stay held.
- is related to
-
LU-21101 lockdep reports in osd-ldiskfs, LNet, lnet_selftest and llite on master
-
- Open
-