Details
-
Bug
-
Resolution: Fixed
-
Major
-
None
-
3,622
-
8049
Description
Local tracking bug for 3622.
Attachments
Issue Links
- is duplicated by
-
-
- Resolved
-
Activity
Chris,
in fact, you issue is client eviction because cancel is not delivered to server.
it may have several different reasons:
1. cancel is lost. it is to be resent - fixed by LU-1565
2. BL AST is lost. BL AST is to be resent - fixed by this patch.
3. CANCEL cannot be sent due to absent connection, re-CONNECT fails with rpc in progress - fixed by LU-793
4. CONNECT cannot be handled by server as all the handling threads are stuck with other RPCs in progress - fixed by LU-1239
5. PING cannot be handled by server as all the handling threads are stuck with other RPCs and client cannot even start re-CONNECT - fixed by LU-1239
Vitaly, I don't understand what that has to do with this ticket. Please expand the explanation, or start a new ticket.
Re the last comment, that would be LU-1239, http://review.whamcloud.com/2355.
Also, LU-793, 'Reconnections should not be refused when there is a request in progress from this client', http://review.whamcloud.com/#change,1616 would also improve the situation by allowing clients to reconnect with RPCs in process.
One aspect of this problem is in the following case:
1. MDS is overloaded with enqueues, consuming all the threads on MDS_REQUEST portal.
2. An rpc times out on a client, leading to it's reconnection. But this client has some locks to cancel, and the MDS is waiting for them.
3. The client sends MDS_CONNECT, but there is no empty thread to handle it.
4. Additionally, other clients are waiting for their enqueue completions; they try to ping MDS, but PING is also sent to MDS_REQUEST portal. Pings are supposed to be high priority rpcs, but since this service has no srv_hqreq_handler we let other low-priority rpc's take the last thread, thus potentially preventing future hp reqs from being serviced.
We've got a patch addressing 3 & 4 in inspection (MRP-455).
This is not a priority for us right now, but we'd be happy to take at a look at patch if you have one.
It is still in working queue. Because of other priority tasks, there is not definite release time. Sorry for that. Thanks for keeping trace this ticket, any updating for that will be posted here.
We are still hitting this issue fairly frequently on our production 1.8.5 clusters. Is anyone still working on the proposed fix?
A updated version is available: (process ldlm callback resend in ptlrpc layer)
Well, no, not my issue. Although my issue may have been lumped in with a bunch of other things. We have specifically been waiting years for issue 3 to be solved.