[LU-7] Reconnect server->client connection

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.7.0, Lustre 2.5.5
• Labels: llnl
• Bugzilla ID: 3,622
• Assignee: nasf (Inactive)
• Reporter: Robert Read

Description

Local tracking bug for 3622.

Issue Links

• is duplicated by: LU-793 Reconnections should not be refused when there is a request in progress from this client.
• is related to: LU-793 Reconnections should not be refused when there is a request in progress from this client.
• is related to: LU-5520 BL AST resend
• is related to: LU-1239 cascading client evictions
• is related to: LU-1565 lost LDLM_CANCEL RPCs

Comments

[Christopher Morrone (Inactive)]

Vitaly, I don't understand what that has to do with this ticket. Please expand the explanation, or start a new ticket.

[Vitaly Fertman]

BL AST resend: http://review.whamcloud.com/9335

[Cory Spitz]

Re the last comment, that would be LU-1239, http://review.whamcloud.com/2355.

Also, LU-793, 'Reconnections should not be refused when there is a request in progress from this client', http://review.whamcloud.com/#change,1616 would also improve the situation by allowing clients to reconnect with RPCs in process.

[Nathan Rutman]

One aspect of this problem is in the following case:
1. MDS is overloaded with enqueues, consuming all the threads on MDS_REQUEST portal.
2. An rpc times out on a client, leading to it's reconnection. But this client has some locks to cancel, and the MDS is waiting for them.
3. The client sends MDS_CONNECT, but there is no empty thread to handle it.
4. Additionally, other clients are waiting for their enqueue completions; they try to ping MDS, but PING is also sent to MDS_REQUEST portal. Pings are supposed to be high priority rpcs, but since this service has no srv_hqreq_handler we let other low-priority rpc's take the last thread, thus potentially preventing future hp reqs from being serviced.
We've got a patch addressing 3 & 4 in inspection (MRP-455).

[Robert Read]

This is not a priority for us right now, but we'd be happy to take at a look at patch if you have one.

[Bruce Korb (Inactive)]

Ping? We have a customer nuisanced by this, too.

[nasf (Inactive)]

It is still in working queue. Because of other priority tasks, there is not definite release time. Sorry for that. Thanks for keeping trace this ticket, any updating for that will be posted here.

[Ned Bass (Inactive)]

We are still hitting this issue fairly frequently on our production 1.8.5 clusters. Is anyone still working on the proposed fix?

[nasf (Inactive)]

A updated version is available: (process ldlm callback resend in ptlrpc layer)

http://review.whamcloud.com/#change,125

[Robert Read]

I have requested Di and Bobi Jam to inspect this patch, instead of me.

[nasf (Inactive)]

According to the normal process, such patch should be inspected internal firstly, but I am not sure Robert has enough time to do that in time, so now it is better to be verified by customer if possible.