|
problem is simple.
GLIMPSE lock is in NEW state, to switch into ENQUEUE state it need to send requests for each stripe assigned to the file.. fine.
but when we send any lock enqueue request we are cancel unused + aged locks on same namespace, so we need to cancel an extent lock with dirty data. Before 2.4 (or better say) ptlrpcd "optimisation" lock enqueue send via normal thread but lock cancel send an BRW request via special brw thread and we have none dependencies between these requests, so brw finished and processed in own thread.. but now..
glimpse reply tried to take an top level cl_lock, but it's blocked by wanting an brw reply.. OK.. same as before.. but one difference.
brw request now in set_new_requests queue in same thread as tried to take top level mutex... and don't have ability to the send request over network.. so.. deadlocked.
Xyratex have a several ideas how it's need fixed.
1) simplest - restore an _brw threads. so that situation was resolved as designed in CLIO. but it's will be block an ptlrpcd threads probably for a long time and may bad in case one of OST's in down.
2( change a ELC/LRU policy to avoid cancelling an extent locks while taking an glimpse lock, or some tuning with ability to cancel locks only if they don't have dirty data and may be canceled locally without sending RPC's.
3) change CLIO to be closed to the older LOV design.. prepare a full set of requests before sending any request over wire. in that case,
glimpse replies will be never received while we hold a top level lock and flush a dirty data will be separated from and sending glimpse reply. so deadlock isn't possible.
We want to discus with Intel before we will be prepare a patch for that issue.
|
|
I think Jinshan and Bobijam are already working on simplifying cl_lock as part of the CLIO cleanup project. Have added them to the CC list of this bug to discuss possible solutions.
|
|
from my point view - we need implement an something like to 2+3.
it's don't need a large changes in cl_lock design - but need divide a any CLIO operation into two parts - preparing and executing.
it's will be close to the transaction on server side where we have declare and execute phases also.
as about 2 - it's changes just to avoid an any recursive dependence in preparing operation. Other solution in that area will be separate lock namespace completely. FLOCK's from IBITS, Extents from Glimpse ...
I think we need invite an Alex and Tappro to that discussion but i don't able to add new watches to that ticket.
|
|
It seems that the 2nd option is a good bet. The glimpse requests should be sent out asap so it makes no sense to write some data back before doing that.
A rule thumb is never to block ptlrpc threads and we have taken this into account in the cl_lock rebuild project.
|
|
Jay,
what about separate a declare and execute phases for a any CLIO (and may be MD) operations ?
|
|
What occurred with glimpse rpc may happen with regular enqueue rpc, so let's separate these problems:
1 - the current issue
2 - glimpse vs. lru_resize optimisation.
regarding 2 - we may want to drop lru cancellation on enqueue rpc or only on glimpse enqueue rpc or even leave it as is. the original idea was that shrinker needs to cancel aged locks from lru and instead of creating new rpc from time to time, we may distribute this job among existing enqueue rpc.
|
|
Vitaly,
i think we need to align LRU policy to avoid cancelling a locks with dirty data. It's will solve both issues - just make glimpse faster and avoid any waiting in lock enqueue. what you think about ?
|
|
That sounds perfect in my opinion, or we can only cancel PR locks. Please take a look at osc_ldlm_weigh_ast() which can be revised to figure out if there is any dirty pages covered by a specific lock.
|
|
Hi Jinshan, just to follow up with you - will you be working on it, or it is ok for Xyratex to provide a patch for you to review? Thanks.
|
|
I'm overwhelmed so I will appreciate if you guys can create a patch for this.
|
|
OK, thanks Jinshan.
|
|
That ticket affect 2.6 also.
|
|
Hi, do you guys have a patch for this issue yet?
|
|
This problem is also impacting ORNL. Does a reproducer exit?
|
|
This is critical bug for us. Alexey if you have a patch can you push it so I can be tested and inspected.
|
|
James, Jinshan,
Cray's been working with Xyratex on this. Unfortunately, they haven't been able to come up with a patch for this issue yet. If someone from Intel is able to look at it, we can ask them to speak to you.
|
|
Ann think simple fix should be enough as workaround, until we will have complex fix. I will push a patch in next two days.
|
|
James, agreed with you. This is critical and we also hit rancoramly client crash due to this issue. As a quick workaround, "echo 0 > /proc/fs/lustre/ldlm/namespaces/*/early_lock_cancel" on clients helped to avoid this issue in our case. Hope this helps in your case too.
|
|
Yes, it would be interesting to know if disabling ELC, the work-around we found for LU-4552, also works there. This is highly possible, since Lustre version, stack-traces, time-out situation are very similar, so, since the very beginning, reading both tickets I had quickly the strong assumption that they could be duplicates.
|
|
please try attached fix and say how it help.
|
|
Pushed the patch to gerrit so it can run through maloo.
http://review.whamcloud.com/#/c/9156
|
|
We also disabled ELC and our problems did stop.
|
|
That patch is a good workaround but it will affect performance. I'm going to work out another solution to not do early cancel for glimpse lock ENQ.
|
|
I pushed a new patch at: http://review.whamcloud.com/9175
I don't run any tests so please run sanity test and then see if it can fix the problem as well.
I will run test tomorrow.
|
|
I'm seeing jobs failing to run with patch 9156.
|
|
failed due to exact the same reason, or a new one?
|
|
Looks like new one. I never saw failures of job before the xyratex patch at small scale. Only large scale did we see problems. Either way I will be try the intel patch next
|
|
Patch has landed to Master.
|
|
b2_5 patch at http://review.whamcloud.com/#/c/9451/
|
|
James and all, I wonder why the patch-set #2/#3 of the b2_5 patch version mentioned (http://review.whamcloud.com/#/c/9451/) only affect osc/osc_request.c file vs much more in its patch-set #1 like in the master version (http://review.whamcloud.com/9175) ?
|
|
The reason is Jinshan pointed the other part of the fix in master is related to the clio cleanup work. For b2_5 we only need the simpler fix.
|
|
For the record - Jay changes in osc_cancel_for_recovery() re-introduced deadlock during recovery.
|
Generated at Sat Feb 10 01:41:29 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
ptlrpcd-rpc.info
rpcs.log