https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4300 Lustre <p>TraveltimeLaunc thread held cl_lock and cause call of cl_lock_cancel. Lock cancel causes rpc OST_WRITE. But OST_WRITE can't be sent because all ptlrpcd are waiting for cl_lock.</p> <p>So, OST_WRITE rpc sticks at new requests queue and can't be sent because ptlrpcd_19 is blocked by osc lock up call.</p> <p>dump from ptlrpcd threads - each thread has a recovery set, and two "production" new and processed.<br/> each rpc is dumped in format<br/> $rpc_ptr : $opc : $rq_status</p> <p>thread 19 have queue<br/> print tread 19 - 0xffff881fe70fae70<br/> new requests<br/> rpc 0xffff880faf27e000 : 101 - 0<br/> rpc 0xffff881feff5d800 : 4 - 0<br/> request in processing</p> <p>All 32 ptlrpcd threads are waiting with this stack trace:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&gt; 2013-10-15T15:14:18.766544-05:00 c0-0c0s5n3 ptlrpcd_0 D 0000000000000000 0 4183 2 0x00000000 &gt; 2013-10-15T15:14:18.791825-05:00 c0-0c0s5n3 ffff881fe7085aa0 0000000000000046 ffff881fe7084010 ffff881fea497040 &gt; 2013-10-15T15:14:18.792017-05:00 c0-0c0s5n3 0000000000010b00 ffff881fe7085fd8 ffff881fe7085fd8 0000000000010b00 &gt; 2013-10-15T15:14:18.817016-05:00 c0-0c0s5n3 0000000000000013 ffff881fea497040 0000000000000000 ffff881fe9800040 &gt; 2013-10-15T15:14:18.817082-05:00 c0-0c0s5n3 Call Trace: &gt; 2013-10-15T15:14:18.817244-05:00 c0-0c0s5n3 [&lt;ffffffff8138311f&gt;] schedule+0x3f/0x60 &gt; 2013-10-15T15:14:18.817388-05:00 c0-0c0s5n3 [&lt;ffffffff81383f92&gt;] __mutex_lock_slowpath+0x102/0x180 &gt; 2013-10-15T15:14:18.842380-05:00 c0-0c0s5n3 [&lt;ffffffff81383993&gt;] mutex_lock+0x23/0x40 &gt; 2013-10-15T15:14:18.842582-05:00 c0-0c0s5n3 [&lt;ffffffffa0375806&gt;] cl_lock_mutex_get+0x76/0xd0 [obdclass] &gt; 2013-10-15T15:14:18.867656-05:00 c0-0c0s5n3 [&lt;ffffffffa07bbc08&gt;] lovsub_parent_lock+0x48/0x120 [lov] &gt; 2013-10-15T15:14:18.867709-05:00 c0-0c0s5n3 [&lt;ffffffffa07bc368&gt;] lovsub_lock_state+0x68/0x1a0 [lov] &gt; 2013-10-15T15:14:18.867880-05:00 c0-0c0s5n3 [&lt;ffffffffa0373558&gt;] cl_lock_state_signal+0x68/0x160 [obdclass] &gt; 2013-10-15T15:14:18.893131-05:00 c0-0c0s5n3 [&lt;ffffffffa0373e7a&gt;] cl_lock_signal+0x5a/0x130 [obdclass] &gt; 2013-10-15T15:14:18.893285-05:00 c0-0c0s5n3 [&lt;ffffffffa072279d&gt;] osc_lock_upcall+0x25d/0x610 [osc] &gt; 2013-10-15T15:14:18.893419-05:00 c0-0c0s5n3 [&lt;ffffffffa07041bd&gt;] osc_enqueue_fini+0x9d/0x240 [osc] &gt; 2013-10-15T15:14:18.918520-05:00 c0-0c0s5n3 [&lt;ffffffffa0706eb7&gt;] osc_enqueue_interpret+0xe7/0x1d0 [osc] &gt; 2013-10-15T15:14:18.918702-05:00 c0-0c0s5n3 [&lt;ffffffffa049733c&gt;] ptlrpc_check_set+0x52c/0x1e20 [ptlrpc] &gt; 2013-10-15T15:14:18.943647-05:00 c0-0c0s5n3 [&lt;ffffffffa04c4acb&gt;] ptlrpcd_check+0x53b/0x560 [ptlrpc] &gt; 2013-10-15T15:14:18.943885-05:00 c0-0c0s5n3 [&lt;ffffffffa04c504b&gt;] ptlrpcd+0x29b/0x3b0 [ptlrpc] &gt; 2013-10-15T15:14:18.943979-05:00 c0-0c0s5n3 [&lt;ffffffff8138de94&gt;] kernel_thread_helper+0x4/0x10 </pre> </div></div> <p>The following application process is waiting for an RPC to be sent:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&gt; 2013-10-15T15:14:31.200904-05:00 c0-0c0s5n3 TraveltimeLaunc S 00000001050e94cc 0 16793 16777 0x00000000 &gt; 2013-10-15T15:14:31.200927-05:00 c0-0c0s5n3 ffff880fe8979308 0000000000000082 ffff880fe8978010 ffff880faca8e040 &gt; 2013-10-15T15:14:31.200967-05:00 c0-0c0s5n3 0000000000010b00 ffff880fe8979fd8 ffff880fe8979fd8 0000000000010b00 &gt; 2013-10-15T15:14:31.226179-05:00 c0-0c0s5n3 0000000000000000 ffff880faca8e040 0000000000000000 ffffffff81667020 &gt; 2013-10-15T15:14:31.226195-05:00 c0-0c0s5n3 Call Trace: &gt; 2013-10-15T15:14:31.226219-05:00 c0-0c0s5n3 [&lt;ffffffff8138311f&gt;] schedule+0x3f/0x60 &gt; 2013-10-15T15:14:31.251476-05:00 c0-0c0s5n3 [&lt;ffffffffa021d74e&gt;] cfs_waitq_wait+0xe/0x10 [libcfs] &gt; 2013-10-15T15:14:31.251533-05:00 c0-0c0s5n3 [&lt;ffffffffa0731e56&gt;] osc_extent_wait+0x576/0x630 [osc] &gt; 2013-10-15T15:14:31.251563-05:00 c0-0c0s5n3 [&lt;ffffffffa0732447&gt;] osc_cache_wait_range+0x537/0x820 [osc] &gt; 2013-10-15T15:14:31.276843-05:00 c0-0c0s5n3 [&lt;ffffffffa0733199&gt;] osc_cache_writeback_range+0xa69/0x1030 [osc] &gt; 2013-10-15T15:14:31.276897-05:00 c0-0c0s5n3 [&lt;ffffffffa071febe&gt;] osc_lock_flush+0x7e/0x260 [osc] &gt; 2013-10-15T15:14:31.302146-05:00 c0-0c0s5n3 [&lt;ffffffffa0720181&gt;] osc_lock_cancel+0xe1/0x1c0 [osc] &gt; 2013-10-15T15:14:31.302187-05:00 c0-0c0s5n3 [&lt;ffffffffa03730d5&gt;] cl_lock_cancel0+0x75/0x160 [obdclass] &gt; 2013-10-15T15:14:31.302229-05:00 c0-0c0s5n3 [&lt;ffffffffa0373e1b&gt;] cl_lock_cancel+0x13b/0x140 [obdclass] &gt; 2013-10-15T15:14:31.327498-05:00 c0-0c0s5n3 [&lt;ffffffffa072162c&gt;] osc_ldlm_blocking_ast+0x20c/0x330 [osc] &gt; 2013-10-15T15:14:31.327566-05:00 c0-0c0s5n3 [&lt;ffffffffa046baab&gt;] ldlm_cancel_callback+0x6b/0x190 [ptlrpc] &gt; 2013-10-15T15:14:31.352863-05:00 c0-0c0s5n3 [&lt;ffffffffa0479d1a&gt;] ldlm_cli_cancel_local+0x8a/0x470 [ptlrpc] &gt; 2013-10-15T15:14:31.352917-05:00 c0-0c0s5n3 [&lt;ffffffffa047cfec&gt;] ldlm_cli_cancel_list_local+0xec/0x290 [ptlrpc] &gt; 2013-10-15T15:14:31.378183-05:00 c0-0c0s5n3 [&lt;ffffffffa047dee5&gt;] ldlm_cancel_lru_local+0x35/0x40 [ptlrpc] &gt; 2013-10-15T15:14:31.378216-05:00 c0-0c0s5n3 [&lt;ffffffffa047f36f&gt;] ldlm_prep_elc_req+0x3df/0x490 [ptlrpc] &gt; 2013-10-15T15:14:31.403528-05:00 c0-0c0s5n3 [&lt;ffffffffa047f448&gt;] ldlm_prep_enqueue_req+0x28/0x30 [ptlrpc] &gt; 2013-10-15T15:14:31.403581-05:00 c0-0c0s5n3 [&lt;ffffffffa0706963&gt;] osc_enqueue_base+0x103/0x570 [osc] &gt; 2013-10-15T15:14:31.403617-05:00 c0-0c0s5n3 [&lt;ffffffffa0720e36&gt;] osc_lock_enqueue+0x506/0x900 [osc] &gt; 2013-10-15T15:14:31.403617-05:00 c0-0c0s5n3 [&lt;ffffffffa0720e36&gt;] osc_lock_enqueue+0x506/0x900 [osc] &gt; 2013-10-15T15:14:31.428880-05:00 c0-0c0s5n3 [&lt;ffffffffa03777bb&gt;] cl_enqueue_try+0xfb/0x320 [obdclass] &gt; 2013-10-15T15:14:31.428980-05:00 c0-0c0s5n3 [&lt;ffffffffa07b4a54&gt;] lov_lock_enqueue+0x1f4/0x890 [lov] &gt; 2013-10-15T15:14:31.429041-05:00 c0-0c0s5n3 [&lt;ffffffffa03777bb&gt;] cl_enqueue_try+0xfb/0x320 [obdclass] &gt; 2013-10-15T15:14:31.454182-05:00 c0-0c0s5n3 [&lt;ffffffffa037869f&gt;] cl_enqueue_locked+0x7f/0x1f0 [obdclass] &gt; 2013-10-15T15:14:31.454296-05:00 c0-0c0s5n3 [&lt;ffffffffa03792ce&gt;] cl_lock_request+0x7e/0x270 [obdclass] &gt; 2013-10-15T15:14:31.454354-05:00 c0-0c0s5n3 [&lt;ffffffffa0883e9f&gt;] cl_glimpse_lock+0x17f/0x490 [lustre] &gt; 2013-10-15T15:14:31.479540-05:00 c0-0c0s5n3 [&lt;ffffffffa088434d&gt;] cl_glimpse_size0+0x19d/0x1c0 [lustre] &gt; 2013-10-15T15:14:31.504845-05:00 c0-0c0s5n3 [&lt;ffffffffa0837650&gt;] ll_inode_revalidate_it+0x1b0/0x1d0 [lustre] &gt; 2013-10-15T15:14:31.504912-05:00 c0-0c0s5n3 [&lt;ffffffffa08376b6&gt;] ll_getattr_it+0x46/0x180 [lustre] &gt; 2013-10-15T15:14:31.504990-05:00 c0-0c0s5n3 [&lt;ffffffffa083782c&gt;] ll_getattr+0x3c/0x40 [lustre] &gt; 2013-10-15T15:14:31.505021-05:00 c0-0c0s5n3 [&lt;ffffffff81147355&gt;] vfs_getattr+0x25/0x50 &gt; 2013-10-15T15:14:31.530127-05:00 c0-0c0s5n3 [&lt;ffffffff81147920&gt;] vfs_fstatat+0x80/0x90 &gt; 2013-10-15T15:14:31.530209-05:00 c0-0c0s5n3 [&lt;ffffffff81147a5b&gt;] vfs_stat+0x1b/0x20 &gt; 2013-10-15T15:14:31.530264-05:00 c0-0c0s5n3 [&lt;ffffffff81147a84&gt;] sys_newstat+0x24/0x50 &gt; 2013-10-15T15:14:31.530321-05:00 c0-0c0s5n3 [&lt;ffffffff8138cdab&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> LU-4300

ptlrpcd threads deadlocked in cl_lock_mutex_get

Bug Blocker Resolved Fixed Jinshan Xiong Sergey Cheremencev MB Fri, 22 Nov 2013 18:38:12 +0000 Fri, 6 Sep 2019 22:44:08 +0000 Fri, 7 Mar 2014 19:11:39 +0000 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.5.2 0 31 <p>problem is simple.</p> <p>GLIMPSE lock is in NEW state, to switch into ENQUEUE state it need to send requests for each stripe assigned to the file.. fine.<br/> but when we send any lock enqueue request we are cancel unused + aged locks on same namespace, so we need to cancel an extent lock with dirty data. Before 2.4 (or better say) ptlrpcd "optimisation" lock enqueue send via normal thread but lock cancel send an BRW request via special brw thread and we have none dependencies between these requests, so brw finished and processed in own thread.. but now..<br/> glimpse reply tried to take an top level cl_lock, but it's blocked by wanting an brw reply.. OK.. same as before.. but one difference.<br/> brw request now in set_new_requests queue in same thread as tried to take top level mutex... and don't have ability to the send request over network.. so.. deadlocked.</p> <p>Xyratex have a several ideas how it's need fixed.<br/> 1) simplest - restore an _brw threads. so that situation was resolved as designed in CLIO. but it's will be block an ptlrpcd threads probably for a long time and may bad in case one of OST's in down.<br/> 2( change a ELC/LRU policy to avoid cancelling an extent locks while taking an glimpse lock, or some tuning with ability to cancel locks only if they don't have dirty data and may be canceled locally without sending RPC's.<br/> 3) change CLIO to be closed to the older LOV design.. prepare a full set of requests before sending any request over wire. in that case,<br/> glimpse replies will be never received while we hold a top level lock and flush a dirty data will be separated from and sending glimpse reply. so deadlock isn't possible.</p> <p>We want to discus with Intel before we will be prepare a patch for that issue. </p> <p>I think Jinshan and Bobijam are already working on simplifying cl_lock as part of the CLIO cleanup project. Have added them to the CC list of this bug to discuss possible solutions.</p> <p>from my point view - we need implement an something like to 2+3.<br/> it's don't need a large changes in cl_lock design - but need divide a any CLIO operation into two parts - preparing and executing.<br/> it's will be close to the transaction on server side where we have declare and execute phases also.</p> <p>as about 2 - it's changes just to avoid an any recursive dependence in preparing operation. Other solution in that area will be separate lock namespace completely. FLOCK's from IBITS, Extents from Glimpse ... </p> <p>I think we need invite an Alex and Tappro to that discussion but i don't able to add new watches to that ticket.</p> <p>It seems that the 2nd option is a good bet. The glimpse requests should be sent out asap so it makes no sense to write some data back before doing that.</p> <p>A rule thumb is never to block ptlrpc threads and we have taken this into account in the cl_lock rebuild project.</p> <p>Jay,</p> <p>what about separate a declare and execute phases for a any CLIO (and may be MD) operations ?</p> <p>What occurred with glimpse rpc may happen with regular enqueue rpc, so let's separate these problems:<br/> 1 - the current issue<br/> 2 - glimpse vs. lru_resize optimisation.</p> <p>regarding 2 - we may want to drop lru cancellation on enqueue rpc or only on glimpse enqueue rpc or even leave it as is. the original idea was that shrinker needs to cancel aged locks from lru and instead of creating new rpc from time to time, we may distribute this job among existing enqueue rpc.</p> <p>Vitaly,</p> <p>i think we need to align LRU policy to avoid cancelling a locks with dirty data. It's will solve both issues - just make glimpse faster and avoid any waiting in lock enqueue. what you think about ?</p> <p>That sounds perfect in my opinion, or we can only cancel PR locks. Please take a look at osc_ldlm_weigh_ast() which can be revised to figure out if there is any dirty pages covered by a specific lock.</p> <p>Hi Jinshan, just to follow up with you - will you be working on it, or it is ok for Xyratex to provide a patch for you to review? Thanks.</p> <p>I'm overwhelmed so I will appreciate if you guys can create a patch for this.</p> <p>OK, thanks Jinshan.</p> <p>That ticket affect 2.6 also.</p> <p>Hi, do you guys have a patch for this issue yet?</p> <p>This problem is also impacting ORNL. Does a reproducer exit?</p> <p>This is critical bug for us. Alexey if you have a patch can you push it so I can be tested and inspected.</p> <p>James, Jinshan,</p> <p>Cray's been working with Xyratex on this. Unfortunately, they haven't been able to come up with a patch for this issue yet. If someone from Intel is able to look at it, we can ask them to speak to you.</p> <p>Ann think simple fix should be enough as workaround, until we will have complex fix. I will push a patch in next two days.</p> <p>James, agreed with you. This is critical and we also hit rancoramly client crash due to this issue. As a quick workaround, "echo 0 &gt; /proc/fs/lustre/ldlm/namespaces/*/early_lock_cancel" on clients helped to avoid this issue in our case. Hope this helps in your case too.</p> <p>Yes, it would be interesting to know if disabling ELC, the work-around we found for <a href="https://jira.whamcloud.com/browse/LU-4552" title="osc_cache.c:899:osc_extent_wait() timeout quite often" class="issue-link" data-issue-key="LU-4552"><del>LU-4552</del></a>, also works there. This is highly possible, since Lustre version, stack-traces, time-out situation are very similar, so, since the very beginning, reading both tickets I had quickly the strong assumption that they could be duplicates.</p> <p>please try attached fix and say how it help.</p> <p>Pushed the patch to gerrit so it can run through maloo.</p> <p><a href="http://review.whamcloud.com/#/c/9156" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9156</a></p> <p>We also disabled ELC and our problems did stop.</p> <p>That patch is a good workaround but it will affect performance. I'm going to work out another solution to not do early cancel for glimpse lock ENQ.</p> <p>I pushed a new patch at: <a href="http://review.whamcloud.com/9175" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9175</a></p> <p>I don't run any tests so please run sanity test and then see if it can fix the problem as well.</p> <p>I will run test tomorrow.</p> <p>I'm seeing jobs failing to run with patch 9156.</p> <p>failed due to exact the same reason, or a new one?</p> <p>Looks like new one. I never saw failures of job before the xyratex patch at small scale. Only large scale did we see problems. Either way I will be try the intel patch next</p> <p>Patch has landed to Master.</p> <p>b2_5 patch at <a href="http://review.whamcloud.com/#/c/9451/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9451/</a></p> <p>James and all, I wonder why the patch-set #2/#3 of the b2_5 patch version mentioned (<a href="http://review.whamcloud.com/#/c/9451/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9451/</a>) only affect osc/osc_request.c file vs much more in its patch-set #1 like in the master version (<a href="http://review.whamcloud.com/9175" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9175</a>) ?</p> <p>The reason is Jinshan pointed the other part of the fix in master is related to the clio cleanup work. For b2_5 we only need the simpler fix.</p> <p>For the record - Jay changes in osc_cancel_for_recovery() re-introduced deadlock during recovery.</p> Duplicate LU-4552 LU-465 Related LU-5727 LU-11518 Development Rank 1|hzw9zb: Rank (Obsolete) 11787 Severity