https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4269 Lustre <p>Running SWL - miranda IO test. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">2013-11-16 17:27:14 LustreError: 105183:0:(ldlm_lock.c:222:ldlm_lock_put()) ASSERTION( (((( lock))-&gt;l_flags &amp; (1ULL &lt;&lt; 50)) != 0) ) failed: 2013-11-16 17:27:14 LustreError: 105183:0:(ldlm_lock.c:222:ldlm_lock_put()) LBUG 2013-11-16 17:27:14 Pid: 105183, comm: miranda_io 2013-11-16 17:27:14 Nov 16 17:27:14 2013-11-16 17:27:14 Call Trace: 2013-11-16 17:27:14 iwc48 kernel: LustreError: 10518 [&lt;ffffffffa056b895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 2013-11-16 17:27:14 3:0:(ldlm_lock.c:222:ldlm_lock_p [&lt;ffffffffa056be97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 2013-11-16 17:27:14 ut()) ASSERTION( (((( lock))-&gt;l_flags &amp; (1ULL &lt;&lt; [&lt;ffffffffa080f94d&gt;] ldlm_lock_put+0x44d/0x560 [ptlrpc] 2013-11-16 17:27:14 50)) != 0) ) failed: 2013-11-16 17:27:14 Nov 16 17:27:14 iwc48 kernel: LustreError [&lt;ffffffffa0822d22&gt;] ldlm_cli_cancel_list+0xf2/0x3e0 [ptlrpc] 2013-11-16 17:27:14 : 105183:0:(ldlm_lock.c:222:ldlm_lock_put()) LBUG 2013-11-16 17:27:14 [&lt;ffffffffa0824952&gt;] ldlm_cli_cancel+0x132/0x360 [ptlrpc] 2013-11-16 17:27:14 [&lt;ffffffffa09fdbce&gt;] osc_lock_cancel+0xfe/0x1c0 [osc] 2013-11-16 17:27:14 [&lt;ffffffffa0708525&gt;] cl_lock_cancel0+0x75/0x160 [obdclass] 2013-11-16 17:27:14 [&lt;ffffffffa07090db&gt;] cl_lock_cancel+0x13b/0x140 [obdclass] 2013-11-16 17:27:14 [&lt;ffffffffa070d7e4&gt;] cl_lock_enqueue_wait+0xc4/0x2d0 [obdclass] 2013-11-16 17:27:14 [&lt;ffffffffa070aa17&gt;] ? cl_lock_mutex_put+0x77/0x90 [obdclass] 2013-11-16 17:27:14 [&lt;ffffffffa0a9126b&gt;] lov_lock_enqueue+0x3fb/0x850 [lov] 2013-11-16 17:27:14 [&lt;ffffffffa070c65c&gt;] cl_enqueue_try+0xfc/0x300 [obdclass] 2013-11-16 17:27:14 [&lt;ffffffffa070da5f&gt;] cl_enqueue_locked+0x6f/0x1f0 [obdclass] 2013-11-16 17:27:14 [&lt;ffffffffa070e6ae&gt;] cl_lock_request+0x7e/0x270 [obdclass] 2013-11-16 17:27:14 [&lt;ffffffffa071391c&gt;] cl_io_lock+0x3cc/0x560 [obdclass] 2013-11-16 17:27:14 [&lt;ffffffffa0713b52&gt;] cl_io_loop+0xa2/0x1b0 [obdclass] 2013-11-16 17:27:14 [&lt;ffffffffa0b0e290&gt;] ll_file_io_generic+0x460/0x610 [lustre] 2013-11-16 17:27:14 [&lt;ffffffffa0701e39&gt;] ? cl_env_get+0x29/0x350 [obdclass] 2013-11-16 17:27:14 [&lt;ffffffffa0b0ecb2&gt;] ll_file_aio_write+0x142/0x2c0 [lustre] 2013-11-16 17:27:14 [&lt;ffffffffa0b0ef9c&gt;] ll_file_write+0x16c/0x2a0 [lustre] 2013-11-16 17:27:14 [&lt;ffffffff81181398&gt;] vfs_write+0xb8/0x1a0 2013-11-16 17:27:14 [&lt;ffffffff81181c91&gt;] sys_write+0x51/0x90 2013-11-16 17:27:14 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 2013-11-16 17:27:14 </pre> </div></div> <p>Lctl dump attached</p> Hyperion/LLNL LU-4269

ldlm_lock_put()) ASSERTION( (((( lock))->l_flags & (1ULL << 50)) != 0) ) failed

Bug Blocker Resolved Fixed Zhenyu Xu Cliff White MB mn4 Mon, 18 Nov 2013 19:17:58 +0000 Thu, 27 Feb 2014 14:21:03 +0000 Fri, 21 Feb 2014 13:23:09 +0000 Lustre 2.6.0 Lustre 2.6.0 Lustre 2.5.1 0 11 <p>What I observed in the log</p> <div class="panel" style="border-width: 1px;"><div class="panelContent"> <p>line 140698, ldlm_cli_cancel()<br/> line 140704, ldlm_cli_cancel_local() ldlm lock ffff880793e54180/0xf128b59d838c9d81 res: <span class="error">&#91;0x9edf8f:0x0:0x0&#93;</span>.0 flags: 0x284'0000'0000 (LVB_READY|CANCELING|CBPENDING)<br/> line 140756, the ldlm lock is destroyed ldlm_lock_destroy_internal(), which set the destroy flag for the ldlm lock <font color="blue">(0x0004000000000000ULL)</font><br/> line 140771, prepare to issue ldlm lock cancel request<br/> line 140781, ldlm_cancel_pack() lock: ffff880793e54180/0xf128b59d838c9d81 flags: 0x8694'0000'0000 (BL_DONE|KMS_IGNORE|LVB_READY|CANCELING|CANCEL|CBPENDING) <font color="red">the destroy flag disappeared</font></p> </div></div> <p>Cliff, what's the latest git commit is this code? Since I've seen cl_lock_discard_pages() while current master branch does not have this function in it, I need to know the exact code base you were using when you hit this problem.</p> <p>We saw exactly same problem on multiple clients during mdtest is running. The client was running with the latest master branch.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;0&gt;LustreError: 10212:0:(ldlm_lock.c:222:ldlm_lock_put()) ASSERTION( (((( lock))-&gt;l_flags &amp; (1ULL &lt;&lt; 50)) != 0) ) failed: &lt;0&gt;LustreError: 10212:0:(ldlm_lock.c:222:ldlm_lock_put()) LBUG &lt;4&gt;Pid: 10212, comm: mdtest &lt;4&gt; &lt;4&gt;Call Trace: &lt;4&gt; [&lt;ffffffffa05c5895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] &lt;4&gt; [&lt;ffffffffa05c5e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] &lt;4&gt; [&lt;ffffffffa0951e27&gt;] ldlm_lock_put+0x547/0x630 [ptlrpc] &lt;4&gt; [&lt;ffffffffa09708f2&gt;] ldlm_cli_cancel_list+0xf2/0x3e0 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0972e0a&gt;] ldlm_prep_elc_req+0x21a/0x4b0 [ptlrpc] &lt;4&gt; [&lt;ffffffffa1043637&gt;] mdc_unlink+0xe7/0x500 [mdc] &lt;4&gt; [&lt;ffffffffa12ade7b&gt;] lmv_unlink+0x1db/0x7a0 [lmv] &lt;4&gt; [&lt;ffffffffa11c4a1e&gt;] ? ll_i2gids+0x2e/0xd0 [lustre] &lt;4&gt; [&lt;ffffffffa11cb228&gt;] ll_unlink+0x158/0x610 [lustre] &lt;4&gt; [&lt;ffffffff8118fec0&gt;] vfs_unlink+0xa0/0xf0 &lt;4&gt; [&lt;ffffffff8118ebfa&gt;] ? lookup_hash+0x3a/0x50 &lt;4&gt; [&lt;ffffffff81192265&gt;] do_unlinkat+0xf5/0x1b0 &lt;4&gt; [&lt;ffffffff810d94e2&gt;] ? unroll_tree_refs+0xe2/0x120 &lt;4&gt; [&lt;ffffffff8100c535&gt;] ? math_state_restore+0x45/0x60 &lt;4&gt; [&lt;ffffffff81192336&gt;] sys_unlink+0x16/0x20 &lt;4&gt; [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b &lt;4&gt; &lt;0&gt;Kernel panic - not syncing: LBUG &lt;4&gt;Pid: 10212, comm: mdtest Not tainted 2.6.32-358.23.2.el6_lustre.ge975b1c.x86_64 #1 &lt;4&gt;Call Trace: &lt;4&gt; [&lt;ffffffff8150deec&gt;] ? panic+0xa7/0x16f &lt;4&gt; [&lt;ffffffffa05c5eeb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] &lt;4&gt; [&lt;ffffffffa0951e27&gt;] ? ldlm_lock_put+0x547/0x630 [ptlrpc] &lt;4&gt; [&lt;ffffffffa09708f2&gt;] ? ldlm_cli_cancel_list+0xf2/0x3e0 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0972e0a&gt;] ? ldlm_prep_elc_req+0x21a/0x4b0 [ptlrpc] &lt;4&gt; [&lt;ffffffffa1043637&gt;] ? mdc_unlink+0xe7/0x500 [mdc] &lt;4&gt; [&lt;ffffffffa12ade7b&gt;] ? lmv_unlink+0x1db/0x7a0 [lmv] &lt;4&gt; [&lt;ffffffffa11c4a1e&gt;] ? ll_i2gids+0x2e/0xd0 [lustre] &lt;4&gt; [&lt;ffffffffa11cb228&gt;] ? ll_unlink+0x158/0x610 [lustre] &lt;4&gt; [&lt;ffffffff8118fec0&gt;] ? vfs_unlink+0xa0/0xf0 &lt;4&gt; [&lt;ffffffff8118ebfa&gt;] ? lookup_hash+0x3a/0x50 &lt;4&gt; [&lt;ffffffff81192265&gt;] ? do_unlinkat+0xf5/0x1b0 &lt;4&gt; [&lt;ffffffff810d94e2&gt;] ? unroll_tree_refs+0xe2/0x120 &lt;4&gt; [&lt;ffffffff8100c535&gt;] ? math_state_restore+0x45/0x60 &lt;4&gt; [&lt;ffffffff81192336&gt;] ? sys_unlink+0x16/0x20 &lt;4&gt; [&lt;ffffffff8100b072&gt;] ? system_call_fastpath+0x16/0x1b </pre> </div></div> <p>we have crashdump as well. Please let me know it helps for debugging.</p> <p>Hi Ihara-San,</p> <p>Do you have debug log of it?</p> <p>Hi Bobijam,</p> <p>vmware-dmesg.txt attached. Please let me know if you need vmware itself. I can upload it as well.</p> <p>besides dmesg, do you have debug log? (lctl dk)</p> <p>The test was run Nov 18th, build information was not in the version: build: 2.5.51--PRISTINE-2.6.32-358.23.2.el6_lustre.x86_64 - so it was likely the 2.5.51 tag. </p> <p>Sysrq -t from hung client.</p> <p>Attempting to attach file again</p> <p>From the feedback from Ihara, it looks like there exists race condition to access the flags of dlm lock.</p> <p>Let's create a debug patch to add a line into ldlm_lock_destroy_internal() to print out what's exact state of the lock</p> <p>debug patch at <a href="http://review.whamcloud.com/8759" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8759</a>, please unset panic_on_lbug (echo 0 &gt; /proc/sys/lnet/panic_on_lbug) and run this debug patched version so that we can collect the debug log when this assertion is hit.</p> <p>I've post a patch for this. However, it is not sure whether it fixes this problem, since we don't have a reliable way to reproduce this problem yet.<br/> <a href="http://review.whamcloud.com/#/c/8772/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8772/</a></p> <p>Thank you LiXi, but the iwc48.lctl.log.txt shows in line 140770</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00010000:00000001:0.0:1384651634.269628:0:105183:0:(ldlm_request.c:1760:ldlm_prepare_lru_list()) Process leaving (rc=0 : 0 : 0) </pre> </div></div> <p>which shows that when we canceled the problematic lock, ldlm_prepare_lru_list() didn't find other lru locks, so I'd think your patch would possibly not affect this issue.</p> <p>Ah, thank you for pointing that out, Zhenyu. I was just wondering why we can call ldlm_lock_remove_from_lru_nolock() without calling lock_res_and_lock() first. Is there anything I missed?</p> <p>ldlm_prepare_lru_list() is to collect unused lock from namespace::ns_unused_list, where only unused locks are linked into it (please refer to ldlm_lock_decref_internal()-&gt;ldlm_lock_add_to_lru()), so in ldlm_prepare_lru_list() we can call ldlm_lock_remove_from_lru_nolock() w/o locking the res and lock since the lock has no readers nor writers referring to it.</p> <p>Ah, thank you for the explaination! Now I understand. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>how often do you guys see this problem? If it's quite often, please apply Bobijam's patch and get some logs; otherwise, I'd like to drop the priority of this ticket.</p> <p>Jinshan</p> <p>Jinshan, Bobijam,</p> <p>Yes, we applied Bobijam's debug patch at the customer site, but didn't reproduce problem yet. I will post logs once we can hit same issue.</p> <p>We got crash again with debug patch. Attached is diagnostic information of crach. </p> <p>do you collect any lctl log from it?</p> <p>no, unfortunately.. client got panic even we have panic_on_lbug=0 setting.. I don't know why we got panic after LBUG..</p> <p>We reproduced this running master + a client lock debug patch on SLES11SP3 while reproducing a client lock bug. I believe we hit this with a special debug flag for cl_lock_trace turned on (So all calls to cl_lock_trace are being logged, but little or nothing else.)</p> <p>I should be able to make the dump and logs available if you think it would be helpful. I'm not sure about reproducing it, with different debug or at all - We run the test I'm using as a reproducer a lot and this is the first we've seen this particular bug. (The test in question is mmstress from the Linux Test Project.)</p> <p>yes please upload logs Patrick, thank you.</p> <p>Console log &amp; Lustre log.<br/> System is master from January 23rd on SLES11SP3, with a special debug patch that enables all calls to cl_lock_trace.</p> <p>Logs attached as noted. I've also uploaded the dump (with logs, vmlinuz, ko files, etc) to ftp.whamcloud.com at uploads/<a href="https://jira.whamcloud.com/browse/LU-4269" title="ldlm_lock_put()) ASSERTION( (((( lock))-&gt;l_flags &amp; (1ULL &lt;&lt; 50)) != 0) ) failed" class="issue-link" data-issue-key="LU-4269"><del>LU-4269</del></a>/<a href="https://jira.whamcloud.com/browse/LU-4269" title="ldlm_lock_put()) ASSERTION( (((( lock))-&gt;l_flags &amp; (1ULL &lt;&lt; 50)) != 0) ) failed" class="issue-link" data-issue-key="LU-4269"><del>LU-4269</del></a>-140123.tar.gz</p> <p>System is master from January 23rd on SLES11SP3, with a special debug patch that enables all calls to cl_lock_trace.</p> <p>Also, we reproduced this by running mmstress from the Linux test project. We ran multiple copies on multiple nodes. (This also hits several cl_lock related bugs, so that may make this harder to find. Those problems are more common than this one when running mmstress.)</p> <p>Hi Jinshan,</p> <p>When a top lock enqueue failed, top lock will go to cl_lock_hold_release() and then unhold a lovsub lock, which will go to osc_lock_cancel(), if at this time the osc lock is used by another IO, then the ldlm lock is not canceled.</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>osc_lock_cancel()</b></div><div class="codeContent panelContent"> <pre class="code-java"> do_cancel = (dlmlock-&gt;l_readers == 0 &amp;&amp; dlmlock-&gt;l_writers == 0); ldlm_set_cbpending(dlmlock); unlock_res_and_lock(dlmlock); <span class="code-keyword">if</span> (do_cancel) result = ldlm_cli_cancel(&amp;olck-&gt;ols_handle, LCF_ASYNC); } </pre> </div></div> <p>and the osc lock is detached from the lobsub lock in cl_lock_hold_release()</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (lock-&gt;cll_holds == 0) { CL_LOCK_ASSERT(lock-&gt;cll_state != CLS_HELD, env, lock); <span class="code-keyword">if</span> (lock-&gt;cll_descr.cld_mode == CLM_PHANTOM || lock-&gt;cll_descr.cld_mode == CLM_GROUP || lock-&gt;cll_state != CLS_CACHED) /* * If lock is still phantom or grouplock when user is * done with it---destroy the lock. */ lock-&gt;cll_flags |= CLF_CANCELPEND|CLF_DOOMED; <span class="code-keyword">if</span> (lock-&gt;cll_flags &amp; CLF_CANCELPEND) { lock-&gt;cll_flags &amp;= ~CLF_CANCELPEND; cl_lock_cancel0(env, lock); } <span class="code-keyword">if</span> (lock-&gt;cll_flags &amp; CLF_DOOMED) { <span class="code-comment">/* no longer doomed: it's dead... Jim. */</span> lock-&gt;cll_flags &amp;= ~CLF_DOOMED; cl_lock_delete0(env, lock); } } </pre> </div></div> <p>Is it possible that the ldlm lock somehow missed the cancel process somewhere so that its last dereference is still missing the destroy flag?</p> <p>I happen to be able to reproduce this issue steadily while I was working another one.</p> <p>After I applied the following patch:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">diff --git a/lustre/ldlm/ldlm_lock.c b/lustre/ldlm/ldlm_lock.c index 214ee8c..2e69ad0 100644 --- a/lustre/ldlm/ldlm_lock.c +++ b/lustre/ldlm/ldlm_lock.c @@ -372,12 +372,14 @@ <span class="code-object">int</span> ldlm_lock_destroy_internal(struct ldlm_lock *lock) LBUG(); } + LDLM_DEBUG(lock, <span class="code-quote">"destroy"</span>); <span class="code-keyword">if</span> (ldlm_is_destroyed(lock)) { LASSERT(cfs_list_empty(&amp;lock-&gt;l_lru)); EXIT; <span class="code-keyword">return</span> 0; } ldlm_set_destroyed(lock); + LDLM_DEBUG(lock, <span class="code-quote">"destroy"</span>); <span class="code-keyword">if</span> (lock-&gt;l_export &amp;&amp; lock-&gt;l_export-&gt;exp_lock_hash) { /* NB: it<span class="code-quote">'s safe to call cfs_hash_del() even lock isn'</span>t @@ -2169,6 +2171,7 @@ void ldlm_lock_cancel(struct ldlm_lock *lock) ldlm_resource_unlink_lock(lock); ldlm_lock_destroy_nolock(lock); + LDLM_DEBUG(lock, <span class="code-quote">"destroyed"</span>); <span class="code-keyword">if</span> (lock-&gt;l_granted_mode == lock-&gt;l_req_mode) ldlm_pool_del(&amp;ns-&gt;ns_pool, lock); diff --git a/lustre/ldlm/ldlm_request.c b/lustre/ldlm/ldlm_request.c index e140e0f..def3ee5 100644 --- a/lustre/ldlm/ldlm_request.c +++ b/lustre/ldlm/ldlm_request.c @@ -1141,6 +1141,7 @@ <span class="code-keyword">static</span> __u64 ldlm_cli_cancel_local(struct ldlm_lock *lock) rc = LDLM_FL_LOCAL_ONLY; } ldlm_lock_cancel(lock); + LDLM_DEBUG(lock, <span class="code-quote">"client-side cancel completed, rc = %lld"</span>, rc); } <span class="code-keyword">else</span> { <span class="code-keyword">if</span> (ns_is_client(ldlm_lock_to_ns(lock))) { LDLM_ERROR(lock, <span class="code-quote">"Trying to cancel local lock"</span>); diff --git a/lustre/llite/llite_internal.h b/lustre/llite/llite_internal.h index 92b278e..e9e7053 100644 --- a/lustre/llite/llite_internal.h +++ b/lustre/llite/llite_internal.h @@ -1469,7 +1469,7 @@ <span class="code-keyword">static</span> inline void cl_isize_unlock(struct inode *inode) <span class="code-keyword">static</span> inline void cl_isize_write_nolock(struct inode *inode, loff_t kms) { - LASSERT(down_trylock(&amp;ll_i2info(inode)-&gt;lli_size_sem) != 0); + <span class="code-comment">//LASSERT(down_trylock(&amp;ll_i2info(inode)-&gt;lli_size_sem) != 0); </span> i_size_write(inode, kms); } diff --git a/lustre/llite/llite_lib.c b/lustre/llite/llite_lib.c index df421d6..ae53f3d 100644 --- a/lustre/llite/llite_lib.c +++ b/lustre/llite/llite_lib.c @@ -1700,25 +1700,29 @@ <span class="code-object">int</span> ll_statfs(struct dentry *de, struct kstatfs *sfs) void ll_inode_size_lock(struct inode *inode) { +#<span class="code-keyword">if</span> 0 struct ll_inode_info *lli; LASSERT(!S_ISDIR(inode-&gt;i_mode)); lli = ll_i2info(inode); LASSERT(lli-&gt;lli_size_sem_owner != current); - down(&amp;lli-&gt;lli_size_sem); + <span class="code-comment">//down(&amp;lli-&gt;lli_size_sem); </span> LASSERT(lli-&gt;lli_size_sem_owner == NULL); lli-&gt;lli_size_sem_owner = current; +#endif } void ll_inode_size_unlock(struct inode *inode) { +#<span class="code-keyword">if</span> 0 struct ll_inode_info *lli; lli = ll_i2info(inode); LASSERT(lli-&gt;lli_size_sem_owner == current); lli-&gt;lli_size_sem_owner = NULL; - up(&amp;lli-&gt;lli_size_sem); + <span class="code-comment">//up(&amp;lli-&gt;lli_size_sem); </span>+#endif } void ll_update_inode(struct inode *inode, struct lustre_md *md) diff --git a/lustre/obdclass/cl_lock.c b/lustre/obdclass/cl_lock.c index d440da9..b8c2e04 100644 --- a/lustre/obdclass/cl_lock.c +++ b/lustre/obdclass/cl_lock.c @@ -493,6 +493,7 @@ <span class="code-keyword">static</span> struct cl_lock *cl_lock_lookup(<span class="code-keyword">const</span> struct lu_env *env, <span class="code-keyword">const</span> struct cl_io *io, <span class="code-keyword">const</span> struct cl_lock_descr *need) { +#<span class="code-keyword">if</span> 0 struct cl_lock *lock; struct cl_object_header *head; @@ -518,6 +519,7 @@ <span class="code-keyword">static</span> struct cl_lock *cl_lock_lookup(<span class="code-keyword">const</span> struct lu_env *env, RETURN(lock); } } +#endif RETURN(NULL); } @@ -1964,6 +1966,8 @@ <span class="code-keyword">static</span> struct cl_lock *cl_lock_hold_mutex(<span class="code-keyword">const</span> struct lu_env *env, cl_lock_hold_mod(env, lock, +1); lu_ref_add(&amp;lock-&gt;cll_holders, scope, source); lu_ref_add(&amp;lock-&gt;cll_reference, scope, source); + + cl_lock_cancel(env, lock); <span class="code-keyword">break</span>; } cl_lock_mutex_put(env, lock); </pre> </div></div> <p>And uses 32 processes to read the same file on the same time. The reading process hit the LBUG at the same place:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Call Trace: [&lt;ffffffffa03fd895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa03fde97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa0774fb7&gt;] ldlm_lock_put+0x547/0x630 [ptlrpc] [&lt;ffffffffa0793df2&gt;] ldlm_cli_cancel_list+0xf2/0x3e0 [ptlrpc] [&lt;ffffffffa0795a22&gt;] ldlm_cli_cancel+0x132/0x360 [ptlrpc] [&lt;ffffffffa0a8e72e&gt;] osc_lock_cancel+0x10e/0x1d0 [osc] [&lt;ffffffffa059c045&gt;] cl_lock_cancel0+0x75/0x160 [obdclass] [&lt;ffffffffa059d9d6&gt;] cl_lock_hold_release+0x1c6/0x2a0 [obdclass] [&lt;ffffffffa059ecc7&gt;] cl_lock_unhold+0x37/0x130 [obdclass] [&lt;ffffffffa0af8a68&gt;] lov_sublock_release+0x1a8/0x280 [lov] [&lt;ffffffffa0afa4fe&gt;] lov_lock_unuse+0x1be/0x290 [lov] [&lt;ffffffffa059c205&gt;] cl_unuse_try_internal+0x55/0xe0 [obdclass] [&lt;ffffffffa059f0a9&gt;] cl_unuse_try+0x199/0x320 [obdclass] [&lt;ffffffffa059f267&gt;] cl_unuse_locked+0x37/0x1a0 [obdclass] [&lt;ffffffffa059f40e&gt;] cl_unuse+0x3e/0x120 [obdclass] [&lt;ffffffffa05a5e57&gt;] cl_io_unlock+0x127/0x2b0 [obdclass] [&lt;ffffffffa05a6aed&gt;] cl_io_loop+0xcd/0x1b0 [obdclass] [&lt;ffffffffa0f7c9d6&gt;] ll_file_io_generic+0x2b6/0x710 [lustre] [&lt;ffffffffa0596db9&gt;] ? cl_env_get+0x29/0x350 [obdclass] [&lt;ffffffffa0f7cf6f&gt;] ll_file_aio_read+0x13f/0x2c0 [lustre] [&lt;ffffffffa0f7d42c&gt;] ll_file_read+0x16c/0x2a0 [lustre] [&lt;ffffffff81181a95&gt;] vfs_read+0xb5/0x1a0 [&lt;ffffffff81181bd1&gt;] sys_read+0x51/0x90 [&lt;ffffffff810dc685&gt;] ? __audit_syscall_exit+0x265/0x290 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b LustreError: dumping log to /tmp/lustre-log.1391207221.6283 </pre> </div></div> <p>However, from the log, I can see this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000020:00010000:5.0:1391207220.451167:0:6294:0:(cl_lock.c:151:cl_lock_trace0()) unuse lock: ffff88081acf48e8@(2 ffff8807e6fd0080 1 3 0 1 1 2)(ffff88080d4a0db8/0/1) at cl_unuse_try():1363 00000020:00010000:5.0:1391207220.451172:0:6294:0:(cl_lock.c:151:cl_lock_trace0()) enclosure lock: ffff88081acf4738@(1 (null) 0 3 0 1 1 2)(ffff88080d4a0db8/1/1) at cl_lock_enclosure():1691 00000020:00010000:5.0:1391207220.451175:0:6294:0:(cl_lock.c:151:cl_lock_trace0()) enclosure lock: ffff88081acf48e8@(3 ffff8807e6fd0080 1 4 0 2 1 2)(ffff88080d4a0db8/0/2) at cl_lock_enclosure():1691 00000020:00010000:5.0:1391207220.451178:0:6294:0:(cl_lock.c:151:cl_lock_trace0()) unuse lock: ffff88081acf4738@(2 ffff8807e6fd0080 1 3 0 1 1 2)(ffff88080d4a0c68/1/0) at cl_unuse_try():1363 00010000:00010000:5.0:1391207220.451185:0:6294:0:(ldlm_lock.c:852:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff8808198eb000 lock: ffff8808152de340/0x6d44b209b7d2fe79 lrc: 4/1,0 mode: PR/PR res: [0x2:0x0:0x0].0 rrc: 15 type: EXT [0-&gt;18446744073709551615] (req 1146880-&gt;1150975) flags: 0x10020000000000 nid: local remote: 0x6d44b209b7d2fe95 expref: -99 pid: 6294 timeout: 0 lvb_type: 1 00010000:00010000:5.0:1391207220.451191:0:6294:0:(ldlm_lock.c:920:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff8808198eb000 lock: ffff8808152de340/0x6d44b209b7d2fe79 lrc: 3/0,0 mode: PR/PR res: [0x2:0x0:0x0].0 rrc: 15 type: EXT [0-&gt;18446744073709551615] (req 1146880-&gt;1150975) flags: 0x10020000000000 nid: local remote: 0x6d44b209b7d2fe95 expref: -99 pid: 6294 timeout: 0 lvb_type: 1 00000020:00010000:5.0:1391207220.451197:0:6294:0:(cl_lock.c:151:cl_lock_trace0()) hold release lock: ffff88081acf4738@(3 ffff8807e6fd0080 1 5 0 2 0 2)(ffff88080d4a0c68/1/0) at cl_lock_hold_release():903 00000020:00010000:5.0:1391207220.451200:0:6294:0:(cl_lock.c:151:cl_lock_trace0()) hold release lock: ffff88081acf4738@(2 ffff8807e6fd0080 1 5 0 1 0 2)(ffff88080d4a0db8/1/2) at cl_lock_hold_release():903 00000080:00200000:5.0:1391207220.451207:0:6294:0:(vvp_io.c:1163:vvp_io_init()) [0x200000400:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:5.0:1391207220.452460:0:6294:0:(vvp_io.c:153:vvp_io_fini()) [0x200000400:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:5.0:1391207220.452485:0:6294:0:(ldlm_request.c:1127:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff8808198eb000 lock: ffff8808152de340/0x6d44b209b7d2fe79 lrc: 3/0,0 mode: PR/PR res: [0x2:0x0:0x0].0 rrc: 15 type: EXT [0-&gt;18446744073709551615] (req 1146880-&gt;1150975) flags: 0x28400000000 nid: local remote: 0x6d44b209b7d2fe95 expref: -99 pid: 6294 timeout: 0 lvb_type: 1 00000020:00010000:5.0:1391207220.452493:0:6294:0:(cl_lock.c:151:cl_lock_trace0()) cancel lock: ffff88081acf4738@(3 ffff8807e6fd0080 2 5 0 0 0 1)(ffff8807e0547920/1/1) at cl_lock_cancel():1836 00000020:00010000:5.0:1391207220.452496:0:6294:0:(cl_lock.c:151:cl_lock_trace0()) delete lock: ffff88081acf4738@(3 ffff8807e6fd0080 2 5 0 0 0 1)(ffff8807e0547920/1/1) at cl_lock_delete():1783 00010000:00010000:5.0:1391207220.452523:0:6294:0:(ldlm_lock.c:375:ldlm_lock_destroy_internal()) ### destroy ns: lustre-OST0000-osc-ffff8808198eb000 lock: ffff8808152de340/0x6d44b209b7d2fe79 lrc: 2/0,0 mode: PR/PR res: [0x2:0x0:0x0].0 rrc: 15 type: EXT [0-&gt;18446744073709551615] (req 1146880-&gt;1150975) flags: 0x10869400000000 nid: local remote: 0x6d44b209b7d2fe95 expref: -99 pid: 6294 timeout: 0 lvb_type: 1 00010000:00010000:5.0:1391207220.452529:0:6294:0:(ldlm_lock.c:382:ldlm_lock_destroy_internal()) ### destroy ns: lustre-OST0000-osc-ffff8808198eb000 lock: ffff8808152de340/0x6d44b209b7d2fe79 lrc: 2/0,0 mode: PR/PR res: [0x2:0x0:0x0].0 rrc: 15 type: EXT [0-&gt;18446744073709551615] (req 1146880-&gt;1150975) flags: 0x10869400000000 nid: local remote: 0x6d44b209b7d2fe95 expref: -99 pid: 6294 timeout: 0 lvb_type: 1 00010000:00010000:5.0:1391207220.452534:0:6294:0:(ldlm_lock.c:2174:ldlm_lock_cancel()) ### destroyed ns: lustre-OST0000-osc-ffff8808198eb000 lock: ffff8808152de340/0x6d44b209b7d2fe79 lrc: 1/0,0 mode: PR/PR res: [0x2:0x0:0x0].0 rrc: 15 type: EXT [0-&gt;18446744073709551615] (req 1146880-&gt;1150975) flags: 0x10869400000000 nid: local remote: 0x6d44b209b7d2fe95 expref: -99 pid: 6294 timeout: 0 lvb_type: 1 00010000:00010000:5.0:1391207220.452544:0:6294:0:(ldlm_request.c:1144:ldlm_cli_cancel_local()) ### client-side cancel completed, rc = 549755813888 ns: lustre-OST0000-osc-ffff8808198eb000 lock: ffff8808152de340/0x6d44b209b7d2fe79 lrc: 1/0,0 mode: --/PR res: [0x2:0x0:0x0].0 rrc: 15 type: EXT [0-&gt;18446744073709551615] (req 1146880-&gt;1150975) flags: 0x869400000000 nid: local remote: 0x6d44b209b7d2fe95 expref: -99 pid: 6294 timeout: 0 lvb_type: 1 00010000:00010000:5.0:1391207220.452554:0:6294:0:(ldlm_request.c:1187:ldlm_cancel_pack()) ### packing ns: lustre-OST0000-osc-ffff8808198eb000 lock: ffff8808152de340/0x6d44b209b7d2fe79 lrc: 1/0,0 mode: --/PR res: [0x2:0x0:0x0].0 rrc: 14 type: EXT [0-&gt;18446744073709551615] (req 1146880-&gt;1150975) flags: 0x869400000000 nid: local remote: 0x6d44b209b7d2fe95 expref: -99 pid: 6294 timeout: 0 lvb_type: 1 00010000:00010000:5.0:1391207220.452559:0:6294:0:(ldlm_request.c:1191:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:5.0:1391207220.452565:0:6294:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff8808198eb000 lock: ffff8808152de340/0x6d44b209b7d2fe79 lrc: 0/0,0 mode: --/PR res: [0x2:0x0:0x0].0 rrc: 14 type: EXT [0-&gt;18446744073709551615] (req 1146880-&gt;1150975) flags: 0x869400000000 nid: local remote: 0x6d44b209b7d2fe95 expref: -99 pid: 6294 timeout: 0 lvb_type: 1 00010000:00040000:5.0:1391207220.452570:0:6294:0:(ldlm_lock.c:222:ldlm_lock_put()) ASSERTION( (((( lock))-&gt;l_flags &amp; (1ULL &lt;&lt; 50)) != 0) ) failed: 00010000:00040000:5.0:1391207220.577549:0:6294:0:(ldlm_lock.c:222:ldlm_lock_put()) LBUG </pre> </div></div> <p>Both line 375 and 382 in were printed. and the source is as follows:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 361 <span class="code-object">int</span> ldlm_lock_destroy_internal(struct ldlm_lock *lock) 362 { 363 ENTRY; 364 365 <span class="code-keyword">if</span> (lock-&gt;l_readers || lock-&gt;l_writers) { 366 LDLM_ERROR(lock, <span class="code-quote">"lock still has references"</span>); 367 LBUG(); 368 } 369 370 <span class="code-keyword">if</span> (!cfs_list_empty(&amp;lock-&gt;l_res_link)) { 371 LDLM_ERROR(lock, <span class="code-quote">"lock still on resource"</span>); 372 LBUG(); 373 } 374 375 LDLM_DEBUG(lock, <span class="code-quote">"destroy"</span>); 376 <span class="code-keyword">if</span> (ldlm_is_destroyed(lock)) { 377 LASSERT(cfs_list_empty(&amp;lock-&gt;l_lru)); 378 EXIT; 379 <span class="code-keyword">return</span> 0; 380 } 381 ldlm_set_destroyed(lock); 382 LDLM_DEBUG(lock, <span class="code-quote">"destroy"</span>); </pre> </div></div> <p>However, the destroyed flag was failed to set. This is beyond of my understanding, and we should escalate this issue.</p> <p>Full log is here</p> <p>I think the prime suspect owuld be some place that updates lock flags without taking lock spinlock. There are several such places.</p> <p>I confirmed that changing the dlm flag operation to bitops fixed the problem.</p> <p>Check the attachment for the concept patch. The real patch should be worked out by modifying lustre/contrib/bit-masks/lustre_dlm_flags.tpl.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">diff --git a/lustre/include/lustre_dlm.h b/lustre/include/lustre_dlm.h index 98e03c9..bf4fb65 100644 --- a/lustre/include/lustre_dlm.h +++ b/lustre/include/lustre_dlm.h @@ -772,7 +772,7 @@ struct ldlm_lock { * Lock state flags. Protected by lr_lock. * \see lustre_dlm_flags.h where the bits are defined. */ - __u64 l_flags; + <span class="code-keyword">volatile</span> __u64 l_flags; /** * Lock r/w usage counters. diff --git a/lustre/include/lustre_dlm_flags.h b/lustre/include/lustre_dlm_flags.h index 283546d..4be532f 100644 --- a/lustre/include/lustre_dlm_flags.h +++ b/lustre/include/lustre_dlm_flags.h @@ -57,91 +57,91 @@ <span class="code-comment">/** extent, mode, or resource changed */</span> #define LDLM_FL_LOCK_CHANGED 0x0000000000000001ULL <span class="code-comment">// bit 0 </span>-#define ldlm_is_lock_changed(_l) LDLM_TEST_FLAG(( _l), 1ULL &lt;&lt; 0) -#define ldlm_set_lock_changed(_l) LDLM_SET_FLAG(( _l), 1ULL &lt;&lt; 0) -#define ldlm_clear_lock_changed(_l) LDLM_CLEAR_FLAG((_l), 1ULL &lt;&lt; 0) +#define ldlm_is_lock_changed(_l) LDLM_TEST_FLAG(( _l), 0) +#define ldlm_set_lock_changed(_l) LDLM_SET_FLAG(( _l), 0) +#define ldlm_clear_lock_changed(_l) LDLM_CLEAR_FLAG((_l), 0) ...... ...... <span class="code-comment">/** test <span class="code-keyword">for</span> ldlm_lock flag bit set */</span> -#define LDLM_TEST_FLAG(_l, _b) (((_l)-&gt;l_flags &amp; (_b)) != 0) +#define LDLM_TEST_FLAG(_l, _b) test_bit(_b, (<span class="code-keyword">volatile</span> unsigned <span class="code-object">long</span> *)&amp;(_l)-&gt;l_flags) <span class="code-comment">/** multi-bit test: are any of mask bits set? */</span> #define LDLM_HAVE_MASK(_l, _m) (((_l)-&gt;l_flags &amp; LDLM_FL_##_m##_MASK) != 0) <span class="code-comment">/** set a ldlm_lock flag bit */</span> -#define LDLM_SET_FLAG(_l, _b) ((_l)-&gt;l_flags |= (_b)) +#define LDLM_SET_FLAG(_l, _b) set_bit(_b, (<span class="code-keyword">volatile</span> unsigned <span class="code-object">long</span> *)&amp;(_l)-&gt;l_flags) <span class="code-comment">/** clear a ldlm_lock flag bit */</span> -#define LDLM_CLEAR_FLAG(_l, _b) ((_l)-&gt;l_flags &amp;= ~(_b)) +#define LDLM_CLEAR_FLAG(_l, _b) clear_bit(_b, (<span class="code-keyword">volatile</span> unsigned <span class="code-object">long</span> *)&amp;(_l)-&gt;l_flags) <span class="code-comment">/** @} subgroup */</span> <span class="code-comment">/** @} group */</span> </pre> </div></div> <p>Jinshan, according to the comment, the flags should be protected by lr_flags. Are there flag changes not protected by this lock?</p> <p>Also, I have some concern that using set_bit() will not set the same bits as the Lustre network protocol on all architectures. Do you know if there is a specific bit ordering used in all cases?</p> <p>The flags should be protected by ldlm_lock::l_lock. When I saw this issue, my first reaction was to check if there are unprotected writing to l_flags and the only place is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">/** * Removes LDLM lock \a lock from LRU. Assumes LRU is already locked. */ <span class="code-object">int</span> ldlm_lock_remove_from_lru_nolock(struct ldlm_lock *lock) { <span class="code-object">int</span> rc = 0; <span class="code-keyword">if</span> (!cfs_list_empty(&amp;lock-&gt;l_lru)) { struct ldlm_namespace *ns = ldlm_lock_to_ns(lock); LASSERT(lock-&gt;l_resource-&gt;lr_type != LDLM_FLOCK); cfs_list_del_init(&amp;lock-&gt;l_lru); ldlm_clear_skipped(lock); LASSERT(ns-&gt;ns_nr_unused &gt; 0); ns-&gt;ns_nr_unused--; rc = 1; } <span class="code-keyword">return</span> rc; } </pre> </div></div> <p>However, it was protected by ns_lock and if the lock is being destroyed, it must have been taken out of LRU list.</p> <p>Hi Jinshan,</p> <p>According to your test results, this problem really looks like a race problem. Would you please try following patch to check that the lock is always held when clearing/setting the flag?</p> <p><a href="http://review.whamcloud.com/#/c/8772/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8772/</a></p> <p>Hi Li Xi,</p> <p>That looks true. Somehow I looked at the code wrong. The dlm lock was set destroyed flag before taking out of LRU, so ldlm_clear_skipped() in ldlm_lock_remove_from_lru_nolock() is indeed the root cause of this problem.</p> <p>Your patch will cause deadlock because we should take res lock first and then ns_lock. My idea to fix this problem is to move ldlm_clear_skipped() out of ldlm_lock_remove_from_lru_nolock() into ldlm_lock_add_to_lru_nolock(). I think this will fix the problem as well.</p> <p>Ah, yeah, the right lock order is lock-&gt;l_lock, res-&gt;lr_lock, ns-&gt;ns_lock. Thanks for poiting that error out. </p> <p>We are able to reproduce the problem steadily too. If you could push a patch, we can check whether it helps. I feel that there are more than one place where lock-&gt;l_flags is changed without the protection of lock-&gt;l_lock.</p> <p>Li, Jinshan - Are either of you able to share how you're reproducing this?</p> <p>The root cause of this issue is pretty clear.</p> <p>Bobijam, can you please create a patch for this?</p> <p>updated patch <a href="http://review.whamcloud.com/#/c/8772/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8772/</a></p> <p>Patch landed to Master. Please reopen ticket if more work is needed.</p> <p>backport to b2_5:<br/> <a href="http://review.whamcloud.com/9346" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9346</a></p> Development Rank 1|hzw9mv: Rank (Obsolete) 11723 Severity