https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12017 Lustre <p>setxattr takes inode lock and sends reint to MDS.<br/> truncate takes MDS_INODELOCK_DOM lock and  wants to acquire inode lock.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 14942 TASK: ffff88007659cf10 CPU: 3 COMMAND: "truncate" #0 [ffff88011f397af8] __schedule at ffffffff816b3de4 #1 [ffff88011f397b88] schedule_preempt_disabled at ffffffff816b5329 #2 [ffff88011f397b98] __mutex_lock_slowpath at ffffffff816b30d7 #3 [ffff88011f397bf0] mutex_lock at ffffffff816b24bf #4 [ffff88011f397c08] vvp_io_setattr_start at ffffffffc118993d [lustre] #5 [ffff88011f397c40] cl_io_start at ffffffffc06e7a25 [obdclass] #6 [ffff88011f397c68] cl_io_loop at ffffffffc06e9e01 [obdclass] #7 [ffff88011f397cd8] cl_setattr_ost at ffffffffc11847ef [lustre] #8 [ffff88011f397d28] ll_setattr_raw at ffffffffc11614d8 [lustre] #9 [ffff88011f397df0] ll_setattr at ffffffffc11617d3 [lustre] #10 [ffff88011f397e00] notify_change at ffffffff81223bc4 #11 [ffff88011f397e48] do_truncate at ffffffff81203445 #12 [ffff88011f397ec0] vfs_truncate at ffffffff8120361c #13 [ffff88011f397ef8] do_sys_truncate at ffffffff8120370c #14 [ffff88011f397f40] sys_truncate at ffffffff812038de </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 15194 TASK: ffff880077f18000 CPU: 1 COMMAND: "setfattr" #0 [ffff88011d33b8b8] __schedule at ffffffff816b3de4 #1 [ffff88011d33b948] schedule at ffffffff816b4409 #2 [ffff88011d33b958] schedule_timeout at ffffffff816b1ca4 #3 [ffff88011d33ba00] ptlrpc_set_wait at ffffffffc09070a0 [ptlrpc] #4 [ffff88011d33baf0] ptlrpc_queue_wait at ffffffffc09074e3 [ptlrpc] #5 [ffff88011d33bb10] mdc_xattr_common at ffffffffc0b52186 [mdc] #6 [ffff88011d33bb90] mdc_setxattr at ffffffffc0b522de [mdc] #7 [ffff88011d33bbd0] lmv_setxattr at ffffffffc0872524 [lmv] #8 [ffff88011d33bc48] ll_xattr_set_common at ffffffffc1175b54 [lustre] #9 [ffff88011d33bcc8] ll_xattr_set_common_3_11 at ffffffffc11769ab [lustre] #10 [ffff88011d33bcd8] generic_setxattr at ffffffff8122c2d8 #11 [ffff88011d33bd10] __vfs_setxattr_noperm at ffffffff8122cb45 #12 [ffff88011d33bd58] vfs_setxattr at ffffffff8122cd45 #13 [ffff88011d33bd98] setxattr at ffffffff8122ce7e #14 [ffff88011d33bef0] sys_setxattr at ffffffff8122d177 </pre> </div></div> <p>MDS locks are for different bits MDS_INODELOCK_UPDATE|MDS_INODELOCK_XATTR vs<br/> MDS_INODELOCK_DOM but they blocks each other if some blocking lock was present earlier because Lustre tries to grant only first lock in the waiting list.</p> LU-12017

Truncate vs setxattr deadlock with DoM

Bug Major Resolved Fixed Mikhail Pershin Andriy Skulysh patch Mon, 25 Feb 2019 20:33:45 +0000 Wed, 29 Apr 2020 05:38:12 +0000 Tue, 27 Aug 2019 02:55:57 +0000 Lustre 2.13.0 Lustre 2.12.3 0 7 <p>I suppose the easiest way to resolve the deadlock is to add a separate lock type for DoM lock.</p> <p>are you able to reproduce this behavior? There is <a href="https://jira.whamcloud.com/browse/LU-11285" title="don&#39;t stop on the first blocked lock in ldlm_reprocess_queue()" class="issue-link" data-issue-key="LU-11285"><del>LU-11285</del></a> which should help</p> <p>I can confirm that <a href="https://jira.whamcloud.com/browse/LU-11285" title="don&#39;t stop on the first blocked lock in ldlm_reprocess_queue()" class="issue-link" data-issue-key="LU-11285"><del>LU-11285</del></a> helps with my reproducer</p> <p>Are you able to share your reproducer, maybe as a testcase?</p> <p>I don't see how deadlock happens there, if truncate and setxattr are not dependent than other third lock can't cause deadlock because it will be granted at some moment and let waiting queue go further. The only case when it cause deadlock is when truncate and setxattr are dependent somehow so one is needed other to be granted. So I also wonder about reproduces to investigate this case more closely.</p> <p><a href="https://jira.whamcloud.com/browse/LU-11285" title="don&#39;t stop on the first blocked lock in ldlm_reprocess_queue()" class="issue-link" data-issue-key="LU-11285"><del>LU-11285</del></a> was made for other reasons, DOM lock as IO lock can be held for significant time and any other DOM lock in waiting queue might block all MD locks in that queue with no reasons. So this is sort of optimization when locks with non-crossing bits are not blocking each other. Strictly speaking that is not deadlock but lock timeout case, increasing lock timeout resolves that which is not possible with real deadlock. So <a href="https://jira.whamcloud.com/browse/LU-11285" title="don&#39;t stop on the first blocked lock in ldlm_reprocess_queue()" class="issue-link" data-issue-key="LU-11285"><del>LU-11285</del></a> is not deadlock resolver, thought may help with some deadlocks, but right way it to fix the source of deadlock.</p> <p>This deadlock caused because CLIO code takes a i_mutex lock after DoM lock is obtained.<br/> Fixing client side - solve this issue completely without ldlm code changes.</p> <p>Exactly, that is what I've meant, solving that at client side is the right thing. Alexey, could you give particular function name where that is happening?</p> <p>ll_setattr_raw - release a i_mutex to take it later.</p> <p>test patch (passed sanity / sanity-dom), including Andriy test for this paticual deadlock.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> diff --git a/lustre/llite/llite_lib.c b/lustre/llite/llite_lib.c index 1839163936..d33d02519e 100644 --- a/lustre/llite/llite_lib.c +++ b/lustre/llite/llite_lib.c @@ -1566,11 +1566,11 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> ll_md_setattr(struct dentry *dentry, struct md_op_data *op_data) /* inode size will be in ll_setattr_ost, can't <span class="code-keyword">do</span> it now since dirty * cache is not cleared yet. */ op_data-&gt;op_attr.ia_valid &amp;= ~(TIMES_SET_FLAGS | ATTR_SIZE); - <span class="code-keyword">if</span> (S_ISREG(inode-&gt;i_mode)) - inode_lock(inode); +<span class="code-comment">// <span class="code-keyword">if</span> (S_ISREG(inode-&gt;i_mode)) </span>+<span class="code-comment">// inode_lock(inode); </span> rc = simple_setattr(dentry, &amp;op_data-&gt;op_attr); - <span class="code-keyword">if</span> (S_ISREG(inode-&gt;i_mode)) - inode_unlock(inode); +<span class="code-comment">// <span class="code-keyword">if</span> (S_ISREG(inode-&gt;i_mode)) </span>+<span class="code-comment">// inode_unlock(inode); </span> op_data-&gt;op_attr.ia_valid = ia_valid; rc = ll_update_inode(inode, &amp;md); @@ -1657,11 +1657,11 @@ <span class="code-object">int</span> ll_setattr_raw(struct dentry *dentry, struct iattr *attr, LTIME_S(attr-&gt;ia_mtime), LTIME_S(attr-&gt;ia_ctime), (s64)ktime_get_real_seconds()); - <span class="code-keyword">if</span> (S_ISREG(inode-&gt;i_mode)) { - <span class="code-keyword">if</span> (attr-&gt;ia_valid &amp; ATTR_SIZE) - inode_dio_write_done(inode); - inode_unlock(inode); - } +<span class="code-comment">// <span class="code-keyword">if</span> (S_ISREG(inode-&gt;i_mode)) { </span>+<span class="code-comment">// <span class="code-keyword">if</span> (attr-&gt;ia_valid &amp; ATTR_SIZE) </span>+<span class="code-comment">// inode_dio_write_done(inode); </span>+<span class="code-comment">// inode_unlock(inode); </span>+<span class="code-comment">// } </span> /* We always <span class="code-keyword">do</span> an MDS RPC, even <span class="code-keyword">if</span> we're only changing the size; * only the MDS knows whether truncate() should fail with -ETXTBUSY */ @@ -1746,7 +1746,7 @@ out: ll_finish_md_op_data(op_data); <span class="code-keyword">if</span> (S_ISREG(inode-&gt;i_mode)) { - inode_lock(inode); +<span class="code-comment">// inode_lock(inode); </span> <span class="code-keyword">if</span> ((attr-&gt;ia_valid &amp; ATTR_SIZE) &amp;&amp; !hsm_import) inode_dio_wait(inode); /* Once we<span class="code-quote">'ve got the i_mutex, it'</span>s safe to set the S_NOSEC diff --git a/lustre/llite/vvp_io.c b/lustre/llite/vvp_io.c index ac2cd0ea20..f9cbf6a9cc 100644 --- a/lustre/llite/vvp_io.c +++ b/lustre/llite/vvp_io.c @@ -712,10 +712,10 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> vvp_io_setattr_start(<span class="code-keyword">const</span> struct lu_env *env, <span class="code-keyword">if</span> (cl_io_is_trunc(io)) { down_write(&amp;lli-&gt;lli_trunc_sem); - inode_lock(inode); +<span class="code-comment">// inode_lock(inode); </span> inode_dio_wait(inode); } <span class="code-keyword">else</span> { - inode_lock(inode); +<span class="code-comment">// inode_lock(inode); </span> } <span class="code-keyword">if</span> (io-&gt;u.ci_setattr.sa_avalid &amp; TIMES_SET_FLAGS) @@ -736,10 +736,10 @@ <span class="code-keyword">static</span> void vvp_io_setattr_end(<span class="code-keyword">const</span> struct lu_env *env, * because osc has already notified to destroy osc_extents. */ vvp_do_vmtruncate(inode, io-&gt;u.ci_setattr.sa_attr.lvb_size); inode_dio_write_done(inode); - inode_unlock(inode); +<span class="code-comment">// inode_unlock(inode); </span> up_write(&amp;lli-&gt;lli_trunc_sem); } <span class="code-keyword">else</span> { - inode_unlock(inode); +<span class="code-comment">// inode_unlock(inode); </span> } } </pre> </div></div> <p>as about deadlock scenario it's deadlock via client side.<br/> truncate releases an i_mutex in ll_setattr_raw and start to take DoM lock, setxattr remove a parent lookup lock, so open start from lookup step but MDT have DOM always open option set, so open have conflict with DoM lock but i_mutex was held. Setattr have a DoM lock and start to take i_mutex..<br/> It looks setxattr have posibility to have deadlock with truncate itself, but he should don't conflict with DoM lock (I don't check it really).</p> <p>OOPS. Deadlock.</p> <p>I wonder why inode lock was released and taken back later in vvp_io_setattr_start(). Probably that was also intended to resolve other lock conflicts. Anyway, thanks for analysis, if it is possible, please attach test for that particular deadlock, so I will be able to use it for testing too</p> <p>Ah, that is because of lli_trunc_sem/i_mutex lock ordering, <a href="https://jira.whamcloud.com/browse/LU-7927" title="Deadlock between ll_setattr() and ll_file_write()-&gt;ll_fsync()" class="issue-link" data-issue-key="LU-7927"><del>LU-7927</del></a>, commit 5d60fd75152d10<br/> Well there are things to think about</p> <p>No. it's not <a href="https://jira.whamcloud.com/browse/LU-7927" title="Deadlock between ll_setattr() and ll_file_write()-&gt;ll_fsync()" class="issue-link" data-issue-key="LU-7927"><del>LU-7927</del></a>. inode_lock was exist before it. This ticket was changed a place, but not add locking where.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> - inode_lock(inode); <span class="code-keyword">if</span> (cl_io_is_trunc(io)) { down_write(&amp;lli-&gt;lli_trunc_sem); + inode_lock(inode); inode_dio_wait(inode); + } <span class="code-keyword">else</span> { + inode_lock(inode); } </pre> </div></div> <p>Andriy Skulysh (c17819@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/35057" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35057</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12017" title="Truncate vs setxattr deadlock with DoM" class="issue-link" data-issue-key="LU-12017"><del>LU-12017</del></a> test: DoM truncate deadlock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2f0a4e2a2073fa4c0fced7f1e155ff7b4372298a</p> <p>Similar deadlock (getattr vs rename) is triggered :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>granted lock: &lt;struct ldlm_lock 0xffff88007905eb40&gt; 0x73f93784e720791e [0x200000402:0x14c8:0x0] pid 20988 MDS_INODELOCK_LOOKUP|MDS_INODELOCK_UPDATE waiting locks: &lt;struct ldlm_lock 0xffff8801243d8000&gt; 0x73f93784e72079b1 [0x200000402:0x14c8:0x0] pid 10035 MDS_INODELOCK_LOOKUP|MDS_INODELOCK_UPDATE|MDS_INODELOCK_PERM &lt;struct ldlm_lock 0xffff880090f23440&gt; 0x73f93784e72079f0 [0x200000402:0x14c8:0x0] pid 20988 MDS_INODELOCK_DOM </pre> </div></div> <p>lock 0xffff88007905eb40 is for mnew in mdt_reint_rename()<br/> lock 0xffff880090f23440 is requested by same thread mdt_reint_rename()-&gt;mdt_dom_discard_data()<br/> so DOM lock can't be  granted and mnew lock can't be unlocked as we need to send the reply</p> <p>Andriy, currently the mdt_dom_discard_data() doesn't wait for lock to be granted, that sort of deadlock should be fixed already by <a href="https://jira.whamcloud.com/browse/LU-11359" title="racer test 1 times out with client hung in dir_create.sh, ls, … and MDS in ldlm_completion_ast()" class="issue-link" data-issue-key="LU-11359"><del>LU-11359</del></a></p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=pjones" class="user-hover" rel="pjones">pjones</a>, can we target this for 2.13.0?</p> <p>I've added L2.13 to the Fix Version.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35057/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35057/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12017" title="Truncate vs setxattr deadlock with DoM" class="issue-link" data-issue-key="LU-12017"><del>LU-12017</del></a> ldlm: DoM truncate deadlock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 2250e072c37855d611aa64027945981fe2c8f4d7</p> <p>Landed for 2.13</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35937" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35937</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12017" title="Truncate vs setxattr deadlock with DoM" class="issue-link" data-issue-key="LU-12017"><del>LU-12017</del></a> ldlm: DoM truncate deadlock<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 3f8d06297971efbf773df70c97edb1a7f96a34ad</p> <p>The problem is not fixed fully, patch helps to avoid some situations with deadlock when IBITS of conflicting locks don't intersect but deadlock still exists and may happen in future when other IBITS combination will occur. The reason of deadlock on client side is not yet fixed.</p> <p>I'd like to keep this ticket opened with lowered severity and priority or create new ticket with link to this one to fix the reason of deadlock on client side</p> <p>As long as there is <em>some</em> value to landing the original patch I would suggest moving remaining work to a distinct ticket.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35937/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35937/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12017" title="Truncate vs setxattr deadlock with DoM" class="issue-link" data-issue-key="LU-12017"><del>LU-12017</del></a> ldlm: DoM truncate deadlock<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 671ece3104edd3342838e2c0eda350e72219dc97</p> <p>Short update for everyone interested. After discussion with Vitaly Fertman there is conclusion about possibility of new deadlocks due to reverse locking for <tt>i_mutex</tt> and <tt>DOM</tt> LDLM lock in truncate path. That still can cause deadlocks in generic case like the following:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> th1: setattr took DOM lock, going to take inode lock th2: wants some lock with DOM|XXX and stuck waiting the th1's DOM lock th3: already have inode lock, enqueue some XXX bits and waits these bits on th2 waiting lock. </pre> </div></div> <p>Due to reverse order of taking inode lock and ldlm lock in th1 and th3, any th2 with DOM and XXX bits which are common with th3 bits can cause deadlock. Ideally that should be resolved by changing truncate for proper lock order, though that can be non-trivial. Until that we can just avoid combining DOM bit with other if they cannot be granted immediately. In most cases it is already so, the only exception is OPEN path, <tt>mdt_object_open_lock()</tt> particularly. In case of similar deadlocks it can be switched to 'trybits' mode, by setting <tt>mdt.*.dom_lock=trylock</tt> parameter</p> <p>Mike,</p> <p>i_mutex is useless for truncate case now. IO path uses an unlocked version with range lock for long time, truncate vs IO protected by separate mutex.</p> <p>This <a href="https://jira.whamcloud.com/browse/LU-12017?focusedCommentId=266947&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-266947" class="external-link" rel="nofollow">comment</a> relates to <a href="https://jira.whamcloud.com/browse/LU-13467" title=" truncate deadlock with DoM files" class="issue-link" data-issue-key="LU-13467"><del>LU-13467</del></a>.</p> Related LU-13467 LU-11285 LU-13456 Development Rank 1|i00ca7: Rank (Obsolete) 9223372036854775807 Severity