https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16958 Lustre <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> PID: 350193 TASK: ffff9bd65af446c0 CPU: 0 COMMAND: <span class="code-quote">"getfattr"</span> #0 [ffff9bd63ffb7950] __schedule at ffffffffba5a232d /tmp/kernel/kernel/sched/core.c: 3109 #1 [ffff9bd63ffb79d8] schedule at ffffffffba5a2748 /tmp/kernel/./arch/x86/include/asm/preempt.h: 84 #2 [ffff9bd63ffb79e8] rwsem_down_write_slowpath at ffffffffba0f41a7 /tmp/kernel/./arch/x86/include/asm/current.h: 15 #3 [ffff9bd63ffb7a88] down_write at ffffffffba5a691a /tmp/kernel/./include/linux/err.h: 36 #4 [ffff9bd63ffb7ac0] vvp_inode_ops at ffffffffc116d57f [lustre] /home/lustre/linux-4.18.0-305.25.1.el8_4/./arch/x86/include/asm/current.h: 15 #5 [ffff9bd63ffb7ae0] cl_object_inode_ops at ffffffffc0454a50 [obdclass] /home/lustre/master-mine/lustre/obdclass/cl_object.c: 442 #6 [ffff9bd63ffb7b18] lov_conf_set at ffffffffc0aa36c4 [lov] /home/lustre/master-mine/lustre/lov/lov_object.c: 1465 #7 [ffff9bd63ffb7b88] cl_conf_set at ffffffffc04542d8 [obdclass] /home/lustre/master-mine/lustre/obdclass/cl_object.c: 299 #8 [ffff9bd63ffb7bb8] ll_layout_conf at ffffffffc111d110 [lustre] /home/lustre/master-mine/lustre/llite/file.c: 5995 #9 [ffff9bd63ffb7c28] ll_layout_refresh at ffffffffc111dad3 [lustre] /home/lustre/master-mine/libcfs/include/libcfs/libcfs_debug.h: 155 #10 [ffff9bd63ffb7cf0] vvp_io_init at ffffffffc116d019 [lustre] /home/lustre/master-mine/lustre/llite/vvp_io.c: 1870 #11 [ffff9bd63ffb7d20] __cl_io_init at ffffffffc045e66f [obdclass] /home/lustre/master-mine/lustre/obdclass/cl_io.c: 134 #12 [ffff9bd63ffb7d58] cl_glimpse_size0 at ffffffffc11642ca [lustre] /home/lustre/master-mine/lustre/llite/glimpse.c: 204 #13 [ffff9bd63ffb7da0] ll_getattr_dentry at ffffffffc111c65d [lustre] /home/lustre/master-mine/lustre/llite/llite_internal.h: 1677 #14 [ffff9bd63ffb7e50] vfs_statx at ffffffffba1d4be9 /tmp/kernel/fs/stat.c: 204 </pre> </div></div> <p>checking the stack on the process above inode was found at 0xffff9bd60367d350:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> crash&gt; p *(struct ll_inode_info *)(0xffff9bd60367d350-0x150) lli_inode_magic = 287116773, ... lli_inode_lock_owner = 0xffff9bd68f51d380 </pre> </div></div> <p>now check task 0xffff9bd68f51d380:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> crash&gt; p *(struct task_struct *)0xffff9bd68f51d380|more ... pid = 348428, ... PID: 348428 TASK: ffff9bd68f51d380 CPU: 1 COMMAND: <span class="code-quote">"lfs"</span> #0 [ffff9bd613c37968] __schedule at ffffffffba5a232d /tmp/kernel/kernel/sched/core.c: 3109 #1 [ffff9bd613c379f0] schedule at ffffffffba5a2748 /tmp/kernel/./arch/x86/include/asm/preempt.h: 84 #2 [ffff9bd613c37a00] schedule_preempt_disabled at ffffffffba5a2a6c /tmp/kernel/./arch/x86/include/asm/preempt.h: 79 #3 [ffff9bd613c37a08] __mutex_lock at ffffffffba5a3a40 /tmp/kernel/kernel/locking/mutex.c: 1038 #4 [ffff9bd613c37ac8] ll_layout_refresh at ffffffffc111d577 [lustre] /home/lustre/master-mine/lustre/llite/llite_internal.h: 1536 #5 [ffff9bd613c37b88] vvp_io_init at ffffffffc116d019 [lustre] /home/lustre/master-mine/lustre/llite/vvp_io.c: 1870 #6 [ffff9bd613c37bb8] __cl_io_init at ffffffffc045e66f [obdclass] /home/lustre/master-mine/lustre/obdclass/cl_io.c: 134 #7 [ffff9bd613c37bf0] ll_ioc_data_version at ffffffffc110c665 [lustre] /home/lustre/master-mine/lustre/llite/file.c: 3193 #8 [ffff9bd613c37c28] ll_migrate at ffffffffc111b244 [lustre] /home/lustre/master-mine/lustre/llite/file.c: 3227 #9 [ffff9bd613c37ca8] ll_dir_ioctl at ffffffffc1105563 [lustre] /home/lustre/master-mine/lustre/llite/dir.c: 2277 #10 [ffff9bd613c37e88] do_vfs_ioctl at ffffffffba1e3199 /tmp/kernel/fs/ioctl.c: 48 </pre> </div></div> <p>it seems this is an locking order issue:<br/> ll_migrate() takes inode lock, then lli_layout_mutex (in ll_layout_refresh()) while other ops (like getfattr) use the reversed order.</p> LU-16958

migrate vs regular ops deadlock

Bug Critical Resolved Fixed Zhenyu Xu Alex Zhuravlev Wed, 12 Jul 2023 13:16:16 +0000 Mon, 20 Nov 2023 14:23:10 +0000 Sat, 18 Nov 2023 21:54:26 +0000 Lustre 2.16.0 0 6 <p>"Zhenyu Xu &lt;bobijam@hotmail.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51641" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51641</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16958" title="migrate vs regular ops deadlock" class="issue-link" data-issue-key="LU-16958"><del>LU-16958</del></a> llite: migrate vs regular ops deadlock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8de4a374979a37d09a057cbcdfd9914775cfc59b</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51641/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51641/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16958" title="migrate vs regular ops deadlock" class="issue-link" data-issue-key="LU-16958"><del>LU-16958</del></a> llite: migrate vs regular ops deadlock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8f2c1592c3bbd0351ab3984a88a3eed7075690c8</p> <p>Landed for 2.16</p> <p>another deadlock found </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>T1: vvp_io_init() -&gt;ll_layout_refresh() &lt;= take lli_layout_mutex -&gt;ll_layout_intent() -&gt;ll_take_md_lock() &lt;= take the CR layout lock ref -&gt;ll_layout_conf() -&gt;vvp_prune() -&gt;vvp_inode_ops() &lt;= release lli_layout_mtex -&gt;vvp_inode_ops() &lt;= try to acquire lli_layout_mutex -&gt; racer wait here T2: -&gt;ll_file_write_iter() -&gt;vvp_io_init() -&gt;ll_layout_refresh() &lt;= take lli_layout_mutex -&gt;ll_layout_intent() &lt;= Request layout from MDT -&gt; racer wait ... T3: occure in PCC-RO attach, It can happen in normal case without PCC-RO. -&gt;pcc_readonly_attach() -&gt;ll_layout_intent_write() -&gt;ll_intent_lock() -&gt; on MDT, it will try to obtain EX layout lock to change layout. but the client T1 hold CR layout lock, and T2's lock request is in lock waiting list to wai for T3 finished, thus cause dealock... </pre> </div></div> <p>I thought deadlock due to this patch , but I reverted the essential part of this patch at <a href="https://review.whamcloud.com/52388" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/52388</a>, and the racer still hang at the server, looks more like <a href="https://jira.whamcloud.com/browse/LU-15491" title="rename deadlock" class="issue-link" data-issue-key="LU-15491">LU-15491</a></p> <p>Found another deadlock for parallel DIO:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> T1: writer Obtain DLM extent lock: L1=PW[0, EOF] T2: DIO reader: 50M data, iosize=64M, max_pages_per_rpc=1024 (4M) max_rpcs_in_flight=8 ll_direct_IO_impl() use all available RPC slots: number of read RPC in flight is 9 on the server side: -&gt;tgt_brw_read() -&gt;tgt_brw_lock() # server side locking -&gt; Try to cancel the conflict locks on client: L1=PW[0, EOF] T3: reader take DLM lock ref on L1=PW[0, EOF] Read-ahead pages (prepare pages); wait <span class="code-keyword">for</span> RPC slots to send the read RPCs to OST deadlock: T2-&gt;T3: T2 is waiting <span class="code-keyword">for</span> T3 to release DLM extent lock L1; T3-&gt;T2: T3 is waiting <span class="code-keyword">for</span> T2 finished to free RPC slots...</pre> </div></div> <p>The possible solution is that when found all RPC slots are used by srvlock DIO, and there are urgent I/O, force to send the I/O RPC to OST? </p> <p>Another patch was pushed under this ticket.</p> <blockquote> <p>I thought deadlock due to this patch , but I reverted the essential part of this patch at <a href="https://review.whamcloud.com/52388" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/52388</a>, and the racer still hang at the server, looks more like <a href="https://jira.whamcloud.com/browse/LU-15491" title="rename deadlock" class="issue-link" data-issue-key="LU-15491">LU-15491</a></p></blockquote> <p>There could definitely be multiple different issues affecting racer testing, so that doesn't mean the above patch is not fixing a problem.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52388/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52388/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16958" title="migrate vs regular ops deadlock" class="issue-link" data-issue-key="LU-16958"><del>LU-16958</del></a> llite: migrate deadlock on not responding lock cancel<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 37646c74bf884c535149d530af840d728814792b</p> <p>Landed for 2.16</p> Related LU-16637 Development Rank 1|i03qcf: Rank (Obsolete) 9223372036854775807 Severity