https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8791 Lustre <p>This problem occurred in soaked test.</p> <p>There are two suspicious call trace which leads me to suspect there exists a deadlock in the code:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LNet: Service thread pid 30124 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: LNet: Skipped 3 previous similar messages Pid: 30124, comm: ll_ost_io00_116 Call Trace: [&lt;ffffffff81178260&gt;] ? cache_alloc_refill+0x1c0/0x240 [&lt;ffffffff81291489&gt;] ? cpumask_next_and+0x29/0x50 [&lt;ffffffff8153cc55&gt;] rwsem_down_failed_common+0x95/0x1d0 [&lt;ffffffff8153cde6&gt;] rwsem_down_read_failed+0x26/0x30 [&lt;ffffffff8129e8f4&gt;] call_rwsem_down_read_failed+0x14/0x30 [&lt;ffffffff8153c2e4&gt;] ? down_read+0x24/0x30 [&lt;ffffffffa0f13917&gt;] osd_attr_get+0x77/0x2e0 [osd_zfs] [&lt;ffffffffa1022301&gt;] ofd_attr_handle_ugid+0x161/0x430 [ofd] [&lt;ffffffffa1027ebd&gt;] ofd_write_attr_set+0xad/0x9e0 [ofd] [&lt;ffffffffa102933f&gt;] ofd_commitrw_write+0x23f/0x10a0 [ofd] [&lt;ffffffffa102df1d&gt;] ? ofd_fmd_find_nolock+0xad/0xd0 [ofd] [&lt;ffffffffa102a75f&gt;] ofd_commitrw+0x5bf/0xb10 [ofd] [&lt;ffffffff81149461&gt;] ? kzfree+0x31/0x40 [&lt;ffffffffa079d5c1&gt;] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] [&lt;ffffffffa0a17ad4&gt;] obd_commitrw+0x114/0x380 [ptlrpc] [&lt;ffffffffa0a20870&gt;] tgt_brw_write+0xc70/0x1530 [ptlrpc] [&lt;ffffffff8105e9b6&gt;] ? enqueue_task+0x66/0x80 [&lt;ffffffff8105ab8d&gt;] ? check_preempt_curr+0x6d/0x90 [&lt;ffffffff810674be&gt;] ? try_to_wake_up+0x24e/0x3e0 [&lt;ffffffffa0974e20&gt;] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [&lt;ffffffffa0a1f0ac&gt;] tgt_request_handle+0x8ec/0x1440 [ptlrpc] [&lt;ffffffffa09cb7b1&gt;] ptlrpc_main+0xd31/0x1800 [ptlrpc] [&lt;ffffffff81539b0e&gt;] ? thread_return+0x4e/0x7d0 [&lt;ffffffffa09caa80&gt;] ? ptlrpc_main+0x0/0x1800 [ptlrpc] [&lt;ffffffff810a138e&gt;] kthread+0x9e/0xc0 [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 [&lt;ffffffff810a12f0&gt;] ? kthread+0x0/0xc0 [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>And then another one:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Call Trace: [&lt;ffffffffa015208c&gt;] ? mlx4_ib_post_send+0x4fc/0x1370 [mlx4_ib] [&lt;ffffffff8153cc55&gt;] rwsem_down_failed_common+0x95/0x1d0 [&lt;ffffffff8153cde6&gt;] rwsem_down_read_failed+0x26/0x30 [&lt;ffffffff8129e8f4&gt;] call_rwsem_down_read_failed+0x14/0x30 [&lt;ffffffff8153c2e4&gt;] ? down_read+0x24/0x30 [&lt;ffffffffa037e9b0&gt;] dmu_buf_hold_array_by_dnode+0x50/0x4a0 [zfs] [&lt;ffffffffa037f85b&gt;] dmu_buf_hold_array_by_bonus+0x6b/0x90 [zfs] [&lt;ffffffffa0f149af&gt;] osd_bufs_get+0x4bf/0xb50 [osd_zfs] [&lt;ffffffffa07c0af6&gt;] ? lu_object_find+0x16/0x20 [obdclass] [&lt;ffffffffa102893f&gt;] ofd_preprw_read+0x14f/0x910 [ofd] [&lt;ffffffffa102b008&gt;] ofd_preprw+0x358/0x14f0 [ofd] [&lt;ffffffffa0a1774f&gt;] obd_preprw+0x10f/0x380 [ptlrpc] [&lt;ffffffffa0a217c5&gt;] tgt_brw_read+0x695/0x11d0 [ptlrpc] [&lt;ffffffffa09e4194&gt;] ? sptlrpc_svc_alloc_rs+0x74/0x360 [ptlrpc] [&lt;ffffffffa09bab8b&gt;] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] [&lt;ffffffffa09bacc6&gt;] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] [&lt;ffffffffa067ec8a&gt;] ? lc_watchdog_touch+0x7a/0x190 [libcfs] [&lt;ffffffffa0a1f0ac&gt;] tgt_request_handle+0x8ec/0x1440 [ptlrpc] [&lt;ffffffffa09cb7b1&gt;] ptlrpc_main+0xd31/0x1800 [ptlrpc] [&lt;ffffffff8106ee50&gt;] ? pick_next_task_fair+0xd0/0x130 [&lt;ffffffff81539896&gt;] ? schedule+0x176/0x3a0 [&lt;ffffffffa09caa80&gt;] ? ptlrpc_main+0x0/0x1800 [ptlrpc] [&lt;ffffffff810a138e&gt;] kthread+0x9e/0xc0 [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 [&lt;ffffffff810a12f0&gt;] ? kthread+0x0/0xc0 [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 Pid: 29297, comm: ll_ost_io00_003 </pre> </div></div> <p>Two more:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ll_ost_io00_0 D 0000000000000003 0 29439 2 0x00000080 ffff8803d6a1f6f0 0000000000000046 0000000000000000 ffff8803d6a1f6b4 0000000000000000 ffff88043fe82400 00006a78551f8e00 ffff8800387f5a00 0000000000002c62 0000000106f6922c ffff8803d6a1bad8 ffff8803d6a1ffd8 Call Trace: [&lt;ffffffffa0314e5d&gt;] cv_wait_common+0x11d/0x130 [spl] [&lt;ffffffff810a1820&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffffa0314ec5&gt;] __cv_wait+0x15/0x20 [spl] [&lt;ffffffffa03757fb&gt;] dbuf_read+0x3bb/0x850 [zfs] [&lt;ffffffff8153afdb&gt;] ? mutex_lock+0x2b/0x50 [&lt;ffffffffa0372a13&gt;] ? dbuf_find+0x193/0x1a0 [zfs] [&lt;ffffffffa0377c78&gt;] dmu_buf_will_dirty+0x58/0xc0 [zfs] [&lt;ffffffffa0377d37&gt;] dbuf_new_size+0x57/0x1c0 [zfs] [&lt;ffffffffa039259c&gt;] dnode_set_blksz+0x30c/0x330 [zfs] [&lt;ffffffffa037cbb8&gt;] dmu_object_set_blocksize+0x48/0x70 [zfs] [&lt;ffffffffa0f15e00&gt;] osd_write_commit+0x770/0x970 [osd_zfs] [&lt;ffffffffa102980b&gt;] ofd_commitrw_write+0x70b/0x10a0 [ofd] [&lt;ffffffffa102a75f&gt;] ofd_commitrw+0x5bf/0xb10 [ofd] [&lt;ffffffff81149461&gt;] ? kzfree+0x31/0x40 [&lt;ffffffffa079d5c1&gt;] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] [&lt;ffffffffa0a17ad4&gt;] obd_commitrw+0x114/0x380 [ptlrpc] [&lt;ffffffffa0a20870&gt;] tgt_brw_write+0xc70/0x1530 [ptlrpc] [&lt;ffffffff8105e9b6&gt;] ? enqueue_task+0x66/0x80 [&lt;ffffffff8105ab8d&gt;] ? check_preempt_curr+0x6d/0x90 [&lt;ffffffff810674be&gt;] ? try_to_wake_up+0x24e/0x3e0 [&lt;ffffffffa0974e20&gt;] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [&lt;ffffffffa0a1f0ac&gt;] tgt_request_handle+0x8ec/0x1440 [ptlrpc] [&lt;ffffffffa09cb7b1&gt;] ptlrpc_main+0xd31/0x1800 [ptlrpc] [&lt;ffffffff8106ee50&gt;] ? pick_next_task_fair+0xd0/0x130 [&lt;ffffffff81539896&gt;] ? schedule+0x176/0x3a0 [&lt;ffffffffa09caa80&gt;] ? ptlrpc_main+0x0/0x1800 [ptlrpc] [&lt;ffffffff810a138e&gt;] kthread+0x9e/0xc0 [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 [&lt;ffffffff810a12f0&gt;] ? kthread+0x0/0xc0 [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ll_ost_io00_1 D 0000000000000005 0 29763 2 0x00000080 ffff88021676b768 0000000000000046 0000000000000000 ffff88021676b72c ffff880400000000 ffff88043fe82800 00006a78551f88c2 ffff8800387f5a00 0000000000002c58 0000000106f6922c ffff8803d27b9ad8 ffff88021676bfd8 Call Trace: [&lt;ffffffffa030db8e&gt;] ? spl_kmem_zalloc+0x8e/0x190 [spl] [&lt;ffffffff8153cc55&gt;] rwsem_down_failed_common+0x95/0x1d0 [&lt;ffffffff8153afce&gt;] ? mutex_lock+0x1e/0x50 [&lt;ffffffff8153cde6&gt;] rwsem_down_read_failed+0x26/0x30 [&lt;ffffffff8129e8f4&gt;] call_rwsem_down_read_failed+0x14/0x30 [&lt;ffffffff8153c2e4&gt;] ? down_read+0x24/0x30 [&lt;ffffffffa0376bb1&gt;] dbuf_dirty+0x381/0x980 [zfs] [&lt;ffffffffa037728a&gt;] dbuf_assign_arcbuf+0xda/0x2b0 [zfs] [&lt;ffffffffa037f679&gt;] dmu_assign_arcbuf+0x149/0x1f0 [zfs] [&lt;ffffffffa0f15ac1&gt;] osd_write_commit+0x431/0x970 [osd_zfs] [&lt;ffffffffa102980b&gt;] ofd_commitrw_write+0x70b/0x10a0 [ofd] [&lt;ffffffffa102a75f&gt;] ofd_commitrw+0x5bf/0xb10 [ofd] [&lt;ffffffff81149461&gt;] ? kzfree+0x31/0x40 [&lt;ffffffffa079d5c1&gt;] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] [&lt;ffffffffa0a17ad4&gt;] obd_commitrw+0x114/0x380 [ptlrpc] [&lt;ffffffffa0a20870&gt;] tgt_brw_write+0xc70/0x1530 [ptlrpc] [&lt;ffffffff8105e9b6&gt;] ? enqueue_task+0x66/0x80 [&lt;ffffffff8105ab8d&gt;] ? check_preempt_curr+0x6d/0x90 [&lt;ffffffff810674be&gt;] ? try_to_wake_up+0x24e/0x3e0 [&lt;ffffffffa0974e20&gt;] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [&lt;ffffffffa0a1f0ac&gt;] tgt_request_handle+0x8ec/0x1440 [ptlrpc] [&lt;ffffffffa09cb7b1&gt;] ptlrpc_main+0xd31/0x1800 [ptlrpc] [&lt;ffffffff81539b0e&gt;] ? thread_return+0x4e/0x7d0 [&lt;ffffffffa09caa80&gt;] ? ptlrpc_main+0x0/0x1800 [ptlrpc] [&lt;ffffffff810a138e&gt;] kthread+0x9e/0xc0 [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 [&lt;ffffffff810a12f0&gt;] ? kthread+0x0/0xc0 [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Due to this issue, other service threads can't move forward because this tx can never complete.</p> <p>The node died when I was trying to collect more information. However, these callback trace are enough to start an initial analysis. </p> LU-8791

Suspicious deadlock on OST stack

Bug Critical Resolved Fixed Jinshan Xiong Jinshan Xiong soak Wed, 2 Nov 2016 19:16:01 +0000 Mon, 7 Nov 2016 20:15:23 +0000 Mon, 7 Nov 2016 20:15:23 +0000 Lustre 2.9.0 Lustre 2.9.0 0 5 <p>At the time of the hang discovery, i dumped debug logs on lola-8 (MGS) lola-5(OSS) and lola-10 (MDS doing recovery) <br/> I then set debug = -1, waited five minutes and re-dumped the OSS and MDS logs. Attached.</p> <p>Lustre version lustre: 2.8.59_62_g165c308 <br/> commit 165c3082079d48f70b6e340ad23514e0e7524978</p> <p>Sequence of events:<br/> Noticed excessive reconnection attempts from lola-10 to lola-5, <br/> Nov 1 22:03:32 lola-5 kernel: Lustre: soaked-OST000f: Export ffff880310f5bc00 already connecting from 192.168.1.110@o2ib10<br/> Nov 1 22:04:22 lola-5 kernel: Lustre: soaked-OST000f: Export ffff880310f5bc00 already connecting from 192.168.1.110@o2ib10<br/> Nov 1 22:05:12 lola-5 kernel: Lustre: soaked-OST000f: Export ffff880310f5bc00 already connecting from 192.168.1.110@o2ib10<br/> Nov 1 22:06:02 lola-5 kernel: Lustre: soaked-OST000f: Export ffff880310f5bc00 already connecting from 192.168.1.110@o2ib10<br/> Nov 1 22:06:52 lola-5 kernel: Lustre: soaked-OST000f: Export ffff880310f5bc00 already connecting from 192.168.1.110@o2ib10<br/> On MDT:<br/> Nov 1 17:02:28 lola-10 kernel: LustreError: 11-0: soaked-MDT0000-lwp-MDT0002: operation mds_connect to node 192.168.1.108@o2ib10 failed: rc = -114<br/> Nov 1 17:03:18 lola-10 kernel: LustreError: 11-0: soaked-MDT0000-lwp-MDT0002: operation mds_connect to node 192.168.1.108@o2ib10 failed: rc = -114<br/> Nov 1 17:04:08 lola-10 kernel: LustreError: 11-0: soaked-MDT0000-lwp-MDT0002: operation mds_connect to node 192.168.1.108@o2ib10 failed: rc = -114<br/> Nov 1 17:04:58 lola-10 kernel: LustreError: 11-0: soaked-MDT0000-lwp-MDT0002: operation mds_connect to node 192.168.1.108@o2ib10 failed: rc = -114<br/> Examination revealed lola-5 in a hung state apparently due to deadlock above.</p> <p>It turned out that the root cause of the deadlock is due to block size change and object write at the same time.</p> <p>Thread 1:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>osd_grow_blocksize() with osd_object::oo_guard held -&gt; dmu_object_set_blocksize() -&gt; dnode_set_blksz(), with dnode_t::dn_struct_rwlock write lock held -&gt; dbuf_new_size() -&gt; dmu_buf_will_dirty() -&gt; dbuf_read() -&gt; wait for the dbuf state to change to DB_CACHED </pre> </div></div> <p>Thread 2:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>osd_write_commit() -&gt; dmu_assign_arcbuf() -&gt; dbuf_assign_arcbuf(), set dbuf state to DB_FILL -&gt; dbuf_dirty() -&gt; try to hold the read lock of dnode_t::dn_struct_rwlock </pre> </div></div> <p>In summary, thread 2 sets dbuf to DB_FILL and then try to grab dn_struct_rwlock to complete the assign; while thread 1 holds dn_struct_rwlock and waits for the dbuf state to change to DB_CACHED. Deadlocked.</p> <p>In order to fix this problem, we can prevent further writes from happening to the same object while it's changing the block size. I would propose to hold read lock of osd_object::oo_guard in osd_write_commit().</p> <p>Patch is coming soon.</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/23550" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23550</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8791" title="Suspicious deadlock on OST stack" class="issue-link" data-issue-key="LU-8791"><del>LU-8791</del></a> osd-zfs: hold oo_guard read lock for object write<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b8811a0a02c801cc85ee07cb7e68b8514a14b031</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/23550/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23550/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8791" title="Suspicious deadlock on OST stack" class="issue-link" data-issue-key="LU-8791"><del>LU-8791</del></a> osd-zfs: hold oo_guard read lock for object write<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: dde6d59ad5c7bb915900bbbe471104ab3c4d3629</p> <p>Landed for 2.9</p> Development Rank 1|hzyu7r: Rank (Obsolete) 9223372036854775807 Severity