https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4977 Lustre <p>I can occasionally see this issue in machines with less memory. The deadlock has the following call stack:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>dd D 0000000000000000 0 2158 1 0x00000004 ffff88010ecc10f8 0000000000000086 ffff8801ffffffff 0000000042a8c635 ffff88010ecc1078 ffff88009ccb68a0 0000000000047e6a ffffffffaca103a3 ffff8800d7bd5058 ffff88010ecc1fd8 000000000000fb88 ffff8800d7bd5058 Call Trace: [&lt;ffffffff810a2431&gt;] ? ktime_get_ts+0xb1/0xf0 [&lt;ffffffff81119e10&gt;] ? sync_page+0x0/0x50 [&lt;ffffffff8150ed93&gt;] io_schedule+0x73/0xc0 [&lt;ffffffff81119e4d&gt;] sync_page+0x3d/0x50 [&lt;ffffffff8150f5fa&gt;] __wait_on_bit_lock+0x5a/0xc0 [&lt;ffffffff81119de7&gt;] __lock_page+0x67/0x70 [&lt;ffffffff81096de0&gt;] ? wake_bit_function+0x0/0x50 [&lt;ffffffffa0f60101&gt;] vvp_page_make_ready+0x271/0x280 [lustre] [&lt;ffffffffa0542999&gt;] cl_page_make_ready+0x89/0x370 [obdclass] [&lt;ffffffffa03b45a1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa0a323b7&gt;] osc_extent_make_ready+0x3b7/0xe50 [osc] [&lt;ffffffff81055ad3&gt;] ? __wake_up+0x53/0x70 [&lt;ffffffffa0a36af6&gt;] osc_io_unplug0+0x1736/0x2130 [osc] [&lt;ffffffff8103c7d8&gt;] ? pvclock_clocksource_read+0x58/0xd0 [&lt;ffffffffa03b45a1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa0a39681&gt;] osc_io_unplug+0x11/0x20 [osc] [&lt;ffffffffa0a3bc86&gt;] osc_cache_writeback_range+0xdb6/0x1290 [osc] [&lt;ffffffffa03b9d47&gt;] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs] [&lt;ffffffffa03b9d47&gt;] ? cfs_hash_bd_lookup_intent+0x37/0x130 [libcfs] [&lt;ffffffffa03b9362&gt;] ? cfs_hash_bd_add_locked+0x62/0x90 [libcfs] [&lt;ffffffffa054a45d&gt;] ? cl_io_sub_init+0x5d/0xc0 [obdclass] [&lt;ffffffffa0a29fd0&gt;] osc_io_fsync_start+0x90/0x360 [osc] [&lt;ffffffffa0547640&gt;] ? cl_io_start+0x0/0x140 [obdclass] [&lt;ffffffffa05476aa&gt;] cl_io_start+0x6a/0x140 [obdclass] [&lt;ffffffffa0a8f18e&gt;] lov_io_call+0x8e/0x130 [lov] [&lt;ffffffffa0a9324c&gt;] lov_io_start+0x10c/0x180 [lov] [&lt;ffffffffa05476aa&gt;] cl_io_start+0x6a/0x140 [obdclass] [&lt;ffffffffa054aea4&gt;] cl_io_loop+0xb4/0x1b0 [obdclass] [&lt;ffffffffa0f02acb&gt;] cl_sync_file_range+0x31b/0x500 [lustre] [&lt;ffffffffa0f2fe7c&gt;] ll_writepages+0x9c/0x220 [lustre] [&lt;ffffffff8112e1b1&gt;] do_writepages+0x21/0x40 [&lt;ffffffff811aca9d&gt;] writeback_single_inode+0xdd/0x290 [&lt;ffffffff811aceae&gt;] writeback_sb_inodes+0xce/0x180 [&lt;ffffffff811ad00b&gt;] writeback_inodes_wb+0xab/0x1b0 [&lt;ffffffff8112d60d&gt;] balance_dirty_pages+0x23d/0x4d0 [&lt;ffffffffa0541768&gt;] ? cl_page_invoid+0x68/0x160 [obdclass] [&lt;ffffffff8112d904&gt;] balance_dirty_pages_ratelimited_nr+0x64/0x70 [&lt;ffffffff8111a86a&gt;] generic_file_buffered_write+0x1da/0x2e0 [&lt;ffffffff81075887&gt;] ? current_fs_time+0x27/0x30 [&lt;ffffffff8111c210&gt;] __generic_file_aio_write+0x260/0x490 [&lt;ffffffffa0a93d9c&gt;] ? lov_lock_enqueue+0xbc/0x170 [lov] [&lt;ffffffff8111c4c8&gt;] generic_file_aio_write+0x88/0x100 [&lt;ffffffffa0f634a2&gt;] vvp_io_write_start+0x102/0x3f0 [lustre] [&lt;ffffffffa05476aa&gt;] cl_io_start+0x6a/0x140 [obdclass] [&lt;ffffffffa054aea4&gt;] cl_io_loop+0xb4/0x1b0 [obdclass] [&lt;ffffffffa0f00297&gt;] ll_file_io_generic+0x407/0x8d0 [lustre] [&lt;ffffffffa05406c9&gt;] ? cl_env_get+0x29/0x350 [obdclass] [&lt;ffffffffa0f00fa3&gt;] ll_file_aio_write+0x133/0x2b0 [lustre] [&lt;ffffffffa0f01279&gt;] ll_file_write+0x159/0x290 [lustre] [&lt;ffffffff81181398&gt;] vfs_write+0xb8/0x1a0 [&lt;ffffffff81181c91&gt;] sys_write+0x51/0x90 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>In balance_dirty_pages(), it tries to write back some dirty pages between after write_end(). However, ll_write_end() can hold the page to add it into commit queue and causes the problem.</p> <p>We can fix the problem by releasing the page in ll_write_end() if the page is already dirty.</p> <p>Patch is coming.</p> LU-4977

Deadlock in balance_dirty_pages()

Bug Blocker Resolved Fixed Jinshan Xiong Jinshan Xiong Tue, 29 Apr 2014 15:41:08 +0000 Thu, 15 May 2014 18:23:54 +0000 Wed, 14 May 2014 15:16:39 +0000 Lustre 2.6.0 Lustre 2.6.0 0 5 <p>patch is located at <a href="http://review.whamcloud.com/10149" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10149</a></p> <p>Landed for 2.6</p> <p>The patch for this issue removes the change made to fix <a href="https://jira.whamcloud.com/browse/LU-4873" title="Lustre client hangs in vvp_page_make_ready" class="issue-link" data-issue-key="LU-4873"><del>LU-4873</del></a>.</p> <p>Can you explain why this change means it's safe to do:<br/> vmpage = grab_cache_page_nowait(mapping, index);</p> <p>for partial page writes?</p> <p>With <a href="https://jira.whamcloud.com/browse/LU-4977" title="Deadlock in balance_dirty_pages()" class="issue-link" data-issue-key="LU-4977"><del>LU-4977</del></a>, we see many tasks failing to exit, stuck in various pieces of the memory management code. While we no longer see <a href="https://jira.whamcloud.com/browse/LU-4873" title="Lustre client hangs in vvp_page_make_ready" class="issue-link" data-issue-key="LU-4873"><del>LU-4873</del></a>, we see many tasks stuck in other locations:</p> <p>Here's a sampling... We saw a small number of this stack trace:<br/> 2014-05-14T23:40:01.626699-05:00 c3-0c0s5n1 &lt;node_health:5.1&gt; APID:13971031 (Application_Exited_Check) STACK: call_rwsem_down_write_failed+0x13/0x20; ll_setattr_raw+0x19a/0x1030 <span class="error">&#91;lustre&#93;</span>; ll_setattr+0x57/0xf0 <span class="error">&#91;lustre&#93;</span>; notify_change+0x120/0x310; file_remove_suid+0x81/0xb0; __generic_file_aio_write+0x209/0x450; generic_file_aio_write+0x59/0xc0; vvp_io_write_start+0xe0/0x3c0 <span class="error">&#91;lustre&#93;</span>; cl_io_start+0x72/0x140 <span class="error">&#91;obdclass&#93;</span>; cl_io_loop+0xac/0x1a0 <span class="error">&#91;obdclass&#93;</span>; ll_file_io_generic+0x452/0x700 <span class="error">&#91;lustre&#93;</span>; ll_file_aio_write+0x23e/0x290 <span class="error">&#91;lustre&#93;</span>; ll_file_write+0x1f2/0x280 <span class="error">&#91;lustre&#93;</span>; vfs_write+0xcb/0x180; sys_write+0x55/0x90; system_call_fastpath+0x16/0x1b; 0x2aaaad4f6730; 0xffffffffffffffff;</p> <p>A similar number with this stack trace:<br/> 2014-05-14T23:24:19.890744-05:00 c6-0c2s2n0 &lt;node_health:5.1&gt; APID:13969099 (Application_Exited_Check) STACK: call_rwsem_down_write_failed+0x13/0x20; do_coredump+0x14e/0xbe0; get_signal_to_deliver+0x23b/0x480; do_notify_resume+0xe0/0x7f0; retint_signal+0x46/0x83; 0x2000feb6; 0xffffffffffffffff;</p> <p>A lot with this stack trace:<br/> 2014-05-14T23:24:20.280631-05:00 c6-0c1s4n0 &lt;node_health:5.1&gt; APID:13969099 (Application_Exited_Check) STACK: call_rwsem_down_write_failed+0x13/0x20; kgni_mm_fini+0x82/0x3e0 <span class="error">&#91;kgni_gem&#93;</span>; kgni_nic_destroy+0x37/0x70 <span class="error">&#91;kgni_gem&#93;</span>; kgni_close+0x164/0x230 <span class="error">&#91;kgni_gem&#93;</span>; fput+0xda/0x200; filp_close+0x63/0x90; put_files_struct+0x84/0xe0; exit_files+0x53/0x70; do_exit+0x1ec/0x980; do_group_exit+0x48/0xc0; get_signal_to_deliver+0x243/0x480; do_notify_resume+0xe0/0x7f0; int_signal+0x12/0x17; 0x2001c111; 0xffffffffffffffff;</p> <p>And the largest number were stuck here:<br/> 2014-05-14T22:44:36.488369-05:00 c4-0c2s4n0 &lt;node_health:5.1&gt; APID:13969066 (Application_Exited_Check) STACK: lu_cache_shrink+0x56/0x2a0 <span class="error">&#91;obdclass&#93;</span>; shrink_slab+0xae/0x2c0; do_try_to_free_pages+0x593/0x6e0; try_to_free_pages+0x116/0x3c0; __alloc_pages_nodemask+0x58c/0x950; alloc_pages_current+0xbe/0x130; alloc_buddy_huge_page+0xf4/0x130; hugetlb_acct_memory+0xf0/0x330; hugetlb_reserve_pages+0x10c/0x1e0; hugetlbfs_file_mmap+0xec/0x150; mmap_region+0x4d2/0x6b0; do_mmap_pgoff+0x367/0x390; sys_mmap_pgoff+0x1fe/0x220; sys_mmap+0x29/0x30; system_call_fastpath+0x16/0x1b; 0x201b5599; 0xffffffffffffffff;</p> <p>Hi Patrick,</p> <p>I don't think the problem you've seen is related to the patch here. Is this the first time you see this issue? Please file a separate ticket.</p> <p>Jinshan</p> <p>Ah, sorry, the setattr_raw issue is a known one. For some reason that fix wasn't on this tree.</p> <p>The others are, as far as I can tell, new instances. I'm going to test again with the latest master...<br/> The processes stuck in lu_cache_shrink eventually completed, but a significant amount of time was spent there first.</p> Related LU-4873 Development Rank 1|hzwle7: Rank (Obsolete) 13790 Severity