https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4873 Lustre <p>Application hangs in vvp_page_make_ready and dumps call traces.<br/> We applied patch of <a href="https://jira.whamcloud.com/browse/LU-4540" title="Test failure sanity-quota test_8: dbench hung in vvp_page_assume" class="issue-link" data-issue-key="LU-4540"><del>LU-4540</del></a>, but still hit this issue.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Apr 8 01:27:04 r21 kernel: Call Trace: Apr 8 01:27:04 r21 kernel: [&lt;ffffffff81055f96&gt;] ? enqueue_task+0x66/0x80 Apr 8 01:27:04 r21 kernel: [&lt;ffffffff810a2431&gt;] ? ktime_get_ts+0xb1/0xf0 Apr 8 01:27:04 r21 kernel: [&lt;ffffffff81119e10&gt;] ? sync_page+0x0/0x50 Apr 8 01:27:04 r21 kernel: [&lt;ffffffff8150e953&gt;] io_schedule+0x73/0xc0 Apr 8 01:27:04 r21 kernel: [&lt;ffffffff81119e4d&gt;] sync_page+0x3d/0x50 Apr 8 01:27:04 r21 kernel: [&lt;ffffffff8150f1ba&gt;] __wait_on_bit_lock+0x5a/0xc0 Apr 8 01:27:04 r21 kernel: [&lt;ffffffff81119de7&gt;] __lock_page+0x67/0x70 Apr 8 01:27:04 r21 kernel: [&lt;ffffffff81096de0&gt;] ? wake_bit_function+0x0/0x50 Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa1055cc1&gt;] vvp_page_make_ready+0x271/0x280 [lustre] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0e89149&gt;] cl_page_make_ready+0x89/0x370 [obdclass] Apr 8 01:27:04 r21 kernel: [&lt;ffffffff81055ad3&gt;] ? __wake_up+0x53/0x70 Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0ca069e&gt;] osc_extent_make_ready+0x34e/0xc00 [osc] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0c9c113&gt;] ? on_list+0x43/0x50 [osc] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0c9c1cd&gt;] ? __osc_list_maint+0xad/0x150 [osc] Apr 8 01:27:04 r21 kernel: [&lt;ffffffff81055ad3&gt;] ? __wake_up+0x53/0x70 Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0ca4826&gt;] osc_io_unplug0+0x15e6/0x1f00 [osc] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0c9e1ad&gt;] ? osc_extent_tree_dump0+0x28d/0xb20 [osc] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0ca6ec1&gt;] osc_io_unplug+0x11/0x20 [osc] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0ca709d&gt;] osc_queue_sync_pages+0x1cd/0x350 [osc] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0c985d7&gt;] osc_io_submit+0x1c7/0x4b0 [osc] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0e92b3e&gt;] cl_io_submit_rw+0x6e/0x160 [obdclass] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0d2c360&gt;] lov_io_submit+0x2d0/0x4b0 [lov] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0d0f874&gt;] ? lov_merge_lvb_kms+0x124/0x530 [lov] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0e92b3e&gt;] cl_io_submit_rw+0x6e/0x160 [obdclass] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0e95ae7&gt;] cl_io_submit_sync+0x87/0x1b0 [obdclass] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa1026adc&gt;] ll_page_sync_io+0x5c/0x110 [lustre] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0d2009e&gt;] ? lov_attr_get+0x1e/0x60 [lov] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa1045561&gt;] ll_write_begin+0x5c1/0x760 [lustre] Apr 8 01:27:04 r21 kernel: [&lt;ffffffff8111a7b3&gt;] generic_file_buffered_write+0x123/0x2e0 Apr 8 01:27:04 r21 kernel: [&lt;ffffffff81075887&gt;] ? current_fs_time+0x27/0x30 Apr 8 01:27:04 r21 kernel: [&lt;ffffffff8111c210&gt;] __generic_file_aio_write+0x260/0x490 Apr 8 01:27:04 r21 kernel: [&lt;ffffffff8111c4c8&gt;] generic_file_aio_write+0x88/0x100 Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa105907b&gt;] vvp_io_write_start+0xdb/0x3d0 [lustre] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0e92dca&gt;] cl_io_start+0x6a/0x140 [obdclass] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0e96f54&gt;] cl_io_loop+0xb4/0x1b0 [obdclass] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0ff9f36&gt;] ll_file_io_generic+0x2b6/0x710 [lustre] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0e86d09&gt;] ? cl_env_get+0x29/0x350 [obdclass] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0ffa4d2&gt;] ll_file_aio_write+0x142/0x2c0 [lustre] Apr 8 01:27:04 r21 kernel: [&lt;ffffffffa0ffad6c&gt;] ll_file_write+0x16c/0x2a0 [lustre] Apr 8 01:27:04 r21 kernel: [&lt;ffffffff81181398&gt;] vfs_write+0xb8/0x1a0 Apr 8 01:27:04 r21 kernel: [&lt;ffffffff81181c91&gt;] sys_write+0x51/0x90 Apr 8 01:27:04 r21 kernel: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b Apr 8 01:29:04 r21 kernel: INFO: task pre:46376 blocked for more than 120 seconds. Apr 8 01:29:04 r21 kernel: "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 8 01:29:04 r21 kernel: pre D 000000000000000c 0 46376 46345 0x00000004 Apr 8 01:29:04 r21 kernel: ffff8807f19f7388 0000000000000086 ffff8807f19f72f8 ffff880b97c9b500 Apr 8 01:29:04 r21 kernel: ffff88085c456700 0000000000000003 ffff8807f19f7328 ffffffff81055f96 Apr 8 01:29:04 r21 kernel: ffff880560820638 ffff8807f19f7fd8 000000000000fb88 ffff880560820638 Apr 8 01:29:04 r21 kernel: Call Trace: </pre> </div></div> LU-4873

Lustre client hangs in vvp_page_make_ready

Bug Blocker Resolved Fixed Niu Yawei Shuichi Ihara Tue, 8 Apr 2014 21:02:57 +0000 Tue, 29 Apr 2014 17:25:45 +0000 Thu, 17 Apr 2014 16:06:24 +0000 Lustre 2.6.0 Lustre 2.6.0 0 5 <p>Niu</p> <p>Could you please comment on this issues?</p> <p>Thanks</p> <p>Peter</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* To avoid deadlock, <span class="code-keyword">try</span> to lock page first. */</span> vmpage = grab_cache_page_nowait(mapping, index); <span class="code-keyword">if</span> (unlikely(vmpage == NULL || PageDirty(vmpage) || PageWriteback(vmpage))) { struct ccc_io *cio = ccc_env_io(env); struct cl_page_list *plist = &amp;cio-&gt;u.write.cui_queue; /* <span class="code-keyword">if</span> the page is already in dirty cache, we have to commit * the pages right now; otherwise, it may cause deadlock * because it holds page lock of a dirty page and request <span class="code-keyword">for</span> * more grants. It's okay <span class="code-keyword">for</span> the dirty page to be the first * one in commit page list, though. */ <span class="code-keyword">if</span> (vmpage != NULL &amp;&amp; plist-&gt;pl_nr &gt; 0) { unlock_page(vmpage); page_cache_release(vmpage); vmpage = NULL; } <span class="code-comment">/* commit pages and then wait <span class="code-keyword">for</span> page lock */</span> result = vvp_io_write_commit(env, io); <span class="code-keyword">if</span> (result &lt; 0) GOTO(out, result); </pre> </div></div> <p>Look into the ll_write_begin(), if the page is in writeback and cui_queue-&gt;pl_nr == 0, it could call vvp_io_write_commit() with page locked? Xiong, is it possible? please take a look, thanks.</p> <p>If plist-&gt;nr is zero, then there is no page in pending, so vvp_io_write_commit() is a null op essentially.</p> <p>the root cause of this issue is as follows:<br/> 1. pages with indices 0,...,254 have already been dirtied and stayed in OSC cache;<br/> 2. thread T tries to write pages from 254 to 255, and the last page(page 255) is a partial page;<br/> 3. T will have to issue a read RPC due to partial page write, with page 254 locked;<br/> 4. T composes a READ RPC and add it into the pending list;<br/> 5. osc_io_unplug() will be called by T to send RPCs;<br/> 6. the osc_extent that contains page <span class="error">&#91;0,...254&#93;</span> is also ready to send;<br/> 7. osc_io_unplug() sends WRITE RPC first, so it will try to compose a WRITE RPC for the above extent, however, the page 254 is already locked by T.</p> <p>Deadlocked.</p> <p>I will create a patch soon.</p> <p>can you please try patch: <a href="http://review.whamcloud.com/9928?" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9928?</a></p> <p>I just tested patch and didn't reproduce same issue so far. Originally, this problem 100% didn't happen, so, I will keep on testing, but it seems patch can solve this problem.</p> <p>more tested, but nothing reproduced issue so far. the patch solved problem. Thanks!</p> <p>patch landed.</p> Related LU-4977 Development Rank 1|hzwjlb: Rank (Obsolete) 13466 Severity