https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12718 Lustre <p>The follow crash message was observed (on the upstream-linux client).</p> <p>If the pre-read needed for a sub-page write triggers read-ahead, this will happen.</p> <p>We should probably just disable read-ahead for writes.</p> <p> [ 1666.576466] LustreError: 3089:0:(cl_io.c:549:cl_io_read_ahead()) ASSERTION( io-&gt;ci_type == CIT_READ || io-&gt;ci_type<br/> [ 1666.578761] LustreError: 3089:0:(cl_io.c:549:cl_io_read_ahead()) LBUG<br/> [ 1666.579807] Pid: 3089, comm: loop3 5.2.0-rc2-00877-gca23fc4b3908 #278 SMP PREEMPT Mon Jul 22 10:32:24 AEST 2019<br/> [ 1666.581132] Call Trace:<br/> [ 1666.581519] libcfs_debug_dumpstack+0x67/0x84<br/> [ 1666.582072] lbug_with_loc+0x33/0x75<br/> [ 1666.582520] cl_io_read_ahead+0x86/0xd6<br/> [ 1666.582979] ll_io_read_page+0x887/0x1089<br/> [ 1666.583457] ll_write_begin+0x4fa/0x69b<br/> [ 1666.583917] generic_perform_write+0xb4/0x193<br/> [ 1666.584448] __generic_file_write_iter+0x12c/0x16c<br/> [ 1666.585011] vvp_io_write_start+0x492/0x60c<br/> [ 1666.585555] cl_io_start+0xef/0x105<br/> [ 1666.586049] cl_io_loop+0xaa/0x1ec<br/> [ 1666.586527] ll_file_io_generic+0x475/0x7ca<br/> [ 1666.587131] ll_file_write_iter+0x224/0x274<br/> [ 1666.587718] do_iter_readv_writev+0x13a/0x176<br/> [ 1666.588308] do_iter_write+0x8b/0xff<br/> [ 1666.588866] lo_write_bvec+0x74/0xf8<br/> [ 1666.589374] loop_queue_work+0x376/0x8bc<br/> [ 1666.589908] kthread_worker_fn+0xc3/0x154<br/> [ 1666.590452] kthread+0x111/0x119<br/> [ 1666.590892] ret_from_fork+0x24/0x30</p> LU-12718

Readahead can trigger an assertion failire in cl_io_read_ahead

Bug Minor Resolved Fixed Neil Brown Neil Brown Fri, 30 Aug 2019 03:26:08 +0000 Sat, 14 Dec 2019 13:58:06 +0000 Sat, 14 Dec 2019 13:58:06 +0000 Lustre 2.14.0 0 6 <p>Neil Brown (neilb@suse.de) uploaded a new patch: <a href="https://review.whamcloud.com/36000" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36000</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12718" title="Readahead can trigger an assertion failire in cl_io_read_ahead" class="issue-link" data-issue-key="LU-12718"><del>LU-12718</del></a> llite: don't trigger read-ahead for a write.<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ff8d8420ebdcd819c10824ca5c95f1a3163c9a80</p> <p>Actually, the readahead for sub-page write was explicitly implemented not so long ago, because this improved performance noticeably. Otherwise, the client may be waiting for two separate reads (first partial page and last partial page) for each write. See patch <a href="https://review.whamcloud.com/27544" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27544</a><br/> "<tt><a href="https://jira.whamcloud.com/browse/LU-9618" title="Connect readahead to prep_partial_page to improve small (&lt; 1 page) write performance" class="issue-link" data-issue-key="LU-9618"><del>LU-9618</del></a> clio: Use readahead for partial page write</tt>" for details. </p> <p>Ahh... maybe it's just the LINVRNT() in cl_io_read_ahead() that is bad then ... but if it is, I would have expected it to show up when this feature was tested.</p> <p>But then, there is a lot that I don't understand yet, so maybe there is a good explanation.</p> <p>I did have the "Use readahead for partial page write" patch in the kernel that crashed.</p> <p> </p> <p>Generally:</p> <p>So, this didn't show up during original testing because the LINVRNT checks are only enabled by a build flag (I think it's config-debug-check-expensive or similar) that isn't part of the normal builds, but I think it's come up before that you use it, Neil.  (This is good!  We should use it in some of our builds as well...)</p> <p>So it didn't hit in the testing for this feature because the check wasn't there.</p> <p>Specifically:</p> <p>Definitely the problem is the LINVRNT - It's just a belt and suspenders, "check what you know to be true" sort of thing, it's not actually enforcing any constraints that I can see.  Certainly <a href="https://jira.whamcloud.com/browse/LU-9618" title="Connect readahead to prep_partial_page to improve small (&lt; 1 page) write performance" class="issue-link" data-issue-key="LU-9618"><del>LU-9618</del></a> has been in for a while and doesn't seem to cause any trouble <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I'll also drop this in Gerrit...</p> <p>Let's talk to Oleg about him enabling LINVRNT in this test rig.</p> <p>I discussed with Oleg about enabling CONFIG_LUSTRE_DEBUG_EXPENSIVE_CHECK in his test rig. We also have this issue with lu_ref as well. Its never tested and Neil has purposed just removing it since its never used and no one seems to be interesting in it.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36000/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36000/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12718" title="Readahead can trigger an assertion failire in cl_io_read_ahead" class="issue-link" data-issue-key="LU-12718"><del>LU-12718</del></a> obdclass: Allow read-ahead for write requests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 514bd936d06141319fb5e8032e4b7fa7bc0d1194</p> <p>Landed for 2.14</p> Related LU-9618 Development Rank 1|i00m47: Rank (Obsolete) 9223372036854775807 Severity