https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6666 Lustre <p>I met this during racer. This looks like <a href="https://jira.whamcloud.com/browse/LU-6227" title="Master testing: (osc_request.c:1219:osc_brw_prep_request()) ASSERTION( i == 0 || pg-&gt;off &gt; pg_prev-&gt;off) " class="issue-link" data-issue-key="LU-6227"><del>LU-6227</del></a>, but not in directIO patch.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> VFS: Error -28 occurred while creating quota. VFS: Error -28 occurred while creating quota. VFS: Error -28 occurred while creating quota. VFS: Error -28 occurred while creating quota. VFS: Error -28 occurred while creating quota. VFS: Error -28 occurred while creating quota. VFS: Error -28 occurred while creating quota. VFS: Error -28 occurred while creating quota. VFS: Error -28 occurred while creating quota. VFS: Error -28 occurred while creating quota. VFS: Error -28 occurred while creating quota. LustreError: 19811:0:(osc_request.c:1101:osc_brw_prep_request()) ASSERTION( page_count == 1 || (ergo(i == 0, poff + pg-&gt;count == PAGE_CACHE_SIZE) &amp;&amp; ergo(i &gt; 0 &amp;&amp; i &lt; page_count - 1, poff == 0 &amp;&amp; pg-&gt;count == PAGE_CACHE_SIZE) &amp;&amp; ergo(i == page_count - 1, poff == 0)) ) failed: i: 6/27 pg: ffff8801efd6d640 off: 24576, count: 3272 LustreError: 19811:0:(osc_request.c:1101:osc_brw_prep_request()) LBUG Pid: 19811, comm: ptlrpcd_4 Call Trace: [&lt;ffffffffa116b875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa116be77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa08a4b55&gt;] osc_brw_prep_request+0xc35/0x10a0 [osc] [&lt;ffffffffa08b4471&gt;] ? osc_req_attr_set+0x1b1/0x740 [osc] [&lt;ffffffffa08a5859&gt;] osc_build_rpc+0x899/0x15c0 [osc] [&lt;ffffffffa08c0eda&gt;] osc_io_unplug0+0x115a/0x1b40 [osc] [&lt;ffffffffa08b9a83&gt;] ? osc_ap_completion+0x213/0x600 [osc] [&lt;ffffffffa156d8bb&gt;] ? lu_object_put+0x12b/0x310 [obdclass] [&lt;ffffffffa08c3e61&gt;] osc_io_unplug+0x11/0x20 [osc] [&lt;ffffffffa08a746f&gt;] brw_interpret+0x9bf/0x1fa0 [osc] [&lt;ffffffffa060eadc&gt;] ? ptlrpc_free_committed+0x56c/0x770 [ptlrpc] [&lt;ffffffffa061bdb2&gt;] ? ptlrpc_unregister_bulk+0xa2/0xac0 [ptlrpc] [&lt;ffffffffa0610772&gt;] ? after_reply+0xcb2/0xeb0 [ptlrpc] [&lt;ffffffffa0614ab1&gt;] ptlrpc_check_set+0x331/0x1c70 [ptlrpc] [&lt;ffffffff81087fdb&gt;] ? try_to_del_timer_sync+0x7b/0xe0 [&lt;ffffffffa0642393&gt;] ptlrpcd_check+0x533/0x550 [ptlrpc] [&lt;ffffffffa06429cb&gt;] ptlrpcd+0x35b/0x430 [ptlrpc] [&lt;ffffffff81064b90&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0642670&gt;] ? ptlrpcd+0x0/0x430 [ptlrpc] [&lt;ffffffff8109e66e&gt;] kthread+0x9e/0xc0 [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 [&lt;ffffffff8109e5d0&gt;] ? kthread+0x0/0xc0 [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 </pre> </div></div> LU-6666

osc_brw_prep_request()) ASSERTION( page_count == 1 || (ergo(i == 0, poff + pg->count == PAGE_CACHE_SIZE)

Bug Minor Resolved Fixed Jinshan Xiong Di Wang Sun, 31 May 2015 05:19:11 +0000 Tue, 4 Jun 2019 17:45:23 +0000 Mon, 30 Nov 2015 17:38:20 +0000 Lustre 2.8.0 Lustre 2.8.0 0 13 <p>Btw: the build already includes the fix in <a href="https://jira.whamcloud.com/browse/LU-6227" title="Master testing: (osc_request.c:1219:osc_brw_prep_request()) ASSERTION( i == 0 || pg-&gt;off &gt; pg_prev-&gt;off) " class="issue-link" data-issue-key="LU-6227"><del>LU-6227</del></a>.</p> <p>in osc_build_rpc(), and in "/* add pages into rpc_list to build BRW rpc */" section, it makes sure that the middle pages has oap_page_off ==0 &amp;&amp; oap_count == PAGE_CACHE_SIZE, and the error log shows that the 7th page has count == 3272 (its offset is PAGE aligned), could it be the page cache has been truncated while the I/O is undergoing? </p> <p>Di, is it reproducible? Can I get the core dump and logs?</p> <p>here you are</p> <p>also need core dump</p> <p>Sorry, I do not have that.</p> <p>Can you provide how to reproduce the issue?</p> <p>Hi, Bobi</p> <p>I tried a few times on my branch. Sorry I can not reproduce it. Maybe it is just temporary.</p> <p>close it for now until we catch it again.</p> <p>Happened again with some another symptoms (LBUGs)</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LustreError: 4713:0:(osc_request.c:1812:osc_build_rpc()) ASSERTION( oap-&gt;oap_page_off == 0 ) failed: LustreError: 4718:0:(osc_request.c:1818:osc_build_rpc()) ASSERTION( oap-&gt;oap_page_off + oap-&gt;oap_brw_page.count == ((1UL) &lt;&lt; 12) ) failed: LustreError: 4718:0:(osc_request.c:1818:osc_build_rpc()) LBUG c0-0c0s10n1: LustreError: 4661:0:(osc_request.c:1212:osc_brw_prep_request()) ASSERTION( page_count == 1 || (ergo(i == 0, poff + pg-&gt;count == PAGE_CACHE_SIZE) &amp;&amp; ergo(i &gt; 0 &amp;&amp; i &lt; page_count - 1, poff == 0 &amp;&amp; pg-&gt;count == PAGE_CACHE_SIZE) &amp;&amp; ergo(i == page_count - 1, poff == 0)) ) failed: i: 1/2 pg: ffff88044b96cf78 off: 7884799, count: 1 LustreError: 4661:0:(osc_request.c:1212:osc_brw_prep_request()) LBUG </pre> </div></div> <p>Some additional information. Thread A<br/> There are two objects</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">struct osc_async_page { ... oap_obj_off = 0x23000, oap_page_off = 0x873, ... count = 0x41d, ... } struct osc_async_page { oap_obj_off = 0x26000, oap_page_off = 0x5eb, ... count = 0xa15, } </pre> </div></div> <p>so LustreError: 4713:0:(osc_request.c:1812:osc_build_rpc()) ASSERTION( oap-&gt;oap_page_off == 0 ) failed: happened<br/> thread B</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">oap_obj_off = 0x1f000, oap_page_off = 0x351, ... count = 0x342, ... and oap_obj_off = 0x4e000, oap_page_off = 0x542, count = 0x1c4, </pre> </div></div> <p>so,</p> <p>LustreError: 4718:0:(osc_request.c:1818:osc_build_rpc()) ASSERTION( oap-&gt;oap_page_off + oap-&gt;oap_brw_page.count == ((1UL) &lt;&lt; 12) ) failed</p> <p>Is it possible that pages from output above can be in one rpc after landing patches from <a href="https://jira.hpdd.intel.com/browse/LU-1669?" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-1669?</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Range lock will be worked out to allow write to a shared file in parallel, <span class="code-keyword">while</span> Posix compliance is still guaranteed. </pre> </div></div> <p>Artem, were you doing direct I/O? Otherwise this looks like a race between osc_refresh_count() and osc_page_touch_at(). Now that you can see pages in RPCs in question, can you post segments of continuous pages?</p> <p>&gt;Now that you can see pages in RPCs in question, can you post segments of continuous pages?</p> <p>Jinshan Xiong, I can't understed what exact information I need to post. Can you give some details.<br/> Post both brw_pages now.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">crash-7.0.1-29bit&gt; search -K 0xffff88044b96cf78 ffff88044a218728: ffff88044b96cf78 ffff88045392d858: ffff88044b96cf78 crash-7.0.1-29bit&gt; brw_page ffff88044c771378 struct brw_page { off = 7872511, pg = 0xffffea000f8698d0, count = 1, flag = 8 } crash-7.0.1-29bit&gt; ffff88044b96cf78 crash-7.0.1-29bit: command not found: ffff88044b96cf78 crash-7.0.1-29bit&gt; brw_page ffff88044b96cf78 struct brw_page { off = 7884799, pg = 0xffffea000f865500, count = 1, flag = 8 } </pre> </div></div> <p>&gt;Artem, were you doing direct I/O?<br/> Can't determine is this direct I/O from crash dump. I believe them have the same code path. </p> <p>Hi Artem,</p> <p>This issue is indeed introduced by <a href="https://jira.whamcloud.com/browse/LU-1669" title="lli-&gt;lli_write_mutex (single shared file performance)" class="issue-link" data-issue-key="LU-1669"><del>LU-1669</del></a>. When OST is out of space, and two processes write disjoint file range that don't overlap by page basis, the two writes can be merged into the same synchronous write RPC, and then hit this issue.</p> <p>This problem can be fixed by adding a mutex in vvp_io_commit_sync() to serialize sync threads.</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15468" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15468</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6666" title="osc_brw_prep_request()) ASSERTION( page_count == 1 || (ergo(i == 0, poff + pg-&gt;count == PAGE_CACHE_SIZE)" class="issue-link" data-issue-key="LU-6666"><del>LU-6666</del></a> osc: Do not merge extents with partial pages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: af031ebb20db32e0ab01a558b05442efcede5dbf</p> <p>If the pages are cached on the client (i.e. not O_DIRECT) then the full page must be cached since it isn't possible to mark only part of a page dirty, so I don't see how there can be multiple partial-page writes to the same file being merged?</p> <p>This can only happen when OST is running out of space, so the victim client can't cache page any more instead it uses sync I/O to write pages. For example, the first thread writes [0, 1024) and the 2nd thread writes [8192, 9216), these writes are not conflicted by range locks, and finally these two writes are picked by the same RPC and then hit this bug.</p> <p>Jinshan - Are you saying the OST must be running out of space for this to happen? I'm almost certain in the instance Artem is talking about (the dump is from Cray), the OST is not low on space.</p> <p>I'm puzzled by why the OST running out of space would impact client caching? Have I misunderstood?</p> <p>One last thing - Why is it specifically unsafe to merge two extents with partial pages? (if I'm reading right, it is safe to merge one extent with a partial page with another that doesn't have a partial page.)</p> <p>Patrick, it might also happen if the user runs out of quota, I believe. </p> <p>The reason it isn't ok to merge multiple fragments is because of LNet RDMA. The way the data is packed into the bulk RPC can't have gaps in the middle due to some RDMA implementations, only a fragment at the end (which is very common).</p> <p>If the pages were cached on the client it would be possible to just expand the write to cover the whole page in the middle of the file, since the client needs to do a read-modify-write of the page. With O_DIRECT that isn't possible. Not knowledgable enough in CLIO to know whether the out-of-space handling could be changed to allow this or not. </p> <p>Andreas, Jinshan - Apologies if I'm misunderstanding, but more questions...</p> <p>You're saying an RPC can't have gaps in the middle. I'm wondering about the meaning of gap. Does that mean it's an issue to have any gap in the data in a bulk RPC, like two non-contiguous pages? Or is the meaning of 'gap' limited to a fragmented page?</p> <p>If the first meaning (including non-contiguous full pages) is correct, I don't see how Jinshan's patch keeps us safe by preventing merging partial pages. (As I don't see how the code enforces merging only contiguous extents, though the comment on get_write_extents implies that's the case.)</p> <p>Patrick, sorry for long delay in replying. Having full-page holes in the middle of an RPC is fine, since a BRW RPC can have multiple niobufs in it. The problem is with partial page writes. That used to happen with liblustre (deprecated userspace library), but I'm not even sure it is possible with llite (kernel VFS client) since always full pages are written unless it is at EOF, or possibly with O_DIRECT?</p> <p>The other question is whether this bug is still being hit with recent builds, or has it been fixed by something else and could be closed?</p> <p>Patrick, Artem, are you able to test the patch <a href="http://review.whamcloud.com/15468" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15468</a> to see if this resolves your problem?</p> <p>I was just able to test this - It <b>does</b> resolve the assertions in our 4 MiB RPC testing. I don't have a specific, well defined test case, however. And I'm still fairly sure we're not out of space or out of quota when hitting this.</p> <p>We're hitting this when running a few different tests. One of them is growfiles from ltp.</p> <p>This stupid little shell script hit for us eventually, but only on nodes with a number of real CPUs - no luck in VMs. (The numbers before the tests are internal Cray test numbers):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#29 ARG1='-W gf29 -b -D 0 -r 1-4096 -R 0-33554432 -i 0 -L 60 -C 1 -u gfsparse-3-$$' #117 ARG2='-W gf117 -b -e 1 -i 0 -L 120 -u -g 5000 -T 100 -t 499990 -C 10 -c 1000 -S 10 -f Lgf03' #309 ARG3=' -W gf309 -b -e 1 -u -r 1-5000 -R 0--1 -i 0 -L 30 -C 1 -I p g_rand12 g_rand12.2' #701 ARG4='-W gf701 -b -e 1 -u -i 0 -L 20 -w -C 1 -I r -T 10 glseek20 glseek20.2' #811 ARG5='-W gf811 -b -e 1 -u -r 1-5000 -i 0 -L 30 -C 1 -I L g_lio15 g_lio15.2' #814 ARG6='-W gf814 -b -e 1 -u -i 0 -L 20 -w -C 1 -T 10 glseek19 glseek19.2' #815 ARG7='-W gf815 -b -e 1 -u -r 1-49600 -I r -u -i 0 -L 120 Lgfile1' (./growfiles $ARG1)&amp; (./growfiles $ARG2)&amp; (./growfiles $ARG3)&amp; (./growfiles $ARG4)&amp; (./growfiles $ARG5)&amp; (./growfiles $ARG6)&amp; (./growfiles $ARG7)&amp; </pre> </div></div> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/15468/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15468/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6666" title="osc_brw_prep_request()) ASSERTION( page_count == 1 || (ergo(i == 0, poff + pg-&gt;count == PAGE_CACHE_SIZE)" class="issue-link" data-issue-key="LU-6666"><del>LU-6666</del></a> osc: Do not merge extents with partial pages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ac1d6ab73c733caf2fa0ccf504955b23d3e572f0</p> <p>Landed for 2.8</p> Duplicate Related LU-6227 LU-6776 Development Rank 1|hzxekf: Rank (Obsolete) 9223372036854775807 Severity