https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3192 Lustre <p>This happens to Lustre 2.3 running with SP1 but we have seen it in SP2 or 2.2(or even 2.1 with different stack trace, see LELUS-41):</p> <p>LustreError: 3821:0:(osc_request.c:819:osc_announce_cached()) dirty 361 - 362 &gt; system dirty_max 2113536<br/> LustreError: 3818:0:(osc_request.c:1308:osc_brw_prep_request()) ASSERTION( i == 0 || pg-&gt;off &gt; pg_prev-&gt;off ) failed: i 1 p_c 151 pg ffffea0004129ce8 <span class="error">&#91;pri 18446612138517887360 ind 30208&#93;</span> off 123731968 prev_pg ffffea000dbc0e18 [pri 0<br/> ind 263316] off 123731968<br/> LustreError: 3818:0:(osc_request.c:1308:osc_brw_prep_request()) LBUG<br/> Pid: 3818, comm: doio_mpi<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffff81007e59&gt;&#93;</span> try_stack_unwind+0x1a9/0x200<br/> <span class="error">&#91;&lt;ffffffff81006625&gt;&#93;</span> dump_trace+0x95/0x300<br/> <span class="error">&#91;&lt;ffffffffa016b8d7&gt;&#93;</span> libcfs_debug_dumpstack+0x57/0x80 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa016be27&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06969c9&gt;&#93;</span> osc_brw_prep_request+0x959/0xe60 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06983bc&gt;&#93;</span> osc_build_rpc+0x90c/0x1180 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06ad617&gt;&#93;</span> osc_send_oap_rpc+0x3c7/0xc20 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06ae24f&gt;&#93;</span> osc_io_unplug+0x3df/0x730 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06a834a&gt;&#93;</span> osc_io_submit+0x1da/0x520 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa02e13e8&gt;&#93;</span> cl_io_submit_rw+0x78/0x190 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07334cd&gt;&#93;</span> lov_io_submit+0x27d/0xc00 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa02e13e8&gt;&#93;</span> cl_io_submit_rw+0x78/0x190 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa02e35c1&gt;&#93;</span> cl_io_read_page+0xd1/0x190 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07ee254&gt;&#93;</span> ll_readpage+0x184/0x210 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff810d3ff0&gt;&#93;</span> generic_file_aio_read+0x230/0x640<br/> <span class="error">&#91;&lt;ffffffffa0819846&gt;&#93;</span> vvp_io_read_start+0x1f6/0x3d0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa02e16ca&gt;&#93;</span> cl_io_start+0x6a/0x130 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa02e597c&gt;&#93;</span> cl_io_loop+0xac/0x1a0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07c5c23&gt;&#93;</span> ll_file_io_generic+0x353/0x530 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07c62c8&gt;&#93;</span> ll_file_aio_read+0x238/0x290 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07c69bf&gt;&#93;</span> ll_file_read+0x20f/0x2b0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81117fe8&gt;&#93;</span> vfs_read+0xc8/0x1a0<br/> <span class="error">&#91;&lt;ffffffff811181b5&gt;&#93;</span> sys_read+0x55/0x90<br/> <span class="error">&#91;&lt;ffffffff8100305b&gt;&#93;</span> system_call_fastpath+0x16/0x1b<br/> <span class="error">&#91;&lt;0000000020139f90&gt;&#93;</span> 0x20139f90<br/> Kernel panic - not syncing: LBUG</p> <p>Here are also some other instances all seem to hit the same page for the RPC:</p> <p>Lustre 2.3.0-trunk-1.0000.40706.58.4-abuild-trunk 27</p> <p>First hit Cname # hits Apid/Roles<br/> LUS: LBUG-ASSERTION( i == 0 || pg-&gt;off &gt; pg_prev-&gt;off ) failed: i 1 p_c 256 pg ffffea000c65a268 <span class="error">&#91;pri 18446612137055183040 ind 26880&#93;</span> off 110100480 prev_pg ffffea000e0e73f0 <span class="error">&#91;pri 0 ind 263338&#93;</span> off 110100480<br/> 13/02/06 01:43:15 c0-0c0s1n2 1 854758</p> <p>854758 doio_mpi</p> <p>LUS: LBUG-ASSERTION( i == 0 || pg-&gt;off &gt; pg_prev-&gt;off ) failed: i 1 p_c 256 pg ffffea00179fdfa0 <span class="error">&#91;pri 18446612160482967680 ind 46909&#93;</span> off 192139264 prev_pg ffffea000bd34598 <span class="error">&#91;pri 0 ind 263425&#93;</span> off 192139264<br/> 13/02/06 02:22:40 c0-0c0s4n3 1 855115</p> <p>855115 doio_mpi</p> <p>LUS: LBUG-ASSERTION( i == 0 || pg-&gt;off &gt; pg_prev-&gt;off ) failed: i 32 p_c 155 pg ffffea001bcb95e0 <span class="error">&#91;pri 0 ind 264145&#93;</span> off 14807040 prev_pg ffffea001bff2bd0 <span class="error">&#91;pri 18446612166806628160 ind 3615&#93;</span> off 14807040<br/> 13/02/06 00:09:44 c0-0c0s7n2 1 854062</p> <p>854062 doio_mpi</p> LU-3192

LBUG:(osc_request.c:1308:osc_brw_prep_request()) ASSERTION( i == 0 || pg->off > pg_prev->off )

Bug Major Resolved Fixed WC Triage Vitaly Fertman patch Thu, 18 Apr 2013 14:19:48 +0000 Fri, 7 Nov 2014 19:27:05 +0000 Fri, 7 Nov 2014 19:27:05 +0000 Lustre 2.7.0 0 12 <p>What is 2.3 SP1 and SP2? Do you see this with the 2.3 branch in the git tree? Or with Master?</p> <p>Keith I think that Vitaly means SLES11 SP1 and SP2</p> <p>this is not correct to kick transient pages off the CLIO cache, because we may get 2 sets of pages in osc_build_req(), cacheable and transient ones, for the same offsets.</p> <p><a href="http://review.whamcloud.com/6096" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6096</a> - revert of <a href="https://jira.whamcloud.com/browse/LU-481" title="sanity test_119d fails (ASSERTION((struct cl_page *)vmpage-&gt;private != slice-&gt;cpl_page) failed)" class="issue-link" data-issue-key="LU-481"><del>LU-481</del></a><br/> <a href="http://review.whamcloud.com/6097" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6097</a> - fix of original <a href="https://jira.whamcloud.com/browse/LU-481" title="sanity test_119d fails (ASSERTION((struct cl_page *)vmpage-&gt;private != slice-&gt;cpl_page) failed)" class="issue-link" data-issue-key="LU-481"><del>LU-481</del></a> issue</p> <p>lustre 2.3 on sles11 sp1/sp2</p> <p>So can you please describe the scenario to hit this and the environment?</p> <p>due to putting out transient pages of the page cache, we may get 2 sets of pages, cacheable and transient, for the same offsets, originated in BIO and DIO. later, when preparing rpc, such pages are gathered for the same rpc, and assertion that 2 pages with the same offset are found is hit.</p> <p>Actually, I just started refresh the fix of <a href="https://jira.whamcloud.com/browse/LU-247" title="Lustre client slow performance on BG/P IONs: unaligned DIRECT_IO" class="issue-link" data-issue-key="LU-247"><del>LU-247</del></a> (<a href="http://review.whamcloud.com/#change,980" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,980</a>) yesterday, which could fix this problem (concurrent dio read &amp; buffered read), but that isn't a small fix, I'm not sure if it can be merged in 2.4.</p> <p>Vitaly, can you please tell us what the workload is that is triggering this problem? Is this a synthetic workload that is generating "difficult" IO patterns with O_DIRECT and buffered IO on the same file offset, or is this a failure seen by a real application running from a user?</p> <p>I would be surprised if any real application will have an IO workload to trigger this, but then again application developers are known to do all kinds of strange things...</p> <p>If this is only from a synthetic workload, then I'm inclined to drop it as a blocker for the 2.4.0 release, since this is extremely unlikely to be hit under real-world usage.</p> <p>Andreas, this is not some real application workload. It is from an internal stress test. </p> <p>Let's mark the transient page as special in IO engine so that it can't be merged with caching pages.</p> <p>Jay,</p> <p>transient pages already marked as special and live in CLIO cache until we have IO finished, and merging with buffered pages exist in CLIO from beginning, so we may say - that is broken for now. Per Nikita`s comments - he tried to present a flat file as offset range with clio pages if it's covered by lock. That allowed to re-stripe it easy just a change mapping a some offset range to new osc object.</p> <p>from other view - OST uses a single page cache to store DIO and buffered IO so we have no differences (buffered IO may read DIO data without access to disk). so we have free to use same view on client side, have a single CLIO page cache to store a data before send over network. That way is better in case we will return to idea to create a p2p communication and client may send own data to new client without asking any data from a server. That is very usefull in excacale as extend buffer cache dramatically.</p> <p>Because of it, i think we need to return to put transient pages into CLIO cache, but remember that is special pages and need disowned after IO finished and don't forget we need to stay in waiting until that data committed on OST side and don't removed a request from sending queue, otherwise we will hit a <a href="https://jira.whamcloud.com/browse/LU-1573" title="avoid data corruption for direct io data" class="issue-link" data-issue-key="LU-1573"><del>LU-1573</del></a>. But until request in sending queue, we a free to read/write in same pages, to reduce a network usage.</p> <blockquote> <p>transient pages already marked as special and live in CLIO cache until we have IO finished, and merging with buffered pages exist in CLIO from beginning, so we may say - that is broken for now.</p></blockquote> <p>It was. But we found a problem in <a href="https://jira.whamcloud.com/browse/LU-481" title="sanity test_119d fails (ASSERTION((struct cl_page *)vmpage-&gt;private != slice-&gt;cpl_page) failed)" class="issue-link" data-issue-key="LU-481"><del>LU-481</del></a> where a transient page can be in cache when trying to add a caching page. We could fix it by awaiting transient pages to go away but I didn't see any benefits there. Any idea?</p> <blockquote> <p>Per Nikita`s comments - he tried to present a flat file as offset range with clio pages if it's covered by lock. That allowed to re-stripe it easy just a change mapping a some offset range to new osc object.</p></blockquote> <p>This turns out too optimistic. Restriping is a case of layout change and right now we have a complete solution for layout change. The safest way to change a file's layout is to clean up file's page cache in prior. Don't bother talking about performance as changing layout is a rare operation <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <blockquote> <p>from other view - OST uses a single page cache to store DIO and buffered IO so we have no differences (buffered IO may read DIO data without access to disk). so we have free to use same view on client side, have a single CLIO page cache to store a data before send over network.</p></blockquote> <p>hmm.. I tend to think DIO shouldn't pollute cache in anyways - this is why it is called Direct IO. Even on the OST side, it should go directly into disks before replying the clients. Also it;s really dangerous to cache write of DIO on the OSTs.</p> <p>Alexey, we've seen quite a few problems caused by putting 'transient' pages in the radix tree:</p> <ul class="alternate" type="square"> <li>Race of buffered io vs. dio described in this ticket &amp; <a href="https://jira.whamcloud.com/browse/LU-481" title="sanity test_119d fails (ASSERTION((struct cl_page *)vmpage-&gt;private != slice-&gt;cpl_page) failed)" class="issue-link" data-issue-key="LU-481"><del>LU-481</del></a>;</li> <li>Unable to support unalinged DIO. Lustre now can only do aligned DIO (buffer address, offset &amp; write count must be page size aligned), the reason we have such unnecessary<br/> restriction is that we put user pages (transient pages) in the radix tree. I think unaligned DIO is very important for DIO users and it also very helpful for lockless io;</li> </ul> <p>All in all, DIO pages are managed by applications, I don't see any reason to store them in the radix tree. (add them in the tree before io then remove on io completion looks only increase overhead).</p> <p>I've refreshed the patch of <a href="https://jira.whamcloud.com/browse/LU-247" title="Lustre client slow performance on BG/P IONs: unaligned DIRECT_IO" class="issue-link" data-issue-key="LU-247"><del>LU-247</del></a> (<a href="http://review.whamcloud.com/#change,980" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,980</a>), but I'm not sure if it can catch the 2.4 window. I think Jingshan's idea of not merging 'transient' page with cached page into same RPC could be a temporariy solution if the patch can't fit in with the 2.4 schedule.</p> <p>&gt; It was. But we found a problem in <a href="https://jira.whamcloud.com/browse/LU-481" title="sanity test_119d fails (ASSERTION((struct cl_page *)vmpage-&gt;private != slice-&gt;cpl_page) failed)" class="issue-link" data-issue-key="LU-481"><del>LU-481</del></a> where a transient page can be in cache when trying to add a caching page. We could fix it by awaiting transient pages to go away but I didn't see any benefits there. Any idea?</p> <p>Vitaly have fix for <a href="https://jira.whamcloud.com/browse/LU-481" title="sanity test_119d fails (ASSERTION((struct cl_page *)vmpage-&gt;private != slice-&gt;cpl_page) failed)" class="issue-link" data-issue-key="LU-481"><del>LU-481</del></a>, it's simple. But with that fix we have avoid doubling IO stream in case mixing IO and (as i say before) we may read from a that pages without sending network requests.</p> <p>&gt;This turns out too optimistic. Restriping is a case of layout change and right now we have a complete solution for layout change. The safest way to change a file's layout is to clean up file's page cache in prior. Don't bother talking about performance as changing layout is a rare operation </p> <p>why? if you change a layout - you just change a translation about offset in file, to data object and offset to object. main question in that case - have a stable point when old layout and new layout have a same contents.<br/> but that is outside of that ticket.</p> <p>&gt; hmm.. I tend to think DIO shouldn't pollute cache in anyways - this is why it is called Direct IO. Even on the OST side, it should go directly into disks before replying the clients. Also it;s really dangerous to cache write of DIO on the OSTs.</p> <p>by POSIX/SUS definition, DIO need bypass only pagecache, but don't see anything about raid cache, disk drive cache or other cache types...</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">http:<span class="code-comment">//pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_chap03.html </span>Direct I/O Historically, direct I/O refers to the system bypassing intermediate buffering, but may be extended to cover implementation-defined optimizations. </pre> </div></div> <p>&gt; Race of buffered io vs. dio described in this ticket &amp; <a href="https://jira.whamcloud.com/browse/LU-481" title="sanity test_119d fails (ASSERTION((struct cl_page *)vmpage-&gt;private != slice-&gt;cpl_page) failed)" class="issue-link" data-issue-key="LU-481"><del>LU-481</del></a>;</p> <p>That ticket isn't race. that ticket is result of "fix" in <a href="https://jira.whamcloud.com/browse/LU-481" title="sanity test_119d fails (ASSERTION((struct cl_page *)vmpage-&gt;private != slice-&gt;cpl_page) failed)" class="issue-link" data-issue-key="LU-481"><del>LU-481</del></a>. When you kill finding a buffered page for DIO access you start able to put two CLIO pages in same offset and OSC send two pages in single bulk with same offset.</p> <p>&gt; Unable to support unalinged DIO. Lustre now can only do aligned DIO (buffer address, offset &amp; write count must be page size aligned), the reason we have such unnecessary</p> <p>That is big question. You need to read full page anyway on OST side and have full page write.<br/> but unaligned write may be not at DIO case, so you will fix a particle case and not a generic.<br/> Why it's don't fix in different way? like send a first page of buffer as part of LOCK ENQ request?<br/> that is need to change a wire protocol, but will solve DIO and buffered unaligned write access.<br/> (CLIO release page should be revised to ability to kill pages only with locks cover it, otherwise we have much more problems with readahead, single page reads inside a large up2date region and other)</p> <p>as about lockless IO.. lock less IO more easy implement as direct access to osc_brw api or other ways but have similar to obdecho client.</p> <p>&gt; All in all, DIO pages are managed by applications, I don't see any reason to store them in the radix tree. (add them in the tree before io then remove on io completion looks only increase overhead).</p> <p>if you don't plan to have it's in radix tree - why it's should be covered by CLIO page? kill these pages from CLIO that is reduce a complexity of CLIO (we will have a single page type). </p> <p><a href="http://review.whamcloud.com/#/c/10930/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10930/</a></p> <p><a href="http://review.whamcloud.com/#/c/10930" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10930</a> landed, so should this bug be closed?</p> <p>Yes it should - thanks!</p> Related LU-247 Development Rank 1|hzvohj: Rank (Obsolete) 7788 Severity