https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2720 Lustre <p>it was already posted to <a href="https://jira.whamcloud.com/browse/LU-1723" title="osc_page_delete()) ASSERTION(0) failed" class="issue-link" data-issue-key="LU-1723"><del>LU-1723</del></a>, but seems to be a different issue from the original one in that ticket, so moving it here.</p> <p>2012-11-06T14:57:10.872333-06:00 c0-0c1s6n0 LustreError: 5270:0:(osc_cache.c:2367:osc_teardown_async_page()) extent ffff88060eedfe58@</p> {[23 -&gt; 23/255], [2|0|-|cache|wi|ffff88020e18f8c8], [4096|1|+|-|ffff8801f9ed9c18|256| (null)]} <p> trunc at 23.<br/> 2012-11-06T14:57:10.872389-06:00 c0-0c1s6n0 LustreError: 5270:0:(osc_page.c:411:osc_page_delete()) page@ffff88080e8c8bc0<span class="error">&#91;2 ffff88020cefad08:23 ^ (null)_ffff88080e8c8b00 4 0 1 (null) (null) 0x0&#93;</span><br/> 2012-11-06T14:57:10.902445-06:00 c0-0c1s6n0 LustreError: 5270:0:(osc_page.c:411:osc_page_delete()) page@ffff88080e8c8b00<span class="error">&#91;1 ffff88020d8c1f58:23 ^ffff88080e8c8bc0_ (null) 4 0 1 (null) (null) 0x0&#93;</span><br/> 2012-11-06T14:57:10.902494-06:00 c0-0c1s6n0 LustreError: 5270:0:(osc_page.c:411:osc_page_delete()) vvp-page@ffff88080e8cf5a0(0:0:0) vm@ffffea001c6a99b8 e00000000000063 7:0 0 23 lru<br/> 2012-11-06T14:57:10.902508-06:00 c0-0c1s6n0 LustreError: 5270:0:(osc_page.c:411:osc_page_delete()) lov-page@ffff8808050a7888<br/> 2012-11-06T14:57:10.958005-06:00 c0-0c1s6n0 LustreError: 5270:0:(osc_page.c:411:osc_page_delete()) osc-page@ffff88080d78ed20: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 94208 0 4096 0x0 0x520 | (null) ffff880804350700 ffff88020e18f8c8 &gt; 3&lt; + ffff8801f8f577b0 0 0 0 &gt; 4&lt; 0 0 8 33824768 - | - - + - &gt; 5&lt; - - + - | 0 - | 1 - -&gt;<br/> 2012-11-06T14:57:10.958049-06:00 c0-0c1s6n0 LustreError: 5270:0:(osc_page.c:411:osc_page_delete()) end page@ffff88080e8c8bc0<br/> 2012-11-06T14:57:10.983504-06:00 c0-0c1s6n0 LustreError: 5270:0:(osc_page.c:411:osc_page_delete()) Trying to teardown failed: -16<br/> 2012-11-06T14:57:10.983536-06:00 c0-0c1s6n0 LustreError: 5270:0:(osc_page.c:412:osc_page_delete()) ASSERTION( 0 ) failed:<br/> 2012-11-06T14:57:10.983549-06:00 c0-0c1s6n0 LustreError: 5270:0:(osc_page.c:412:osc_page_delete()) LBUG<br/> 2012-11-06T14:57:10.983570-06:00 c0-0c1s6n0 Pid: 5270, comm: fsx-linux<br/> 2012-11-06T14:57:10.983582-06:00 c0-0c1s6n0 Call Trace:<br/> 2012-11-06T14:57:10.983605-06:00 c0-0c1s6n0 <span class="error">&#91;&lt;ffffffff810063b1&gt;&#93;</span> try_stack_unwind+0x161/0x1a0<br/> 2012-11-06T14:57:11.009114-06:00 c0-0c1s6n0 <span class="error">&#91;&lt;ffffffff81004bf9&gt;&#93;</span> dump_trace+0x89/0x440<br/> 2012-11-06T14:57:11.009138-06:00 c0-0c1s6n0 <span class="error">&#91;&lt;ffffffffa014e887&gt;&#93;</span> libcfs_debug_dumpstack+0x57/0x80 <span class="error">&#91;libcfs&#93;</span><br/> 2012-11-06T14:57:11.009161-06:00 c0-0c1s6n0 <span class="error">&#91;&lt;ffffffffa014ede7&gt;&#93;</span> lbug_with_loc+0x47/0xc0 <span class="error">&#91;libcfs&#93;</span><br/> 2012-11-06T14:57:11.009173-06:00 c0-0c1s6n0 <span class="error">&#91;&lt;ffffffffa0671d41&gt;&#93;</span> osc_page_delete+0x2d1/0x2e0 <span class="error">&#91;osc&#93;</span><br/> 2012-11-06T14:57:11.034700-06:00 c0-0c1s6n0 <span class="error">&#91;&lt;ffffffffa02b0095&gt;&#93;</span> cl_page_delete0+0xd5/0x4e0 <span class="error">&#91;obdclass&#93;</span><br/> 2012-11-06T14:57:11.034737-06:00 c0-0c1s6n0 <span class="error">&#91;&lt;ffffffffa02b04e2&gt;&#93;</span> cl_page_delete+0x42/0x120 <span class="error">&#91;obdclass&#93;</span><br/> 2012-11-06T14:57:11.034762-06:00 c0-0c1s6n0 <span class="error">&#91;&lt;ffffffffa07f2e2d&gt;&#93;</span> ll_invalidatepage+0x8d/0x170 <span class="error">&#91;lustre&#93;</span><br/> 2012-11-06T14:57:11.034774-06:00 c0-0c1s6n0 <span class="error">&#91;&lt;ffffffffa07ea290&gt;&#93;</span> ll_page_mkwrite+0x7c0/0x840 <span class="error">&#91;lustre&#93;</span><br/> 2012-11-06T14:57:11.034820-06:00 c0-0c1s6n0 <span class="error">&#91;&lt;ffffffff81107cb7&gt;&#93;</span> __do_fault+0xe7/0x570<br/> 2012-11-06T14:57:11.034833-06:00 c0-0c1s6n0 <span class="error">&#91;&lt;ffffffff811081e4&gt;&#93;</span> handle_pte_fault+0xa4/0xcd0<br/> 2012-11-06T14:57:11.060561-06:00 c0-0c1s6n0 <span class="error">&#91;&lt;ffffffff81108fbe&gt;&#93;</span> handle_mm_fault+0x1ae/0x240<br/> 2012-11-06T14:57:11.060588-06:00 c0-0c1s6n0 <span class="error">&#91;&lt;ffffffff81025471&gt;&#93;</span> do_page_fault+0x191/0x410<br/> 2012-11-06T14:57:11.060600-06:00 c0-0c1s6n0 <span class="error">&#91;&lt;ffffffff81301b5f&gt;&#93;</span> page_fault+0x1f/0x30<br/> 2012-11-06T14:57:11.060647-06:00 c0-0c1s6n0 <span class="error">&#91;&lt;00000000200422b3&gt;&#93;</span> 0x200422b3<br/> 2012-11-06T14:57:11.060660-06:00 c0-0c1s6n0 Kernel panic - not syncing: LBUG</p> LU-2720

osc_page_delete()) ASSERTION(0) failed

Bug Blocker Resolved Fixed Keith Mannthey Vitaly Fertman LB patch Thu, 31 Jan 2013 07:35:38 +0000 Fri, 15 Apr 2016 17:19:11 +0000 Sat, 23 Feb 2013 14:28:52 +0000 Lustre 2.4.0 Lustre 2.4.0 0 8 <p>1. The ENODATA handling code in ll_page_mkwrite0 writes the page and invalidates it, but the page could have been re-added to the cache in between these 2 steps.<br/> 2. lock could be busy, so we get ENODATA on trylock, whereas a racy ll_fault waits for lock mutex and re-adds the page to the cache;</p> <p>At the same time, it is not clear why we wanted to just PEEK the lock, not waiting here for a new one. we want a writable page, thus we need a lock. if old one is cancelled, we will have to request a new one anyway. i.e.<br/> 1. if to be cancelled and to request a new lock, the new one will be granted only after canceling the old one on ost. thus, page will not get under old lock and will not be written, won't it ? </p> <p>2. whereas mkwrite will finish must faster with no new lock request, later we still want to make the page writable and anyway we have to request a new lock - so not faster in general, isn't it?</p> <p>so the PEEK "optimisation" is not clear and troublesome, so the patch I send drops it.</p> <p><a href="http://review.whamcloud.com/5222" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5222</a></p> <p>Vitaly, can you please include some information about how this problem was initially hit (e.g. test load, frequency of being hit, etc).</p> <p>We hit this often when running fsx-linux from LTP. Usually it happens during a stress run in an hour or two. We haven't seen this bug after applying this patch together with <a href="https://jira.whamcloud.com/browse/LU-2722" title="LustreError: 30217:0:(rw26.c:485:ll_direct_IO_26()) ASSERTION( obj-&gt;cob_transient_pages == 0 ) failed:" class="issue-link" data-issue-key="LU-2722"><del>LU-2722</del></a>/2723 patches fixing the DIO issues.</p> <p>after a talk to Jay, we decided not to change cl_lock_peek, because it may return REPEAT only for lock canceling or glimpse ast. at the same time, lock canceling may be long and we do not sleep here, so this looping will consume CPU resources. as after this patch, cl_lock_peek is used for SOM only, it may result the only ioepoch holder does not provide attibute update in done_writing and mds re-asks for them. if there will be a need in minimizing the amount of these RPCs, a sleeping version of cl_lock_peek is to be implemented.</p> <p>Are you going to drop <a href="http://review.whamcloud.com/5222" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5222</a> ? </p> <p>Landed for 2.4</p> <p>Can someone please comment about change #5222 landing considering the comments and questions from 08/Feb/13? Cray is seeing significant CPU spinning in 2.4 RC testing, but Wally would have to confirm if change #5222 is the cause.</p> <p>Vitaly comments relates to patchset #2, he since added patchset #3 that fixes the potential issue discussed.<br/> As for the cpu spinning you are seeing, we need more details to make any educated guess there.</p> <p>Ah, thanks Oleg. Sure, we'll keep looking at our spin issue.</p> <p>Corey</p> <p>It would be best to open a new ticket with details of the CPU spinning issue that you are seeing</p> <p>Peter</p> <p>Thanks, it is <a href="https://jira.whamcloud.com/browse/LU-3217" title="Client hung in cl_io_loop/cl_io_lock path" class="issue-link" data-issue-key="LU-3217"><del>LU-3217</del></a>.</p> Related LU-2557 LU-2788 Development Rank 1|hzviav: Rank (Obsolete) 6614 Severity