https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5138 Lustre <p>This process has been waiting in osc_lru_reserve for a very long time:</p> <p>PID: 22025 TASK: ffff88017aeba480 CPU: 5 COMMAND: "reads"<br/> #0 <span class="error">&#91;ffff88017b2ff838&#93;</span> schedule at ffffffff8145ec7b<br/> #1 <span class="error">&#91;ffff88017b2ff980&#93;</span> osc_lru_reserve at ffffffffa0e16ee5 <span class="error">&#91;osc&#93;</span><br/> #2 <span class="error">&#91;ffff88017b2ffa00&#93;</span> osc_page_init at ffffffffa0e1710d <span class="error">&#91;osc&#93;</span><br/> #3 <span class="error">&#91;ffff88017b2ffa40&#93;</span> lov_page_init_raid0 at ffffffffa0ea48b0 <span class="error">&#91;lov&#93;</span><br/> #4 <span class="error">&#91;ffff88017b2ffaa0&#93;</span> cl_page_alloc at ffffffffa0aae632 <span class="error">&#91;obdclass&#93;</span><br/> #5 <span class="error">&#91;ffff88017b2ffae0&#93;</span> cl_page_find at ffffffffa0aae91b <span class="error">&#91;obdclass&#93;</span><br/> #6 <span class="error">&#91;ffff88017b2ffb30&#93;</span> ll_write_begin at ffffffffa0f96f8d <span class="error">&#91;lustre&#93;</span><br/> #7 <span class="error">&#91;ffff88017b2ffb90&#93;</span> generic_perform_write at ffffffff810f8242<br/> #8 <span class="error">&#91;ffff88017b2ffc10&#93;</span> generic_file_buffered_write at ffffffff810f83a1<br/> #9 <span class="error">&#91;ffff88017b2ffc60&#93;</span> __generic_file_aio_write at ffffffff810fb336<br/> #10 <span class="error">&#91;ffff88017b2ffd10&#93;</span> generic_file_aio_write at ffffffff810fb57c<br/> #11 <span class="error">&#91;ffff88017b2ffd50&#93;</span> vvp_io_write_start at ffffffffa0faae48 <span class="error">&#91;lustre&#93;</span><br/> #12 <span class="error">&#91;ffff88017b2ffda0&#93;</span> cl_io_start at ffffffffa0ab65f9 <span class="error">&#91;obdclass&#93;</span><br/> #13 <span class="error">&#91;ffff88017b2ffdd0&#93;</span> cl_io_loop at ffffffffa0aba123 <span class="error">&#91;obdclass&#93;</span><br/> #14 <span class="error">&#91;ffff88017b2ffe00&#93;</span> ll_file_io_generic at ffffffffa0f46af1 <span class="error">&#91;lustre&#93;</span><br/> #15 <span class="error">&#91;ffff88017b2ffe70&#93;</span> ll_file_aio_write at ffffffffa0f47037 <span class="error">&#91;lustre&#93;</span><br/> #16 <span class="error">&#91;ffff88017b2ffec0&#93;</span> ll_file_write at ffffffffa0f47a00 <span class="error">&#91;lustre&#93;</span><br/> #17 <span class="error">&#91;ffff88017b2fff10&#93;</span> vfs_write at ffffffff8115aeae<br/> #18 <span class="error">&#91;ffff88017b2fff40&#93;</span> sys_write at ffffffff8115b023<br/> #19 <span class="error">&#91;ffff88017b2fff80&#93;</span> system_call_fastpath at ffffffff81468d92<br/> RIP: 00002aaaaad99630 RSP: 00007fffffffc568 RFLAGS: 00010246<br/> RAX: 0000000000000001 RBX: ffffffff81468d92 RCX: 00007fffffffc510<br/> RDX: 0000000000010000 RSI: 0000000000603040 RDI: 0000000000000003<br/> RBP: 0000000000010000 R8: 0000000000000000 R9: 0101010101010101<br/> R10: 00007fffffffc3b0 R11: 0000000000000246 R12: 0000000000010000<br/> R13: 0000000000000001 R14: 00000000063b0000 R15: 00000000063c0000<br/> ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b</p> <p>While testing for <a href="https://jira.whamcloud.com/browse/LU-4856" title="osc_lru_reserve()) ASSERTION( atomic_read(cli-&gt;cl_lru_left) &gt;= 0 ) failed" class="issue-link" data-issue-key="LU-4856"><del>LU-4856</del></a>, the bug described in <a href="https://jira.whamcloud.com/browse/LU-5123" title="lprocfs_write_frac_u64_helper does not respect multiplier" class="issue-link" data-issue-key="LU-5123"><del>LU-5123</del></a> caused sanity 101a to run with ccc_lru_max = 32 (pages), I have not tried, but it should be possible to reproduce this in master by modifying 101a to set max_dirty_mb to 128k.</p> <p>This is a pathological condition, but I think it exposed a real bug. Namely, it appears that the wakeup from the sleep in osc_lru_reserve can be incidental - causes by another process that just happens to do something that triggers an osc_lru_shrink, rather than the deliberate and specific process of the conditions which caused the sleep being addressed when it becomes possible to do so. </p> <p>I have a core and debug log from a system in this state, and will attach the debug log, and paste my notes in a comment.</p> patch for <a href="https://jira.whamcloud.com/browse/LU-4856" title="osc_lru_reserve()) ASSERTION( atomic_read(cli-&gt;cl_lru_left) &gt;= 0 ) failed" class="issue-link" data-issue-key="LU-4856"><strike>LU-4856</strike></a> + without a patch for <a href="https://jira.whamcloud.com/browse/LU-5123" title="lprocfs_write_frac_u64_helper does not respect multiplier" class="issue-link" data-issue-key="LU-5123"><strike>LU-5123</strike></a> LU-5138

hang in osc_lru_reserve despite recoverable state

Bug Minor Resolved Won't Fix Peter Jones Stephen Champion Mon, 2 Jun 2014 18:26:02 +0000 Mon, 17 Nov 2014 19:03:34 +0000 Mon, 17 Nov 2014 19:03:34 +0000 0 5 <p>The last we hear from the hung process is that it has no free pages. It goes to sleep immediately afterwards. At this point in time. there are no free slots. All lru items are busy:</p> <p> 00000008:00000020:5.0:1400133972.928506:0:22025:0:(osc_page.c:833:osc_lru_reclaim()) accfs1-OST0002-osc-ffff88022017e000: cli ffff880431ea5538 no free slots, pages: 0, busy: 30.<br/> 00000008:00000020:5.0:1400133972.928507:0:22025:0:(osc_page.c:849:osc_lru_reclaim()) accfs1-OST0003-osc-ffff88022017e000: cli ffff8803653cd578 LRU pages: 0, busy: 0.<br/> 00000008:00000020:5.0:1400133972.928509:0:22025:0:(osc_page.c:849:osc_lru_reclaim()) accfs1-OST0001-osc-ffff88022017e000: cli ffff8804316575f8 LRU pages: 0, busy: 2.<br/> 00000008:00000020:5.0:1400133972.928510:0:22025:0:(osc_page.c:849:osc_lru_reclaim()) accfs1-OST0000-osc-ffff88022017e000: cli ffff880430d09538 LRU pages: 0, busy: 0.<br/> 00000020:00001000:5.0:1400133972.928511:0:22025:0:(cl_object.c:966:cl_env_put()) 1@ffff880179bdee58<br/> 00000008:00000020:5.0:1400133972.928513:0:22025:0:(osc_page.c:867:osc_lru_reclaim()) accfs1-OST0000-osc-ffff88022017e000: cli ffff880430d09538 freed 0 pages.</p> <p>Then a sixteen page extent is completed.</p> <p> 00000008:00000020:12.0:1400134007.925975:0:19741:0:(osc_cache.c:810:osc_extent_finish()) extent ffff8803803091e0@</p> {[0 -&gt; 15/255], [2|0|-|rpc|wiuY|ffff880352ecfc48], [65536|16|+|-|ffff8803823a0660|256|ffff880166486640]} <p> extent finished.</p> <p>And the lru_queue work is requested for ptlrpc_0</p> <p> 00000100:00000040:12.0:1400134007.925983:0:19741:0:(ptlrpcd.c:271:ptlrpcd_add_req()) @@@ add req <span class="error">&#91;ffff880347956000&#93;</span> to pc <span class="error">&#91;ptlrpcd_0:0&#93;</span> req@ffff880347956000 x1468144665523584/t0(0) o-1-&gt;accfs1-OST0002-osc-ffff88022017e000@192.168.21.8@o2ib1:0/0 lens 0/0 e 0 to 0 dl 1400134027 ref 2 fl Interpret:N/ffffffff/ffffffff rc 0/-1</p> <p>The req is processed</p> <p> 00000008:00000020:8.0:1400134007.926058:0:19727:0:(osc_page.c:568:lru_queue_work()) Run LRU work for client obd ffff880431ea5538.</p> <p>It goes to shrink</p> <p> 00000008:00000001:8.0:1400134007.926061:0:19727:0:(osc_page.c:687:osc_lru_shrink()) Process entered</p> <p>It leaves shrink with nothing</p> <p> 00000008:00000001:8.0:1400134007.926066:0:19727:0:(osc_page.c:801:osc_lru_shrink()) Process leaving (rc=0 : 0 : 0)</p> <p>And finishes the req</p> <p> 00000008:00000001:8.0:1400134007.926068:0:19727:0:(osc_page.c:573:lru_queue_work()) Process leaving (rc=0 : 0 : 0)<br/> 00000100:00000001:8.0:1400134007.926069:0:19727:0:(client.c:1888:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1)</p> <p>It should of had 16 pages to shrink, if they are not referenced.</p> <p>Currently, all of the pages have a ref count of 1, and would be shrunk. This may not have been the case the last time we called the shrink, though...</p> <p>The last time we called shrink was <br/> 00000008:00000001:8.0:1400134007.926066:0:19727:0:(osc_page.c:801:osc_lru_shrink()) Process leaving (rc=0 : 0 : 0)</p> <p>Since then, we've put 32 pages.</p> <p> 00000020:00000001:12.0:1400134007.926074:0:19741:0:(cl_page.c:385:cl_page_put()) page@ffff8803772e6800<span class="error">&#91;2 ffff880381c2c0f0 0 0 1 (null) (null) 0x0&#93;</span><br/> 00000020:00000001:12.0:1400134007.926075:0:19741:0:(cl_page.c:385:cl_page_put()) 2<br/> 00000020:00000001:12.0:1400134007.926103:0:19741:0:(cl_page.c:385:cl_page_put()) page@ffff8803772e6400<span class="error">&#91;2 ffff880381c2c0f0 0 0 1 (null) (null) 0x0&#93;</span><br/> 00000020:00000001:12.0:1400134007.926104:0:19741:0:(cl_page.c:385:cl_page_put()) 2<br/> 00000020:00000001:12.0:1400134007.926130:0:19741:0:(cl_page.c:385:cl_page_put()) page@ffff88036f5c8000<span class="error">&#91;2 ffff880381c2c0f0 0 0 1 (null) (null) 0x0&#93;</span><br/> 00000020:00000001:12.0:1400134007.926131:0:19741:0:(cl_page.c:385:cl_page_put()) 2<br/> 00000020:00000001:12.0:1400134007.926157:0:19741:0:(cl_page.c:385:cl_page_put()) page@ffff88036f5c8800<span class="error">&#91;2 ffff880381c2c0f0 0 0 1 (null) (null) 0x0&#93;</span><br/> 00000020:00000001:12.0:1400134007.926158:0:19741:0:(cl_page.c:385:cl_page_put()) 2<br/> 00000020:00000001:12.0:1400134007.926185:0:19741:0:(cl_page.c:385:cl_page_put()) page@ffff88036f5c8400<span class="error">&#91;2 ffff880381c2c0f0 0 0 1 (null) (null) 0x0&#93;</span><br/> 00000020:00000001:12.0:1400134007.926185:0:19741:0:(cl_page.c:385:cl_page_put()) 2<br/> 00000020:00000001:12.0:1400134007.926212:0:19741:0:(cl_page.c:385:cl_page_put()) page@ffff880377565c00<span class="error">&#91;2 ffff880381c2c0f0 0 0 1 (null) (null) 0x0&#93;</span><br/> 00000020:00000001:12.0:1400134007.926213:0:19741:0:(cl_page.c:385:cl_page_put()) 2<br/> 00000020:00000001:12.0:1400134007.926239:0:19741:0:(cl_page.c:385:cl_page_put()) page@ffff8803820d9800<span class="error">&#91;2 ffff880381c2c0f0 0 0 1 (null) (null) 0x0&#93;</span><br/> 00000020:00000001:12.0:1400134007.926239:0:19741:0:(cl_page.c:385:cl_page_put()) 2<br/> 00000020:00000001:12.0:1400134007.926266:0:19741:0:(cl_page.c:385:cl_page_put()) page@ffff88037856b000<span class="error">&#91;2 ffff880381c2c0f0 0 0 1 (null) (null) 0x0&#93;</span><br/> 00000020:00000001:12.0:1400134007.926267:0:19741:0:(cl_page.c:385:cl_page_put()) 2<br/> 00000020:00000001:12.0:1400134007.926293:0:19741:0:(cl_page.c:385:cl_page_put()) page@ffff880380d12400<span class="error">&#91;2 ffff880381c2c0f0 0 0 1 (null) (null) 0x0&#93;</span><br/> 00000020:00000001:12.0:1400134007.926294:0:19741:0:(cl_page.c:385:cl_page_put()) 2<br/> 00000020:00000001:12.0:1400134007.926320:0:19741:0:(cl_page.c:385:cl_page_put()) page@ffff8803811ef400<span class="error">&#91;2 ffff880381c2c0f0 0 0 1 (null) (null) 0x0&#93;</span><br/> 00000020:00000001:12.0:1400134007.926321:0:19741:0:(cl_page.c:385:cl_page_put()) 2<br/> 00000020:00000001:12.0:1400134007.926347:0:19741:0:(cl_page.c:385:cl_page_put()) page@ffff8803811ef000<span class="error">&#91;2 ffff880381c2c0f0 0 0 1 (null) (null) 0x0&#93;</span><br/> 00000020:00000001:12.0:1400134007.926348:0:19741:0:(cl_page.c:385:cl_page_put()) 2<br/> 00000020:00000001:12.0:1400134007.926374:0:19741:0:(cl_page.c:385:cl_page_put()) page@ffff88037f49f800<span class="error">&#91;2 ffff880381c2c0f0 0 0 1 (null) (null) 0x0&#93;</span><br/> 00000020:00000001:12.0:1400134007.926375:0:19741:0:(cl_page.c:385:cl_page_put()) 2<br/> 00000020:00000001:12.0:1400134007.926401:0:19741:0:(cl_page.c:385:cl_page_put()) page@ffff880362cf4000<span class="error">&#91;2 ffff880381c2c0f0 0 0 1 (null) (null) 0x0&#93;</span><br/> 00000020:00000001:12.0:1400134007.926402:0:19741:0:(cl_page.c:385:cl_page_put()) 2<br/> 00000020:00000001:12.0:1400134007.926428:0:19741:0:(cl_page.c:385:cl_page_put()) page@ffff880376c65c00<span class="error">&#91;2 ffff880381c2c0f0 0 0 1 (null) (null) 0x0&#93;</span><br/> 00000020:00000001:12.0:1400134007.926429:0:19741:0:(cl_page.c:385:cl_page_put()) 2<br/> 00000020:00000001:12.0:1400134007.926455:0:19741:0:(cl_page.c:385:cl_page_put()) page@ffff880386096000<span class="error">&#91;2 ffff880381c2c0f0 0 0 1 (null) (null) 0x0&#93;</span><br/> 00000020:00000001:12.0:1400134007.926456:0:19741:0:(cl_page.c:385:cl_page_put()) 2<br/> 00000020:00000001:12.0:1400134007.926482:0:19741:0:(cl_page.c:385:cl_page_put()) page@ffff880431d7e800<span class="error">&#91;2 ffff880381c2c0f0 0 0 1 (null) (null) 0x0&#93;</span><br/> 00000020:00000001:12.0:1400134007.926483:0:19741:0:(cl_page.c:385:cl_page_put()) 2</p> <p>And all of them had a refcnt of 2 at the time, so they were unshrinkable at the last shrink, are now shrinkable, but nobody has called shrink since the pages were put...</p> <p>So how did put get called?</p> <p>Here is the last call exiting, followed by it's call chain:</p> <p> 00000020:00000001:12.0:1400134007.926483:0:19741:0:(cl_page.c:400:cl_page_put()) Process leaving<br/> 00000020:00000001:12.0:1400134007.926484:0:19741:0:(cl_page.c:985:cl_page_completion()) Process leaving<br/> 00000008:00000001:12.0:1400134007.926484:0:19741:0:(osc_cache.c:1316:osc_completion()) Process leaving (rc=0 : 0 : 0)<br/> 00000008:00000001:12.0:1400134007.926486:0:19741:0:(osc_cache.c:1896:osc_ap_completion()) Process leaving<br/> [ ... ]<br/> 00000008:00000001:12.0:1400134007.926499:0:19741:0:(osc_cache.c:851:osc_extent_finish()) Process leaving (rc=0 : 0 : 0)</p> <p>So... osc_extent_finish() called osc_lru_add_batch, responsible for moving pages from the busy list the regular list - then scheduling a shrink - right off the bat. Since then, the pages have been put, and could be shrunk, but nobody has requested a shrink.</p> <p>That seems wrong to me.</p> <p>Extra notes, with structure dumps, disassembly, etc.</p> <p>Indeed, we should schedule a shrink outside of osc_lru_add_batch(), it should be in osc_extent_finish() after osc_ap_completion() is called.</p> <p>Can you please create a patch for this, Stephen?</p> <p>The other part is adding a test, which should just be a modified sanity 101a. I may be able to cook something up this week.</p> <p>Do you have an opinion on whether the shrink should be inline or a via ptlrpcd request? I'm thinking this can be inline:</p> <p>if osc_cache_too_much<br/> osc_lru_shrink</p> <p>but there may be advantages to queuing a request instead:</p> <p>if osc_cache_too_much<br/> ptlrpcd_queue_work cl_lru_work</p> <p>Just an update:</p> <p>I wrote a test for this, which worked. It also failed (in a different way) with the patch I wrote.<br/> The patch I wrote also failed on another test... So there's a good bit of work to do understanding these cases.</p> <p>I intend to get back this, but have some pressing customer issues to deal with.</p> <p>Hello Stephen,<br/> Is this still a live issue for you?<br/> If so, can you please give us an update on where you are with it.<br/> Many thanks,<br/> ~ jfc.</p> <p>This is not really a problem for us - it's just a bug I stumbled across. I wrote a test that triggers another bug that I have not fully investigated, and a fix that crashes in a later acceptance test.</p> <p>Our customers suffer from too much memory, not too little, so I am unlikely to be able to spend additional time on this. I'd be happy to pass my incomplete work along for someone to pick up.</p> <p>Thank you Stephen.<br/> We'll leave the ticket open for the time being and please let us know if it rises in priority in the future.<br/> ~ jfc.</p> <p>We can reopen this ticket if this issue becomes a priority.</p> <p>~ jfc.</p> Development Rank 1|hzwnlj: Rank (Obsolete) 14168 Severity