https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3416 Lustre <p>It looks like <a href="https://jira.whamcloud.com/browse/LU-2576" title="Hangs in osc_enter_cache due to dirty pages not being flushed" class="issue-link" data-issue-key="LU-2576"><del>LU-2576</del></a> is back again. The problem went away for a while, seemingly thanks to the patch from <a href="https://jira.whamcloud.com/browse/LU-2576" title="Hangs in osc_enter_cache due to dirty pages not being flushed" class="issue-link" data-issue-key="LU-2576"><del>LU-2576</del></a>. I note that a later fix for stack overflow in <a href="https://jira.whamcloud.com/browse/LU-2859" title="Stack overflow issues on x86_64 clients in memory reclaim during writes" class="issue-link" data-issue-key="LU-2859"><del>LU-2859</del></a> changed the same lines where the fix was applied, so perhaps that reintroduced the problem?</p> <p>We are seeing hangs during writes on BG/Q hardware. We find tasks that appear to be stuck sleeping indefinitely here:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2013-05-29 16:15:36.363547 sysiod S 00000fffa85363bc 0 4111 3070 0x00000000 2013-05-29 16:15:36.363582 Call Trace: 2013-05-29 16:15:36.363617 [c0000002e1e12440] [c000000302e95cc8] 0xc000000302e95cc8 (unreliable) 2013-05-29 16:15:36.363653 [c0000002e1e12610] [c000000000008de0] .__switch_to+0xc4/0x100 2013-05-29 16:15:36.363688 [c0000002e1e126a0] [c00000000044dc68] .schedule+0x858/0x9c0 2013-05-29 16:15:36.363723 [c0000002e1e12950] [80000000004820a0] .cfs_waitq_wait+0x10/0x30 [libcfs] 2013-05-29 16:15:36.363758 [c0000002e1e129c0] [80000000015b3ccc] .osc_enter_cache+0xb6c/0x1410 [osc] 2013-05-29 16:15:36.363793 [c0000002e1e12ba0] [80000000015bbf30] .osc_queue_async_io+0xcd0/0x2690 [osc] 2013-05-29 16:15:36.363828 [c0000002e1e12db0] [8000000001598598] .osc_page_cache_add+0xf8/0x2a0 [osc] 2013-05-29 16:15:36.363863 [c0000002e1e12e70] [8000000000a04248] .cl_page_cache_add+0xf8/0x420 [obdclass] 2013-05-29 16:15:36.363898 [c0000002e1e12fa0] [800000000179ed28] .lov_page_cache_add+0xc8/0x340 [lov] 2013-05-29 16:15:36.363934 [c0000002e1e13070] [8000000000a04248] .cl_page_cache_add+0xf8/0x420 [obdclass] 2013-05-29 16:15:36.363968 [c0000002e1e131a0] [8000000001d2ac74] .vvp_io_commit_write+0x464/0x910 [lustre] 2013-05-29 16:15:36.364003 [c0000002e1e132c0] [8000000000a1df6c] .cl_io_commit_write+0x11c/0x2d0 [obdclass] 2013-05-29 16:15:36.364038 [c0000002e1e13380] [8000000001cebc00] .ll_commit_write+0x120/0x3e0 [lustre] 2013-05-29 16:15:36.364074 [c0000002e1e13450] [8000000001d0f134] .ll_write_end+0x34/0x80 [lustre] 2013-05-29 16:15:36.364109 [c0000002e1e134e0] [c000000000097238] .generic_file_buffered_write+0x1f4/0x388 2013-05-29 16:15:36.364143 [c0000002e1e13620] [c000000000097928] .__generic_file_aio_write+0x374/0x3d8 2013-05-29 16:15:36.364178 [c0000002e1e13720] [c000000000097a04] .generic_file_aio_write+0x78/0xe8 2013-05-29 16:15:36.364213 [c0000002e1e137d0] [8000000001d2df00] .vvp_io_write_start+0x170/0x3b0 [lustre] 2013-05-29 16:15:36.364248 [c0000002e1e138a0] [8000000000a1849c] .cl_io_start+0xcc/0x220 [obdclass] 2013-05-29 16:15:36.364283 [c0000002e1e13940] [8000000000a202a4] .cl_io_loop+0x194/0x2c0 [obdclass] 2013-05-29 16:15:36.364317 [c0000002e1e139f0] [8000000001ca0780] .ll_file_io_generic+0x4f0/0x850 [lustre] 2013-05-29 16:15:36.364352 [c0000002e1e13b30] [8000000001ca0f64] .ll_file_aio_write+0x1d4/0x3a0 [lustre] 2013-05-29 16:15:36.364387 [c0000002e1e13c00] [8000000001ca1280] .ll_file_write+0x150/0x320 [lustre] 2013-05-29 16:15:36.364422 [c0000002e1e13ce0] [c0000000000d4328] .vfs_write+0xd0/0x1c4 2013-05-29 16:15:36.364458 [c0000002e1e13d80] [c0000000000d4518] .SyS_write+0x54/0x98 2013-05-29 16:15:36.364492 [c0000002e1e13e30] [c000000000000580] syscall_exit+0x0/0x2c </pre> </div></div> <p>This was with Lustre <a href="https://github.com/chaos/lustre/tree/2.4.0-RC1_3chaos" class="external-link" target="_blank" rel="nofollow noopener">2.4.0-RC1_3chaos</a>.</p> Lustre 2.4.0-RC1_3chaos. <a href="https://github.com/chaos/lustre/tree/2.4.0-RC1_3chaos,">https://github.com/chaos/lustre/tree/2.4.0-RC1_3chaos,</a> ZFS servers LU-3416

Hangs on write in osc_enter_cache()

Bug Critical Resolved Fixed Niu Yawei Christopher Morrone mq313 Thu, 30 May 2013 00:07:14 +0000 Thu, 20 Oct 2022 13:42:31 +0000 Thu, 29 Aug 2013 07:00:03 +0000 Lustre 2.4.0 Lustre 2.4.1 Lustre 2.5.0 Lustre 2.4.2 0 12 <p>Just as reported in <a href="https://jira.whamcloud.com/browse/LU-2576" title="Hangs in osc_enter_cache due to dirty pages not being flushed" class="issue-link" data-issue-key="LU-2576"><del>LU-2576</del></a>, the processes will wake up and make progress if a sysadmin issues:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>echo 3 &gt; /proc/sys/vm/drop_caches</pre> </div></div> <p>Hi, Chris</p> <ul class="alternate" type="square"> <li>Is the fix of <a href="https://jira.whamcloud.com/browse/LU-2909" title="Failure on test suite sanity-benchmark test_fsx: fsx bus error, core dumped" class="issue-link" data-issue-key="LU-2909"><del>LU-2909</del></a> also applied?</li> </ul> <ul class="alternate" type="square"> <li>Is there any lli_write_mutex changes in your branch, like <a href="https://jira.whamcloud.com/browse/LU-3261" title="Threads stuck in osc_enter_cache when testing LU-1669" class="issue-link" data-issue-key="LU-3261"><del>LU-3261</del></a>?</li> </ul> <ul class="alternate" type="square"> <li>What's the 'cur_dirty_bytes' &amp; 'max_dirty_mb' of each OSC? (this can be checked by the OSC proc file)</li> </ul> <ul class="alternate" type="square"> <li>What's the 'dirty_expire_centisecs' of your system? Is the thread being stuck longer than 'dirty_expire_centisecs' (30 seconds by defatult)?</li> </ul> <p>Thanks.</p> <blockquote><p>Is the fix of <a href="https://jira.whamcloud.com/browse/LU-2909" title="Failure on test suite sanity-benchmark test_fsx: fsx bus error, core dumped" class="issue-link" data-issue-key="LU-2909"><del>LU-2909</del></a> also applied?</p></blockquote> <p>I pointed you to the exact tag that we are running. Our patches in the branch are clearly based at 2.4.0-RC1. So yes, of course we have that fix applied. You guys applied it.</p> <blockquote><p>Is there any lli_write_mutex changes in your branch, like <a href="https://jira.whamcloud.com/browse/LU-3261" title="Threads stuck in osc_enter_cache when testing LU-1669" class="issue-link" data-issue-key="LU-3261"><del>LU-3261</del></a>?</p></blockquote> <p>Again, see the tag that I included in the bug report. This branch does not include any experimental work to improve the <a href="https://jira.whamcloud.com/browse/LU-1669" title="lli-&gt;lli_write_mutex (single shared file performance)" class="issue-link" data-issue-key="LU-1669"><del>LU-1669</del></a> problem, which is what Prakash was talking about in <a href="https://jira.whamcloud.com/browse/LU-3261" title="Threads stuck in osc_enter_cache when testing LU-1669" class="issue-link" data-issue-key="LU-3261"><del>LU-3261</del></a>. I also ran:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>git diff 2.4.0-RC1 2.4.0-RC1_3chaos | grep lli_write_mutex</pre> </div></div> <p>and it did not report any results, so no direct changes that I am aware of.</p> <blockquote><p>What's the 'cur_dirty_bytes' &amp; 'max_dirty_mb' of each OSC? (this can be checked by the OSC proc file)</p></blockquote> <p>max_dirty_mb is 32 for all OSC. I didn't get cur_dirty_bytes. I did capture dump_page_cache. See attached. Keep in mind that BG/Q is ppc64, and pages are 64K.</p> <p>Looks like there were 7129 dirty pages. At 64K each, that is 456,256 KiB. There are 16 OSTs, so on average there there were 28,516 KiB dirty. It seems pretty likely that some of the OSTs were at the max_dirty_mb limit.</p> <blockquote><p>What's the 'dirty_expire_centisecs' of your system? Is the thread being stuck longer than 'dirty_expire_centisecs' (30 seconds by defatult)?</p></blockquote> <p>3000</p> <p>And yes, the threads are stuck for much, much longer than the 30 second dirty_export_centiseconds value. As I said, they appear to be stuck indefinitely. My guess is that they will never make progress without external intervention. They appeared to be stuck for at least 2 hours before I intervened by triggering drop_caches.</p> <p>A "sync" command on the command line also successfully clears the outstanding dirty data.</p> <p>Here is cur_dirty_bytes for one of the clients that is hanging on write:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>osc.lsrzb-OST0001-osc-c0000003c6ed3400.cur_dirty_bytes=27852800 osc.lsrzb-OST0002-osc-c0000003c6ed3400.cur_dirty_bytes=25755648 osc.lsrzb-OST0003-osc-c0000003c6ed3400.cur_dirty_bytes=28246016 osc.lsrzb-OST0004-osc-c0000003c6ed3400.cur_dirty_bytes=26935296 osc.lsrzb-OST0005-osc-c0000003c6ed3400.cur_dirty_bytes=26411008 osc.lsrzb-OST0006-osc-c0000003c6ed3400.cur_dirty_bytes=29229056 osc.lsrzb-OST0007-osc-c0000003c6ed3400.cur_dirty_bytes=26083328 osc.lsrzb-OST0008-osc-c0000003c6ed3400.cur_dirty_bytes=27459584 osc.lsrzb-OST0009-osc-c0000003c6ed3400.cur_dirty_bytes=27787264 osc.lsrzb-OST000a-osc-c0000003c6ed3400.cur_dirty_bytes=29163520 osc.lsrzb-OST000b-osc-c0000003c6ed3400.cur_dirty_bytes=27656192 osc.lsrzb-OST000c-osc-c0000003c6ed3400.cur_dirty_bytes=27131904 osc.lsrzb-OST000d-osc-c0000003c6ed3400.cur_dirty_bytes=25362432 osc.lsrzb-OST000e-osc-c0000003c6ed3400.cur_dirty_bytes=27525120 osc.lsrzb-OST000f-osc-c0000003c6ed3400.cur_dirty_bytes=27197440 osc.lsrzb-OST0010-osc-c0000003c6ed3400.cur_dirty_bytes=27262976 </pre> </div></div> <p>The dirty data is actually pretty evenly spread. There were 512 files being written through this client at the time of the hang.</p> <p>I also note that the flush-lustre-1 thread is waking up regularly, but does not seem to result in any of the dirty data being written. I enabled full lustre debugging watched flush-lustre-1 run for a bit in top, and then dumped the lustre log. See attached file <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/12969/12969_rzuseqio14_lustre_log.txt" title="rzuseqio14_lustre_log.txt attached to LU-3416">rzuseqio14_lustre_log.txt^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span>. The flush-lustre-1 process was PID 10295 at the time of the log dump.</p> <p>I haven't stepped through the code and logs long enough yet to understand why it is doing so much but failing to trigger the sending of the dirty data.</p> <p>The log (rzuseqio14_lustre_log.txt) shows that ll_writepages() didn't trigger any dirty flush at all:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">... 00000008:00000001:21.0:1369956379.455717:3872:10295:0:(osc_io.c:611:osc_io_fsync_start()) <span class="code-object">Process</span> entered 00000008:00000001:21.0:1369956379.455724:4368:10295:0:(osc_cache.c:2914:osc_cache_writeback_range()) <span class="code-object">Process</span> entered 00000008:00000020:21.0:1369956379.455732:4368:10295:0:(osc_cache.c:3010:osc_cache_writeback_range()) obj c0000002eacf46e0 ready 0|-|- wr 15|-|- rd 0|- cache page out. 00000008:00000001:21.0:1369956379.455744:4496:10295:0:(osc_cache.c:3011:osc_cache_writeback_range()) <span class="code-object">Process</span> leaving (rc=0 : 0 : 0) ... </pre> </div></div> <p>I think these dirty pages are supposed to be on the oo_urgent_exts, but seems they were not on either oo_hp_exts or oo_urgent_exts. Jinshan, any idea how this could happen?</p> <p>Obviously there existed an active extent so that osc_io_fsync_start() didn't move it into urgent list.</p> <p>Hi Chris, just to clarify, both `echo 3 &gt; drop_caches' and sync will trigger it out of hang state?</p> <p>It looks like this issue can only be seen on ppc clients, so let's drop the priority to not block 2.4 release while we're working on it.</p> <blockquote><p>Hi Chris, just to clarify, both `echo 3 &gt; drop_caches' and sync will trigger it out of hang state?</p></blockquote> <p>Yes, that is correct.</p> <blockquote><p>It looks like this issue can only be seen on ppc clients, so let's drop the priority to not block 2.4 release while we're working on it.</p></blockquote> <p>I disagree. Just because we discovered it on ppc clients doesn't mean it is a ppc-only problem.</p> <p>Hi Chris, if you see this issue again, try if `echo 1 &gt; drop_caches' can also release the hang state. Also it will be helpful to check how much grants are there at the corresponding osc.</p> <p>here is another global dirty page parameter: obd_dirty_pages and obd_max_dirty_pages. Can you please check it as well?</p> <p>Chris, how frequent are the hangs, and how many systems does this affect? We've had several days of SWL runs on recent Lustre tags without seeing anything similar on x86_64 clients. While I don't want to minimize the importance of this issue, the main question is whether this problem is serious and widespread enough that it needs to block 2.4.0 from being released to any users, or whether it is possible to fix this after the 2.4.0 release?</p> <p>LLNL will be running a patched version of Lustre regardless of whether this bug is fixed before 2.4.0 or afterward, so if this is more prevalent with PPC clients or ZFS servers it will not affect a majority of Lustre users. Also, by making Lustre 2.4.0 available to a wider user base the 2.4.0 code will get more testing and fixes, which helps LLNL as well. This bug is the only one holding the 2.4.0 release at this point, so we'd really prefer to drop it from the blockers list and release 2.4.0.</p> <p>Andreas</p> <p>I just gave Chris a call and he agrees that, while this may not necessarily be a PPC issue, that the amount of testing elsewhere demonstrates that this is a rare issue to hit and one that we can defer to 2.4.1.</p> <p>Peter</p> <p>Where would I find those?</p> <p>Here's what I found in proc:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rzuseqio16-ib0@root:lctl get_param 'osc.*.cur_dirty_bytes' osc.lsrzb-OST0001-osc-c0000003c737a400.cur_dirty_bytes=27656192 osc.lsrzb-OST0002-osc-c0000003c737a400.cur_dirty_bytes=27983872 osc.lsrzb-OST0003-osc-c0000003c737a400.cur_dirty_bytes=30146560 osc.lsrzb-OST0004-osc-c0000003c737a400.cur_dirty_bytes=28508160 osc.lsrzb-OST0005-osc-c0000003c737a400.cur_dirty_bytes=28901376 osc.lsrzb-OST0006-osc-c0000003c737a400.cur_dirty_bytes=30277632 osc.lsrzb-OST0007-osc-c0000003c737a400.cur_dirty_bytes=29294592 osc.lsrzb-OST0008-osc-c0000003c737a400.cur_dirty_bytes=27394048 osc.lsrzb-OST0009-osc-c0000003c737a400.cur_dirty_bytes=29884416 osc.lsrzb-OST000a-osc-c0000003c737a400.cur_dirty_bytes=28835840 osc.lsrzb-OST000b-osc-c0000003c737a400.cur_dirty_bytes=29753344 osc.lsrzb-OST000c-osc-c0000003c737a400.cur_dirty_bytes=27262976 osc.lsrzb-OST000d-osc-c0000003c737a400.cur_dirty_bytes=29818880 osc.lsrzb-OST000e-osc-c0000003c737a400.cur_dirty_bytes=27590656 osc.lsrzb-OST000f-osc-c0000003c737a400.cur_dirty_bytes=28049408 osc.lsrzb-OST0010-osc-c0000003c737a400.cur_dirty_bytes=27983872 rzuseqio16-ib0@root:lctl get_param 'osc.*.cur_grant_bytes' osc.lsrzb-OST0001-osc-c0000003c737a400.cur_grant_bytes=33554432 osc.lsrzb-OST0002-osc-c0000003c737a400.cur_grant_bytes=51511296 osc.lsrzb-OST0003-osc-c0000003c737a400.cur_grant_bytes=41746432 osc.lsrzb-OST0004-osc-c0000003c737a400.cur_grant_bytes=36831232 osc.lsrzb-OST0005-osc-c0000003c737a400.cur_grant_bytes=38010880 osc.lsrzb-OST0006-osc-c0000003c737a400.cur_grant_bytes=35848192 osc.lsrzb-OST0007-osc-c0000003c737a400.cur_grant_bytes=33488896 osc.lsrzb-OST0008-osc-c0000003c737a400.cur_grant_bytes=37093376 osc.lsrzb-OST0009-osc-c0000003c737a400.cur_grant_bytes=63897600 osc.lsrzb-OST000a-osc-c0000003c737a400.cur_grant_bytes=35454976 osc.lsrzb-OST000b-osc-c0000003c737a400.cur_grant_bytes=37683200 osc.lsrzb-OST000c-osc-c0000003c737a400.cur_grant_bytes=33488896 osc.lsrzb-OST000d-osc-c0000003c737a400.cur_grant_bytes=34275328 osc.lsrzb-OST000e-osc-c0000003c737a400.cur_grant_bytes=33751040 osc.lsrzb-OST000f-osc-c0000003c737a400.cur_grant_bytes=35454976 osc.lsrzb-OST0010-osc-c0000003c737a400.cur_grant_bytes=40828928 rzuseqio16-ib0@root:lctl get_param 'osc.*.cur_lost_grant_bytes' osc.lsrzb-OST0001-osc-c0000003c737a400.cur_lost_grant_bytes=0 osc.lsrzb-OST0002-osc-c0000003c737a400.cur_lost_grant_bytes=0 osc.lsrzb-OST0003-osc-c0000003c737a400.cur_lost_grant_bytes=0 osc.lsrzb-OST0004-osc-c0000003c737a400.cur_lost_grant_bytes=1073803264 osc.lsrzb-OST0005-osc-c0000003c737a400.cur_lost_grant_bytes=0 osc.lsrzb-OST0006-osc-c0000003c737a400.cur_lost_grant_bytes=1073803264 osc.lsrzb-OST0007-osc-c0000003c737a400.cur_lost_grant_bytes=0 osc.lsrzb-OST0008-osc-c0000003c737a400.cur_lost_grant_bytes=1073803264 osc.lsrzb-OST0009-osc-c0000003c737a400.cur_lost_grant_bytes=0 osc.lsrzb-OST000a-osc-c0000003c737a400.cur_lost_grant_bytes=0 osc.lsrzb-OST000b-osc-c0000003c737a400.cur_lost_grant_bytes=0 osc.lsrzb-OST000c-osc-c0000003c737a400.cur_lost_grant_bytes=1073803264 osc.lsrzb-OST000d-osc-c0000003c737a400.cur_lost_grant_bytes=1073803264 osc.lsrzb-OST000e-osc-c0000003c737a400.cur_lost_grant_bytes=2147606528 osc.lsrzb-OST000f-osc-c0000003c737a400.cur_lost_grant_bytes=0 osc.lsrzb-OST0010-osc-c0000003c737a400.cur_lost_grant_bytes=0 rzuseqio16-ib0@root:lctl get_param 'osc.*.max_dirty_mb' osc.lsrzb-OST0001-osc-c0000003c737a400.max_dirty_mb=32 osc.lsrzb-OST0002-osc-c0000003c737a400.max_dirty_mb=32 osc.lsrzb-OST0003-osc-c0000003c737a400.max_dirty_mb=32 osc.lsrzb-OST0004-osc-c0000003c737a400.max_dirty_mb=32 osc.lsrzb-OST0005-osc-c0000003c737a400.max_dirty_mb=32 osc.lsrzb-OST0006-osc-c0000003c737a400.max_dirty_mb=32 osc.lsrzb-OST0007-osc-c0000003c737a400.max_dirty_mb=32 osc.lsrzb-OST0008-osc-c0000003c737a400.max_dirty_mb=32 osc.lsrzb-OST0009-osc-c0000003c737a400.max_dirty_mb=32 osc.lsrzb-OST000a-osc-c0000003c737a400.max_dirty_mb=32 osc.lsrzb-OST000b-osc-c0000003c737a400.max_dirty_mb=32 osc.lsrzb-OST000c-osc-c0000003c737a400.max_dirty_mb=32 osc.lsrzb-OST000d-osc-c0000003c737a400.max_dirty_mb=32 osc.lsrzb-OST000e-osc-c0000003c737a400.max_dirty_mb=32 osc.lsrzb-OST000f-osc-c0000003c737a400.max_dirty_mb=32 osc.lsrzb-OST0010-osc-c0000003c737a400.max_dirty_mb=32 </pre> </div></div> <blockquote><p>Chris, how frequent are the hangs, and how many systems does this affect?</p></blockquote> <p>Like Peter said, I don't particularly care about if this problem gets fixed in 2.4.0. But it is a very frequent problem. In fact, it may have never been fixed like we thought a few months ago. I've learned that a sysadmins was running "echo 3 &gt; /proc/sys/vm/drop_caches" to get around the problem. But I was not aware that it was still a problem.</p> <p>It affects all of our production systems that are running pre-2.4. Of course, we're only doing that on BG/Q systems with ZFS. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> I just don't want to call this a ppc issue until we understand it better. I kind of suspect that it has more to do with the scale of simultaneous writers or something, rather than being endianness or page size related. I could be wrong. Too soon to say, really.</p> <p>Chris, I'm just on my phone so can't verify, but I believe the parameters Jinshan is referencing are global parameters in llite or at the top level. </p> <p>This is from the same useqio16 node as last Friday. Still hung in the same place after the full weekend. The obd_dirty_pages number exactly matches the sum of the 16 cur_dirty_bytes numbers (keeping in mind that pages are 64K on this platform): 459,341,824 bytes.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; print obd_dirty_pages $8 = { counter = 7009 } crash&gt; print obd_max_dirty_pages $9 = 131071 </pre> </div></div> <p>Any ideas?</p> <p>Chirs, I was looking at related code these days, but didn't find any clue so far. btw, were there many truncate operations?</p> <p>I can take a look at this issue if it's okay for you guys.</p> <p>Yes please Jinshan!</p> <p>Please apply patch: <a href="http://review.whamcloud.com/6554" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6554</a> to your tree to print accurate information when this problem happens. I tend to suspect a wakeup is missed somewhere but need to verify.</p> <p>Ok, I'll apply that and get the info.</p> <p>But there certainly seems to be an additional problem here. The Lustre client seems to be rather poorly integrated with the kernel's way of doing things. The kernel is clearly trying to write out these dirty pages, and Lustre is completely ignores the kernel. That is Bad. I would like to see that problem fixed as well.</p> <p>Hi Chris, not sure I understand. You mean the dirty page won't be written back if EDQUOT is returned? Actually in vvp_io_commit_write(), if EDQUOT is seen, Lustre will try to write the page in sync mode.</p> <p>I don't know what EDQUOT is, so no. I am referring to the kernel's normal BDI mechanism for regularly pushing dirty to disk. The calls are being made into Lustre, but Lustre decides to ignore them. I don't understand the whole call stack yet (by a long shot), but clearly the kernel's normal mechanisms get the Lustre code into osc_cache_writeback_range, where it notes that dirty pages exist. However, lustre chooses to do absolutely nothing with that information, rather than obeying the kernel's request to write out the dirty pages. That has got to have all kinds of bad implications for low memory situations on nodes. And not too surprising, we <em>have</em> all kinds of memory problems with the new Lustre client.</p> <p>Do you mean the call path of -&gt;writepage()? In this case, it will mark the dirty page as urgent and then wait for it to be written ll_writepage().</p> <p>Yes, I agree that the current client has problem of memory and performance, and I will give you an explanation soon.</p> <p>Jinshan, your patch does hit ten minutes after the hang. Here are the 6 lines that popped out on the console from one of the clients:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2013-06-06 15:05:48.739336 {DefaultControlEventListener} [mmcs]{22}.1.1: LustreError: 4820:0:(osc_cache.c:1559:osc_enter_cache()) lsrzb-OST000d-osc-c0000003c7540c00: { dirty: 33554432/33554432 dirty_pages: 6441/131071 unstable_pages: 0/131071 dropped: 2147606528 avail: 27459584, reserved: 0, flight: 0 } try to reserve 65536. 2013-06-06 15:05:48.740174 {DefaultControlEventListener} [mmcs]{22}.4.0: LustreError: 4726:0:(osc_cache.c:1559:osc_enter_cache()) lsrzb-OST000d-osc-c0000003c7540c00: { dirty: 33554432/33554432 dirty_pages: 6441/131071 unstable_pages: 0/131071 dropped: 2147606528 avail: 27459584, reserved: 0, flight: 0 } try to reserve 65536. 2013-06-06 15:05:48.740565 {DefaultControlEventListener} [mmcs]{22}.2.3: LustreError: 4751:0:(osc_cache.c:1559:osc_enter_cache()) lsrzb-OST000d-osc-c0000003c7540c00: { dirty: 33554432/33554432 dirty_pages: 6441/131071 unstable_pages: 0/131071 dropped: 0 avail: 27459584, reserved: 0, flight: 0 } try to reserve 65536. 2013-06-06 15:05:48.740946 {DefaultControlEventListener} [mmcs]{22}.7.2: LustreError: 4733:0:(osc_cache.c:1559:osc_enter_cache()) lsrzb-OST000d-osc-c0000003c7540c00: { dirty: 33554432/33554432 dirty_pages: 6441/131071 unstable_pages: 0/131071 dropped: 0 avail: 27459584, reserved: 0, flight: 0 } try to reserve 65536. 2013-06-06 15:05:48.741355 {DefaultControlEventListener} [mmcs]{22}.13.0: LustreError: 4830:0:(osc_cache.c:1559:osc_enter_cache()) lsrzb-OST000d-osc-c0000003c7540c00: { dirty: 33554432/33554432 dirty_pages: 6441/131071 unstable_pages: 0/131071 dropped: 0 avail: 27459584, reserved: 0, flight: 0 } try to reserve 65536. 2013-06-06 15:05:48.741649 {DefaultControlEventListener} [mmcs]{22}.10.0: LustreError: 4724:0:(osc_cache.c:1559:osc_enter_cache()) lsrzb-OST000d-osc-c0000003c7540c00: { dirty: 33554432/33554432 dirty_pages: 6441/131071 unstable_pages: 0/131071 dropped: 0 avail: 27459584, reserved: 0, flight: 1 } try to reserve 65536. </pre> </div></div> <p>You patch seems to keep data flowing. But with 10 minute pauses here and there, of course. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Do you understand how to fix this, then?</p> <p>Also, be aware that we are using the following settings on our BG/Q clients:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>sysctl vm.min_free_kbytes=1048576 /usr/sbin/lctl set_param "llite.*.max_cached_mb=4096" /usr/sbin/lctl set_param "llite.*.max_read_ahead_mb=10" </pre> </div></div> <p>About the cache, another useful information would be:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lctl get_param osc.*.osc_cached_mb </pre> </div></div> <p>so that I know how busy it is.</p> <p>Apparently I should have adjusted cl_dirty in osc_unreserve_grant() as well, working on a patch...</p> <p>Can you please try patch set 2 of <a href="http://review.whamcloud.com/6554?" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6554?</a></p> <p>Yes, I'll try it out. Thanks, Jinshan.</p> <p>The problem seems to have disappeared with Patch Set 2 in place.</p> <p>But now write performance is unacceptably bad. Four BG/Q clients are only pushing a <em>total</em> of around 720 MB/s (180MB/S each). That number should be closer to 1GB/s per client.</p> <p>this patch should have nothing to do with the performance. Maybe you can try to roll back to unpatched version and see if it's due to env settings. Of course, we can simplify the problem by using dd to check performance.</p> <p>If you can still see the performance drop, please try to collect a log with D_CACHE enable (lctl set_param debug=cache), I will take a look to make sure it's in the correct path.</p> <p>I double checked, and I am certain that the patch is the only difference. FYI, I've moved to Lustre version <a href="https://github.com/chaos/lustre/tree/2.4.0-RC2_2chaos" class="external-link" target="_blank" rel="nofollow noopener">2.4.0-RC2_2chaos</a> on both clients and servers.</p> <p>Without the patch, I see write throughput burst to 4.4GB/s intially (as seen in ltop), and then taper off to a fairly constant rate in the low 3GB/s range.</p> <p>With the patch, I see 720 MB/s. (These are aggregate throughput for 4 BG/Q clients.)</p> <p>Keep in mind that our tree has the unstable page tracking changes. I don't know if there is a relation there or not, but you should be aware.</p> <p>Without your patch, at the beginning of an IOR write phase, you might see something like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rzuseqio13-ib0@morrone:lctl get_param "osc.*.cur_dirty_bytes" osc.lsrzb-OST0001-osc-c0000003e18f6400.cur_dirty_bytes=33554432 osc.lsrzb-OST0002-osc-c0000003e18f6400.cur_dirty_bytes=33554432 osc.lsrzb-OST0003-osc-c0000003e18f6400.cur_dirty_bytes=33554432 osc.lsrzb-OST0004-osc-c0000003e18f6400.cur_dirty_bytes=33554432 osc.lsrzb-OST0005-osc-c0000003e18f6400.cur_dirty_bytes=33554432 osc.lsrzb-OST0006-osc-c0000003e18f6400.cur_dirty_bytes=33161216 osc.lsrzb-OST0007-osc-c0000003e18f6400.cur_dirty_bytes=33554432 osc.lsrzb-OST0008-osc-c0000003e18f6400.cur_dirty_bytes=32899072 osc.lsrzb-OST0009-osc-c0000003e18f6400.cur_dirty_bytes=33554432 osc.lsrzb-OST000a-osc-c0000003e18f6400.cur_dirty_bytes=33554432 osc.lsrzb-OST000b-osc-c0000003e18f6400.cur_dirty_bytes=33554432 osc.lsrzb-OST000c-osc-c0000003e18f6400.cur_dirty_bytes=33030144 osc.lsrzb-OST000d-osc-c0000003e18f6400.cur_dirty_bytes=33554432 osc.lsrzb-OST000e-osc-c0000003e18f6400.cur_dirty_bytes=12582912 osc.lsrzb-OST000f-osc-c0000003e18f6400.cur_dirty_bytes=33554432 osc.lsrzb-OST0010-osc-c0000003e18f6400.cur_dirty_bytes=33554432 </pre> </div></div> <p>And that seems fairly normal.</p> <p>But <em>with</em> patch set 2 of <a href="http://review.whamcloud.com/6554" class="external-link" target="_blank" rel="nofollow noopener">6554</a> applied I see these bogus numbers:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rzuseqlac2:~/BG-191$ rsh rzuseqio13 '/usr/sbin/lctl get_param "osc.*.cur_dirty_bytes"' osc.lsrzb-OST0001-osc-c0000003ec957c00.cur_dirty_bytes=18446744073707454464 osc.lsrzb-OST0002-osc-c0000003ec957c00.cur_dirty_bytes=18446744073707454464 osc.lsrzb-OST0003-osc-c0000003ec957c00.cur_dirty_bytes=18446744073707454464 osc.lsrzb-OST0004-osc-c0000003ec957c00.cur_dirty_bytes=18446744073703260160 osc.lsrzb-OST0005-osc-c0000003ec957c00.cur_dirty_bytes=18446744073707454464 osc.lsrzb-OST0006-osc-c0000003ec957c00.cur_dirty_bytes=18446744073707454464 osc.lsrzb-OST0007-osc-c0000003ec957c00.cur_dirty_bytes=18446744073706930176 osc.lsrzb-OST0008-osc-c0000003ec957c00.cur_dirty_bytes=18446744073707454464 osc.lsrzb-OST0009-osc-c0000003ec957c00.cur_dirty_bytes=18446744073707454464 osc.lsrzb-OST000a-osc-c0000003ec957c00.cur_dirty_bytes=18446744073707454464 osc.lsrzb-OST000b-osc-c0000003ec957c00.cur_dirty_bytes=18446744073706536960 osc.lsrzb-OST000c-osc-c0000003ec957c00.cur_dirty_bytes=18446744073707454464 osc.lsrzb-OST000d-osc-c0000003ec957c00.cur_dirty_bytes=18446744073707454464 osc.lsrzb-OST000e-osc-c0000003ec957c00.cur_dirty_bytes=18446744073707454464 osc.lsrzb-OST000f-osc-c0000003ec957c00.cur_dirty_bytes=18446744073707454464 osc.lsrzb-OST0010-osc-c0000003ec957c00.cur_dirty_bytes=18446744073707454464 </pre> </div></div> <p>I'm guessing that those large numbers are simply an arithmetic problem, where we're allowing the number to become negative. That somehow results in very slow write throughput, perhaps because all operations are not completely synchronous?</p> <p>ah yes, indeed. The dirty bytes are actually negative numbers. I'll check it.</p> <p>I found the problem. Working on a patch.</p> <p>Will you please try patch set 3 and see if it helps.</p> <p>Performance (and presumably the negative number) problem is gone, but it is still hanging for ten minutes like with patch one. I'll get post some info tomorrow.</p> <p>Here's output from one of the debug messages before I leave for the day:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 3709:0:(osc_cache.c:1563:osc_enter_cache()) lsrzb-OST0004-osc-c0000003c70ca400: grant { dirty: 33554432/33554432 dirty_pages: 6934/131071 unstable_pages: 0/131071 dropped: 0 avail: 45613056, reserved: 0, flight: 0 } lru {in list: 2407, left: 515, waiters: 0 }try to reserve 65536. LustreError: 3736:0:(osc_cache.c:1563:osc_enter_cache()) lsrzb-OST0004-osc-c0000003c70ca400: grant { dirty: 33554432/33554432 dirty_pages: 6934/131071 unstable_pages: 0/131071 dropped: 0 avail: 45613056, reserved: 0, flight: 0 } lru {in list: 2407, left: 515, waiters: 0 }try to reserve 65536. </pre> </div></div> <p>This is awkward - I thought it must be sleeping for LRU slots while holding some dirty pages. The problem is obvious: cl_dirty couldn't be deducted for some reason, one possible reason would the ptlrpc threads were blocked somewhere so brw_interpret couldn't be called, or the processes are waiting for something while holding active osc_extents. Can you please take a look at backtrace of running processes and tell me if there exists any other process is blocking on something?</p> <p>Also, I worked out patch set 4 which dumps the extent tree when this problem happens. This can help us isolate the root cause.</p> <p>See new attachment <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/13038/13038_rzuseqio15_console.txt.bz2" title="rzuseqio15_console.txt.bz2 attached to LU-3416">rzuseqio15_console.txt.bz2<sup><img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/></sup></a></span>. It shows backtraces from all processes after a hang began, and then the messages that were printed by patch set 4.</p> <p>Can you please try patch set 5?</p> <p>Will do.</p> <p>I have been running the reproducer, and haven't had a single hang yet with patch set 5 applied. I will leave it running over night.</p> <p>No news assumes good news, no?</p> <p>If this patch can survive, can you please do me a favor to roll back this patch, make the hang happen, and collect logs when doing `echo 3 &gt; drop_caches', I don't understand why this can make it go forward?</p> <p>Yes, I haven't been able to reproduce the problem again. Performance is in the accepatble range.</p> <p>I should be able to get that log tomorrow.</p> <p>See the attached rzuseq*_drop_caches.bz2 files. When I got a write hange, I enabled full lustre debugging, issued an 'echo 2 &gt; drop_caches', and waited a few seconds, and then dumped the logs on all four clients.</p> <p>Hopefully there is something useful in there.</p> <p>Hi Chris, can you please try the last patch set? If this works, I will land it into master.</p> <p>I took a look at the log but unfortunately I didn't find anything interesting.</p> <p>For example, for log rzuseqio15_drop_caches.txt, obviously process 15336 is the drop_caches process and 15302 is flush process. So I tried to dig activities other than these two processes by:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[jinxiong@intel tmp]$ cat rzuseqio15_drop_caches.txt |grep -v <span class="code-quote">'15336:'</span> |grep -v <span class="code-quote">':15302:'</span> |grep cl_io [jinxiong@intel tmp]$ </pre> </div></div> <p>but nothing is printed out. I don't know why this can happen because the pages are already existing but no process is related to them. It will be helpful to figure out which process was stuck before dropping caches.</p> <p>The purpose of doing this is to fully understand why dropping cache can make the hung process go forward. Though we can fix the problem by latest patch, it's really bad if we don't understand a code path clearly.</p> <p>It takes quite a while to run drop_caches. I fear that getting the log of the stuck processes will be quite difficult with the other threads swamping the logs. But I'll keep this as a background task when I have time.</p> <p>In the mean time, I'll update to the latest patch.</p> <p>Jinshan, testing of change <a href="http://review.whamcloud.com/6554" class="external-link" target="_blank" rel="nofollow noopener">6554</a> patch set 7 went well. No failures, and no new performance problems noted.</p> <p>The commit message will need to be rewritten before landing.</p> <p>patch landed for 2.5</p> Related LU-2576 LU-2859 Development Rank 1|hzvs8f: Rank (Obsolete) 8471 Severity