https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2816 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/91d8ae78-755b-11e2-bf59-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/91d8ae78-755b-11e2-bf59-52540035b04c</a>.</p> <p>The sub-test test_bonnie failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>From the stdout log, I found the test ran extremely slow compare to the success run:</p> <p>It took almost 45 minutes for the failed test:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>16:44:39:== sanity-benchmark test bonnie: bonnie++ == 16:44:30 (1360629870) 16:44:39:min OST has 1809000kB available, using 3845408kB file size 16:44:39:debug=0 16:44:39:running as uid/gid/euid/egid 500/500/500/500, groups: 16:44:39: [touch] [/mnt/lustre/d0_runas_test/f16388] 16:44:39:running as uid/gid/euid/egid 500/500/500/500, groups: 16:44:39: [bonnie++] [-f] [-r] [0] [-s3755] [-n] [10] [-u500:500] [-d/mnt/lustre/d0.bonnie] 16:44:39:Using uid:500, gid:500. 16:47:56:Writing intelligently...done 17:29:18:Rewriting...done </pre> </div></div> <p>Only 9 minutes for the success test:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>01:31:59:== sanity-benchmark test bonnie: bonnie++ == 01:31:52 (1359624712) 01:31:59:min OST has 1783256kB available, using 3845408kB file size 01:31:59:debug=0 01:31:59:running as uid/gid/euid/egid 500/500/500/500, groups: 01:31:59: [touch] [/mnt/lustre/d0_runas_test/f22196] 01:31:59:running as uid/gid/euid/egid 500/500/500/500, groups: 01:31:59: [bonnie++] [-f] [-r] [0] [-s3755] [-n] [10] [-u500:500] [-d/mnt/lustre/d0.bonnie] 01:31:59:Using uid:500, gid:500. 01:35:21:Writing intelligently...done 01:40:56:Rewriting...done </pre> </div></div> LU-2816

sanity-benchmark test_bonnie slow after 4MB BRW RPC patch

Bug Blocker Resolved Fixed Jinshan Xiong Maloo HB Thu, 14 Feb 2013 21:47:00 +0000 Wed, 13 Mar 2013 17:19:06 +0000 Wed, 13 Mar 2013 15:30:27 +0000 Lustre 2.4.0 Lustre 2.4.0 0 9 <p>A test run that was fast before the 4MB patch landing (commit b7ee9ce814f37475cdfc6d4def83906b8b41c1d8):<br/> <a href="https://maloo.whamcloud.com/sub_tests/e3aca9ba-7665-11e2-bc2f-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/e3aca9ba-7665-11e2-bc2f-52540035b04c</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP client-24vm2. 3755M 23662 6 12159 7 21428 5 64.2 23 Latency 4517ms 3199ms 2569ms 655ms </pre> </div></div> <p>After 4MB patch landing the read performance is much slower (commit 86097b23c959c50a81f7b60ce969bcfece8839aa):<br/> <a href="https://maloo.whamcloud.com/sub_tests/cbd6a396-73f8-11e2-9df1-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/cbd6a396-73f8-11e2-9df1-52540035b04c</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP client-29vm6. 3753M 26102 37 1526 4 1858 3 858.0 63 Latency 430ms 1412ms 59478us 45500us </pre> </div></div> <p>I'm not sure why rewrite would be slower, when the initial write performance is actually faster. It might have something to do with the returned st_blocksize, or possibly tying the readahead chunk size (RAS_INCREASE_STEP) to st_blocksize instead of hard-coding it at 1MB.</p> <p>Hi Jinshan,</p> <p>This is a high blocker. Can you determine if this is something you can look at and if not, let me know.</p> <p>The below stuck out at me on a quick review of this LU. </p> <p>Both report (with a little spacing help):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>OST has 1809000kB available, using 3845408kB file size </pre> </div></div> <p>We are creating a file that will not fit in the OST Space for this test? </p> <p>It seem this is case for good and bad runs both but it stuck me as odd. </p> <p>Also on the OST from the system log it looks like memory pressure was on. </p> <p>In the backtraces pte_fault is pretty common path for programs to be stuck in and a few seconds after the systems showstate</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Feb 11 17:46:06 client-16vm4 kernel: cannot allocate a tage (399) Feb 11 17:46:06 client-16vm4 kernel: cannot allocate a tage (399) Feb 11 17:46:06 client-16vm4 kernel: cannot allocate a tage (399) Feb 11 17:46:06 client-16vm4 kernel: cannot allocate a tage (399) Feb 11 17:46:06 client-16vm4 kernel: cannot allocate a tage (399) Feb 11 17:46:06 client-16vm4 kernel: cannot allocate a tage (399) </pre> </div></div> <p>This system may have just been under memory pressure and swapping the whole time. It is hard to say from the logs for sure it was not OOM but things can slow down alot before that point is reached. </p> <p>Keith, note that the OST size reported is for one OST, but there are 3 OSTs in total, hence 3x file size. </p> <p>I agree that memory pressure can slow things down, and the 4MB patch will chase larger allocations. Alex has a patch that could help this. </p> <p>I'm also wary about the readahead changes I had to make for this patch. They may be causing larger reads than needed. It might also relate to a larger st_blocksize reported to userspace for files (4MB vs. 2MB).</p> <p>also seen in interop test between 2.3.0 server and 2.4 client:<br/> <a href="https://maloo.whamcloud.com/test_sets/2746a972-7572-11e2-93d9-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2746a972-7572-11e2-93d9-52540035b04c</a></p> <p>I'm looking at this issue.</p> <p>Performance test with 1MB vs 4MB RPC and writethrough cache settings</p> <p>Please check the following picture for 1MB vs. 4MB RPC w/ Writethrough settings:</p> <p><span class="image-wrap" style=""><img src="http://jira.whamcloud.com/secure/attachment/12286/Untitled.png" style="border: 0px solid black" /></span></p> <p>From this testing, if writethrough_cache is disabled on the OST side, the performance is not dropped significantly with 4MB RPC size, 10% less I would say. However, if write through_cache is enabled, I can see huge performance drop with both settings - but 4MB cache is worse.</p> <p>So I guess this is all about memory pressure, the memory is almost used up if write through is enabled.</p> <p>About the testing env: This test is run on rosso, with 16 CPUs and 48G memory, I used memory disk on OST and applied patch 5164 to saturate QDR network. Iozone was used to generate data w/ 4MB block size.</p> <p>Initially I was trying to reproduce this issue but I found the results are interesting, so I completed the performance under different settings.</p> <p>I will continue working on this issue itself.</p> <p>I suspect that 4MB RPC size will be most interesting for DDN RAID-6 devices and similar. Running on RAM disk will not show any of the overhead of seeking or better streaming IO. </p> <p>Jinshan, also note that the bug isn't about how to get 4MB performance to be the same or better than 1MB performance, since the default is going to stay at 1MB for the 2.4 release. Rather, the performance at 1MB has gone down after the 4MB path landed, even though the RPC size stayed at 1MB. Also, it may be that the performance loss is for reads and not writes.</p> <p>Please compare performance from before and after the patch, and look at the test times for before/after in Maloo.</p> <p>Maybe Alex's patch to dynamically allocate the OST memory has removed this problem already? I'm not sure if it recently landed, or is still to be landed...</p> <p>I see. I will perform the test. I reset the tree to before commit 0f8b7f951e7f43dbf389a431afea2091388a805e(<a href="https://jira.whamcloud.com/browse/LU-1431" title="Support for larger than 1MB sequential I/O RPCs" class="issue-link" data-issue-key="LU-1431"><del>LU-1431</del></a> ptlrpc: Support for over 1MB bulk I/O RPC) but didn't see much difference on write performance with 1MB RPC. I will try read.</p> <p>Jinshan, note that if you do find a read performance drop with the 4MZb patch, it is worthwhile to see if Alex's <a href="http://review.whamcloud.com/5444" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5444</a> patch will fix it. This should reduce the memory pressure significantly. </p> <p>Also, it is worthwhile to do a Maloo search for the sanity-benchmark test-bonnie results and compare the times. It may be that the performance got worse for a time and the got better. </p> <p>I confirmed that there is significant read performance degradation after 4MB RPC patch is landed, the read performance dropped from ~3GB/s to ~1GB/s. The patch 5444 didn't help. I'm working on it.</p> <p>Also, I didn't see significant performance loss for write case.</p> <p>There were some changes to the readahead code as a result of the 4MB patch. In particular, the changes in <a href="http://review.whamcloud.com/5230" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5230</a> were done by me because of PTLRPC_MAX_BRW_PAGES increasing to a large value. Instead of allowing the readahead window to grow immediately to potentially 32MB, the RAS_INCREASE_STEP is limited to (1 &lt;&lt; inode-&gt;i_blkbits) = (1 &lt;&lt; LL_MAX_BLKSIZE_BITS) = 4MB, instead of the previous 1MB value.</p> <p>It would be a simple first test to see if changing LL_MAX_BLKSIZE_BITS to 20 or 21 fixes the read performance issue. This can't be the final fix, since this will affect st_blksize and the IO size that "cp" and other tools use, but it will at least help narrow down the problem.</p> <p>Probably a better long-term solution is to change inode-&gt;i_blkbits to be (2 * lmm_stripe_size) instead of being tied to PTLRPC_MAX_BRW_PAGES or LL_MAX_BLKSIZE_BITS. This was originally working in Lustre (lsm_xfersize), but was changed because of some minor NFS issues in <a href="https://bugzilla.lustre.org/show_bug.cgi?id=6062" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=6062</a>, but should be based on lmm_stripe_size again. The main difference compared to the old code would be the default lmm_xfersize should be 2x the default stripe size, so that it will match under normal usage (i.e. for SPEC SFS NFS testing), but it can be tuned on a per-file basis for applications that are badly behaved (e.g. <a href="https://bugzilla.lustre.org/show_bug.cgi?id=12739" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=12739</a>).</p> <blockquote> <p>There were some changes to the readahead code as a result of the 4MB patch. In particular, the changes in <a href="http://review.whamcloud.com/5230" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5230</a> were done by me because of PTLRPC_MAX_BRW_PAGES increasing to a large value. Instead of allowing the readahead window to grow immediately to potentially 32MB, the RAS_INCREASE_STEP is limited to (1 &lt;&lt; inode-&gt;i_blkbits) = (1 &lt;&lt; LL_MAX_BLKSIZE_BITS) = 4MB, instead of the previous 1MB value.</p></blockquote> <p>I reverted this change but it didn't help. I'm now on the way to revert change of ptlrpc desc change, I guess this would be the major cause of performance loss. Stay tuned.</p> <p>Do you have the parameters you were using for iozone? Do you have the raw data for all of the iozone output? </p> <p>Also you say your disk was a ram disk? I guess that explains why cached is so much slower (that is non-intuitive for spindles). </p> <p>Also you could add the data for pre 4MB BRW patch data so we know what things looked like before the changes? </p> <p>I've known the root cause of this problem - it's a simple one:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">diff --git a/lustre/llite/rw.c b/lustre/llite/rw.c index cbd1c91..8cfa70e 100644 --- a/lustre/llite/rw.c +++ b/lustre/llite/rw.c @@ -576,7 +576,7 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> ll_read_ahead_page(<span class="code-keyword">const</span> struct lu_env *env, struct cl_io *io, * know what the actual RPC size is. If <span class="code-keyword">this</span> needs to change, it makes more * sense to tune the i_blkbits value <span class="code-keyword">for</span> the file based on the OSTs it is * striped over, rather than having a constant value <span class="code-keyword">for</span> all files here. */ -#define RAS_INCREASE_STEP(inode) (1UL &lt;&lt; inode-&gt;i_blkbits) +#define RAS_INCREASE_STEP(inode) (1UL &lt;&lt; (inode-&gt;i_blkbits - CFS_PAGE_SHIFT)) <span class="code-keyword">static</span> inline <span class="code-object">int</span> stride_io_mode(struct ll_readahead_state *ras) { </pre> </div></div> <p>But there is still a 10%~20% performance loss on read after applying this patch, I'd like to spend more time on it.</p> <p>I did some research and it turned out that we need to adjust the max_read_ahead_mb and max_read_ahead_per_file_mb to match 4MB size RPC and 4MB increase step. Otherwise the readahead budget will be used up and produce huge number of RA_STAT_MAX_IN_FLIGHT. After I increased those two parameters I didn't see significant performance loss on read.</p> <p>However, adjusting those two parameters need lot of work to do to make sure they are okay with other user cases. So I'd like to set RAS_INCREASE_STEP back to 1MB and after 4MB RPC is enabled by default, we should spend more time on that.</p> <p>patch is at: <a href="http://review.whamcloud.com/5691" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5691</a></p> <p>Let's fix it for now. Readahead performance tweak will be addressed in another ticket.</p> Related LU-2957 LU-1431 Development Rank 1|hzvj67: Rank (Obsolete) 6820 Severity