https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8964 Lustre <p>On machines with slow single thread performance like KNL the bottleneck of I/O performance moved into code which just copy memory from one buffer to other (from user space to kernel or vice versa). In current Lustre implementation all I/O performs in single thread and this is become an issue for KNL. Significantly improve performance can be with solution which do parallel memory transfer of large buffers.</p> <p> </p> LU-8964

use parallel I/O to improve performance on machines with slow single thread performance

New Feature Major Resolved Duplicate James A Simmons Dmitry Eremin Wed, 21 Dec 2016 15:02:02 +0000 Tue, 5 Nov 2019 05:41:12 +0000 Thu, 7 Mar 2019 23:39:04 +0000 0 26 <p>Dmitry Eremin (dmitry.eremin@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/24474" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/24474</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> libcfs: Introduce parallel tasks framework<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3806af960d1cee93e7643040bcee8eb23404ec3d</p> <p><del>Dmitry Eremin (dmitry.eremin@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/24475" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/24475</a></del><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> llite: Parallelize generic I/O</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: 88d7234f1a26678aebcd3adb0aeb2c17630d8965</del></p> <p><del>Dmitry Eremin (dmitry.eremin@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/24476" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/24476</a></del><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> llite: Parallelize readahead and make it async</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: 86a24764485b980d0707a996b0cf58582a585fd3</del></p> <p>Dmitry Eremin (dmitry.eremin@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/24933" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/24933</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> llite: Make readahead async<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 941395a2d9c93275d5fac10bf3a2cc1ee50d6fe8</p> <p>Dmitry Eremin (dmitry.eremin@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/25046" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25046</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> clio: Parallelize generic I/O<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3414d26cb7d3bfe40311780fce60dd491d7cc445</p> <p>Hi Dmitry,<br/> Which is the latest patchset series?<br/> I saw two, and they conflict with each other.<br/> <a href="https://review.whamcloud.com/#/c/24933/4" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/24933/4</a><br/> <a href="https://review.whamcloud.com/#/c/25046/3" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/25046/3</a></p> <p>Thanks,<br/> Gu</p> <p>Hi Gu,<br/> Those patches are different. The first one makes read ahead async and that is all. The second one makes user I/O (read/write) parallel. So, those patches are independent but beneficially to have them both.</p> <p>Hi Dmitry,<br/> Good to know, thanks.<br/> These series are really exciting, hope them can be merged soon.<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Dmitry,</p> <p>As promised, here's my readahead microbenchmark. It's pretty basic and not very well commented.</p> <p>You can turn on and off the 1 page offset I'm using to "break" the strided pattern (see comments on the results below) by uncommenting and commenting out the code on lines 84 and 85 of the file.</p> <p>With strided read pattern:<br/> <span class="error">&#91;root@centclient02 reverse&#93;</span># vim read_readahead_test.c; gcc read_readahead_test.c ; echo 3 &gt; /proc/sys/vm/drop_caches ; lctl set_param debug=0; time ./a.out<br/> debug=0<br/> pos 0<br/> pos 1048576, read rc: 1048576<br/> pos 210763776, read rc: 1048576<br/> pos 420478976, read rc: 1048576<br/> pos 630194176, read rc: 1048576<br/> pos 839909376, read rc: 1048576<br/> pos 1049624576, read rc: 1048576<br/> pos 1259339776, read rc: 1048576<br/> pos 1469054976, read rc: 1048576<br/> pos 1678770176, read rc: 1048576<br/> pos 1888485376, read rc: 1048576<br/> pos 2098200576, read rc: 1048576<br/> amount read (0) != amount requested (1048576), stopping<br/> iterations: 1021, read 1071644672 total bytes</p> <p>real 0m1.178s<br/> user 0m0.002s<br/> sys 0m0.977s<br/> <span class="error">&#91;root@centclient02 reverse&#93;</span># cat /proc/fs/lustre/llite/centss03-ffff88013d20c000/read_ahead_stats<br/> snapshot_time 1486349623.499066 secs.usecs<br/> hits 261333 samples <span class="error">&#91;pages&#93;</span><br/> misses 45 samples <span class="error">&#91;pages&#93;</span><br/> readpage not consecutive 1021 samples <span class="error">&#91;pages&#93;</span><br/> failed grab_cache_page 4 samples <span class="error">&#91;pages&#93;</span><br/> zero length file 1 samples <span class="error">&#91;pages&#93;</span><br/> zero size window 260356 samples <span class="error">&#91;pages&#93;</span><br/> failed to reach end 509 samples <span class="error">&#91;pages&#93;</span><br/> <span class="error">&#91;root@centclient02 reverse&#93;</span># echo c &gt; /proc/fs/lustre/llite/centss03-ffff88013d20c000/read_ahead_stats</p> <p>With slightly broken strided read pattern (1 page offset on every second read), to simulate disabling strided readahead:<br/> <span class="error">&#91;root@centclient02 reverse&#93;</span># vim read_readahead_test.c; gcc read_readahead_test.c ; echo 3 &gt; /proc/sys/vm/drop_caches ; lctl set_param debug=0; time ./a.out<br/> debug=0<br/> pos 0<br/> pos 1052672, read rc: 1048576<br/> pos 210972672, read rc: 1048576<br/> pos 420892672, read rc: 1048576<br/> pos 630812672, read rc: 1048576<br/> pos 840732672, read rc: 1048576<br/> pos 1050652672, read rc: 1048576<br/> pos 1260572672, read rc: 1048576<br/> pos 1470492672, read rc: 1048576<br/> pos 1680412672, read rc: 1048576<br/> pos 1890332672, read rc: 1048576<br/> pos 2100252672, read rc: 1048576<br/> amount read (0) != amount requested (1048576), stopping<br/> iterations: 1020, read 1070596096 total bytes</p> <p>real 0m5.100s<br/> user 0m0.003s<br/> sys 0m1.416s<br/> <span class="error">&#91;root@centclient02 reverse&#93;</span># cat /proc/fs/lustre/llite/centss03-ffff88013d20c000/read_ahead_stats <br/> snapshot_time 1486349643.260326 secs.usecs<br/> hits 260098 samples <span class="error">&#91;pages&#93;</span><br/> misses 1022 samples <span class="error">&#91;pages&#93;</span><br/> readpage not consecutive 1020 samples <span class="error">&#91;pages&#93;</span><br/> failed grab_cache_page 1020 samples <span class="error">&#91;pages&#93;</span><br/> zero length file 1 samples <span class="error">&#91;pages&#93;</span><br/> zero size window 258574 samples <span class="error">&#91;pages&#93;</span><br/> read-ahead to EOF 1 samples <span class="error">&#91;pages&#93;</span><br/> hit max r-a issue 1 samples <span class="error">&#91;pages&#93;</span><br/> failed to reach end 1 samples <span class="error">&#91;pages&#93;</span></p> <p>Note the huge difference in time, even though the bytes read are almost exactly the same.<br/> The misses in the strided case are 45, in the slightly off case, they are 1022.</p> <p>(Attachment coming in a moment)</p> <p>Also, I should mention - It prints out the pos and bytes_read on only every 100th iteration, not every iteration.</p> <p>Patrick,<br/> Do you expect the same behavior from new version? Briefly I got the following results on my VM machine:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>strided read pattern: 108 pages miss iterations: 512, read 537919488 total bytes real 0m1.279s user 0m0.002s sys 0m0.283s </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>broken strided read pattern: 471 pages miss iterations: 511, read 536870912 total bytes real 0m1.524s user 0m0.002s sys 0m0.229s </pre> </div></div> <p>So, there is no big difference between two patterns in terms of speed but in first case there are low read pages miss.</p> <p>Is this acceptable for you or you still will complain about regression?</p> <p>With bigger file (1772392448 bytes (1.8 GB)) the proportion become the same. Asynchronous nature of read ahead become page miss less critical than it was before.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>strided read pattern: 137 pages miss iterations: 845, read 887095296 total bytes real 0m2.217s user 0m0.000s sys 0m0.499s </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>broken strided read pattern: 756 pages miss iterations: 844, read 886046720 total bytes real 0m2.571s user 0m0.001s sys 0m0.408s </pre> </div></div> <p>Hmm, that might be fine. I was also looking at implementing reverse readahead, though I don't have a prototype yet or anything. (That's why I had that test program around.) It looks like we can't do anything like that when using the kernel readahead implementation, so that would be a shame. Reverse reading of anything other than really big blocks is very slow.</p> <p>About the performance, two things to try, just to see what they do.</p> <p>I think we should probably try a much bigger stride size, in case we're just reading across the gap <span class="error">&#91;i.e. we&#39;re just reading ahead covering both the pages in the gap and the next section we really read... I don&#39;t think so, but probably worth checking.&#93;</span>.</p> <p>So, instead of:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">if</span> (strided) { pos = lseek(fd, pos+read_bytes, SEEK_SET); /* ATTN here: Uncomment these lines to add a one </pre> </div></div> <p>This:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">if</span> (strided) { pos = lseek(fd, pos+read_bytes*10, SEEK_SET); /* ATTN here: Uncomment these lines to add a one </pre> </div></div> <p>Giving us that larger gap between reads.</p> <p>I'd also be curious to see what happens with larger read sizes. But I expect that larger read sizes will look better with the async implementation, so that's good.</p> <p>I'm wondering about Jinshan's comment that readahead is already async? It does not look like that's the case currently (at least not async the way I would mean it) - Looking at logs here, it looks like the read syscall does not complete until all of the reading readahead is doing is complete, even if readahead is reading beyond what the read from userspace requested.</p> <p>Is readahead async in that way with your patch? i.e., if I read a 1 MiB chunk of the file but readahead wants to read 5 MiB, assuming RPC size of 1 MiB, will my read call return once the 1 MiB is present, without waiting for the rest?</p> <p>I'm struggling with single stream read performance on ZFS OSTs and tested this patch set. I like it a lot because it simplifies and cleans up the code very much. But without further tunables it will IMO harm single stream sequential read performance on decent ZFS OSTs. The problem is basically that ZFS prefetch is very sensitive to the patterns it receives.</p> <p>My setup has OSTs which are capable of 1.8GB/s read with one dd directly on the OST (zpool, ZFS filesystem). With the old Lustre prefetcher (without the patchsets of this ticket) I get best performance if I limit the max_rpcs_in_flight to 1 or 2, thus allow for only few ll_ost_io_* threads to do the actual reads on the OSS side. Performance then is close to 1GB/s. Setting max_rpcs_in_flight to 16, 32, or even 256 spoils the performance massively, it ends up in the range of 300-500MB/s.</p> <p>With 2.9.0 + <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> my results are:</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rpcs              dd 1M in                reads flight        bandwidth MB/s                zfs prefetch              disabled   enabled ------------------------------------------------------ 1             302        553 2             383        638 4             456        630 8             568        520 16            668        568 32            731        533 64            764        476 128           751        479 256           689        542 </pre> </div></div> <p>I don't get to 1GB/s, that I could reach before, and best results are with disabled ZFS prefetch! Which I don't like, thinking of the 1.8GB/s that it actually is capable of reaching.</p> <p>The problems I see:</p> <ul> <li>reads are evenly distributed across ptasks (56 in my case, big CPUs), which creates more randomness in the read pattern that ZFS is seeing on the OST. A mechanism for adjusting or tuning the number of read threads per stream of per OSC-object would be ideal.</li> <li>the size of the reads is the same for all ptasks, larger than 256, but often an odd number (eg. 2341 pages), which leads probably to some large and some small RPCs. I'm playing with the values and am able to get much better results, at least in the single stream sequential case. Will report when my "scan" is done.</li> </ul> <p>Clearly this patch is very helpful for finally improving read-ahead, but I suppose it needs somehow two modes, one for ZFS and one for ldiskfs.</p> <p>Erich,</p> <p>Thanks for the testing. For what it's worth, ldiskfs is not insensitive to read patterns either. I think splitting the I/O to these weird sizes is inherently problematic - It should be on more logical boundaries.</p> <p>I am really surprised to hear you say that you get better results with 1-2 max_rpcs_in_flight. That seems like then the server would cope <b>extremely</b> poorly with reads from multiple clients. Is that correct? That seems like something is wrong, beyond just ZFS.</p> <p>I am testing with normal clients, not KNL, by the way.</p> <p>With many clients, each with its own streams, things look okay. I couldn't test yet with this async readahead patchset, but the plain old Lustre prefetcher with many rpcs in flight and ZFS OSTs work okay. Even if slower, each stream contributes its share to the total bandwidth and they add up quite well to the more or less expected peak. Some annoying tweaking of max_read_ahead_per_file_mb is necessary, though, and there is no setting that is good for all workloads.</p> <p>Erich Focht (efocht@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/25943" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25943</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> llite: tunable ptasks number with optimized IO size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 371d804834fe72078dec82906c04e057346abbdd</p> <p>Just uploaded a patch on top of the series from Dmitry.</p> <p>It changes the size of the IOs handled by each ptask in ll_readpages(). The original version had the potential to create "odd" npages values (like 276, for example), which translated to a number of large RPCs plus one small one. Worse than the RPC problem are the offsets received by the prefetcher, which (I believe) are rather unaligned in the original case. ZFS prefetcher doesn't deal well with those, especially if they get scheduled in a different order on the OSS.</p> <p>Results: single stream read with dd, 1MB block size, MB/s</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> zfs-0.6.5.7        max_read_ahead_per_file_mb=256 rpcs                                                in      LU-8964      LU-8964+EF  flight                ptasks=1          ---------------------------------   1         553          860             2         638          728             4         630          677             8         520          595             16        568          525             32        533          485             64        476          452             128       479          484             256       542          593             </pre> </div></div> <p> </p> <p>Switching to zfs-0.7.0rc3 improves things dramatically, because the prefetcher is much better:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>zfs-0.7.0rc3 rpcs     ra_p_file=256 ra_p_file=64 |  ra_p_file=64                            in        LU-8964       LU-8964    |  LU-8964+EF     LU-8964+EF  LU-8964+EF flight                              |  ptasks=1       ptasks=14   ptasks=28   ------------------------------------|-------------------------------------    1           449           433       |   1200           1100        1100       2           511           491       |   1600           1700        1600       4           585           582       |   1700           1900        1700       8           674           680       |   1400           1800         943       16          744           797       |   1200           1700        1600       32          847           881       |   1200           1600        1400       64          860           856       |   1100           1500        1400       128         933           819       |   1000           1300        1300       256         868           791       |    747           1300        1200       </pre> </div></div> <p> </p> <p> </p> <p>Erich, thanks for posting your results. Definitely 0.7.0 seems very attractive for this workload.</p> <p>It looks like your patch is improving the performance significantly as well. Is the 56-code client machine NUMA, and if yes how many sockets, 4? I'm not against changing the default ptask count to get better performance out-of-the-box, based on your test results, so long as we can determine what the best setting should be.</p> <p>How does ptasks=14 relate to the socket count, and is there a rule we can make there? It seems like having a thread for each CPU is probably the wrong default, unless they are each working with a different OST?</p> <p>Thanks Erich. This is a good improveement for my patch. But I have other idea how to implement it. I'm going to update my patch soon. I hope you will test it and review.</p> <p>Hi Andreas,</p> <p>the client is a dual socket E5-2680 v4 with 14 cores per CPU and HT enabled. In /proc/cpuinfo I see 56 cores. Experiments are on top of Centos 7.2 3.10.0-514.2.2.el7 kernel.</p> <p>The read ahead engine is initialized with cpu_online_map:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">vvp_raengine = cfs_ptengine_init(<span class="code-quote">"read_ahead"</span>, cpu_online_mask); </pre> </div></div> <p>so I would have expected to be able to chose up to 56 ptasks, but I can only set it to 28. I'm missing some detail here. The highest active kworker thread I see is <span class="error">&#91;kworker/27:1&#93;</span> so I suppose we stay away from the hyperthreads.</p> <p>Tests are with Omnipath, so additional CPU resources for the network are beneficial, that's maybe why ptasks=14 is better than ptasks=28. Rule of thumb right now: ptasks=1 for zfs-0.6.5.7, ptasks=half of maximum for zfs-0.7.0. ptasks=maximum for KNL with ldiskfs OSTs (Dmitry's choice).</p> <p> </p> <p>Erich, I have special logic during initialization.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">if</span> (cpumask_empty(par_mask) || cpumask_equal(par_mask, cpu_online_mask)) { cpumask_copy(all_mask, cpu_online_mask); cpumask_clear(par_mask); <span class="code-keyword">while</span> (!cpumask_empty(all_mask)) { <span class="code-object">int</span> cpu = cpumask_first(all_mask); cpumask_set_cpu(cpu, par_mask); cpumask_andnot(all_mask, all_mask, topology_sibling_cpumask(cpu)); } } </pre> </div></div> <p>This means if CPU mask is empty or full use all available CPUs without HT. So, only pure cores will be used. This is done intentional. Using HT for this will disrupt performance.</p> <p>Ah! Thanks Dmitry, now it's clear <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Dmitry Eremin (dmitry.eremin@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/26314" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26314</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> libcfs: Introduce parallel tasks framework<br/> Project: fs/lustre-release<br/> Branch: pfl<br/> Current Patch Set: 1<br/> Commit: 4b91a6c4999ade6d1d04c89f06bc9ed535616e69</p> <p>Dmitry Eremin (dmitry.eremin@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/26315" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26315</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> clio: Parallelize generic I/O<br/> Project: fs/lustre-release<br/> Branch: pfl<br/> Current Patch Set: 1<br/> Commit: ebd9846fafbe07ba576e288e28a7a6847adc5321</p> <p>Dmitry Eremin (dmitry.eremin@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/26350" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26350</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> llite: Make readahead async<br/> Project: fs/lustre-release<br/> Branch: pfl<br/> Current Patch Set: 1<br/> Commit: 84d8bc4dcbc8f7b187f967af8dc264a6e834b9e3</p> <p>Dmitry Eremin (dmitry.eremin@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/26468" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26468</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> clio: Parallelize generic I/O<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: dfd26cbb6f41a956658b975b68576b8fccbbd342</p> <p>Dmitry Eremin (dmitry.eremin@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/26469" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26469</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> llite: Make readahead async<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d624cc9a301d1014cca088dff014176c7f61d5fc</p> <p>I decided to write down my concern and known problems here. I wrote it on the gerrit but it was missed in the refresh and rebase.</p> <p>1. head of block - when a subtask is blocked due to long holding lock, for example group lock, the tasks behind that one will be blocked that means nobody could move forward;<br/> 2. for pdata implementation right now, once a task has been decided to put into a queue, it has to be stuck in that queue. To some extent, the overhead performance now is limited by the slowest OST;<br/> 3. lack of cancel API. This poses a problem if an error has already seen in a previous subtask, it's expected to cancel all the other pending subtasks that belong to the same I/O because it'll be just wasting of time by continuing doing those;<br/> 4. pdata uses per cpu work queue. This is not perfect fit to drive Lustre I/O task because most of the time the tasks are in sleep state waiting for some resources, which means the CPU could be used to drive the other subtasks. It would be extremely difficult to work out something like ZIO pipeline in ZFS but it'll be easier to just spawn more I/O threads than the number of CPUs;<br/> 5. Lack of NUMA support. This should be really easy to fix.</p> <p>It's not necessary to block the landing of the patch as is. However, this is something we should proceed to work post landing.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/24474/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/24474/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> libcfs: Introduce parallel tasks framework<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: eca949bf83dfec5be33dc6230f55823376aef65a</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/26468/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26468/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> clio: Parallelize generic I/O<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: db59ecb5d1d0284fb918def6348a11e0966d7767</p> <p>Dmitry Eremin (dmitry.eremin@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29540" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29540</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> llite: Make readahead async<br/> Project: fs/lustre-release<br/> Branch: flr<br/> Current Patch Set: 1<br/> Commit: a3d49f1db481c9e4ae72693eade7bcf8729483ce</p> <p>here is quick test resutls of patch <a href="https://review.whamcloud.com/#/c/26469/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/26469/</a> (patch set 37)</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lctl set_param osc.*.max_pages_per_rpc=16M lctl set_param osc.*.max_rpcs_in_flight=16 lctl set_param osc.*.max_dirty_mb=512 lctl set_param llite.*.max_read_ahead_mb=2048 lctl set_param osc.*.checksums=0 </pre> </div></div> <p>40 x OST, 1 x client(2 x E5-2650v4, 128GB memory, 1 x FDR)</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># lfs setstripe -c -1 -S 16M /scratch0/dir # IOR -w -k -F -e -t 1m -b 256g -vv -o /scratch0/dir/file # sync;echo 3 &gt; /proc/sys/vm/drop_caches (client and OSSs) # IOR -r -F -e -t 1m -b 256g -vv -o /scratch0/dir/file </pre> </div></div> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>2.7.22.1</td> <td class='confluenceTd'>2.10.55</td> <td class='confluenceTd'>2.10.55/w patch</td> </tr> <tr> <td class='confluenceTd'>897.34</td> <td class='confluenceTd'>881.34MB/s</td> <td class='confluenceTd'>911.55MB/s</td> </tr> </tbody></table> </div> <p>I didn't see big performance improvments with patches when if max_pages_per_rpc=16M.</p> <p>A few questions.</p> <p>1. What's the best-case bandwidth of an individual OST?<br/> 2. You're only striping to one OST, what happens if you stripe wider?<br/> 3. How many cores on that client?</p> <blockquote> <p>1. What's the best-case bandwidth of an individual OST?</p></blockquote> <p>Here is single OST's performance. this is same IOR test above, but no striping. Just an OST.<br/> 832.16MB/s</p> <blockquote> <p>2. You're only striping to one OST, what happens if you stripe wider?</p></blockquote> <p>No, first results were striping acrossing all OSTs (40 x OST) with "lfs setstripe -c -1".</p> <blockquote> <p>3. How many cores on that client?</p></blockquote> <p>As I've described above 2 x E5-2650v4 means 24 CPU cores total (12 CPU cores each socket)</p> <p>Have you collected rpc_stats from client side to see if it actually sends full size RPC?</p> <p>Firstly, I think one problem is that IOR is being run with a 1Mb transfer size (“<tt>-t 1m</tt>”) so the kernel-side parallelism doesn’t get used. Try running with “<tt>-t 16m</tt>” so that the kernel threads can help do the copies from userspace and RPCs in parallel.</p> <p>Also, how many IOR threads running on the client? Just one? </p> <p>One thing I was wondering is if the 16MB stripe_size is potentially hurting the parallelism? That means multiple threads are doing the copying, but they are all sending to the same OST. Try with the 1MB stripe size so that the kernel threads can send to multiple OSTs in parallel. </p> <blockquote> <p>Have you collected rpc_stats from client side to see if it actually sends full size RPC?</p></blockquote> <p>this is single OST testing. I see some small 4K reads.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">osc.scratch0-OST0009-osc-ffff88203a33e000.rpc_stats= snapshot_time: 1512103327.914410097 (secs.nsecs) read RPCs in flight: 0 write RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 3358 16 16 | 0 0 0 2: 1 0 16 | 0 0 0 4: 0 0 16 | 0 0 0 8: 0 0 16 | 0 0 0 16: 0 0 16 | 0 0 0 32: 2 0 16 | 0 0 0 64: 7 0 16 | 0 0 0 128: 15 0 16 | 0 0 0 256: 27 0 16 | 9 0 0 512: 54 0 17 | 7 0 0 1024: 139 0 17 | 2 0 0 2048: 281 1 19 | 5 0 0 4096: 16441 80 100 | 16384 99 100 read write rpcs in flight rpcs % cum % | rpcs % cum % 0: 0 0 0 | 0 0 0 1: 5854 28 28 | 38 0 0 2: 2654 13 41 | 14182 86 86 3: 2652 13 54 | 1753 10 97 4: 2634 12 67 | 140 0 98 5: 1523 7 75 | 113 0 98 6: 1472 7 82 | 59 0 99 7: 1463 7 89 | 34 0 99 8: 1428 7 96 | 11 0 99 9: 248 1 98 | 5 0 99 10: 178 0 98 | 4 0 99 11: 139 0 99 | 3 0 99 12: 67 0 99 | 4 0 99 13: 8 0 99 | 3 0 99 14: 3 0 99 | 3 0 99 15: 1 0 99 | 3 0 99 16: 1 0 100 | 3 0 99 17: 0 0 100 | 27 0 99 18: 0 0 100 | 22 0 100 read write offset rpcs % cum % | rpcs % cum % 0: 1 0 0 | 1 0 0 1: 1 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 0 0 0 | 0 0 0 32: 0 0 0 | 0 0 0 64: 0 0 0 | 0 0 0 128: 0 0 0 | 0 0 0 256: 0 0 0 | 0 0 0 512: 0 0 0 | 0 0 0 1024: 0 0 0 | 0 0 0 2048: 0 0 0 | 0 0 0 4096: 1 0 0 | 1 0 0 8192: 3 0 0 | 2 0 0 16384: 5 0 0 | 4 0 0 32768: 11 0 0 | 8 0 0 65536: 21 0 0 | 16 0 0 131072: 35 0 0 | 32 0 0 262144: 69 0 0 | 64 0 0 524288: 139 0 1 | 128 0 1 1048576: 280 1 2 | 256 1 3 2097152: 559 2 5 | 512 3 6 4194304: 1114 5 11 | 1024 6 12 8388608: 2241 11 22 | 2052 12 24 16777216: 4622 22 44 | 4110 25 50 33554432: 11223 55 100 | 8197 49 100 </pre> </div></div> <p>Please see attached. <a href="#attachment-28818" target="_blank" rel="noopener">40ost_rpc_stats.txt</a> <br/> this is rpc_stats for all 40 OSTs. Please keep in mind, all my testing are single IOR thread.<br/> there are a lot of small read on some of OSCs.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> osc.scratch0-OST0000-osc-ffff88203a33e000.rpc_stats= snapshot_time: 1512106460.010766948 (secs.nsecs) read RPCs in flight: 0 write RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 3764 81 81 | 0 0 0 2: 0 0 81 | 0 0 0 4: 0 0 81 | 0 0 0 8: 0 0 81 | 0 0 0 16: 0 0 81 | 0 0 0 32: 24 0 82 | 0 0 0 64: 9 0 82 | 0 0 0 128: 18 0 82 | 0 0 0 256: 54 1 83 | 0 0 0 512: 62 1 85 | 0 0 0 1024: 125 2 87 | 0 0 0 2048: 160 3 91 | 0 0 0 4096: 398 8 100 | 0 0 0 </pre> </div></div> <p>majority of RPCs are large (4MB, 8MB or 16MB, but not all 16MB) on some of other OSCs.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> osc.scratch0-OST0022-osc-ffff88203a33e000.rpc_stats= snapshot_time: 1512106460.012210601 (secs.nsecs) read RPCs in flight: 0 write RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 33 3 3 | 0 0 0 2: 0 0 3 | 0 0 0 4: 2 0 3 | 0 0 0 8: 0 0 3 | 0 0 0 16: 4 0 4 | 0 0 0 32: 23 2 7 | 0 0 0 64: 11 1 8 | 0 0 0 128: 15 1 9 | 0 0 0 256: 52 5 15 | 0 0 0 512: 62 7 22 | 0 0 0 1024: 115 12 35 | 0 0 0 2048: 169 19 54 | 0 0 0 4096: 399 45 100 | 0 0 0 </pre> </div></div> <p>16% 4K read should be problematic.</p> <blockquote> <p>Firstly, I think one problem is that IOR is being run with a 1Mb transfer size (“-t 1m”) so the kernel-side parallelism doesn’t get used. Try running with “-t 16m” so that the kernel threads can help do the copies from userspace and RPCs in parallel.</p></blockquote> <p>I haven't never seen any behefits with -t 16M unless we use O_DIRECT. I just dobule checked -t 16mb with patched client, </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">IOR -a POSIX -r -k -e -vv -t 16m -b 256g -F -o /scratch0/dir/file Max Read: 779.65 MiB/sec (817.53 MB/sec) </pre> </div></div> <blockquote> <p>Also, how many IOR threads running on the client? Just one?</p></blockquote> <p>Just one thread.</p> <blockquote> <p>One thing I was wondering is if the 16MB stripe_size is potentially hurting the parallelism? That means multiple threads are doing the copying, but they are all sending to the same OST. Try with the 1MB stripe size so that the kernel threads can send to multiple OSTs in parallel.</p></blockquote> <p>1MB stripe size is even worse.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># lfs setstripe -S 1M -c -1 /scratch0/dir # mpirun -np 1 IOR -a POSIX -w -k -e -vv -t 1m -b 256g -F -o /scratch0/dir/file # echo 3 &gt; /proc/sys/vm/drop_caches (on OSS/MDS and client) # mpirun -np 1 IOR -a POSIX -r -k -e -vv -t 1m -b 256g -F -o /scratch0/dir/file Summary: api = POSIX test filename = /scratch0/dir/file access = file-per-process pattern = segmented (1 segment) ordering in a file = sequential offsets ordering inter file= no tasks offsets clients = 1 (1 per node) repetitions = 1 xfersize = 1 MiB blocksize = 256 GiB aggregate filesize = 256 GiB Using Time Stamp 1512108853 (0x5a20f335) <span class="code-keyword">for</span> Data Signature [RANK 000] open <span class="code-keyword">for</span> reading file /scratch0/dir/file.00000000 XXCEL Commencing read performance test. Fri Dec 1 15:14:13 2017 access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---------- --------- -------- -------- -------- -------- ---- read 111.75 268435456 1024.00 0.000296 2345.81 0.001064 2345.81 0 XXCEL Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize --------- --------- --------- ---------- ------- --------- --------- ---------- ------- -------- read 111.75 111.75 111.75 0.00 111.75 111.75 111.75 0.002345.80832 1 1 1 1 0 1 0 0 1 274877906944 1048576 274877906944 -1 POSIX EXCEL Max Read: 111.75 MiB/sec (117.18 MB/sec) </pre> </div></div> <p>I thought I'd try a quick, simple test, and I'm seeing significant issues.</p> <p>Here's the file, a single stripe on OST 0, around 7 GB in size:<br/> <span class="error">&#91;root@cent7c01 cent7s02&#93;</span># ls -la ost0<br/> <del>rw-r</del><del>r</del>- 1 root root 7345393664 Dec 7 13:04 ost0<br/> <span class="error">&#91;root@cent7c01 cent7s02&#93;</span># lfs getstripe ost0<br/> ost0<br/> lmm_stripe_count: 1<br/> lmm_stripe_size: 1048576<br/> lmm_pattern: raid0<br/> lmm_layout_gen: 0<br/> lmm_stripe_offset: 0<br/> obdidx objid objid group<br/> 0 579 0x243 0</p> <p>When I read it (after clearing my rpc stats) without this patch, here's what that looks like:<br/> <span class="error">&#91;root@cent7c02 cent7s02&#93;</span># cat ost0 &gt; /dev/null; cat /proc/fs/lustre/osc/cent7s02-OST0000-osc-ffff880034222000/rpc_stats<br/> snapshot_time: 1512669699.623744 (secs.usecs)<br/> read RPCs in flight: 0<br/> write RPCs in flight: 0<br/> pending write pages: 0<br/> pending read pages: 0</p> <p> read write<br/> pages per rpc rpcs % cum % | rpcs % cum %<br/> 1: 1 0 0 | 0 0 0<br/> 2: 0 0 0 | 0 0 0<br/> 4: 0 0 0 | 0 0 0<br/> 8: 0 0 0 | 0 0 0<br/> 16: 0 0 0 | 0 0 0<br/> 32: 0 0 0 | 0 0 0<br/> 64: 0 0 0 | 0 0 0<br/> 128: 0 0 0 | 0 0 0<br/> 256: 0 0 0 | 0 0 0<br/> 512: 1 0 0 | 0 0 0<br/> 1024: 1751 99 100 | 0 0 0</p> <p>There is 1 single page RPC, which is pretty good. A few more would be acceptable too, of course.</p> <p>Here's what it looks like with this patch:<br/> <span class="error">&#91;root@cent7c01 cent7s02&#93;</span># cat /proc/fs/lustre/osc/cent7s02-OST0000-osc-ffff8800b24b7000/rpc_stats<br/> snapshot_time: 1512670149.589299551 (secs.nsecs)<br/> read RPCs in flight: 0<br/> write RPCs in flight: 0<br/> pending write pages: 0<br/> pending read pages: 0</p> <p> read write<br/> pages per rpc rpcs % cum % | rpcs % cum %<br/> 1: 22218 92 92 | 0 0 0<br/> 2: 0 0 92 | 0 0 0<br/> 4: 0 0 92 | 0 0 0<br/> 8: 0 0 92 | 0 0 0<br/> 16: 2 0 92 | 0 0 0<br/> 32: 1 0 92 | 0 0 0<br/> 64: 0 0 92 | 0 0 0<br/> 128: 0 0 92 | 0 0 0<br/> 256: 0 0 92 | 0 0 0<br/> 512: 2 0 92 | 0 0 0<br/> 1024: 1729 7 100 | 0 0 0</p> <p>That's 92% 4K RPCs. That's... definitely not OK.</p> <p>Here's a quick test with dd, with 1 MiB reads, just in case cat was the problem:<br/> <span class="error">&#91;root@cent7c01 cent7s02&#93;</span># dd if=./ost0 of=/dev/null bs=1M<br/> 7005+1 records in<br/> 7005+1 records out<br/> 7345393664 bytes (7.3 GB) copied, 26.8776 s, 273 MB/s<br/> <span class="error">&#91;root@cent7c01 cent7s02&#93;</span># cat /proc/fs/lustre/osc/cent7s02-OST0000-osc-ffff8800b24b7000/rpc_stats<br/> snapshot_time: 1512670262.741210431 (secs.nsecs)<br/> read RPCs in flight: 0<br/> write RPCs in flight: 0<br/> pending write pages: 0<br/> pending read pages: 0</p> <p> read write<br/> pages per rpc rpcs % cum % | rpcs % cum %<br/> 1: 20687 92 92 | 0 0 0<br/> 2: 0 0 92 | 0 0 0<br/> 4: 0 0 92 | 0 0 0<br/> 8: 4 0 92 | 0 0 0<br/> 16: 0 0 92 | 0 0 0<br/> 32: 0 0 92 | 0 0 0<br/> 64: 0 0 92 | 0 0 0<br/> 128: 0 0 92 | 0 0 0<br/> 256: 0 0 92 | 0 0 0<br/> 512: 1 0 92 | 0 0 0<br/> 1024: 1731 7 100 | 0 0 0</p> <p>Note that the performance #s aren't really relevant, these are relatively slow VMs. But the RPC size #s very much are.</p> <p>Not to downplay the significance of the 4KB RPCs, since there are a lot of them, but it is worthwhile to note that the bulk of the IO is still being transferred by 4MB RPCs - 1729/1731 in the patched case vs 1751 in the unpatched case. So 98% of the data is still using 4MB RPCs, vs 2% of the data is using 4KB RPCs.</p> <p>My first guess is that the small RPC sizes are caused by a smaller readahead window at the start being split across multiple threads/RPCs, and then as the readahead window grows it starts using larger reads? It may also be that a small number of 4KB RPCs are being handled separately for each 4MB RPC, but we don't see this because of binning in the histogram. That would have to be checked in the client debug logs with <tt>lctl set_param debug=+reada</tt>.</p> <p>I see what you're getting at, but then I would expect the % of small RPCs to <b>drop</b> as we go up in file size, but I in fact see the opposite. Something goes badly wrong for me around 4 GiB in size, which is 50% of memory on this VM node (which is unlikely to be a coincidence).</p> <p>Examples - 1000 MiB:<br/> <span class="error">&#91;root@cent7c01 cent7s02&#93;</span># ls -la ost0; cat ost0 &gt; /dev/null; cat /proc/fs/lustre/osc/cent7s02-OST0000-osc-ffff88022437c800/rpc_stats<br/> <del>rw-r</del><del>r</del>- 1 root root 1048576000 Dec 7 16:52 ost0<br/> snapshot_time: 1512683598.957290341 (secs.nsecs)<br/> read RPCs in flight: 0<br/> write RPCs in flight: 0<br/> pending write pages: 0<br/> pending read pages: 0</p> <p> read write<br/> pages per rpc rpcs % cum % | rpcs % cum %<br/> 1: 22 8 8 | 0 0 0<br/> 2: 0 0 8 | 0 0 0<br/> 4: 0 0 8 | 0 0 0<br/> 8: 0 0 8 | 0 0 0<br/> 16: 0 0 8 | 0 0 0<br/> 32: 0 0 8 | 0 0 0<br/> 64: 0 0 8 | 0 0 0<br/> 128: 0 0 8 | 0 0 0<br/> 256: 0 0 8 | 0 0 0<br/> 512: 0 0 8 | 0 0 0<br/> 1024: 250 91 100 | 0 0 0</p> <p>2000 MiB:<br/> <span class="error">&#91;root@cent7c01 cent7s02&#93;</span># dd if=/dev/zero bs=1M count=2000 of=./ost0; echo 3 &gt; /proc/sys/vm/drop_caches ; echo clear &gt; /sys/fs/lustre/ldlm/namespaces/cent7s02-OST0000-osc-ffff88022437c800/lru_size; echo c &gt; /proc/fs/lustre/osc/cent7s02-OST0000-osc-ffff88022437c800/rpc_stats; cat ost0 &gt; /dev/null; cat /proc/fs/lustre/osc/cent7s02-OST0000-osc-ffff88022437c800/rpc_stats<br/> 2000+0 records in<br/> 2000+0 records out<br/> 2097152000 bytes (2.1 GB) copied, 7.99026 s, 262 MB/s<br/> snapshot_time: 1512683716.923342172 (secs.nsecs)<br/> read RPCs in flight: 0<br/> write RPCs in flight: 0<br/> pending write pages: 0<br/> pending read pages: 0</p> <p> read write<br/> pages per rpc rpcs % cum % | rpcs % cum %<br/> 1: 37 6 6 | 0 0 0<br/> 2: 0 0 6 | 0 0 0<br/> 4: 0 0 6 | 0 0 0<br/> 8: 0 0 6 | 0 0 0<br/> 16: 0 0 6 | 0 0 0<br/> 32: 0 0 6 | 0 0 0<br/> 64: 0 0 6 | 0 0 0<br/> 128: 0 0 6 | 0 0 0<br/> 256: 0 0 6 | 0 0 0<br/> 512: 0 0 6 | 0 0 0<br/> 1024: 500 93 100 | 0 0 0</p> <p>3000 MiB:<br/> <span class="error">&#91;root@cent7c01 cent7s02&#93;</span># dd if=/dev/zero bs=1M count=3000 of=./ost0; echo 3 &gt; /proc/sys/vm/drop_caches ; echo clear &gt; /sys/fs/lustre/ldlm/namespaces/cent7s02-OST0000-osc-ffff88022437c800/lru_size; echo c &gt; /proc/fs/lustre/osc/cent7s02-OST0000-osc-ffff88022437c800/rpc_stats; cat ost0 &gt; /dev/null; cat /proc/fs/lustre/osc/cent7s02-OST0000-osc-ffff88022437c800/rpc_stats<br/> 3000+0 records in<br/> 3000+0 records out<br/> 3145728000 bytes (3.1 GB) copied, 10.6623 s, 295 MB/s<br/> snapshot_time: 1512683893.046695412 (secs.nsecs)<br/> read RPCs in flight: 0<br/> write RPCs in flight: 0<br/> pending write pages: 0<br/> pending read pages: 0</p> <p> read write<br/> pages per rpc rpcs % cum % | rpcs % cum %<br/> 1: 53 6 6 | 0 0 0<br/> 2: 0 0 6 | 0 0 0<br/> 4: 0 0 6 | 0 0 0<br/> 8: 0 0 6 | 0 0 0<br/> 16: 0 0 6 | 0 0 0<br/> 32: 0 0 6 | 0 0 0<br/> 64: 0 0 6 | 0 0 0<br/> 128: 0 0 6 | 0 0 0<br/> 256: 0 0 6 | 0 0 0<br/> 512: 0 0 6 | 0 0 0<br/> 1024: 750 93 100 | 0 0 0</p> <p>4000 MiB:<br/> <span class="error">&#91;root@cent7c01 cent7s02&#93;</span># dd if=/dev/zero bs=1M count=4000 of=./ost0; echo 3 &gt; /proc/sys/vm/drop_caches ; echo clear &gt; /sys/fs/lustre/ldlm/namespaces/cent7s02-OST0000-osc-ffff88022437c800/lru_size; echo c &gt; /proc/fs/lustre/osc/cent7s02-OST0000-osc-ffff88022437c800/rpc_stats; cat ost0 &gt; /dev/null; cat /proc/fs/lustre/osc/cent7s02-OST0000-osc-ffff88022437c800/rpc_stats<br/> 4000+0 records in<br/> 4000+0 records out<br/> 4194304000 bytes (4.2 GB) copied, 13.1352 s, 319 MB/s<br/> snapshot_time: 1512683761.337612432 (secs.nsecs)<br/> read RPCs in flight: 0<br/> write RPCs in flight: 0<br/> pending write pages: 0<br/> pending read pages: 0</p> <p> read write<br/> pages per rpc rpcs % cum % | rpcs % cum %<br/> 1: 8074 89 89 | 0 0 0<br/> 2: 0 0 89 | 0 0 0<br/> 4: 0 0 89 | 0 0 0<br/> 8: 0 0 89 | 0 0 0<br/> 16: 0 0 89 | 0 0 0<br/> 32: 0 0 89 | 0 0 0<br/> 64: 0 0 89 | 0 0 0<br/> 128: 0 0 89 | 0 0 0<br/> 256: 1 0 89 | 0 0 0<br/> 512: 0 0 89 | 0 0 0<br/> 1024: 992 10 100 | 0 0 0<br/> &#8212;</p> <p>Larger sizes remain problematic. So when I hit the cached mb limit on the node, something goes totally off the rails, I think. Perhaps we're getting that from the kernel, but it's still a major degradation.</p> <p>Looking at when these smaller reads happen, they're clustered in the middle of the job, to the point where there are <b>no</b> 1024 page reads for a while (I used the D_INODE debug in osc_build_rpc for this).</p> <p>This is for the 4000 MiB case.</p> <p>This is the first RPC, which is naturally enough 1 page:<br/> 00000008:00000002:0.0F:1512684358.524801:0:5975:0:(osc_request.c:2073:osc_build_rpc()) @@@ 1 pages, aa ffff880034753170. now 1r/0w in flight req@ffff8</p> <p>That's followed by a large # of 1024 page RPCs, though with 1 page RPCs mixed in every so often, which seems weird. It looks like there is no point at which we hit a steady state of only 1024 page RPCs.</p> <p>Then, here's the first of the set of only 1 page RPCs:<br/> 00000008:00000002:2.0:1512684366.335774:0:5973:0:(osc_request.c:2073:osc_build_rpc()) @@@ 1 pages, aa ffff88023602c170. now 1r/0w in flight req@ffff88</p> <p>There are then NO 1024 page RPCs for some thousands of RPCs. Weirdly, at the end it seems to recover and we do some 1024 page RPCs again. Here's the first of those:<br/> 00000008:00000002:0.0:1512684370.524090:0:5974:0:(osc_request.c:2073:osc_build_rpc()) @@@ 1024 pages, aa ffff8800a506c770. now 1r/0w in flight req@fff</p> <p>And here's the last RPC period:<br/> 00000008:00000002:1.0:1512684371.129494:0:5972:0:(osc_request.c:2073:osc_build_rpc()) @@@ 1022 pages, aa ffff88021f8c1c70. now 2r/0w in flight req@fff</p> <p>So spend 8 second sending mostly large RPCs, then 4 seconds sending only 4 KiB RPCs, then another ~1 second sending large RPCs again.</p> <p>That means that, as you said, we're only sending a few % of the data in those RPCs - about 3% in this case.<br/> But they're taking about 30% of the total time, and it's all in one big lump.</p> <p>Something's wrong.</p> <p>This is regreassion of last version of my patch when I turned off async read ahead if PIO flag is not enabled. I'm going to fix this. But anyway I cannot avoid several 1K requests. I think we should not fix this because of this leed low latency in requests. This happens when user request miss page and async read ahead is initiated but minwhile it request a single page only and then unlock user application. In the next loop of reading next page will be available from async read ahead.</p> <p>The last version of patch don't have an issue with RPC splitting. For reading in my VM machine I have the following:</p> <p>with PIO disabled:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> read write pages per rpc rpcs % cum % | rpcs % cum % 1: 3 4 4 | 0 0 0 2: 0 0 4 | 0 0 0 4: 0 0 4 | 0 0 0 8: 0 0 4 | 0 0 0 16: 0 0 4 | 0 0 0 32: 0 0 4 | 0 0 0 64: 0 0 4 | 0 0 0 128: 0 0 4 | 0 0 0 256: 0 0 4 | 0 0 0 512: 1 1 6 | 0 0 0 1024: 62 93 100 | 0 0 0 </pre> </div></div> <p>with PIO enabled:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> read write pages per rpc rpcs % cum % | rpcs % cum % 1: 2 2 2 | 0 0 0 2: 0 0 2 | 0 0 0 4: 0 0 2 | 0 0 0 8: 0 0 2 | 0 0 0 16: 0 0 2 | 0 0 0 32: 0 0 2 | 0 0 0 64: 0 0 2 | 0 0 0 128: 0 0 2 | 0 0 0 256: 1 1 4 | 0 0 0 512: 4 5 10 | 0 0 0 1024: 61 89 100 | 0 0 0 </pre> </div></div> <p>Great <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p><a href="https://www.eofs.eu/_media/events/devsummit17/patrick_farrell_laddevsummit_pio.pdf" class="external-link" target="_blank" rel="nofollow noopener">https://www.eofs.eu/_media/events/devsummit17/patrick_farrell_laddevsummit_pio.pdf</a></p> <p>This is old and out of date, but I wanted to make sure these slides were seen.  I think the performance of the readahead code would probably be helped a lot by changes to the parallelization framework (as would the performance of pio itself).</p> <p>So slides 8, 9, and 10 would probably be of particular interest here.  There are significant performance improvements available for PIO just by going from padata to something simpler.  Also, the CPU binding behavior of padata is pretty bad - Binding explicitly to one CPU is problematic.  Padata seems to assume the whole machine is dedicated, which is not a friendly assumption.  (I discovered its CPU binding behavior because I saw performance problems - A particular CPU would be busy and the work assigned to that CPU would be delayed, which delays the completion of the whole i/o.  At this time, other CPUs were idle, and not binding to a specific CPU would have allowed one of them to be used.)</p> <p>Also, apologies for not posting these last year.</p> <p>Thanks for slides. I will loop at them carefully. But for now I disagree that padata API have a big overhead. It's mostly negligible comparing with other overhead to pass work into different thread. But having many threads will leads a sheduler delay to switch under heavy loads. So, I think padata will work more stable and predictable in this case.</p> <p>Thanks Patrick for the heads up on ktask. I will be watching it closely and give it a spin under this ticket.</p> <p><a href="https://jira.whamcloud.com/browse/LU-12403" title="add e2fsprog support for RHEL-8" class="issue-link" data-issue-key="LU-12403"><del>LU-12403</del></a> will do this work correctly.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=simmonsja" class="user-hover" rel="simmonsja">simmonsja</a>, for the record, you mean <a href="https://jira.whamcloud.com/browse/LU-12043" title="improve Lustre single thread read performances" class="issue-link" data-issue-key="LU-12043"><del>LU-12043</del></a>. <a href="https://jira.whamcloud.com/browse/LU-12403" title="add e2fsprog support for RHEL-8" class="issue-link" data-issue-key="LU-12403"><del>LU-12403</del></a> is "add e2fsprog support for RHEL-8". <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> Duplicate LU-1056 Related LU-6658 LU-8709 LU-11069 LU-10367 LU-9618 LU-11825 LU-12043 Development Rank 1|hzyz1b: Rank (Obsolete) 9223372036854775807