https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8709 Lustre <p>Tracker for parallel async readahead improvement from DDN, as described in <a href="http://www.eofs.eu/_media/events/lad16/19_parallel_readahead_framework_li_xi.pdf" class="external-link" target="_blank" rel="nofollow noopener">http://www.eofs.eu/_media/events/lad16/19_parallel_readahead_framework_li_xi.pdf</a></p> <p>Note that it would be very desirable in the single thread case if the copy_to_user was also handled in parallel, as this is a major CPU overhead on many-core systems and if it can be parallelized it may increase the peak read performance.</p> <p>As for lockahead integration with readahead, I agree that this is possible to do this, but it is only useful if the client doesn't get full-file extent locks. It would also be interesting if the write code detected sequential or strided writes and did lockahead at write time.</p> LU-8709

parallel asynchronous readahead

Improvement Minor Resolved Duplicate Li Xi Andreas Dilger perf_optimization Fri, 14 Oct 2016 23:03:15 +0000 Wed, 10 Feb 2021 19:57:18 +0000 Wed, 10 Feb 2021 19:57:18 +0000 Lustre 2.10.0 0 17 <p>About lockahead/readahead integration. There's not much use for lockahead in the "read" case, since locks can overlap. At least, I've never come up with a realistic scenario where it's relevant/helpful.</p> <p>Assuming it's not totally invalid to tie this stuff together, we could do possibly do something positive by recognizing strided writing. There are some complexities there - For example, we'd need to make a non-blocking lockahead lock request for at least the first one, to cancel the full file locks the clients are exchanging (normal lockahead locks must be non-blocking, so they can be requested many at a time). There's also some danger around the different clients not being coordinated when they go in to lockahead mode.</p> <p>I can think of how to probably make it work, but I'm not totally convinced it's worth the effort.</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/23552" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23552</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8709" title="parallel asynchronous readahead" class="issue-link" data-issue-key="LU-8709"><del>LU-8709</del></a> llite: implement parrallel asynchronous readahead<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5629bfc1afbcf8913d738118f9085d3e5901f831</p> <p>The pushed patch is definitely not the final version, and I am going to test and optimize it further. I really needs your review and feedback in this progress. Thank you in advance!</p> <p>Can you explain in more details what benchmarks you used for testing? It's not clear for me why we need new readahead algorithm instead of using standard from Linux kernel. I just make it asynchronous and parallelize for Lustre needs. This allows me significantly increase a read performance without implementing new algorithm.</p> <p> </p> <p>Closing as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a>.</p> <p>I am repopening this ticket because the patch inLU-8960 doesn't really improve performance as much as we can get from this patch.</p> <p>Here is performance resutls and comapring of patch for <a href="https://jira.whamcloud.com/browse/LU-8709" title="parallel asynchronous readahead" class="issue-link" data-issue-key="LU-8709"><del>LU-8709</del></a> and <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a>.<br/> Unfortunately, I don't see any performance improvements from patch <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> and it's even worse than without patch.<br/> In fact, <a href="https://jira.whamcloud.com/browse/LU-8709" title="parallel asynchronous readahead" class="issue-link" data-issue-key="LU-8709"><del>LU-8709</del></a> (patch <a href="http://review.whamcloud.com/23552" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23552</a>) is diferent approch and 2.3x performance improvements from patch.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># pdsh -w oss[01-06],mds[11-12],dcr-vm[1-4],c[01-32] 'echo 3 &gt; /proc/sys/vm/drop_caches ' # lfs setstripe -S 16m -c -1 /scratch1/out # mpirun -np 1 /work/tools/bin/IOR -w -k -t 1m -b 256g -e -F -vv -o /scratch1/out/file # pdsh -w oss[01-06],mds[11-12],dcr-vm[1-4],c[01-32] 'echo 3 &gt; /proc/sys/vm/drop_caches' # mpirun -np 1 /work/tools/bin/IOR -r -k -E -t 1m -b 256g -e -F -vv -o /scratch1/out/file </pre> </div></div> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <th class='confluenceTh'>Branch</th> <th class='confluenceTh'>Single Thread Read Performance(MB/sec)</th> </tr> <tr> <td class='confluenceTd'>b2_10</td> <td class='confluenceTd'>1,591</td> </tr> <tr> <td class='confluenceTd'>b2_10+<a href="https://jira.whamcloud.com/browse/LU-8709" title="parallel asynchronous readahead" class="issue-link" data-issue-key="LU-8709"><del>LU-8709</del></a></td> <td class='confluenceTd'>3,774</td> </tr> <tr> <td class='confluenceTd'>master</td> <td class='confluenceTd'>1,793</td> </tr> <tr> <td class='confluenceTd'>master+<a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a></td> <td class='confluenceTd'>1,000</td> </tr> <tr> <td class='confluenceTd'>master+<a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a>(pio=1)</td> <td class='confluenceTd'>1,594</td> </tr> <tr> <td class='confluenceTd'>master+<a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a>(cpu_npartitions=10,pio=0)</td> <td class='confluenceTd'>1,000</td> </tr> <tr> <td class='confluenceTd'>master+<a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a>(cpu_npartitions=10,pio=1)</td> <td class='confluenceTd'>1,820</td> </tr> </tbody></table> </div> <p> <span class="image-wrap" style=""><a id="30068_thumb" href="https://jira.whamcloud.com/secure/attachment/30068/30068_Lustre-SingleThread.png" title="Lustre-SingleThread.png" file-preview-type="image" file-preview-id="30068" file-preview-title="Lustre-SingleThread.png"><img src="https://jira.whamcloud.com/secure/thumbnail/30068/_thumb_30068.png" style="border: 0px solid black" role="presentation"/></a></span> </p> <p>We are more investigating what's going on b2_10+<a href="https://jira.whamcloud.com/browse/LU-8709" title="parallel asynchronous readahead" class="issue-link" data-issue-key="LU-8709"><del>LU-8709</del></a>. Atatched is capture cleint performance every 1sec during IOR is running.<br/> When read started on cient and the performance is pretty good (getting ~5.5GB/sec), but once memory usages reached to close to max_cache_mb, performance was getting very nustable. (e.g. sometimes we see more than 5GB/sec, but sometimes less than 3GB/sec)</p> <p>The patch <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> was not used in full speed because of you use small 1Mb transfer buffer (<b>-t 1m</b>). As I mentoined bebofe in common case the transfer buffer is split by strip size and transfer in parallel.</p> <p>Which version of patch <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> was used? In Patch Set 56 I significantly improve algorithm. It should more agressive read ahead.</p> <blockquote> <p>The patch <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> was not used in full speed because of you use small 1Mb transfer buffer (-t 1m). As I mentoined bebofe in common case the transfer buffer is split by strip size and transfer in parallel.</p></blockquote> <p>In fact, 1m is not small. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> And,after patch, it should keep same performance for all operations against before patch unless there are any trade off.</p> <blockquote> <p>Which version of patch <a href="https://jira.whamcloud.com/browse/LU-8964" title="use parallel I/O to improve performance on machines with slow single thread performance" class="issue-link" data-issue-key="LU-8964"><del>LU-8964</del></a> was used? In Patch Set 56 I significantly improve algorithm. It should more agressive read ahead.</p></blockquote> <p>Sure, I can try latest your patch, but please tell us what exactly configurations (e.g. pio, number of partition, RA size, etc.) you prefer.</p> <p>Yeah, 1M is probably more common than any size larger than it.  We should be very concerned with improving 1 MiB i/o.  (It's one thing I would like to change about the PIO code - That it is all implemented on stripe boundaries.  8 MiB stripes are very common these days, and we could benefit from parallelizing i/o smaller than 8 MiB.)</p> <p>Hmm. This is good point to pay more attention to 1Mb size buffer. But splitting into small chunks don't provide benefit because of additional overhead of parallelization. Unfortunatelly we have many restrictions in current Lustre I/O pipeline which prevent me from better parallelization and asyncronizim.</p> <p>I discovered that I had best performance with splitting CPUs into several partitions in the way to have 2 HW cores (whith several hyper threads) into one partition. For example, it you have a machine with 8 HW cores with 2 hyper threds in each core (16 logical CPUs) the best configuration will be split it into 4 partitions. If you have several NUMA nodes it would be good to split each of this node in the same way.</p> <p>I think the 8Mb size transfer buffer is good enough to see the benefits from PIO code.</p> <p>We would like to give this patch a try in a mixed-workload environment on our test filesystem. Is it possible to get this patch for current b2_10 (2.10.7) ?</p> <p>Lukasz, the patch on this ticket is not currently being developed. The patch on <a href="https://jira.whamcloud.com/browse/LU-12043" title="improve Lustre single thread read performances" class="issue-link" data-issue-key="LU-12043"><del>LU-12043</del></a> is the one that will be landing on 2.13.</p> <p>Andreas, thank you for update, <a href="https://jira.whamcloud.com/browse/LU-12043" title="improve Lustre single thread read performances" class="issue-link" data-issue-key="LU-12043"><del>LU-12043</del></a> looks very promising. <br/> I had a quick look at changes and they are all on client side - can we assume that lustre-client 2.13 may have better performance on 2.10 servers ?</p> <p>Correct, with the caveat that <a href="https://jira.whamcloud.com/browse/LU-12043" title="improve Lustre single thread read performances" class="issue-link" data-issue-key="LU-12043"><del>LU-12043</del></a> patch is mainly improving single-threaded read performance. If there are many threads on the client the performance may not be very different, so it depends on your workload.</p> Blocker LU-8726 Duplicate LU-6 Related LU-8413 LU-8964 LU-11416 LU-12043 Development Rank 1|hzyrxz: Rank (Obsolete) 9223372036854775807