https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13799 Lustre <p>There is a large degree of inefficiency/wasted time in the DIO/AIO path.  This does not show up for DIO normally because of the waiting model. but it shows up easily in AIO.</p> <p>This ticket is to cover a set of improvements to DIO/AIO performance, which will also improve DIO performance once the waiting model is adjusted.  (More on this in <a href="https://jira.whamcloud.com/browse/LU-13798" title="Improve direct i/o performance with multiple stripes: Submit all stripes of a DIO and then wait" class="issue-link" data-issue-key="LU-13798"><del>LU-13798</del></a>.)</p> <p>There is a grab bag of patches to be submitted here, and some further proposals that will probably end up in other tickets.</p> <p>The essence of the improvements is that all pages in a DIO submission are the same, and therefore much of the work done on a per-page basis is irrelevant and can be skipped for DIO.  Note this statement is still compatible with unaligned DIO if it can be implemented in the future - In the case of unaligned DIO, only the first and last pages are different, and that can still be handled.</p> <p>Patches and benchmarks on each patch forthcoming.</p> <p>The total effect of the initial set of patches on my testbed is to raise AIO/DIO performance from around 5 GiB/s to around 9 GiB/s.  I'll get more in to what else can be done shortly. </p> LU-13799

DIO/AIO efficiency improvements

Bug Minor Resolved Fixed Patrick Farrell Patrick Farrell Fri, 17 Jul 2020 17:05:56 +0000 Wed, 9 Mar 2022 17:23:53 +0000 Wed, 26 Jan 2022 19:19:16 +0000 Lustre 2.15.0 0 16 <p>Patrick Farrell (farr0186@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/39437" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39437</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> osc: Don't get time for each page<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 344b38006506252f3dfd5494b555a8891b16bf70</p> <p>Patrick Farrell (farr0186@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/39438" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39438</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> llite: Do not get/put DIO pages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7983c6fff30afb43259afdcf76fbda13193588df</p> <p>Patrick Farrell (farr0186@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/39439" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39439</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> clio: Implement real list splice<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 46d18b197f24cb44b9a905449906c832e028a2f3</p> <p>Patrick Farrell (farr0186@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/39440" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39440</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> osc: Simplify clipping for transient pages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a075a2267f0ee3cbd67059d83136b019bae6e46e</p> <p>There are more patches coming in this series - Will update shortly.</p> <p>Eventual performance result is around 10 GiB/s from one thread, but there's still headroom to go higher.</p> <p>Patrick Farrell (farr0186@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/39441" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39441</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> llite: Remove transient page counting<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4f9ffa2f7bd575aa448b254a87d02277cf5b3059</p> <p>Patrick Farrell (farr0186@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/39442" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39442</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> llite: Modify AIO/DIO reference counting<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c967bb0af12957e533008d6ceea98f112f4d9c99</p> <p>Patrick Farrell (farr0186@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/39443" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39443</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> llite: Move free user pages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0518cec5ee189fb3911a3b4e686fe5de37592c0d</p> <p>Patrick Farrell (farr0186@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/39445" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39445</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> lov: Cache stripe offset calculation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2f7966118b379d670621ef35639465a6a6ab8b9e</p> <p>Patrick Farrell (farr0186@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/39448" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39448</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> clio: Skip prep for transients<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 51f72ba24d56201908282b68c47bb39ff232ac3c</p> <p>Patrick Farrell (farr0186@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/39446" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39446</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> lov: Improve DIO submit<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 83fcd181493256d325f7c9386d1638a3ca6a8dc3</p> <p>Patrick Farrell (farr0186@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/39447" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39447</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> llite: Adjust dio refcounting<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 345ddb5bed66ac0f9169c9c4663a3fb7239bd226</p> <p>With all of the patches here, for a 64 MiB DIO, we get:<br/> mpirun -np 1 $IOR -w -r -t 64M -b 64G -o ./iorfile --posix.odirect<br/> write    8155 MiB/s</p> <p>read     8566 MiB/s</p> <p>For a very large DIO, such as 1 GiB, we get:<br/> mpirun -np 1 $IOR -w -r -t 1GM -b 64G -o ./iorfile --posix.odirect</p> <p>write    9033 MiB/s</p> <p>read    9817 MiB/s</p> <p>This is getting near the limit of my testbed hardware, at least with one interface.  It's possible to use two if you configure with just one CPT though, so...</p> <p>At this point, there is still actually a huge amount of overhead in the DIO/AIO path.</p> <p>If you look at perf with all of these patches in place, cl_page allocation is now over 50% of CPU time.  Converting this from single calls to batch calls and removing a few per-page actions that can be done once per DIO, it's possible to get that down significantly.  This leads to a DIO performance of something like 15 GiB/s.</p> <p>I don't have a patch ready for that - I'm going to open a separate ticket to discuss/describe what else I think can be done to improve this further. </p> <p>Patrick, let me state that you rock. That is all. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Patrick, I've also filed <a href="https://jira.whamcloud.com/browse/LU-13801" title="Enable io_uring interface for Lustre client" class="issue-link" data-issue-key="LU-13801"><del>LU-13801</del></a> for implementing the io_uring interface, which is the successor to libaio, which is a bit sad since we just got libaio working well. </p> <p>(Moved from <a href="https://jira.whamcloud.com/browse/LU-13798" title="Improve direct i/o performance with multiple stripes: Submit all stripes of a DIO and then wait" class="issue-link" data-issue-key="LU-13798"><del>LU-13798</del></a>)<br/> Patrick Farrell (farr0186@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/39482" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39482</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> osc: Improve osc_queue_sync_pages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2671920f69c1ff9c89cc7d390e9692acc809e5dd</p> <p>Patrick Farrell (farr0186@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/43835" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43835</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> osc: Update shrink interval less often<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 644ab49701d6305fb79c41e1a5eed9406b3d30a9</p> <p>Patrick Farrell (farr0186@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/43838" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43838</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> osc: Update shrink interval per i/o<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d5560fc6b5f518992aadf5cf3e8861cbadcc6d86</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39437/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39437/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> osc: Don't get time for each page<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 485976ab451dd6708d4d46bce3bbed9991f5d356</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39439/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39439/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> clio: Implement real list splice<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: dfe2d225b86d4215cbd09e863e8de7547f0d4dae</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39440/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39440/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> osc: Simplify clipping for transient pages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b64b9646f17b771c415e4b39cb8babcdc7541b30</p> <p>Current status on these patches, now that we've started landing some of this.</p> <p>The following patches (in order) are ready, modulo completion of testing (errors appear unrelated) and review:<br/> <a href="https://review.whamcloud.com/39442" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39442</a><br/> <a href="https://review.whamcloud.com/39441" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39441</a><br/> <a href="https://review.whamcloud.com/39446" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39446</a><br/> <a href="https://review.whamcloud.com/39447" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39447</a><br/> <a href="https://review.whamcloud.com/39448" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39448</a><br/> <a href="https://review.whamcloud.com/39482" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39482</a></p> <p>Several of those could use a second review; they are almost all pretty simple.</p> <p>These three patches are <b>not</b> complete:<br/> <a href="https://review.whamcloud.com/39445" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39445</a> (bugs; still trying to work it out...)</p> <p><a href="https://review.whamcloud.com/39443" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39443</a> (reworking implementation)<br/> <a href="https://review.whamcloud.com/39438" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39438</a> (Depends on 39443)</p> <p>Update on status:</p> <p>I've rebased this series on <a href="https://review.whamcloud.com/#/c/44131/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/44131/</a> (<a href="https://jira.whamcloud.com/browse/LU-14805" title="Do not do parallel DIO except when lockless" class="issue-link" data-issue-key="LU-14805"><del>LU-14805</del></a>), which was causing what I thought were bugs in <a href="https://review.whamcloud.com/39445" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39445</a></p> <p>It appears, though, that <a href="https://review.whamcloud.com/39445" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39445</a> is OK and should be ready for review.</p> <p>So, updated status:</p> <p>The following patches (in order) are ready, modulo completion of testing (errors appear unrelated) and review:<br/> <a href="https://review.whamcloud.com/39442" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39442</a><br/> <a href="https://review.whamcloud.com/39441" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39441</a><br/> <a href="https://review.whamcloud.com/39446" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39446</a><br/> <a href="https://review.whamcloud.com/39447" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39447</a><br/> <a href="https://review.whamcloud.com/39448" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39448</a><br/> [https://review.whamcloud.com/39482<br/> ] <a href="https://review.whamcloud.com/39445" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39445</a></p> <p>These patches are <b>not</b> complete:</p> <p><a href="https://review.whamcloud.com/39443" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39443</a> (reworking implementation)<br/> <a href="https://review.whamcloud.com/39438" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39438</a> (Depends on 39443)</p> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/44153" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44153</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> osc: Always set aio in anchor<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e1c9ec5fef42f45748174177be8b7f285103ffba</p> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/44154" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44154</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> llite: Simplify cda_no_aio_complete use<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a485a314dc9f8740ee886c64952ccb422feeac99</p> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/44201" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44201</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> llite: Simplify sync_io references<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ad546d08251c9fc6afe66916154644d19c57315e</p> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/44209" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44209</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> llite: Implement lower/upper aio<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 27fba688585261d4002b170060ac462eceac0fd8</p> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/44268" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44268</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> llite: Remove ldp_file_offset<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 36f0b45793b3ae37c4facb46163062a6636ef023</p> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/44293" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44293</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> llite: Remove unnecessary page get/put<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9b4deec3c7e79688f261f668680cf91135681c4b</p> <p>I wanted to update the on the overall status of this series and related patches.</p> <p>The patches associated with this ticket should be ready for landing, but they're often failing testing due to bugs which have patches in flight.</p> <p>So here's a summary of how I'm imagining getting things landed.</p> <p>First, several bug fixes (in this order):<br/> <a href="https://jira.whamcloud.com/browse/LU-14805" title="Do not do parallel DIO except when lockless" class="issue-link" data-issue-key="LU-14805"><del>LU-14805</del></a> llite: No locked parallel DIO<br/> [https://review.whamcloud.com/44131/<br/> ]<br/> Supporting patches for the next fix:<br/> <a href="https://jira.whamcloud.com/browse/LU-14838" title="Remove old lockless code: Truncate &amp; contention based lockless i/o" class="issue-link" data-issue-key="LU-14838"><del>LU-14838</del></a> osc: Remove lockless truncate<br/> <a href="https://review.whamcloud.com/44204/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44204/</a><br/> <a href="https://jira.whamcloud.com/browse/LU-14838" title="Remove old lockless code: Truncate &amp; contention based lockless i/o" class="issue-link" data-issue-key="LU-14838"><del>LU-14838</del></a> osc: Remove client contention support<br/> [https://review.whamcloud.com/44205/<br/> ]The actual fix:</p> <p><a href="https://jira.whamcloud.com/browse/LU-14814" title="sanity test_398b: osc_cache_writeback_range ASSERTION( hp == 0 &amp;&amp; discard == 0 ) failed:" class="issue-link" data-issue-key="LU-14814"><del>LU-14814</del></a> osc: osc: Do not flush on lockless cancel<br/> <a href="https://review.whamcloud.com/44152/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44152/</a></p> <p>Strictly speaking, <a href="https://jira.whamcloud.com/browse/LU-14838" title="Remove old lockless code: Truncate &amp; contention based lockless i/o" class="issue-link" data-issue-key="LU-14838"><del>LU-14838</del></a> osc: Remove lockless truncate is not <b>required</b>.  It's just that the required patch - <a href="https://jira.whamcloud.com/browse/LU-14838" title="Remove old lockless code: Truncate &amp; contention based lockless i/o" class="issue-link" data-issue-key="LU-14838"><del>LU-14838</del></a> osc: Remove client contention support - has a context dependency on it.  I could resolve that but I think we're going to land the lockless truncate patch anyway, so I didn't bother.  (If anyone feels strongly, I'm happy to rearrange.)</p> <p>That fixes all the known bugs, and frees us up to consider the patches under this ticket, <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a>.</p> <p>That should make it <b>much</b> easier to get those patches through testing.</p> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/44321" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44321</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> tests: Improve test 398b<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d3c6b0b8d1c3c2032164f63989b18a8f0933e8ed</p> <p>Current status...</p> <p>There's a series of patches ready for Oleg &amp; landing, starting here:<br/> <a href="https://review.whamcloud.com/#/c/44131/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44131/</a><br/> (The second patch is this one: <a href="https://review.whamcloud.com/39442/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39442/</a> and I trust Oleg to just follow the sequence from there... heh)</p> <p>These patches are waiting for review:<br/> <a href="https://review.whamcloud.com/#/c/39445/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39445/</a><br/> <a href="https://review.whamcloud.com/44154/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44154/</a><br/> <a href="https://review.whamcloud.com/44153/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44153/</a></p> <p>The following patches also need review &amp; landing (not attached to this ticket, but fix related bugs):<br/> <a href="https://review.whamcloud.com/44204/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44204/</a><br/> <a href="https://review.whamcloud.com/44205" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44205</a><br/> <a href="https://review.whamcloud.com/44152" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44152</a></p> <p>There are also a bunch of patches which are reviewed and waiting for Oleg to land.</p> <p>Once these are in, there is a further set, starting here:<br/> <a href="https://review.whamcloud.com/44209/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44209/</a></p> <p>But I would like to get the other patches landed and that patch &amp; the ones after it rebased, etc, rather than try to include them in one giant drop.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39442/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39442/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> llite: Modify AIO/DIO reference counting<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b3de247b76b410101e166b024d65e2bf23d401ba</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39441/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39441/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> llite: Remove transient page counting<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 587e5aa8342980f761930235714add1cca80686b</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39446/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39446/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> lov: Improve DIO submit<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d31647c017a390c9553a38d82c02fe7001a33a05</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39447/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39447/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> llite: Adjust dio refcounting<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1e4d10af3909452b0eee1f99010d80aeb01d42a7</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39448/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39448/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> clio: Skip prep for transients<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b8553978789ad3dd0776c0543dea4641804d0ac5</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39482/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39482/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> osc: Improve osc_queue_sync_pages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 87c4535f7a5d239aad4e936545a72d0199ccd9ba</p> <p>Oleg,</p> <p>Thanks for getting those merged in.</p> <p>The latest drop to master covers all of the fixes associated with <b>other</b> tickets that I think are required, so this series should be good to go now.</p> <p>Here's the current status of the patches here...  I am still <b>hoping</b> to get everything here in to 2.15.  There should not be any more patches <b>added</b> to this ticket at this point.</p> <p>This patch needs review again, but hasn't much changed:<br/> <a href="https://review.whamcloud.com/39445/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39445/</a></p> <p>After that, patches get a little more complicated.  I've rebased all the remaining patches on to tip of master + the test 398b improvement, to get them some more testing.  (398b improvement: <a href="https://review.whamcloud.com/44321/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44321/</a>)</p> <p>Here's the set:<br/> <a href="https://review.whamcloud.com/44154/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44154/</a><br/> <a href="https://review.whamcloud.com/44153/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44153/</a><br/> <a href="https://review.whamcloud.com/44209/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44209/</a></p> <p><a href="https://review.whamcloud.com/39443/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39443/</a></p> <p><a href="https://review.whamcloud.com/39438/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39438/</a></p> <p><a href="https://review.whamcloud.com/44268/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44268/</a></p> <p><a href="https://review.whamcloud.com/44293/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44293/</a></p> <p> </p> <p>A comment on performance.  I have not retested this recently, so definitely take this with some salt.</p> <p>The currently landed set of patches should put us at around 7 GiB/s.  The lov caching patch should take that to around 8 GiB/s.</p> <p>The remaining set of patches should push the rest of the way to around 10 GiB/s.</p> <p>Then that's where this ticket stops and the work is picked up in other tickets.</p> <p><a href="https://jira.whamcloud.com/secure/attachment/40203/LU-13799.xlsx" class="external-link" rel="nofollow">https://jira.whamcloud.com/secure/attachment/40203/LU-13799.xlsx</a><br/> This is the performance comparison of master vs master + patch <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> series (<a href="https://review.whamcloud.com/#/c/44445/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/44445/</a>) on the real storage system.<br/> 1.6GB/sec vs 13GB/sec Write, 1.5GB/sec vs 14GB/sec Read at xfersize=64m in Lustre stripesize=1m.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sihara" class="user-hover" rel="sihara">sihara</a>:<br/> <a href="https://review.whamcloud.com/#/c/44445/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/44445/</a> is one of the EXA6 versions of these patches.  Did you mean to link a different one?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sihara" class="user-hover" rel="sihara">sihara</a> how many OSTs did you have in your system and how many stripes did you use?</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>lctl set_param osc.<b>.max_pages_per_rpc=16M osc.</b>.checksums=0 osc.*.max_rpcs_in_flight=16 </td> </tr> </tbody></table> </div> <p> </p> <p>I'm just curious because it's possible some of these were limited by max_rpcs_in_flight.</p> <p>Still, that performance is excellent - significantly better than I expected.</p> <p>OK, I see in the spreadsheet you used:<br/> <a href="https://review.whamcloud.com/#/c/44293/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/44293/</a></p> <p>That is missing:<br/> <a href="https://review.whamcloud.com/#/c/39445/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/39445/</a></p> <p>Because I did not put it in the series.  If possible, you might want to retest with that patch added.</p> <blockquote><p>I'm just curious because it's possible some of these were limited by max_rpcs_in_flight.</p></blockquote> <p>Yeah, i also thought that helps, but didn't help very much. because it still doesn't reach to max_rpcs_in_flight limit somehow. see example.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl set_param osc.*.rpc_stats=clear # lfs setstripe -c 8 -S 1M /ai400x2/ior.out/ # mpirun -np 1 ior -w -r -t 256m -b 8G -o /ai400x2/ior.out/file --posix.odirect -e # lctl get_param osc.*.rpc_stats read write pages per rpc rpcs % cum % | rpcs % cum % 1: 0 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 0 0 0 | 0 0 0 32: 0 0 0 | 0 0 0 64: 0 0 0 | 0 0 0 128: 0 0 0 | 0 0 0 256: 1024 100 100 | 1024 100 100 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 142 13 13 | 856 83 83 2: 873 85 99 | 152 14 98 3: 5 0 99 | 9 0 99 4: 2 0 99 | 6 0 99 5: 1 0 99 | 1 0 100 6: 1 0 100 | 0 0 100 </pre> </div></div> <p>98-99% of rpcs is still 2 rpcs in flight here.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/39445/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39445/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> lov: Cache stripe offset calculation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 14db1faa0fbe813fed616435303753d390f45827</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/44154/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44154/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> llite: Simplify cda_no_aio_complete use<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b60bd21ec5d5f34ed79c63158860b6f5e948dba2</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/44153/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44153/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> osc: Always set aio in anchor<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: eadccb33ac4bbe54a01da5168f6170702f9b2629</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/44209/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44209/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> llite: Implement lower/upper aio<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 46ff76137160b66f1d4437b3443859027faae9c4</p> <p>"Patrick Farrell &lt;pfarrell@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/46135" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46135</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> tests: Reduce I/O sizes<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0bd3f71bdced8c80513199a6ffebed96a4d0fb54</p> <p>"Patrick Farrell &lt;pfarrell@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/46139" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46139</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> tests: Use fallocate to fill OSTs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 53aa72a20e7af4cdbfd825a8f35100d901266903</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/39443/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39443/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> llite: Move free user pages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7f9b8465bc1125e51aa29cdc27db9a9d6fdc0b89</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/39438/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39438/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> llite: Do not get/put DIO pages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 881b4c722296ff7ac22c6fd7988363f2cdad9f1e</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/44293/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44293/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13799" title="DIO/AIO efficiency improvements" class="issue-link" data-issue-key="LU-13799"><del>LU-13799</del></a> llite: Remove unnecessary page get/put<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c2e94f08cf3ff000b350faf61b6d25ebbad7970e</p> <p>All of the patches have landed to master for this.  Moved the last few test changes to <a href="https://jira.whamcloud.com/browse/LU-15843" title="Crash when umount mdt targets lnet with llstat running." class="issue-link" data-issue-key="LU-15843">LU-15843</a>.</p> <p>&gt; Moved the last few test changes to <a href="https://jira.whamcloud.com/browse/LU-15843" title="Crash when umount mdt targets lnet with llstat running." class="issue-link" data-issue-key="LU-15843">LU-15843</a><br/> Typo?<br/> <a href="https://jira.whamcloud.com/browse/LU-15843" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-15843</a> gives "404 - Not found" and <a href="https://review.whamcloud.com/#/q/LU-15843" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/q/LU-15843</a> doesn't turn up anything either.</p> <p>I think that it's <a href="https://jira.whamcloud.com/browse/LU-15483" title="Minor DIO test improvements" class="issue-link" data-issue-key="LU-15483">LU-15483</a> </p> Related LU-13802 LU-13805 LU-13814 LU-15483 LU-13801 LU-13798 Development Rank 1|i015fz: Rank (Obsolete) 9223372036854775807 Severity