https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12518 Lustre <p>Currently, Lustre works well for aligned IO, but performance is pretty bad for unaligned IO stride read, we might need take some efforts to improve this situation.</p> <p>One of the main problem with current stride read is it is based on Page Index, so if<br/> we hit unaligned page case, Stride Read detection will not work well.</p> <p>To support unaligned page stride read, we might change page index to bytes offset thus<br/> stride read pattern detection work well and we won't hit many small pages RPC and readahead window reset.</p> <p>At the same time, we shall keep as much as performances for existed cases and make<br/> sure there won't be obvious regressions for aligned-stride and sequential read.</p> LU-12518

improve Lustre unaligned IO read performances

Improvement Minor Resolved Fixed Wang Shilong Wang Shilong Mon, 8 Jul 2019 10:43:02 +0000 Wed, 13 Sep 2023 04:01:46 +0000 Fri, 14 Feb 2020 13:45:46 +0000 Lustre 2.14.0 0 7 <p>Proposal patch pushed here(will be refreshed with correct LU title)<br/> <a href="https://review.whamcloud.com/#/c/35437/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/35437/</a></p> <p>Here is quick test resutls of patch on ior_hard_read workload(SSF, 47kb, strided read) From Ihara:</p> <p>10 client, 240 process(24/client)</p> <ol> <li>ior -r -R -s 132000 -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /cache1/io500.out/ior_hard/IOR_file -O stoneWallingStatusFile=/cache1/io500.out/ior_hard/stonewall</li> </ol> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>master Max Read: 7373.80 MiB/sec (7731.98 MB/sec) patch Max Read: 19784.09 MiB/sec (20745.12 MB/sec) </pre> </div></div> <p>patch significant improves performance, but there are still some a bit strange behaviors we need to investigate.<br/> As far as I observe stats during IOR.<br/> 1. I see 28GB/sec at beginning of IOR, but when memory reclaims (later read size &gt; llite.*.cached_mb) started, performance dropped and swing.<br/> 2. tail of IOR is a bit longer. some mpi ranks finished earlier, but other ranks were not finished at same time.<br/> That's two thing why IOR reports 20GB/sec even we are getting 28GB/sec at middle of IOR.</p> <p>Here is more test results.</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <th class='confluenceTh'> </th> <th class='confluenceTh'>FPP Read(MB/sec)</th> <th class='confluenceTh'>SSF Read(MB/sec)</th> </tr> <tr> <td class='confluenceTd'>master</td> <td class='confluenceTd'> 44,636</td> <td class='confluenceTd'> 7,731</td> </tr> <tr> <td class='confluenceTd'>master+patch35437<br/> (overstriping, -C 240 -S 16M)</td> <td class='confluenceTd'> 44,318</td> <td class='confluenceTd'> 20,745</td> </tr> </tbody></table> </div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>FPP # ior -r -R -s 132000 -F -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /cache1/io500.out2/ior_hard/IOR_file -O stoneWallingStatusFile=/cache1/io500.out2/ior_hard/stonewall SSF # ior -r -R -s 132000 -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /cache1/io500.out/ior_hard/IOR_file -O stoneWallingStatusFile=/cache1/io500.out/ior_hard/stonewall </pre> </div></div> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35457" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35457</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12518" title="improve Lustre unaligned IO read performances" class="issue-link" data-issue-key="LU-12518"><del>LU-12518</del></a> llite: Accept EBUSY for unaligned stride<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c4ab79e65acadc16ef2e2c9f5eb73ed94f16c976</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35829" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35829</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12518" title="improve Lustre unaligned IO read performances" class="issue-link" data-issue-key="LU-12518"><del>LU-12518</del></a> readahead: convert stride page index to byte<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ac5cded17af7882d30f3dcda9394f618f5a5d6ad</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35893" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35893</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12518" title="improve Lustre unaligned IO read performances" class="issue-link" data-issue-key="LU-12518"><del>LU-12518</del></a> llite: fix stride window increase<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3cabf63b9aba215e702ddbca30fccc0d3a87c800</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35829/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35829/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12518" title="improve Lustre unaligned IO read performances" class="issue-link" data-issue-key="LU-12518"><del>LU-12518</del></a> readahead: convert stride page index to byte<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0923e405511605b66e73a99dda12ed961ca9e30b</p> <p>Shilong, are all of the patches in this ticket still needed?</p> <p>Yup, patches needed for bursting performances up, i am not sure for lustre-wc repo though.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35437/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35437/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12518" title="improve Lustre unaligned IO read performances" class="issue-link" data-issue-key="LU-12518"><del>LU-12518</del></a> llite: support page unaligned stride readahead<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 91d2645515087df3f912b285419cfff73d9fca9e</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37248" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37248</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12518" title="improve Lustre unaligned IO read performances" class="issue-link" data-issue-key="LU-12518"><del>LU-12518</del></a> llite: proper names/types for offset/pages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 46fae0d64ea26ce743eea6265e169c1a8eb7c187</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37248/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37248/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12518" title="improve Lustre unaligned IO read performances" class="issue-link" data-issue-key="LU-12518"><del>LU-12518</del></a> llite: proper names/types for offset/pages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 83d8dd1d7c30c41e837b07b97198ad77bd903eea</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35457/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35457/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12518" title="improve Lustre unaligned IO read performances" class="issue-link" data-issue-key="LU-12518"><del>LU-12518</del></a> llite: Accept EBUSY for page unaligned read<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b9c155065d2ca4a6037a0ca4bfc788d6961fdc8e</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35893/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35893/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12518" title="improve Lustre unaligned IO read performances" class="issue-link" data-issue-key="LU-12518"><del>LU-12518</del></a> llite: fix stride window increase<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0cc48c4adbe540f8e529f80e4262b6ff47649e7c</p> <p>It looks like all this work has now landed for 2.14</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38154" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38154</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12518" title="improve Lustre unaligned IO read performances" class="issue-link" data-issue-key="LU-12518"><del>LU-12518</del></a> llite: rename count and nob variables to bytes<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 921aa269e749d09b9d2bc5827f5f9ffe02b586b8</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/38154/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/38154/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12518" title="improve Lustre unaligned IO read performances" class="issue-link" data-issue-key="LU-12518"><del>LU-12518</del></a> llite: rename count and nob variables to bytes<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 271f838c5cd1c539f4a7de5008dfc7ffebb156c0</p> Related LU-12043 Development Rank 1|i00jbj: Rank (Obsolete) 9223372036854775807