https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9906 Lustre <p>When Lustre releases a lot of cached pages at once, it still calls page_release, instead of pagevec_release. When clearing OST ldlm lock lrus, the ldlm_bl threads end up spending much of their time contending for the zone lock taken by page_release.</p> <p>With many namespaces and parallel lru clearing (as Cray does at the end of each job), this can be a significant time sink. Using pagevec release is much better. Patch coming shortly.</p> LU-9906

Allow Lustre page dropping to use pagevec_release

Bug Minor Resolved Fixed Patrick Farrell Patrick Farrell performance Wed, 23 Aug 2017 19:52:59 +0000 Thu, 19 Dec 2019 00:55:07 +0000 Wed, 21 Nov 2018 05:35:42 +0000 Lustre 2.12.0 Lustre 2.10.7 0 9 <p>Patrick Farrell (paf@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/28667" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28667</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9906" title="Allow Lustre page dropping to use pagevec_release" class="issue-link" data-issue-key="LU-9906"><del>LU-9906</del></a> clio: use pagevec_release for many pages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: da6033fa0d989c5a8ff5a0bf9d1a8d1f4350a0b1</p> <p>Quoting Andreas in <a href="https://jira.whamcloud.com/browse/LU-9920" title="Use pagevec for marking pages dirty" class="issue-link" data-issue-key="LU-9920"><del>LU-9920</del></a>:<br/> "Patrick, a similar issue exists when pages are dropped from cache upon lock cancellation. It would be useful to clean this up to use invalidate_page_range() or similar to drop pages from cache (at least in stripe_size chunks) instead of doing it one page at a time as it does today."</p> <p>Invalidate_page_range is something else, but I think this does what you're talking about. I don't think we can drop pages in such large chunks, pagevec_release is the best I'm aware of without writing our own. (And I wonder about holding the relevant lock long enough to drop stripe_size chunks.)</p> <p>Patrick Farrell (paf@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/30531" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30531</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9906" title="Allow Lustre page dropping to use pagevec_release" class="issue-link" data-issue-key="LU-9906"><del>LU-9906</del></a> osd: use pagevec for putting pages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f48a740f3bad58ffda44d268454775a4fd26d5a6</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30531/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30531/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9906" title="Allow Lustre page dropping to use pagevec_release" class="issue-link" data-issue-key="LU-9906"><del>LU-9906</del></a> osd: use pagevec for putting pages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 2a2adfd04245a24148d8de29b8558cd98c92bffa</p> <p>patch <a href="https://review.whamcloud.com/#/c/28667" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28667</a> gives huge contributions for single client performance improvements.<br/> In fact, today, there is a single performance b/w limit if network b/w is higher than IB EDR bandwidth. (e.g. 2 x IB EDR with MR on client)<br/> This is not LNET/MR problem, but we confirmed this is because of overhead on lru reclaim in CLIO.<br/> Using pagevec for lru reclaim in addition original patch 28667 shows 32% write and ~60% read performance gains.</p> <p>Here is test results. <br/> I've tested with both 1MB buffered IO and 16MB O_DIRECT to make sure no LNET/MR issue and saturate network bandwith without IO pass of buffered IO.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1 x client (2 x Intel Platinum 8160 CPU @ 2.10GHz, 192GB Memory) parameter lctl set_param osc.*.max_pages_per_rpc=16M osc.*.max_rpcs_in_flight=16 osc.*.max_dirty_mb=512 osc.*.checksums=0 llite.*.max_read_ahead_mb=2048 IOR command mpirun -np 48 ior -w -r -t 16m -b 16g -F -e -vv -o /scratch0/file -i 1 -B (O_DIRECT) mpirun -np 48 ior -w -r -t 16m -b 16g -F -e -vv -o /scratch0/file -i 1 (buffered) </pre> </div></div> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <th class='confluenceTh'> </th> <th class='confluenceTh'>mode</th> <th class='confluenceTh'>write(GB/s)</th> <th class='confluenceTh'>read(GB/s)</th> </tr> <tr> <td class='confluenceTd'>master</td> <td class='confluenceTd'>O_DIRECT</td> <td class='confluenceTd'>20.8</td> <td class='confluenceTd'>21.8</td> </tr> <tr> <td class='confluenceTd'>master+patch28667</td> <td class='confluenceTd'>O_DIRECT</td> <td class='confluenceTd'>20.7</td> <td class='confluenceTd'>22.2</td> </tr> <tr> <td class='confluenceTd'>master</td> <td class='confluenceTd'>Buffered</td> <td class='confluenceTd'>11.6</td> <td class='confluenceTd'>12.3</td> </tr> <tr> <td class='confluenceTd'>master+patch28667</td> <td class='confluenceTd'>Buffered</td> <td class='confluenceTd'>15.3</td> <td class='confluenceTd'>19.6</td> </tr> </tbody></table> </div> <p>That's <b>really</b> impressive.</p> <p>What kernel version are you running there?  I'm curious specifically if you have queued spinlocks.  I haven't looked at lru_reclaim specifically, but the other areas affected by this patch got much better with new kernel versions.  (ie the patch is less important if you have queued spinlocks)</p> <p>i'm testing on 3.10.0-693.21.1.el7.x86_64.<br/> please see two attached flamegraph for ior read.<br/> <a href="https://jira.whamcloud.com/secure/attachment/31475/master-read.svg" class="external-link" rel="nofollow">https://jira.whamcloud.com/secure/attachment/31475/master-read.svg</a> (without patch)<br/> <a href="https://jira.whamcloud.com/secure/attachment/31474/master-patch28667-read.svg" class="external-link" rel="nofollow">https://jira.whamcloud.com/secure/attachment/31474/master-patch28667-read.svg</a> (with patch 28667)</p> <p>cost reduction at discard_pagevec() is from 57.59% to 17.48% after patch.</p> <p>Huh!  Thank you for the detailed look.  I am surprised it's so large with the queued spinlocks, but I'm glad it's helping so much.  Nice find.</p> <p>This is great. It shows that the performance is nearly identical for buffered and unbuffered large reads. </p> <p>It would seem like the next big user is <tt>osc_lru_alloc()</tt>, but it may be that looks like it is taking a lot of time because there is an enforced wait when there are not enough pages. Given that we are very close to peak performance for the reads, it probably makes more sense to focus on improving the write side.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/28667/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28667/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9906" title="Allow Lustre page dropping to use pagevec_release" class="issue-link" data-issue-key="LU-9906"><del>LU-9906</del></a> clio: use pagevec_release for many pages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b4a959eb61bc7e6a64261c704f3f3f5e220c2f02</p> <p>Landed for 2.12</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33988" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33988</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9906" title="Allow Lustre page dropping to use pagevec_release" class="issue-link" data-issue-key="LU-9906"><del>LU-9906</del></a> osd: use pagevec for putting pages<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: e380923f87494519f8a9281ace0c53054f8aab5c</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33988/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33988/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9906" title="Allow Lustre page dropping to use pagevec_release" class="issue-link" data-issue-key="LU-9906"><del>LU-9906</del></a> osd: use pagevec for putting pages<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 76f01221aaf3c4a65a4f1b9af1363838921843a1</p> <p>Landing just the OSD side patch to b2_10 is good here - It was required for some kernel compatibility changes (<a href="https://jira.whamcloud.com/browse/LU-10565" title="Handle some small features in upstream" class="issue-link" data-issue-key="LU-10565"><del>LU-10565</del></a>), and is trivial.</p> <p>There is no need to land the other patch from this ticket - <a href="https://review.whamcloud.com/28667/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28667/</a> <a href="https://jira.whamcloud.com/browse/LU-9906" title="Allow Lustre page dropping to use pagevec_release" class="issue-link" data-issue-key="LU-9906"><del>LU-9906</del></a> clio: use pagevec_release for many pages.  The two patches here are independent, and the clio one is non-trivial.  Not a good candidate for a maintenance branch.</p> Related LU-9920 Development Rank 1|hzziw7: Rank (Obsolete) 9223372036854775807 Severity