https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11775 Lustre <p>After landing <a href="https://jira.whamcloud.com/browse/LU-9906" title="Allow Lustre page dropping to use pagevec_release" class="issue-link" data-issue-key="LU-9906"><del>LU-9906</del></a> the read performance improved to an acceptable level. Write performance also increased but still needs some improvements to reach the same level. This ticket tracks the work.</p> LU-11775

buffered write single client improvements

Improvement Minor Resolved Fixed Dongyang Li Dongyang Li Fri, 14 Dec 2018 04:33:45 +0000 Sat, 13 Jun 2020 11:25:25 +0000 Sat, 13 Jun 2020 11:25:25 +0000 Lustre 2.13.0 0 6 <p>Li Dongyang (dongyangli@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/33858" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33858</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11775" title="buffered write single client improvements" class="issue-link" data-issue-key="LU-11775"><del>LU-11775</del></a> osc: reduce lock contention in osc_unreserve_grant<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ffcc069b2b654f6437b090b8ff8c67b5d1ac2ea6</p> <p>Li Dongyang (dongyangli@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/33859" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33859</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11775" title="buffered write single client improvements" class="issue-link" data-issue-key="LU-11775"><del>LU-11775</del></a> osc: reduce atomic ops in osc_enter_cache_try<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 95d0af71efca35ea610bbddec2025100b052cdf5</p> <p>Li Dongyang (dongyangli@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/33860" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33860</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11775" title="buffered write single client improvements" class="issue-link" data-issue-key="LU-11775"><del>LU-11775</del></a> osc: check if opg is in lru list without locking<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4225477391bd60784151a6102f1fc989e11f7a92</p> <p>Li Dongyang (dongyangli@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/33861" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33861</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11775" title="buffered write single client improvements" class="issue-link" data-issue-key="LU-11775"><del>LU-11775</del></a> obdclass: protect imp_sec using rwlock_t<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 87b1a916e6955c10943d6ee6b9b4d9af393c93cc</p> <p>Li Dongyang (dongyangli@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/33862" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33862</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11775" title="buffered write single client improvements" class="issue-link" data-issue-key="LU-11775"><del>LU-11775</del></a> osc: check imp_invalid without imp_lock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: db8dda7213403e2b2caedf33b55e4fa5f2e62db6</p> <p>As we can see <a href="https://jira.whamcloud.com/browse/LU-9920" title="Use pagevec for marking pages dirty" class="issue-link" data-issue-key="LU-9920"><del>LU-9920</del></a> can reduce the overhead in set_page_dirty/write_commit_callback.</p> <p>There is still __ _raw_spin_lock from osc_page_cache_add which I can not figure out the exact location.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> mpirun -np 32 ior -w -t 16m -b 16g -F -e -vv -o /cache1/file -k</pre> </div></div> <p>master:</p> <p>Max Write: 15237.92 MiB/sec (15978.12 MB/sec)</p> <p>master + <a href="https://jira.whamcloud.com/browse/LU-11775" title="buffered write single client improvements" class="issue-link" data-issue-key="LU-11775"><del>LU-11775</del></a>:</p> <p>Max Write: 17045.11 MiB/sec (17873.09 MB/sec)</p> <p>master + <a href="https://jira.whamcloud.com/browse/LU-11775" title="buffered write single client improvements" class="issue-link" data-issue-key="LU-11775"><del>LU-11775</del></a> + <a href="https://jira.whamcloud.com/browse/LU-9920" title="Use pagevec for marking pages dirty" class="issue-link" data-issue-key="LU-9920"><del>LU-9920</del></a>:</p> <p>Max Write: 17476.77 MiB/sec (18325.72 MB/sec)</p> <p>Li,</p> <p>Looking at your graphics, I think maybe osc_lru_use has been inlined in to osc_page_cache_add.  I haven't dug in to "<a href="https://jira.whamcloud.com/browse/LU-11775" title="buffered write single client improvements" class="issue-link" data-issue-key="LU-11775"><del>LU-11775</del></a> osc: check if opg is in lru list without locking" yet, but I don't think it would stop us from hitting that lock entirely.</p> <p>If you'd like to avoid inlining, the easiest way I've found is calling that function via a function pointer.  Then you could see it easily in your traces again.  (There are various "don't inline" directives, but I've never been able to get them to work when building Lustre.)</p> <p>Patrick, thanks for the nice tip!</p> <p>Yes I've been using noinline directive but apparently it did not work.</p> <p>I will give it a shot when I can get to the environment, and provide an update on <a href="https://jira.whamcloud.com/browse/LU-9920" title="Use pagevec for marking pages dirty" class="issue-link" data-issue-key="LU-9920"><del>LU-9920</del></a> as well.</p> <p>Cheers</p> <p>DY</p> <blockquote> <p>master:<br/> Max Write: 15237.92 MiB/sec (15978.12 MB/sec)</p> <p>master + <a href="https://jira.whamcloud.com/browse/LU-11775" title="buffered write single client improvements" class="issue-link" data-issue-key="LU-11775"><del>LU-11775</del></a>:<br/> Max Write: 17045.11 MiB/sec (17873.09 MB/sec)</p> <p>master + <a href="https://jira.whamcloud.com/browse/LU-11775" title="buffered write single client improvements" class="issue-link" data-issue-key="LU-11775"><del>LU-11775</del></a> + <a href="https://jira.whamcloud.com/browse/LU-9920" title="Use pagevec for marking pages dirty" class="issue-link" data-issue-key="LU-9920"><del>LU-9920</del></a>:<br/> Max Write: 17476.77 MiB/sec (18325.72 MB/sec)</p></blockquote> <p>This information should go into the commit comments of the various patches, like <a href="https://review.whamcloud.com/28667" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28667</a></p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33859/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33859/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11775" title="buffered write single client improvements" class="issue-link" data-issue-key="LU-11775"><del>LU-11775</del></a> osc: reduce atomic ops in osc_enter_cache_try<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8b364fbd6bd9e0088440e6d6837861a641b923a0</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33860/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33860/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11775" title="buffered write single client improvements" class="issue-link" data-issue-key="LU-11775"><del>LU-11775</del></a> osc: check if opg is in lru list without locking<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b3af0798682b24d0f39424a7db12c9710a0e64b4</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33858/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33858/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11775" title="buffered write single client improvements" class="issue-link" data-issue-key="LU-11775"><del>LU-11775</del></a> osc: reduce lock contention in osc_unreserve_grant<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8a1ae45a3e4f828346c890bf4ccacbd68ea6752c</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33861/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33861/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11775" title="buffered write single client improvements" class="issue-link" data-issue-key="LU-11775"><del>LU-11775</del></a> obdclass: protect imp_sec using rwlock_t<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8ed361345154138473707855b2030bad371e96dd</p> <p>Dongyang, is there still work to be done for this ticket? It looks like the patches are included in 2.13.</p> <p>No more patches and I think this ticket can be closed now</p> Related LU-9920 LU-11942 Development Rank 1|i0080f: Rank (Obsolete) 9223372036854775807