https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5621 Lustre <p>We found a performance regression in master branch on our metadata scalability and regression testing.<br/> Lustre-2.6 server is more than 50% slower against lustre-2.5 server on file creation/removal to shared directory. Here is test results.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">mdtest -n $((1280000/NP)) -i 3 -p 10 -d /lustre_{0-$NP}/mdtest.out -F -C -r </pre> </div></div> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>File creation(Shared directory)</td> <td class='confluenceTd'>2.5 server</td> <td class='confluenceTd'> 2.6 server</td> </tr> <tr> <td class='confluenceTd'>32 threads</td> <td class='confluenceTd'>69035</td> <td class='confluenceTd'>33354</td> </tr> <tr> <td class='confluenceTd'>64 threads</td> <td class='confluenceTd'>80798</td> <td class='confluenceTd'>30619</td> </tr> <tr> <td class='confluenceTd'>128 threads</td> <td class='confluenceTd'>69896</td> <td class='confluenceTd'>29956</td> </tr> <tr> <td class='confluenceTd'>256 threads</td> <td class='confluenceTd'>64860</td> <td class='confluenceTd'>29951</td> </tr> <tr> <td class='confluenceTd'>512 threads</td> <td class='confluenceTd'>70733</td> <td class='confluenceTd'>30070</td> </tr> <tr> <td class='confluenceTd'>1024 threads</td> <td class='confluenceTd'>73064</td> <td class='confluenceTd'>30540</td> </tr> </tbody></table> </div> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>File removal(Shared directory)</td> <td class='confluenceTd'>2.5 server</td> <td class='confluenceTd'> 2.6 server</td> </tr> <tr> <td class='confluenceTd'>32 threads</td> <td class='confluenceTd'>61087</td> <td class='confluenceTd'>52609</td> </tr> <tr> <td class='confluenceTd'>64 threads</td> <td class='confluenceTd'>82981</td> <td class='confluenceTd'>46370</td> </tr> <tr> <td class='confluenceTd'>128 threads</td> <td class='confluenceTd'>72802</td> <td class='confluenceTd'>37697</td> </tr> <tr> <td class='confluenceTd'>256 threads</td> <td class='confluenceTd'>78388</td> <td class='confluenceTd'>35324</td> </tr> <tr> <td class='confluenceTd'>512 threads</td> <td class='confluenceTd'>81033</td> <td class='confluenceTd'>41058</td> </tr> <tr> <td class='confluenceTd'>1024 threads</td> <td class='confluenceTd'> 79155</td> <td class='confluenceTd'>40544</td> </tr> </tbody></table> </div> lustre-2.6.52 LU-5621

Performance regression in 2.6 branch on file operations to shared directory

Bug Major Resolved Won't Fix Peter Jones Shuichi Ihara Mon, 15 Sep 2014 07:35:20 +0000 Wed, 21 Mar 2018 15:20:01 +0000 Wed, 21 Mar 2018 15:20:01 +0000 Lustre 2.7.0 0 9 <p>What is the client version? same for two tests?</p> <p>tested with 2.5 clients for both testing. I also have tested 2.6 client for 2.6 server testing(not 2.5 server test), but it was same results.</p> <p>John Hammond has ever submitted a patch about this but I don't know the current status. Please refresh my memory, John.</p> <p>Not sure it is a known issue or not. If it is not, it is worth to run mds-survey to see whether the problem is in LOD/OSD. </p> <p>OK, but "git bisect" would be easy way to find what exactly commit affects this regression. I will do it.</p> <p>Once we see the results of the git bisect I'll work out who to assign this to</p> <p>Jinshan, do you mean the following?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 708d85a652a77f85153790e6cca1b7a2b91947cf Author: John L. Hammond &lt;john.hammond@intel.com&gt; Date: Thu Jan 30 11:07:13 2014 -0600 LU-4398 mdt: acquire an open lock for write or execute In mdt_object_open_lock() opens for write or execute will always acquire an open lock of the appropriate mode so that any conflicting cached open locks on other clients will be canceled. Add a regression test to sanityn.sh. Signed-off-by: John L. Hammond &lt;john.hammond@intel.com&gt; Change-Id: I8092bca4c418ec99a25584abdfb635ffec19a26e Reviewed-on: http://review.whamcloud.com/9063 Reviewed-by: Jinshan Xiong &lt;jinshan.xiong@intel.com&gt; Tested-by: Jenkins Tested-by: Maloo &lt;hpdd-maloo@intel.com&gt; Reviewed-by: Mike Pershin &lt;mike.pershin@intel.com&gt; Reviewed-by: Oleg Drokin &lt;oleg.drokin@intel.com&gt; </pre> </div></div> <p>That was landed after 2.5.0 and reverted before 2.6.0.</p> <p>This issue may be related to <a href="https://jira.whamcloud.com/browse/LU-5663" title="mds-survey performance regress on master" class="issue-link" data-issue-key="LU-5663"><del>LU-5663</del></a>.</p> <p>Ihara, could you please try the patch in <a href="https://jira.whamcloud.com/browse/LU-5663" title="mds-survey performance regress on master" class="issue-link" data-issue-key="LU-5663"><del>LU-5663</del></a> to see whether it can improve the performance of your test. That patch eliminates the overhead of striped directory (brought in 2.6) in the create object patch, then I can see whether striped directory cause this performance regression here.</p> <p>Iharah, althogh this patch does not show any difference in Jinshan's test, it does not mean it would not help yours. Because his test is more about single (or few) thread on each directory. This temporary patch will probably be more helpful for shared directory. Thank you!</p> <p><a href="http://review.whamcloud.com/#/c/12195" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12195</a> Not sure if this will help much, but worth to try.</p> <p>is it possible to get profiling for the both versions?</p> <p>I think that this issue is now out of date</p> Related LU-5663 Development End date Fri, 24 Oct 2014 07:35:20 +0000 Epic metadata performance Rank 1|hzww8f: Rank (Obsolete) 15726 Severity Start date Mon, 15 Sep 2014 07:35:20 +0000