https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6800 Lustre <p>Durding our performance testing, we found siginicant metadata performance regression with <a href="https://jira.whamcloud.com/browse/LU-5264" title="ASSERTION( info-&gt;oti_r_locks == 0 ) at OST umount" class="issue-link" data-issue-key="LU-5264"><del>LU-5264</del></a> on master.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mpirun -np 128 -ppn 4 -hostfile ./hostfile /work/tools/bin/mdtest -n 1000 -p 10 -i 5 -d /scratch1/mdtest.out </pre> </div></div> <p>master </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>SUMMARY: (of 5 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 39552.671 33039.129 37024.828 2875.617 Directory stat : 33462.417 29340.691 31662.586 1384.330 Directory removal : 40938.777 40238.677 40571.960 283.701 File creation : 17696.663 17209.531 17542.185 171.470 File stat : 33892.041 33429.312 33680.603 170.577 File read : 11284.121 11012.694 11220.417 104.978 File removal : 39718.200 39449.348 39556.254 90.590 Tree creation : 4583.939 700.335 3652.356 1487.449 Tree removal : 170.563 156.738 162.935 5.172 </pre> </div></div> <p>keep client version, but revert patch 42fdf8355791cb682c6120f7950bb2ecd50f97aa (<a href="https://jira.whamcloud.com/browse/LU-5264" title="ASSERTION( info-&gt;oti_r_locks == 0 ) at OST umount" class="issue-link" data-issue-key="LU-5264"><del>LU-5264</del></a> obdclass: fix race during key quiescency) on servers.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>SUMMARY: (of 5 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 44937.511 42117.095 43780.402 1335.927 Directory stat : 135310.427 129560.951 133625.293 2077.128 Directory removal : 51525.499 46852.534 49965.297 1611.759 File creation : 42978.506 41435.145 42413.409 586.294 File stat : 135882.699 133344.886 134466.144 977.577 File read : 121788.787 111332.613 116374.190 3351.730 File removal : 84827.815 78120.995 80378.741 2522.662 Tree creation : 4650.004 3788.893 4268.099 336.241 Tree removal : 198.059 129.234 179.980 25.563 </pre> </div></div> master LU-6800

Significant performance regression with patch LU-5264

Bug Critical Resolved Fixed Bruno Faccini Shuichi Ihara patch Sat, 4 Jul 2015 23:35:03 +0000 Wed, 13 Jun 2018 17:02:43 +0000 Mon, 14 Sep 2015 18:00:59 +0000 Lustre 2.8.0 0 13 <p>I am trying to setup a test platform to understand the impact of my original patch from <a href="https://jira.whamcloud.com/browse/LU-5264" title="ASSERTION( info-&gt;oti_r_locks == 0 ) at OST umount" class="issue-link" data-issue-key="LU-5264"><del>LU-5264</del></a> on lu_keys_guard lock usage.</p> <p>I noticed this performance regression as well. In fact so far when testing a DNE2 directory striped across 2 MDS it performance worst than when just using one MDS.</p> <p>Gu Zheng (gzheng@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/15558" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15558</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6800" title="Significant performance regression with patch LU-5264" class="issue-link" data-issue-key="LU-6800"><del>LU-6800</del></a> obdclass: change spinlock of key to rwlock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 56ee901b05afbe3eab2fda3eb2a055025f3a7779</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/15558/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15558/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6800" title="Significant performance regression with patch LU-5264" class="issue-link" data-issue-key="LU-6800"><del>LU-6800</del></a> obdclass: change spinlock of key to rwlock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 96d773d993cd48a069da4098b87da7d9ef0dd52e</p> <p>Landed for 2.8</p> <p>As far as we know, <a href="http://review.whamcloud.com/15558/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15558/</a> is not perfect. It helps to get perforamnce back on most of metadata operation, but the file read operation is still slow before appled <a href="https://jira.whamcloud.com/browse/LU-5264" title="ASSERTION( info-&gt;oti_r_locks == 0 ) at OST umount" class="issue-link" data-issue-key="LU-5264"><del>LU-5264</del></a>.<br/> I will post benchmark resutls soon.</p> <p>FYI, at CEA, we faced heavy load on MDT with several codes. This was introducing bad performance and instability on the filesystem, so we decided to revert the patch from <a href="https://jira.whamcloud.com/browse/LU-5264" title="ASSERTION( info-&gt;oti_r_locks == 0 ) at OST umount" class="issue-link" data-issue-key="LU-5264"><del>LU-5264</del></a> for now, until we get something better.</p> <p>Hi Aurelien,</p> <p>Did you test with or without 15558? Does it help or still have the same problem?</p> <p>Unfortunately, we did not test with 15558. Not sure we will be able to do this on the production system.</p> <p>Grégoire Pichon (gregoire.pichon@bull.net) uploaded a new patch: <a href="http://review.whamcloud.com/15648" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15648</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6800" title="Significant performance regression with patch LU-5264" class="issue-link" data-issue-key="LU-6800"><del>LU-6800</del></a> obdclass: change spinlock of key to rwlock<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: 5adcce4242802b6be3441b425220bc422926a822</p> <p>Aurélien,</p> <p>The issue in the build for bullx has already been reported in duplicate <a href="https://jira.whamcloud.com/browse/LU-6823" title="Performance regression on servers with LU-5264" class="issue-link" data-issue-key="LU-6823"><del>LU-6823</del></a>. Bull is currently looking at <a href="https://jira.whamcloud.com/browse/LU-6800" title="Significant performance regression with patch LU-5264" class="issue-link" data-issue-key="LU-6800"><del>LU-6800</del></a> carrefully.</p> <p>Please re-open <a href="https://jira.whamcloud.com/browse/LU-6800" title="Significant performance regression with patch LU-5264" class="issue-link" data-issue-key="LU-6800"><del>LU-6800</del></a>, we understood <a href="http://review.whamcloud.com/15558" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15558</a> helps a lot, but still not all performance back. Here is test resutls. 32 clients, 128 mdtest process.</p> <p>test1 : master (commit-id: fe60e0135ee2334440247cde167b707b223cf11d) branch (includes <a href="https://jira.whamcloud.com/browse/LU-5264" title="ASSERTION( info-&gt;oti_r_locks == 0 ) at OST umount" class="issue-link" data-issue-key="LU-5264"><del>LU-5264</del></a> and patch 15558 )</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mpirun -np 128 -ppn 4 -hostfile ./hostfile /work/tools/bin/mdtest -i 3 -n 1000 -d /scratch1/mdtest.out Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 45237.210 36692.398 40159.293 3669.695 Directory stat : 132371.575 129820.230 131383.164 1118.004 Directory removal : 53873.775 50985.149 52790.576 1285.107 File creation : 42732.503 37298.342 40070.221 2219.840 File stat : 131527.304 129333.170 130765.529 1013.515 File read : 87588.987 67919.964 80344.389 8825.741 File removal : 84046.477 80418.268 82668.050 1604.248 Tree creation : 4364.520 4032.985 4164.502 143.755 Tree removal : 203.587 194.749 200.008 3.799 </pre> </div></div> <p>test2 : master + revert 15558</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mpirun -np 128 -ppn 4 -hostfile ./hostfile /work/tools/bin/mdtest -i 3 -n 1000 -d /scratch1/mdtest.out Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 40422.683 20650.668 30457.842 8072.661 Directory stat : 33032.600 27110.270 30459.575 2479.308 Directory removal : 41611.362 39640.289 40887.059 885.442 File creation : 17622.819 17537.572 17581.070 34.824 File stat : 33991.557 33935.386 33959.396 23.645 File read : 11241.112 10994.112 11104.383 102.558 File removal : 40024.327 39973.169 39998.669 20.886 Tree creation : 4185.932 3705.216 4007.822 215.092 Tree removal : 170.327 164.689 167.062 2.386 </pre> </div></div> <p>test3 : master + revert 15558 + revert 13103</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mpirun -np 128 -ppn 4 -hostfile ./hostfile /work/tools/bin/mdtest -i 3 -n 1000 -d /scratch1/mdtest.out Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 46423.406 37490.161 43188.774 4041.792 Directory stat : 134178.816 126241.328 130085.996 3245.214 Directory removal : 53737.981 44389.098 50171.405 4125.732 File creation : 44199.169 37398.927 40834.020 2776.628 File stat : 135524.181 130626.893 132934.894 2009.179 File read : 100767.654 76374.732 91483.603 10776.519 File removal : 86318.162 82618.862 85021.870 1700.945 Tree creation : 4634.590 3557.510 4167.598 451.208 Tree removal : 201.814 194.397 197.894 3.043 </pre> </div></div> <p>If we compare test3 and test2 resutls, test2 results are significant bad which means patch 13103 caused this performance regression.<br/> GuZhang at DDN pushed patch 15558 and as far as we can see test1 results, perforamnce was back expect "file read' operation.<br/> So, patch 15558 helps a lot, but even that, we still see perforamnce regression on "file read" operation. We need more investigate on this to back everything performance back.</p> <p>Since I am the creator of patch for <a href="https://jira.whamcloud.com/browse/LU-5264" title="ASSERTION( info-&gt;oti_r_locks == 0 ) at OST umount" class="issue-link" data-issue-key="LU-5264"><del>LU-5264</del></a> and thus the unfortunate guilty of this situation, and based on the fact that DDN team has already produced a very good but partial fix, I would like to work more actively and fix this last read performance regression.</p> <p>Aurelien, Bruno, since the multi-client competition seems to be the main cause to trigger the issue, could it be possible for me to directly work with you on a site where you heavily hit this problem ?</p> <p>We have removed patch for <a href="https://jira.whamcloud.com/browse/LU-5264" title="ASSERTION( info-&gt;oti_r_locks == 0 ) at OST umount" class="issue-link" data-issue-key="LU-5264"><del>LU-5264</del></a> from all our file systems. We will discuss the ability to give a try with the current fix for <a href="https://jira.whamcloud.com/browse/LU-6800" title="Significant performance regression with patch LU-5264" class="issue-link" data-issue-key="LU-6800"><del>LU-6800</del></a> by the end of the month on a test file system.</p> <p>I will keep you in touch.</p> <p>First tests running with patch #15558, at TGCC site, does not show the same read perfs regression.<br/> Site will soon provide their numbers for this ticket.<br/> More instrumentations will be done.</p> <p>mdtest have run in restricted 2 and restricted 3 state, respectively without patch and with all patches....</p> <p>In actual state (revert patch <a href="https://jira.whamcloud.com/browse/LU-5264" title="ASSERTION( info-&gt;oti_r_locks == 0 ) at OST umount" class="issue-link" data-issue-key="LU-5264"><del>LU-5264</del></a> and <a href="https://jira.whamcloud.com/browse/LU-6049" title="General Protection Fault at echo_session_key_fini+0xa9" class="issue-link" data-issue-key="LU-6049"><del>LU-6049</del></a>)</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">$ mpirun -n 128 xx/mdtest -n 1000 -p 10 -i 5 -d xx/run_MDTest_repro3 -- started at 08/13/2015 15:26:03 -- mdtest-1.9.3 was launched with 128 total task(s) on 8 node(s) Command line used: ./mdtest -n 1000 -p 10 -i 5 -d ./run_MDTest_repro3 Path: xxxxxxxxxxx FS: 155.1 TiB Used FS: 7.9% Inodes: 154.1 Mi Used Inodes: 0.3% 128 tasks, 128000 files/directories SUMMARY: (of 5 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 12044.248 3622.254 7693.558 2787.514 Directory stat : 29808.509 28605.578 29428.433 434.277 Directory removal : 16316.172 15596.360 16069.509 271.041 File creation : 8304.285 2475.372 5950.888 2378.499 File stat : 28493.314 28090.363 28265.330 130.886 File read : 15694.955 15170.723 15435.999 181.395 File removal : 15253.714 14426.384 14981.075 305.384 Tree creation : 3077.259 1170.939 1855.926 653.607 Tree removal : 95.066 61.637 77.186 11.245 -- finished at 08/13/2015 15:34:26 -- </pre> </div></div> <p>With <a href="https://jira.whamcloud.com/browse/LU-5264" title="ASSERTION( info-&gt;oti_r_locks == 0 ) at OST umount" class="issue-link" data-issue-key="LU-5264"><del>LU-5264</del></a>, <a href="https://jira.whamcloud.com/browse/LU-6049" title="General Protection Fault at echo_session_key_fini+0xa9" class="issue-link" data-issue-key="LU-6049"><del>LU-6049</del></a> and <a href="https://jira.whamcloud.com/browse/LU-6800" title="Significant performance regression with patch LU-5264" class="issue-link" data-issue-key="LU-6800"><del>LU-6800</del></a>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">$ mpirun -n 128 xx/mdtest -n 1000 -p 10 -i 5 -d xx/run_MDTest_repro2 -- started at 08/13/2015 15:04:09 -- mdtest-1.9.3 was launched with 128 total task(s) on 8 node(s) Command line used: ./mdtest -n 1000 -p 10 -i 5 -d ./run_MDTest_repro2 Path: xxxxxxxxxxxxx FS: 155.1 TiB Used FS: 7.9% Inodes: 154.0 Mi Used Inodes: 0.3% 128 tasks, 128000 files/directories SUMMARY: (of 5 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 11815.599 6041.768 8297.495 2021.031 Directory stat : 29708.108 29290.724 29475.438 147.864 Directory removal : 16459.019 16182.934 16283.041 93.778 File creation : 8561.213 8407.310 8496.989 57.227 File stat : 28579.728 28018.041 28328.611 184.786 File read : 15066.452 14786.594 14943.476 98.652 File removal : 14821.486 14289.802 14645.054 190.972 Tree creation : 2746.761 1234.708 1675.881 558.742 Tree removal : 63.032 51.565 58.417 3.900 -- finished at 08/13/2015 15:11:16 -- </pre> </div></div> <p>We do not observe significant difference but the tests were launched with 8 nodes only.<br/> We expect a test with 32 nodes and more by the end of the month.</p> <p>Ihara, looking at your test results it seems that the mean performance of the original results (before <a href="https://jira.whamcloud.com/browse/LU-5264" title="ASSERTION( info-&gt;oti_r_locks == 0 ) at OST umount" class="issue-link" data-issue-key="LU-5264"><del>LU-5264</del></a>) and the results after the <a href="https://jira.whamcloud.com/browse/LU-6800" title="Significant performance regression with patch LU-5264" class="issue-link" data-issue-key="LU-6800"><del>LU-6800</del></a> patch are very close, within the standard deviation for the tests:<br/> <b>BEFORE</b></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>master + revert 15558 + revert 13103 # mpirun -np 128 -ppn 4 -hostfile ./hostfile /work/tools/bin/mdtest -i 3 -n 1000 -d /scratch1/mdtest.out Operation Mean Std Dev --------- ---- ------- Directory creation: 40159.293 3669.695 Directory stat : 131383.164 1118.004 Directory removal : 52790.576 1285.107 File creation : 40070.221 2219.840 File stat : 130765.529 1013.515 File read : 80344.389 8825.741 File removal : 82668.050 1604.248 Tree creation : 4164.502 143.755 Tree removal : 200.008 3.799 </pre> </div></div> <p><b>AFTER</b></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>master (commit-id: fe60e0135ee2334440247cde167b707b223cf11d, includes LU-5264 and patch 15558) # mpirun -np 128 -ppn 4 -hostfile ./hostfile /work/tools/bin/mdtest -i 3 -n 1000 -d /scratch1/mdtest.out Operation Mean Std Dev --------- ---- ------- Directory creation: 43188.774 4041.792 Directory stat : 130085.996 3245.214 Directory removal : 50171.405 4125.732 File creation : 40834.020 2776.628 File stat : 132934.894 2009.179 File read : 91483.603 10776.519 File removal : 85021.870 1700.945 Tree creation : 4167.598 451.208 Tree removal : 197.894 3.043 </pre> </div></div> <p>The mean <b>Directory removal</b> and <b>Directory stat</b> operations are somewhat slower, but this is within the standard deviation of the three test runs. Conversely, the <b>Directory create</b>, <b>File create</b>, and <b>File removal</b> operations are faster, but are also within the standard deviation of the three test runs. </p> <p>For the <b>File read</b> it appears that the results are highly variable (stddev more than 10% of the mean). Is this performance loss seen with IO benchmarks like IOR or only the mdtest? What size of files is mdtest using?</p> <p>We only hit this performance regression on mdtest and all test file size are zero byte.<br/> And, we agreed patch <a href="http://review.whamcloud.com/15558" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15558</a> helped and the performance was back even with patch <a href="https://jira.whamcloud.com/browse/LU-5264" title="ASSERTION( info-&gt;oti_r_locks == 0 ) at OST umount" class="issue-link" data-issue-key="LU-5264"><del>LU-5264</del></a>, but except "file read" operation.<br/> We still don't know why read operation doesn't come back with patch 15558.</p> <p>ok then let's close this ticket for now and if we need to make future improvements to read operations track that separately</p> <p>With the potential move to rhashtable which have lockless lookups we might be able to resolve these performance issues.</p> Duplicate Related LU-5264 LU-8130 LU-6823 LU-11089 Development Rank 1|hzxh87: Rank (Obsolete) 9223372036854775807 Severity