https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10005 Lustre <p>There is a MDS and two MDTs. both MDT's hardware setup is symmetric.<br/> This is DNE1 setup and static MDT allocation to each directory below.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@c01 ~]# lfs mkdir -i 0 /scratch0/dir0 [root@c01 ~]# lfs mkdir -i 1 /scratch0/dir1 </pre> </div></div> <p>If it run mdtest to each MDT separately, File creation to slave MDT (MDT0001) is much slower than primary MDT (MDT0000).<br/> Here is quick summary</p> <p>1. MDT0000 on MDT0 : 154K ops/sec <br/> 2. MDT0001 on MDT1 : 94K ops/sec</p> <p>Also tested MDT1 device as MDT0000. reformated MDT1 device as MDT0000 and also reformated MDT0 device as MDT0001. (which means swapped MDT0 and MDT1 device)</p> <p>3. MDT0000 on original MDT1 devcide : 151K ops/sec <br/> 4. MDT0001 on original MDT0 devcide : 106K ops/sec</p> <p>From those benchmark results, MDT device and backend storage are no problem and it doesn't master. In any case, file creation to MDT0001 is slower than MDT0000.</p> <p>Here is full mutest results.<br/> mpirun -np 128 /work/tools/bin/mdtest -n 10000 -v -d /scratch0/dir1 -i 3 -p 60 -F -u</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Format MDT0 device with MDT0000 SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 155589.552 152790.159 154009.399 1170.995 File stat : 454932.894 444775.351 449009.216 4315.516 File read : 233628.858 230038.744 232081.775 1507.029 File removal : 188460.588 184435.235 186712.008 1685.251 Tree creation : 551.714 444.141 493.856 44.292 Tree removal : 19.593 18.601 18.984 0.436 V-1: Entering timestamp... Format MDT1 device with MDT0001 SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 97428.133 92734.086 94657.494 2007.797 File stat : 463844.746 439627.133 450037.946 10174.240 File read : 234910.249 232565.024 233533.717 999.923 File removal : 186289.259 181171.423 184208.010 2195.839 Tree creation : 476.266 32.866 325.249 206.784 Tree removal : 19.429 14.144 17.055 2.191 V-1: Entering timestamp... Reformat MDT1 device as MDT0000 SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 155432.973 145656.215 151697.151 4311.335 File stat : 436363.906 420914.320 428509.377 6309.935 File read : 231848.424 229879.273 230823.486 805.926 File removal : 189856.501 186441.697 187710.599 1525.794 Tree creation : 564.044 432.872 504.217 54.166 Tree removal : 18.839 17.053 17.802 0.757 V-1: Entering timestamp... Reformat MDT0 device as MDT0001 SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 110312.440 103512.106 106042.905 3036.285 File stat : 443284.493 425246.521 435923.695 7728.341 File read : 226239.692 225898.388 226067.629 139.351 File removal : 186702.519 181944.612 184773.293 2043.883 Tree creation : 533.233 28.863 342.123 223.290 Tree removal : 17.901 17.260 17.650 0.280 V-1: Entering timestamp... </pre> </div></div> b2_10 LU-10005

File creation to slave MDT is much slower than primary MDT on DNE1 configuration

Bug Major Resolved Fixed Lai Siyao Shuichi Ihara Tue, 19 Sep 2017 11:17:12 +0000 Wed, 5 Sep 2018 18:34:48 +0000 Sun, 17 Dec 2017 16:19:10 +0000 Lustre 2.11.0 Lustre 2.10.4 0 8 <p>Hi Ihara,<br/> I'm wondering if there is some extra overhead in looking up the <tt>scratch0-&gt;dir1</tt> directory entry that is causing the MDT0001 operations to be slower? It would be useful to run mdtest inside the <tt>dir1</tt> directory to avoid the extra lookup, and determine if that is the cause of the slowdown:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># cd /scratch/dir1 # mpirun -np 128 /work/tools/bin/mdtest -n 10000 -v -d . -i 3 -p 60 -F -u </pre> </div></div> <p>The <tt>scratch</tt> directory should always be cached on the client, but I'm wondering if there is some problem with the locking on <tt>dir1</tt> that is preventing it from being cached?</p> <p>Another possible cause is that the default lov stripping cache does not work correctly, which might cause each file open/create (on non-root MDT) tries to get default striping from root MDT (extra RPC). See lod_ah_init()-&gt;lod_get_default_lov_striping(). </p> <p>I am not sure OSP cache works in this case, I will check.</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29078" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29078</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10005" title="File creation to slave MDT is much slower than primary MDT on DNE1 configuration" class="issue-link" data-issue-key="LU-10005"><del>LU-10005</del></a> osp: cache non-exist EA<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8a7e2c23d4bd35d0d544e91bdc51ce0abff895ac</p> <p>Ihara, please try this patch, thanks. </p> <p>Thanks WangDi<br/> patch makes better results.</p> <p>MDT0000</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 145472.642 137093.283 140274.791 3706.089 File stat : 443154.312 431557.793 436764.649 4807.570 File read : 233326.068 229897.041 231796.549 1424.131 File removal : 186842.911 186376.008 186627.793 192.368 Tree creation : 572.336 436.418 499.243 55.961 Tree removal : 19.798 18.276 19.165 0.647 V-1: Entering timestamp... </pre> </div></div> <p>MDT0001</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 153350.909 135825.706 143892.288 7222.027 File stat : 462687.460 449961.013 457977.445 5697.362 File read : 230307.880 224196.385 226475.436 2726.092 File removal : 192887.031 187816.726 189799.023 2212.682 Tree creation : 514.550 399.017 451.076 47.852 Tree removal : 18.876 17.538 18.222 0.546 V-1: Entering timestamp... </pre> </div></div> <p>btw, I didn't see such performance differences with IEEL3.0. somethjing we did in lusre-2.7, but missed or changed after lustre-2.7 and showed up this issue?</p> <p>BTW, after applied patch <a href="https://review.whamcloud.com/29078" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29078</a>.<br/> Overall, average file creation performance to primary MDT (MDT0000) drops.</p> <p>run mdtest 10 times with 60 sec interval<br/> mpirun --allow-run-as-root /work/tools/bin/mdtest -n 10000 -v -d /scratch0/dir0 -i 10 -p 60 -u -F </p> <p>without patch </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@c01 ~]# grep 'V-1: File creation' mdtest-default-dir0-loop.log V-1: File creation : 8.723 sec, 146739.418 ops/sec V-1: File creation : 8.855 sec, 144543.110 ops/sec V-1: File creation : 8.829 sec, 144978.787 ops/sec V-1: File creation : 8.803 sec, 145404.742 ops/sec V-1: File creation : 8.637 sec, 148192.295 ops/sec V-1: File creation : 9.084 sec, 140911.792 ops/sec V-1: File creation : 8.837 sec, 144853.049 ops/sec V-1: File creation : 9.288 sec, 137808.205 ops/sec V-1: File creation : 9.046 sec, 141502.448 ops/sec V-1: File creation : 9.392 sec, 136287.278 ops/sec </pre> </div></div> <p>with patch </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@c01 ~]# grep 'V-1: File creation' mdtest-LU10005-dir0-loop.log V-1: File creation : 8.874 sec, 144246.332 ops/sec V-1: File creation : 8.552 sec, 149675.160 ops/sec V-1: File creation : 9.211 sec, 138957.104 ops/sec V-1: File creation : 9.058 sec, 141315.265 ops/sec V-1: File creation : 9.297 sec, 137673.095 ops/sec V-1: File creation : 9.263 sec, 138185.327 ops/sec V-1: File creation : 9.469 sec, 135184.898 ops/sec V-1: File creation : 9.266 sec, 138134.736 ops/sec V-1: File creation : 9.373 sec, 136563.934 ops/sec V-1: File creation : 9.486 sec, 134930.710 ops/sec </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>btw, I didn't see such performance differences with IEEL3.0. somethjing we did in lusre-2.7, but missed or changed after lustre-2.7 and showed up this issue? </pre> </div></div> <p>I think this is brought in by <a href="https://jira.whamcloud.com/browse/LU-8998" title="Progressive File Layout (PFL)" class="issue-link" data-issue-key="LU-8998"><del>LU-8998</del></a> <a href="https://review.whamcloud.com/#/c/24823" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/24823</a>, which is landed in 2.10, so 2.7 should be fine.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>BTW, after applied patch https://review.whamcloud.com/29078. Overall, average file creation performance to primary MDT (MDT0000) drops. </pre> </div></div> <p>Hmm, I did not touch anything in the path of local file open/creation. Is it repeatable? This drop seems unlikely related with the patch, IMHO. I will check again. thanks</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/29078/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29078/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10005" title="File creation to slave MDT is much slower than primary MDT on DNE1 configuration" class="issue-link" data-issue-key="LU-10005"><del>LU-10005</del></a> osp: cache non-exist EA<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b2fa448050aff0b5230c8cc94e8baf848fbb4ded</p> <p>Landed for 2.11</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30585" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30585</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10005" title="File creation to slave MDT is much slower than primary MDT on DNE1 configuration" class="issue-link" data-issue-key="LU-10005"><del>LU-10005</del></a> osp: cache non-exist EA<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 96e35818b3c09978f7f05020a8f2641e0de0c92c</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30643" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30643</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-10005" title="File creation to slave MDT is much slower than primary MDT on DNE1 configuration" class="issue-link" data-issue-key="LU-10005"><del>LU-10005</del></a> osp: cache non-exist EA"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8da4d30a18ce1371624730225ad1b324e5128db1</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/30585/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30585/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10005" title="File creation to slave MDT is much slower than primary MDT on DNE1 configuration" class="issue-link" data-issue-key="LU-10005"><del>LU-10005</del></a> osp: cache non-exist EA<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 3a59349d931250c3ea008a68f8f0121500d984a4</p> Related LU-10406 Development Rank 1|hzzkev: Rank (Obsolete) 9223372036854775807 Severity