https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9972 Lustre <p>There is a performance regression on dir removal.</p> <p>Server and client : RHEL7.3<br/> Lustre version : 2.10.52<br/> Backend filesystem: ldiskfs</p> <p>mpirun --allow-run-as-root /work/tools/bin/mdtest -n 5000 -v -d /scratch0/mdtest.out -D -i 3 -p 10 -w 0 -u</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 89757.381 65618.928 74607.900 10774.356 Directory stat : 320946.433 319888.242 320294.264 465.749 Directory removal : 19028.569 17837.487 18351.200 499.838 Tree creation : 434.446 158.826 318.943 116.860 Tree removal : 27.018 25.210 26.281 0.775 </pre> </div></div> 2.10 (and 2.11) LU-9972

Performance regressions on unique directory removal

Bug Major Resolved Fixed Alex Zhuravlev Shuichi Ihara Mon, 11 Sep 2017 23:56:18 +0000 Thu, 1 Mar 2018 20:38:08 +0000 Tue, 6 Feb 2018 04:57:42 +0000 Lustre 2.11.0 Lustre 2.10.4 0 15 <p>Compared to which version/kernel?</p> <p>For example lustre-2.7(IEEL3.0)/CentOS7.3</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 46577.991 42249.894 44871.081 1881.494 Directory stat : 373243.136 367643.706 370043.774 2354.791 Directory removal : 78530.701 66152.245 72781.092 5091.584 File creation : 107283.764 96953.405 103118.187 4447.973 File stat : 385082.155 375112.919 379387.910 4191.828 File read : 185463.654 177089.199 182367.310 3750.818 File removal : 127467.768 113218.809 122566.251 6612.256 Tree creation : 349.409 91.996 262.234 120.388 Tree removal : 20.765 18.039 19.132 1.176 </pre> </div></div> <p>I'm going to test lustre-2.9 to compare.</p> <p>Sorry delay response. I needed to change hardware configuration, but here is new results on b2_10 (2.10.1_RC1).<br/> rmdir to unique directories is obviously slow compared to same benchmark to a shared directory. </p> <p>mpirun -np 128 mdtest -n 5000 -v -d /scratch0/mdtest.out -i 3 -p 30 -D (for shared directory )<br/> mpirun -np 128 mdtest -n 5000 -v -d /scratch0/mdtest.out -i 3 -p 30 -D -u (for unique directory )<br/> 32 clients, 128 processes. both of them were collected on exact same hardware configuration.</p> <p>Here is a directory operations to a shared directory.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 91979.485 69249.863 79842.797 9343.315 Directory stat : 197008.811 180039.999 189342.439 7023.422 Directory removal : 140527.764 128798.718 133567.639 5032.803 Tree creation : 5462.720 1034.229 3084.207 1822.788 Tree removal : 92.639 74.702 86.019 8.041 </pre> </div></div> <p>And here is unique directory's results.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 84094.691 75575.177 80444.764 3583.407 Directory stat : 463370.743 431285.266 448299.685 13170.724 Directory removal : 18722.965 18461.182 18558.573 116.903 Tree creation : 593.577 310.356 472.213 119.117 Tree removal : 37.275 33.999 35.691 1.340 V-1: Entering timestamp... </pre> </div></div> <p>Cliff, do we have similar mdtest results from the performance test cluster, in particular 2.10.0/1, 2.10.52/53, and 2.9.x? That would give us a ballpark of where this performance regression has been introduced, and allow git bisect to narrow it down to a particular patch.</p> <p>I think the problem has been exist in b2_9 at least.<br/> Here is results of same test on b2_9. (keep 2.10.1 for client, but just chnged server with b2_9)</p> <p> mpirun -np 128 mdtest -n 5000 -v -d /scratch0/mdtest.out -i 3 -p 30 -D -u (unique directory)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 91409.935 72314.781 84242.568 8491.267 Directory stat : 184806.326 183688.542 184367.927 487.111 Directory removal : 20718.518 20303.157 20555.893 181.147 Tree creation : 552.285 400.441 473.117 62.160 Tree removal : 40.413 29.341 35.321 4.563 V-1: Entering timestamp... </pre> </div></div> <p>mpirun -np 128 mdtest -n 5000 -v -d /scratch0/mdtest.out -i 3 -p 30 -D (shared directory)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 70310.000 45717.790 58926.161 10122.282 Directory stat : 178080.331 175913.598 176783.485 934.667 Directory removal : 86194.900 72838.446 79018.261 5498.119 Tree creation : 5527.274 2804.821 3744.496 1261.231 Tree removal : 80.959 24.059 61.936 26.784 V-1: Entering timestamp... </pre> </div></div> <p>Our hardware config has changed a bit since 2.9, we have seen noticeable improvements since changing the tuned-adm profile. Of course all our old results are on Sharepoint: <br/> If you look at the most current spreadsheet, you will see the jump in Dir rm with the tuned-adm change:<br/> <a href="http://tinyurl.com/ydzx7gxp" class="external-link" target="_blank" rel="nofollow noopener">http://tinyurl.com/ydzx7gxp</a></p> <p>If you look at our last EE 3.0 runs from June 2017, you will see Dir rm is 4x better. (b_ieel3_0 build 214) So I would look at some deltas there: <a href="http://tinyurl.com/yanedznq" class="external-link" target="_blank" rel="nofollow noopener">http://tinyurl.com/yanedznq</a></p> <p>Results from master builds (18 threads 16 client mdtestfpp) :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Build Version Dir create Dir stat Dir rm master-3596 2.9.58_22 21514 188773 11822 master-3598 2.9.58_57 21570 209599 11653 master-3601 2.9.59 21063 223101 11879 master-2607 2.8.59-35 19328 211813 11797 master-3637 2.10.52_83 25987 234954 15033 </pre> </div></div> <p>Results from EE builds:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Build Version Dir create Dir stat Dir rm b_ieel3_0-105 2.7.18 21239 167421 73112 b_ieel3_0-89 2.7.16.1 19758 169050 77119 b_ieel3_0-204 2.7.19.12 28330 288267 59444 b_ieel3_0-214 2.7.20.2 28136 331563 60515 </pre> </div></div> <p>Saurabh</p> <p>Please can you narrow down where the change occurred?</p> <p>Thanks</p> <p>Peter</p> <p>Discussed this with Saurabh and Cliff. Cliff thinks the problem may date back to DNE2 landings, since EE 2.7 predates the DNE2 changes, and they appeared as early as 2.8.0.</p> <p>Saurabh will try a <tt>git bisect</tt> starting with v2_7_50 (== 2.7.0) to see if that has good performance on our test cluster (good ~= 70k rmdir/sec) and go from there. We would like to keep the kernel version the same, at RHEL 7.4, to avoid potential interference with the results from changing the kernel or other configuration options.</p> <p>Saurabh Tandan (saurabh.tandan@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29126" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29126</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9972" title="Performance regressions on unique directory removal " class="issue-link" data-issue-key="LU-9972"><del>LU-9972</del></a> tests: Build required for <a href="https://jira.whamcloud.com/browse/LU-9972" title="Performance regressions on unique directory removal " class="issue-link" data-issue-key="LU-9972"><del>LU-9972</del></a><br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 708fec1e34058ec735be819217498f8cc90aa924</p> <p>Any progress on finding regression point?</p> <p>Still working on it Shuichi, will soon post some findings.</p> <p>Alex</p> <p>I daresay that Saurabh may elaborate but I understand that he has found that your patch <a href="https://jira.whamcloud.com/browse/LU-7053" title="Do not use osd_object_find in osd_index_ea_insert" class="issue-link" data-issue-key="LU-7053"><del>LU-7053</del></a> (osd: don't lookup object at insert <a href="https://review.whamcloud.com/#/c/17092/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/17092/</a>) is the one that introduced the performance regression with directory removal</p> <p>Do you have any ideas on how to avoid this?</p> <p>Peter</p> <p>There was approximately a drop of 90% in performance fir "Dir removal" for "mdtestfpp" results from tag 2.7.65. Following is the data for all the runs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Tag Dir removal 2.7.56 18298 2.7.57 121954 2.7.61 64849 2.7.64 111655 good 2.7.64-g63a3e412 (LU-7419) 74384 good 2.7.64-gc965fc8a (LU-7450) 72374 good 2.7.64-g6765d785 (LU-7408) 92029 good 2.7.64-g9ae3a289 (LU-7053) 11517 bad 2.7.64-g0d3a07a8 (LU-7430) 15114 bad 2.7.64-g959f8f78 (LU-7573) 11530 bad 2.7.65 11375 bad 2.7.66 11403 bad 2.10.53 12473 2.10.54 9649 </pre> </div></div> <p>There must have been more runs than just these if you were able to isolate <a href="https://review.whamcloud.com/#/c/17092/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/17092/</a>.</p> <p>I've updated the results to show the commit-order test results for the bisect (not the bisect order), to show there is a clear break between <a href="https://jira.whamcloud.com/browse/LU-7408" title="multislot RPC support didn&#39;t declare write for reply_data object" class="issue-link" data-issue-key="LU-7408"><del>LU-7408</del></a> and the next patch <a href="https://jira.whamcloud.com/browse/LU-7053" title="Do not use osd_object_find in osd_index_ea_insert" class="issue-link" data-issue-key="LU-7053"><del>LU-7053</del></a>.</p> <p>I tried to revert <a href="https://jira.whamcloud.com/browse/LU-7053" title="Do not use osd_object_find in osd_index_ea_insert" class="issue-link" data-issue-key="LU-7053"><del>LU-7053</del></a> on master and got no difference for rmdir:<br/> with <a href="https://jira.whamcloud.com/browse/LU-7053" title="Do not use osd_object_find in osd_index_ea_insert" class="issue-link" data-issue-key="LU-7053"><del>LU-7053</del></a>: total: 100000 unlinks in 16 seconds: 6250.000000 unlinks/second<br/> without <a href="https://jira.whamcloud.com/browse/LU-7053" title="Do not use osd_object_find in osd_index_ea_insert" class="issue-link" data-issue-key="LU-7053"><del>LU-7053</del></a>: total: 100000 unlinks in 16 seconds: 6250.000000 unlinks/second<br/> the load was simulated with createmany/unlinkmany</p> <p>going to proceed with mdtest, but that will take some time.</p> <p>Alex Zhuravlev (alexey.zhuravlev@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29709" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29709</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9972" title="Performance regressions on unique directory removal " class="issue-link" data-issue-key="LU-9972"><del>LU-9972</del></a> osd: cache OI mapping in dt_ref_</p> {del|add} <p>()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ccb8803749e9234e4bd6e5e451a3333d6b17e594</p> <p>please, try with the patch above.</p> <p>Performance results with patch above:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>MDTEST RESULTS 000: SUMMARY: (of 3 iterations) 000: Operation Max Min Mean Std Dev 000: --------- --- --- ---- ------- 000: Directory creation: 24200.233 15693.923 18542.903 4000.371 000: Directory stat : 235384.333 230225.191 233266.050 2204.926 000: Directory removal : 82809.228 30178.490 51633.084 22559.256 000: File creation : 50508.075 33979.054 41331.172 6870.202 000: File stat : 232294.752 226222.165 228931.104 2521.978 000: File read : 124620.945 110484.961 119398.361 6333.672 000: File removal : 128159.287 77027.802 106534.012 21605.386 000: Tree creation : 124.922 65.746 103.702 26.901 000: Tree removal : 9.709 7.841 8.650 0.783 </pre> </div></div> <p>Results for dir removal with this patch have surely improved but still not to the mark where they were before <a href="https://jira.whamcloud.com/browse/LU-7053" title="Do not use osd_object_find in osd_index_ea_insert" class="issue-link" data-issue-key="LU-7053"><del>LU-7053</del></a> was landed.<br/> Test run for Lustre version just before landing of <a href="https://jira.whamcloud.com/browse/LU-7053" title="Do not use osd_object_find in osd_index_ea_insert" class="issue-link" data-issue-key="LU-7053"><del>LU-7053</del></a> showed performance number for Dir removal as "92029.569" (mean) .</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Build Version Dir create Dir stat Dir rm b_ieel3_0-89 2.7.16.1 19758 169050 77119 b_ieel3_0-105 2.7.18 21239 167421 73112 b_ieel3_0-214 2.7.20.2 28136 331563 60515 master-3601 2.9.59 21063 223101 11879 master-2607 2.8.59-35 19328 211813 11797 master-3637 2.10.52_83 25987 234954 15033 master 2.10.54 18676 210767 10282 review-51618 2.10.54-20-g66bb2d1 18542 233266 51633 </pre> </div></div> <p>So it looks like the rmdir performance is significantly improved, but the mkdir performance is down a bit from 2.10.53, but about on par with 2.10.54. It looks like the file create/stat/unlink performance is a bit lower vs. 2.10.54, but definitely still a lot better than 2.7.64.</p> <p>Alex, would it make sense to only add directory objects to the cache, instead of adding all objects?  That may give us the best of both worlds.</p> <p>Andreas, at the moment I don't quite understand how the cache can decrease performance as it's TLS, lockless, tiny and it replaces a lookup in LU cache which in contrast much larger and needs locking. I keep investigating..</p> <p>please, benchmark master with <a href="https://review.whamcloud.com/#/c/29821/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/29821/</a><br/> this patch reverts just <a href="https://jira.whamcloud.com/browse/LU-7053" title="Do not use osd_object_find in osd_index_ea_insert" class="issue-link" data-issue-key="LU-7053"><del>LU-7053</del></a>.</p> <p>Before <a href="https://jira.whamcloud.com/browse/LU-7053" title="Do not use osd_object_find in osd_index_ea_insert" class="issue-link" data-issue-key="LU-7053"><del>LU-7053</del></a>, when calling osd_index_declare_ea_insert() from orph_declare_index_insert(), we only looked up the FID of mdd-&gt;mdd_orphans in the FLDB to determine if it was local or remote (so that we could declare additional credits in the remote case). Now we do osd_idc_find_or_init() which included OI lookup in the local case. I understand that we may need the extra information if osd_index_ea_insert() gets called to add the directory to the orphan dir. But I assume that we are not hitting this case from mdtest. So maybe the difference between FLDB lookup and full IDC initialization (and/or contention effects) could explain the difference.</p> <p>IDC is lockless and per-thread while FLDB is a shared structure.</p> <p>Yes, but I'm not talking about the cost of accessing the IDC cache. I mean the extra cost of OI lookup to initialize the IDC entry.</p> <p>usually it's initialized from other preceding methods (like osd_declare_ref_</p> {add|del} <p> in <a href="https://review.whamcloud.com/29709" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29709</a><br/> with no extra lookups.</p> <p>to verify that I added printk() to osd_idc_find_or_init() and got zero calls to osd_remote_fid() and osd_oi_lookup() during rmdir.<br/> there are few calls to osd_remote_fid() (which I hope to fix in a separate patch), but those weren't introduced with <a href="https://jira.whamcloud.com/browse/LU-7053" title="Do not use osd_object_find in osd_index_ea_insert" class="issue-link" data-issue-key="LU-7053"><del>LU-7053</del></a></p> <p>Results of patch <a href="https://review.whamcloud.com/#/c/29821/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/29821/</a> which reverts just <a href="https://jira.whamcloud.com/browse/LU-7053" title="Do not use osd_object_find in osd_index_ea_insert" class="issue-link" data-issue-key="LU-7053"><del>LU-7053</del></a>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>MIB RESULTS MDTEST RESULTS 000: SUMMARY: (of 3 iterations) 000: Operation Max Min Mean Std Dev 000: --------- --- --- ---- ------- 000: Directory creation: 21177.729 16443.602 18498.262 1982.554 000: Directory stat : 232409.626 229897.859 230874.272 1098.955 000: Directory removal : 111897.307 40906.454 75055.964 29044.331 000: File creation : 44061.464 38438.052 41863.997 2454.596 000: File stat : 225941.782 193254.640 210777.086 13448.211 000: File read : 146598.308 86669.775 126562.125 28208.247 000: File removal : 155512.154 106886.927 136397.357 21168.452 000: Tree creation : 120.178 56.129 96.113 28.468 000: Tree removal : 8.906 8.122 8.620 0.354 000: </pre> </div></div> <p>Dir removal for this run has been "75055.964" (mean) approx 45% higher in performance in comparison to the patch <a href="https://review.whamcloud.com/29709" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29709</a> under exactly same conditions and number of iterations.</p> <p>thanks for the data. I've updated <a href="https://review.whamcloud.com/29709" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29709</a><br/> please benchmark master with that. the patch collects additional stats and dump it at umount in the following form:<br/> + if (atomic_read(&amp;o-&gt;od_idc_remotes) ||<br/> + atomic_read(&amp;o-&gt;od_idc_oi))<br/> + printk("%s: %d checks for remote, %d OI lookups\n",<br/> + o-&gt;od_svname,<br/> + atomic_read(&amp;o-&gt;od_idc_remotes),<br/> + atomic_read(&amp;o-&gt;od_idc_oi));</p> <p>please attach it here if printed.<br/> thanks in advance.</p> <p>I'm working on a followup patch to optimize calls to osd_remote_fid(), this isn't directly related to <a href="https://jira.whamcloud.com/browse/LU-7053" title="Do not use osd_object_find in osd_index_ea_insert" class="issue-link" data-issue-key="LU-7053"><del>LU-7053</del></a>, but hopefully can fix the issue.</p> <p>please, try with the updated patch.</p> <p>Hi Ihara,</p> <p>Have you been able to confirm that Alex's patch resolves the issue?</p> <p>Thanks.<br/> Joe</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/29709/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29709/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9972" title="Performance regressions on unique directory removal " class="issue-link" data-issue-key="LU-9972"><del>LU-9972</del></a> osd: cache OI mapping in dt_declare_ref_add<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b6e718def348c53759a12afee9450207fc7ab56f</p> <p>Landed for 2.11</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31211" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31211</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9972" title="Performance regressions on unique directory removal " class="issue-link" data-issue-key="LU-9972"><del>LU-9972</del></a> osd: cache OI mapping in dt_declare_ref_add<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: d334fd8aa117d9a957365bdfb6792bcaaf6533cc</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/31211/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31211/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9972" title="Performance regressions on unique directory removal " class="issue-link" data-issue-key="LU-9972"><del>LU-9972</del></a> osd: cache OI mapping in dt_declare_ref_add<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 025d0412599ed9381be4a0ab84d190b59fc2c451</p> Related LU-7053 Development Rank 1|hzzjz3: Rank (Obsolete) 9223372036854775807 Severity