https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13520 Lustre <p>'git bisect' pointed out a performance regression on directory creation and commit "b7cd65a <a href="https://jira.whamcloud.com/browse/LU-12988" title="improve mount time on huge ldiskfs filesystem" class="issue-link" data-issue-key="LU-12988"><del>LU-12988</del></a> ldiskfs: mballoc to prefetch groups" caused regression.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mpirun -np 640 mdtest -D -C -r -u -n 1000 -vv -p 10 -i 3 -d /scratch/mdt0 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>master 52b5f4a LU-10401 tests: fix error from 'tr -d=' SUMMARY rate: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation : 44270.444 42144.617 43138.707 873.040 Directory stat : 0.000 0.000 0.000 0.000 Directory removal : 46058.435 44429.196 45452.138 726.962 Tree creation : 39.058 30.573 35.586 3.631 Tree removal : 32.583 31.615 32.014 0.413 V-1: Entering PrintTimestamp... </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>b7cd65a LU-12988 ldiskfs: mballoc to prefetch groups SUMMARY rate: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation : 44626.926 43321.707 43810.517 580.641 Directory stat : 0.000 0.000 0.000 0.000 Directory removal : 46902.932 45311.241 46091.593 649.716 Tree creation : 39.966 25.251 34.092 6.363 Tree removal : 32.608 25.252 30.093 3.424 V-1: Entering PrintTimestamp... </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>59a3d72 LU-12043 llite: move tunable params to sysfs_memparse() SUMMARY rate: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation : 76243.849 62728.925 69264.981 5525.345 Directory stat : 0.000 0.000 0.000 0.000 Directory removal : 47895.918 47533.099 47709.202 147.805 Tree creation : 43.638 38.047 41.193 2.336 Tree removal : 33.540 27.030 31.193 2.951 V-1: Entering PrintTimestamp... </pre> </div></div> LU-13520

a performance regression on directory creation

Bug Minor Resolved Fixed Alex Zhuravlev Shuichi Ihara Wed, 6 May 2020 08:02:25 +0000 Sat, 6 Jun 2020 15:02:51 +0000 Sat, 6 Jun 2020 15:02:51 +0000 Lustre 2.13.0 Lustre 2.14.0 0 4 <p>can you please try with disabled prefetching, on MDS:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">echo 0 &gt;/sys/fs/ldiskfs/&lt;mdt-device&gt;mb_prefetch_limit</pre> </div></div> <p>attached are two framegraphes on commit 59a3d72(master) 52b5f4a(an commit "b7cd65a <a href="https://jira.whamcloud.com/browse/LU-12988" title="improve mount time on huge ldiskfs filesystem" class="issue-link" data-issue-key="LU-12988"><del>LU-12988</del></a> ldiskfs: mballoc to prefetch groups") and it pointed out the latest master spent a lot of CPU time on ldiskfs_read_block_bitmap_nowait() with spinlock which doesn't see same CPU time on commit 52b5f4a.</p> <p>which kernel did you use?<br/> (I'll try to cook a quick patch for that kernel)</p> <p>looks better after "echo 0 &gt;/sys/fs/ldiskfs/&lt;mdt-device&gt;mb_prefetch_limit"</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>master 52b5f4a LU-10401 tests: fix error from 'tr -d=' with "echo 0 &gt;/sys/fs/ldiskfs/&lt;mdt-device&gt;mb_prefetch_limit" SUMMARY rate: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation : 72279.489 62814.460 67811.718 3881.214 Directory stat : 0.000 0.000 0.000 0.000 Directory removal : 47547.120 47419.048 47503.194 58.836 Tree creation : 53.445 40.462 47.929 5.477 Tree removal : 32.093 27.697 30.574 2.036 V-1: Entering PrintTimestamp... </pre> </div></div> <blockquote> <p>which kernel did you use?</p></blockquote> <p>I'm testing RHEL7.7 kernel.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Directory creation : 76243.849 62728.925 69264.981 5525.345 ... Directory creation : 72279.489 62814.460 67811.718 3881.214 </pre> </div></div> <p>even with mb_prefetch_limit=0 it's still behind ? can you please try again with fs remount? thanks a lot</p> <p>also, how empty/full MDT was? probably just created?</p> <p>I think performance with mb_prefetch_limit=0 is fine. here is resutls after remounted MDT.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>SUMMARY rate: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation : 75747.972 64178.225 70424.704 4767.309 Directory stat : 0.000 0.000 0.000 0.000 Directory removal : 50648.239 44288.365 47869.443 2657.031 Tree creation : 44.080 29.518 37.821 6.118 Tree removal : 32.954 28.309 30.811 1.913 V-1: Entering PrintTimestamp... </pre> </div></div> <blockquote> <p>also, how empty/full MDT was? probably just created?</p></blockquote> <p>MDT is emply and yes, it was just created.</p> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38513" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38513</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13520" title="a performance regression on directory creation" class="issue-link" data-issue-key="LU-13520"><del>LU-13520</del></a> ldiskfs: fastpath in prefetching<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e4267c5f83166bc31475eac7977a1a8ef1070dff</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sihara" class="user-hover" rel="sihara">sihara</a> can you please try the patch above? unfortunately I can test it with master branch only.</p> <p>patch solves problem even with default mb_prefetch_limit parameter.<br/> master (52b5f4a) + 38513</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>SUMMARY rate: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation : 83171.845 71796.197 77274.256 4652.121 Directory stat : 0.000 0.000 0.000 0.000 Directory removal : 55310.503 50699.606 52626.180 1956.720 Tree creation : 38.032 26.759 33.917 5.081 Tree removal : 32.981 30.756 31.565 1.004 V-1: Entering PrintTimestamp... </pre> </div></div> <p>thanks! making patches for other kernels..</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38513/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38513/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13520" title="a performance regression on directory creation" class="issue-link" data-issue-key="LU-13520"><del>LU-13520</del></a> ldiskfs: fastpath in bitmap prefetching<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6a15798f51a9dcb29333e8107a59c5551f607fff</p> <p>Landed for 2.14</p> Related Development Rank 1|i00zlz: Rank (Obsolete) 9223372036854775807 Severity