https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15692 Lustre <p>performance regressions in stripe directory on 2.15.0 (commit;4d93fd7) were found against b2_14(commit:d4b9557).<br/> Here is configuration.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>4 x MDS (1 x MDT per MDS) 4 x OSS (2 x OSS per OSS) 40 x client [root@ec01 ~]# mkdir -p /exafs/d0/d1/d2/mdt_stripe/ [root@ec01 ~]# lfs setdirstripe -c 4 -D /exafs/d0/d1/d2/mdt_stripe/ [root@ec01 ~]# salloc -p 40n -N 40 --ntasks-per-node=16 mpirun --allow-run-as-root -oversubscribe -mca btl_openib_if_include mlx5_1:1 -x UCX_NET_DEVICES=mlx5_1:1 /work/tools/bin/mdtest -n 2000 -F -i 3 -p 10 -v -d /exafs/d0/d1/d2/mdt_stripe/ </pre> </div></div> <p>Here is test resutls.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>server: version=2.15.0_RC2_22_g4d93fd7 client: version=2.15.0_RC2_22_g4d93fd7 SUMMARY rate: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation 103733.203 76276.410 93728.713 15168.101 File stat 693152.731 656461.448 671671.960 19132.425 File read 259081.462 247951.008 253393.168 5569.308 File removal 145137.390 142142.699 143590.068 1499.846 Tree creation 48.035 1.922 17.475 26.467 Tree removal 35.643 15.861 24.045 10.323 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>server: version=2.14.0_21_gd4b9557 client: version=2.14.0_21_gd4b9557 SUMMARY rate: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation 138939.425 81336.388 117014.695 31167.261 File stat 1678888.952 1580356.340 1645190.276 56162.463 File read 569731.788 528830.155 546121.363 21170.387 File removal 191837.291 186597.900 188595.661 2832.527 Tree creation 120.108 0.986 51.078 61.778 Tree removal 40.863 33.203 37.987 4.171 </pre> </div></div> <p>As far as I observed this, it seems to be server side regression since because performance with lustre-2.15 clients + lustre-2.14 was ok below.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>server: version=2.14.0_21_gd4b9557 client: version=2.15.0_RC2_22_g4d93fd7 SUMMARY rate: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation 132009.360 74074.615 106514.108 29585.056 File stat 1570754.679 1457120.401 1532703.082 65457.038 File read 563710.286 540228.432 553871.772 12194.544 File removal 189557.092 186065.253 187536.946 1809.374 Tree creation 54.678 1.883 19.576 30.399 Tree removal 42.065 41.677 41.875 0.194 </pre> </div></div> <p>I am running 'git bisect', can hopefully find an commit where started regression soon.</p> LU-15692

performance regressions for files in stripe directory

Bug Blocker Resolved Fixed Lai Siyao Shuichi Ihara Fri, 25 Mar 2022 00:21:55 +0000 Wed, 14 Sep 2022 06:24:15 +0000 Sat, 2 Apr 2022 15:05:49 +0000 Lustre 2.15.0 Lustre 2.15.0 0 8 <p>Tentatively adding 2.15.0 fix version for tracking until we understand this better.</p> <p>it seems that the following patch where regressions started.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> LU-14459 lmv: change default hash type to crush Change the default hash type to CRUSH to minimize the number of directory entries that need to be migrated. </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>server: version=2.14.51_197_gf269497 client: version=2.15.0_RC2_22_g4d93fd7 SUMMARY rate: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation 148072.690 87600.145 127000.919 34149.618 File stat 1523849.471 1388808.972 1441253.182 72393.681 File read 562840.721 505515.837 538333.864 29552.364 File removal 197259.873 191117.823 194934.244 3331.372 Tree creation 111.869 1.707 39.426 62.755 Tree removal 44.113 30.518 36.562 6.922 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>server: version=2.14.2.14.51_198_gbb60caa client: version=2.15.0_RC2_22_g4d93fd7 SUMMARY rate: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation 86531.781 63506.794 72790.003 12142.761 File stat 808075.643 746570.771 784071.104 32898.551 File read 260064.500 249212.881 256291.924 6135.058 File removal 159592.539 155603.788 157752.556 2012.224 Tree creation 120.060 1.138 41.069 68.410 Tree removal 37.780 37.263 37.450 0.287 V-1: Entering PrintTimestamp... </pre> </div></div> <p>Lustre client didn't change version and server version only changed, but does it make sense patch <a href="https://review.whamcloud.com/#/c/43684/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/43684/</a> impact for server as well?</p> <p>It looks like <tt>LMV_HASH_TYPE_DEFAULT</tt> is used on both the client and server, so patch <a href="https://review.whamcloud.com/43684" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43684</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-14459" title="DNE3: directory auto split during create" class="issue-link" data-issue-key="LU-14459">LU-14459</a> lmv: change default hash type to crush</tt>" would definitely affect behavior of the server if the client is not specifying the directory hash type (which it is not).</p> <p>Ihara, can you collect flamegraph on MDS for both cases?</p> <p>Shuichi, do you have any idea of why this is slower on the server? Higher CPU usage, uneven file distribution across MDTs, something else? If high MDS CPU usage, would you be able to collect a flame graph of the before/after the patch? The CRUSH hash is a bit more CPU intensive, but I wouldn't think it would hurt performance by 50%, but it would be good to know where the performance is lost so possibly it can be optimized.</p> <p>Before collecting flamegraph or other information in detail, I just found MDT load balancing seems to be not working well after patch. It's unbalanced file distribution across MDTs at create.<br/> For instnace, here is just file creation test in a stirpe directory.</p> <p>Before patch (commit:f269497)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mpirunp -np 640 mdtest -n 2000 -F -C -i 1 -p 10 -v -d /exafs/d0/d1/d2/mdt_stripe/ [root@ec01 ~]# lfs df -i | grep MDT exafs-MDT0000_UUID      83050496      320298    82730198   1% /exafs[MDT:0]  exafs-MDT0001_UUID      83050496      320283    82730213   1% /exafs[MDT:1]  exafs-MDT0002_UUID      83050496      320334    82730162   1% /exafs[MDT:2]  exafs-MDT0003_UUID      83050496      320293    82730203   1% /exafs[MDT:3]  </pre> </div></div> <p>After patch (commit:bb60caa)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ec01 ~]# lfs df -i | grep MDT exafs-MDT0000_UUID 83050496 192404 82858092 1% /exafs[MDT:0] exafs-MDT0001_UUID 83050496 190698 82859798 1% /exafs[MDT:1] exafs-MDT0002_UUID 83050496 177266 82873230 1% /exafs[MDT:2] exafs-MDT0003_UUID 83050496 720852 82329644 1% /exafs[MDT:3] </pre> </div></div> <p>That's why mdtest's numbers was slower since one of MDS/MDT (MDT3 in this case) is more working longer than others. Eventually, mdtest's elapsed time is longer than balanced case.</p> <p>Lai, can you please look into the CRUSH hash to see if this is causing imbalance, and why. It would be possibly to just create a bunch of names in the format of mdtest and count the distribution of files. </p> <p>One thing that might be happening is that the format of the mdtest filenames may be triggering the "no rename across MDTs" heuristic for temp files if they are of the form .xxxxxxxx at the end? </p> <p>Shuichi, it should be possible to see if changing the hash function back to the old one solves the problem. Use:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs mkdir -H fnv_1a_64 /exafs/d0/d1/d2/mdt_stripe # lfs setdirstripe -H fnv_1a_64 -D /exafs/d0/d1/d2/mdt_stripe </pre> </div></div> <p>It is easy enough to make a patch that will change the default hash function back to <tt>fnv_1a_64</tt> for 2.15.0, but it would be <b>preferable</b> to understand why the performance/balance is so bad because this hash function <em>should</em> be better for directory migration/restriping.</p> <p>Also, did you give patch <a href="https://review.whamcloud.com/46696" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46696</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-15546" title="Shared Directory File Creates regression seen in 2.15 when comparing to 2.12.6" class="issue-link" data-issue-key="LU-15546"><del>LU-15546</del></a> mdt: keep history of mdt_reint_open() lock</tt>" a try to see if this further restores the file create performance?</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/46950" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46950</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15692" title="performance regressions for files in stripe directory" class="issue-link" data-issue-key="LU-15692"><del>LU-15692</del></a> lmv: change default hash back to fnv_1a_64<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a70be6f15e5f742443c2ca2e854d0bdab53a3dc5</p> <p>If the testing system is fresh new, /exafs/d0/d1/d2/mdt_stripe/ will be created on MDT3 (because max-inherit-rr is 3 by default, d0 will be created on MDT0, d1 on MDT1, d2 on MDT2), and the new directories created by mdtest will all be located on MDT3 (master inode) because at this level max-inherit-rr doesn't work any more. This explains why there are far more inodes used on MDT3. But I don't understand why the inodes are used equally before this patch, will you count the number of directories on each MDT? You can get it via "for i in 0 1 2 3; do lfs getstripe -m /exafs/d0/d1/d2/mdt_stripe/* | grep -c $i; done".</p> <p><del>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch:</del> <a href="https://review.whamcloud.com/46958" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46958</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-15692" title="performance regressions for files in stripe directory" class="issue-link" data-issue-key="LU-15692"><del>LU-15692</del></a> tests: clean up sanity test_316/test_319</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: c65ed52cf04219ef247e7764c5c45e36add6f825</del></p> <p>Andreas, "fnv_1a_64" worked fine below.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ec01 ~]# lfs mkdir -H fnv_1a_64 /exafs/d0/d1/d2/mdt_stripe [root@ec01 ~]# lfs setdirstripe -c 4 -H fnv_1a_64 -D /exafs/d0/d1/d2/mdt_stripe [root@ec01 ~]# lfs getdirstripe /exafs/d0/d1/d2/mdt_stripe lmv_stripe_count: 0 lmv_stripe_offset: 2 lmv_hash_type: none [root@ec01 ~]# lfs getdirstripe -D /exafs/d0/d1/d2/mdt_stripe lmv_stripe_count: 4 lmv_stripe_offset: -1 lmv_hash_type: fnv_1a_64 lmv_max_inherit: 3 lmv_max_inherit_rr: 0 [root@ec01 ~]# mpirun -np 640 mdtest -n 2000 -F -i 1 -v -d /exafs/d0/d1/d2/mdt_stripe/ -C SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation 146900.918 146900.918 146900.918 0.000 File stat 0.000 0.000 0.000 0.000 File read 0.000 0.000 0.000 0.000 File removal 0.000 0.000 0.000 0.000 Tree creation 34.638 34.638 34.638 0.000 Tree removal 0.000 0.000 0.000 0.000 [root@ec01 ~]# lfs df -i | grep MDT exafs-MDT0000_UUID 83050496 320295 82730201 1% /exafs[MDT:0] exafs-MDT0001_UUID 83050496 320290 82730206 1% /exafs[MDT:1] exafs-MDT0002_UUID 83050496 320285 82730211 1% /exafs[MDT:2] exafs-MDT0003_UUID 83050496 320284 82730212 1% /exafs[MDT:3] </pre> </div></div> <p>When I did test "fnv_1a_64" before to check, it didn't work becouse I did run another "lfs setdirstirpe -D" after "lfs setdirstripe -H fnv_1a_64". e.g.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ec01 ~]# lfs mkdir -H fnv_1a_64 /exafs/d0/d1/d2/mdt_stripe [root@ec01 ~]# lfs setdirstripe -H fnv_1a_64 -D /exafs/d0/d1/d2/mdt_stripe [root@ec01 ~]# lfs setdirstripe -c 4 -D /exafs/d0/d1/d2/mdt_stripe [root@ec01 ~]# lfs getdirstripe -D /exafs/d0/d1/d2/mdt_stripe lmv_stripe_count: 4 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 </pre> </div></div> <p>stripe count is configured properly, but hash_type changed back to default CRUSH algorithm.<br/> all combinations is needed when if '-D' (default) option is required.</p> <blockquote> <p>Also, did you give patch <a href="https://review.whamcloud.com/46696" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46696</a> "<a href="https://jira.whamcloud.com/browse/LU-15546" title="Shared Directory File Creates regression seen in 2.15 when comparing to 2.12.6" class="issue-link" data-issue-key="LU-15546"><del>LU-15546</del></a> mdt: keep history of mdt_reint_open() lock" a try to see if this further restores the file create performance?</p></blockquote> <p>let me test patch too.</p> <blockquote><p>If the testing system is fresh new, /exafs/d0/d1/d2/mdt_stripe/ will be created on MDT3 (because max-inherit-rr is 3 by default, d0 wil</p></blockquote> <p>Lai, /exafs/d0/d1/d2/mdt_stripe/ is a striped directory on across all MDTs. I don't think inherit-rr is related in this case, no?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ec01 ~]# lfs setdirstripe -c 4 -D /exafs/d0/d1/d2/mdt_stripe/ [root@ec01 ~]# lfs getdirstripe -D /exafs/d0/d1/d2/mdt_stripe lmv_stripe_count: 4 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ec01 ~]# mkdir -p /exafs/d0/d1/d2/mdt_stripe/ </pre> </div></div> <p>According to the description, mdt_stripe is created with "mkdir", it should be a plain directory. And it has a default LMV to stripe to all MDTs.</p> <p>I don't think the MDT/striping of <tt>/exafs/d0/d1/d2/mdt_stripe/</tt> itself is not so critical, because it is only holding another level of subdirectories in it, like <tt>/exafs/d0/d1/d2/mdt_stripe/test-dir.0-0/mdtest_tree.M.N/</tt> (for different client nodes and MPI ranks), and those <em>should</em> be striped across all MDTs.</p> <p>Rather than checking the MDT index of specific directories, it would probably be better to check all subdirectories like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs find /exafs/d0/d1/d2/mdt_stripe -type d | xargs lfs getdirstripe -m | sort | uniq -c </pre> </div></div> <p>That would at least print the starting MDT index for each directory, and presumably all of those directories should be striped over all 4 MDTs. This would be useful to check in the <tt>crush</tt> hash test if the imbalance is because of <b>directory</b> imbalance (e.g. 3/4 of subdirectories are created on MDT0003), or if it is a <b>file</b> imbalance within a single directory (e.g. 3/4 of files in <b>every</b> directory are created on MDT0003).</p> <p>The MDT distribution of files within each directory could probably be checked something like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># find /exafs/d0/d1/d2/mdt_stripe -mindepth 2 -maxdepth 2 -type d | while read D; do echo $D find $D -type f | xargs lfs getstripe -m | sort | uniq -c done </pre> </div></div> <p>though this may take a few minutes if there are a lot of files.</p> <p>This is mdtest to a single shared dir, not unique dir operation. So, all files are cretead in a sub directory.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ec01 ~]# lfs find /exafs/d0/d1/d2/mdt_stripe -type d /exafs/d0/d1/d2/mdt_stripe [root@ec01 ~]# mpirun -np 640 mdtest -n 2000 -F -i 1 -v -d /exafs/d0/d1/d2/mdt_stripe/ -C [root@ec01 ~]# lfs df -i | grep MDT exafs-MDT0000_UUID 83050496 720861 82329635 1% /exafs[MDT:0] exafs-MDT0001_UUID 83050496 192389 82858107 1% /exafs[MDT:1] exafs-MDT0002_UUID 83050496 190695 82859801 1% /exafs[MDT:2] exafs-MDT0003_UUID 83050496 177213 82873283 1% /exafs[MDT:3] [root@ec01 ~]# lfs find /exafs/d0/d1/d2/mdt_stripe -type d /exafs/d0/d1/d2/mdt_stripe /exafs/d0/d1/d2/mdt_stripe/test-dir.0-0 /exafs/d0/d1/d2/mdt_stripe/test-dir.0-0/mdtest_tree.0 [root@ec01 ~]# lfs find /exafs/d0/d1/d2/mdt_stripe -type d | xargs lfs getdirstripe -m 1 1 1 </pre> </div></div> <p>And, unbalanced distibution is not associated to particular a MDT.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ec01 ~]# lfs df -i | grep MDT exafs-MDT0000_UUID 83050496 301 83050195 1% /exafs[MDT:0] exafs-MDT0001_UUID 83050496 281 83050215 1% /exafs[MDT:1] exafs-MDT0002_UUID 83050496 281 83050215 1% /exafs[MDT:2] exafs-MDT0003_UUID 83050496 280 83050216 1% /exafs[MDT:3] [root@ec01 ~]# mpirun -np 640 mdtest -n 2000 -F -i 1 -v -d /exafs/d0/d1/d2/mdt_stripe/ -C [root@ec01 ~]# lfs df -i | grep MDT exafs-MDT0000_UUID 83050496 190707 82859789 1% /exafs[MDT:0] exafs-MDT0001_UUID 83050496 192389 82858107 1% /exafs[MDT:1] exafs-MDT0002_UUID 83050496 177215 82873281 1% /exafs[MDT:2] exafs-MDT0003_UUID 83050496 720848 82329648 1% /exafs[MDT:3] [root@ec01 ~]# mpirun -np 640 mdtest -n 2000 -F -i 1 -v -d /exafs/d0/d1/d2/mdt_stripe/ -r [root@ec01 ~]# mpirun -np 640 mdtest -n 2000 -F -i 1 -v -d /exafs/d0/d1/d2/mdt_stripe/ -C [root@ec01 ~]# lfs df -i | grep MDT exafs-MDT0000_UUID 83050496 720861 82329635 1% /exafs[MDT:0] exafs-MDT0001_UUID 83050496 192389 82858107 1% /exafs[MDT:1] exafs-MDT0002_UUID 83050496 190703 82859793 1% /exafs[MDT:2] exafs-MDT0003_UUID 83050496 177214 82873282 1% /exafs[MDT:3] </pre> </div></div> <p>This should be because files are treated as temp file, which are created on the same MDT. All digit suffix filename (if the suffix length after "." is 8) are treated as temp file with crush hash type.</p> <p>Ihara, can you list some typical filename in the bottom level?</p> <p>Lai - two things about the "temp" filenames:</p> <ul class="alternate" type="square"> <li>files <em>should</em> still be created on same MDT as the "non-temp" filename. However, if the filename is like "<tt>foo.12345678</tt>" then the hashing code will only ever use "<tt>foo</tt>" to determine the "proper" MDT index.</li> <li>the "temp filename" code should <b>not</b> consider suffixes with <b>only</b> numbers as a temp filename. That is specifically to avoid putting all "<tt>foo.nnnnnnnn</tt>" filenames on the same MDT. However, if there is a mix of numbers and letters (e.g. hex suffix?) then it might be doing the wrong thing.</li> </ul> <p>The version of mdtest that I'm using locally only has numbers in the suffix:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># ls mdtest-easy/test-dir.0-0/mdtest_tree.0.0 file.mdtest.1.127 file.mdtest.1.128 file.mdtest.1.129 file.mdtest.1.13 file.mdtest.1.130 file.mdtest.1.131 file.mdtest.1.132 </pre> </div></div> <p>However, it might be getting confused by the extra '<tt>.</tt>' in the name if there are more files, like "<tt>file.mdtest.1.345678</tt>" or "<tt>file.mdtest.12.45678</tt>"? This would <b>incorrectly</b> fail the "<tt>(digit &gt;= suffixlen -1)</tt>" check because the second '<tt>.</tt>' is not counted in <tt>digit</tt> or <tt>upper</tt> or <tt>lower</tt>. There should probably be an additional check that there aren't non-alphanumeric characters in the suffix:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> ((digit &gt;= suffixlen - 1 &amp;&amp; !isdigit(name[namelen - suffixlen])) || upper == suffixlen || lower == suffixlen) <span class="code-keyword">return</span> <span class="code-keyword">false</span>; <span class="code-keyword">if</span> (type == LMV_HASH_TYPE_CRUSH2 &amp;&amp; digit + upper + lower != suffixlen) <span class="code-keyword">return</span> <span class="code-keyword">false</span>; </pre> </div></div> <p>Unfortunately, this changes the hash function subtly, so a new "<tt>LMV_HASH_TYPE_CRUSH2</tt>" hash type is needed for the new behavior. Otherwise, clients may <em>think</em> they know which MDT a particular filename is on but it would be wrong.</p> <p>I'm working on a patch to add the CRUSH2 hash, but for now we are planning to land patch <a href="https://review.whamcloud.com/46950" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46950</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-15692" title="performance regressions for files in stripe directory" class="issue-link" data-issue-key="LU-15692"><del>LU-15692</del></a> lmv: change default hash back to fnv_1a_64</tt>" for 2.15.0 since that is the lowest risk option. I don't think the current CRUSH hash is totally broken, but it unfortunately has imbalance in some important use cases.</p> <p>Andreas, how about retry lookup on client? if the filename looks to be a temp file in above format, if it's not located on the MDT with name like "foo", then lookup again with full name hash?</p> <blockquote> <p>how about retry lookup on client?</p></blockquote> <p>that would only work for new clients that are patched to repeat the lookup, and at that point the clients could also be patched to understand CRUSH2. Older clients will <b>not</b> repeat the lookup, so they would not be able to find the filename on the MDT where it should be located (based on older CRUSH hash). That might also cause the same filename to be created twice in the same directory (old clients on one MDT, new clients on a different MDT). Having a different hash function that old clients do not understand <b>should</b> result in them looking for the filename on <b>all</b> MDTs, and prevent them from creating new files in that directory (return <tt>-EBADFD</tt>).</p> <p>I'm having a hard time to convince myself this is code correct. The comment in the commit message says:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LU-13481 dne: improve temp file name check Previously if all but two characters in file name suffix are digit, it's not treated as temp file, as is too strict if suffix length is short, e.g. 6. Change it to allow one character, and this non-digit character should not be the starting character. </pre> </div></div> <p>Besides the problem with "<tt>.-_</tt>" characters in the suffix (which would make count of digits/upper/lower too small and fail the <tt>suffixlen</tt> check), it doesn't look like the <tt>isdigit()</tt> check is correct:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> ((digit &gt;= suffixlen - 1 &amp;&amp; !isdigit(name[namelen - suffixlen])) || upper == suffixlen || lower == suffixlen) <span class="code-keyword">return</span> <span class="code-keyword">false</span>; </pre> </div></div> <p>If "<tt>digit &gt;= suffixlen -1</tt>" (say <tt>name = "foo.12345678"</tt>, <tt>digit = 8</tt>, <tt>suffixlen = 8</tt>) this check will fail (and return "true" for the temp filename check) because "<tt>1</tt>" is a digit. I think this is supposed to be just "<tt>isdigit(name[])</tt>" (no '<tt>!</tt>').</p> <p>It's in accordance with the comment, isn't it? The comment says "foo.a1234567" shouldn't be treated as temp file, but "foo.12a34567" and "foo.12345678" should be. IMO "foo.123.4567" shouldn't be treated as a temp file, that is, isdot should be checked here, if suffix contains ".", it should return false.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/46950/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46950/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15692" title="performance regressions for files in stripe directory" class="issue-link" data-issue-key="LU-15692"><del>LU-15692</del></a> lmv: change default hash back to fnv_1a_64<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0090b6f6f6cfd65fcacfa4dda23bee2cec11cb70</p> <p>Definitely "foo.12345678" should <b>not</b> be considered a temp file, since this is a common case (eg file.YYYYMMDD). The chance of a 6-number temp file is 1/40k, and an 8-number temp file being hit randomly is less than 1/2M. </p> <p>The original code even considered 6/6, 5/6 and 4/6 numbers to not be temp files (ie. "digit &gt;= suffixlen - 2") , but 4/6 numbers was too easily hit by mktemp. It was <b>supposed</b> to keep 8/8 and 7/8 as non-temp files as long as 7/8 was like "file.f1234567". The problem is that the 8/8 case also fails because the first char <b>is</b> a digit, so "<tt>!isdigit(name<span class="error">&#91;namelen-suffixlen&#93;</span>)</tt>" fails, and it doesn't matter if the "<tt>(digit &gt;= suffixlen - 1)</tt>" part is true or not because the "false" check is not met, and "true" is returned. </p> <p>The proper check should be something like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (digit == suffixlen || upper == suffixlen || lower == suffixlen || (digit == suffixlen - 1 &amp;&amp; !isdigit(name[namelen - suffixlen]))) <span class="code-keyword">return</span> <span class="code-keyword">false</span>; </pre> </div></div> <p>I believe that the part intend for 2.15 has now landed. There is still some outstanding work to track to work out the issues with the new hash function so we can switch back to that but that can be tracked under a new JIRA for 2.16/2.15.x</p> Cloners LU-15720 Related LU-15479 LU-15502 LU-15546 LU-13481 LU-11025 LU-14459 Development Rank 1|i02lpj: Rank (Obsolete) 9223372036854775807 Severity