https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11854 Lustre <p>Directory operations (create and removal) on stripe dir with DNE2 has been significantly slower (more than 50x) than non DNE configuration and DNE1. Here is test results.</p> <p>Non DNE configuration</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> [root@c01 ~]# mkdir /scratch1/nodne [root@c01 ~]# salloc -N 32 --ntasks-per-node=24 mpirun -np 768 --allow-run-as-root /work/tools/bin/mdtest -n 1000 -u -vv -d /scratch1/nodne SUMMARY: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 97259.037 97259.037 97259.037 0.000 Directory stat : 347749.775 347749.775 347749.775 0.000 Directory removal : 96064.306 96064.306 96064.306 0.000 File creation : 111509.920 111509.920 111509.920 0.000 File stat : 317489.856 317489.856 317489.856 0.000 File read : 183132.719 183132.719 183132.719 0.000 File removal : 166205.620 166205.620 166205.620 0.000 Tree creation : 29.571 29.571 29.571 0.000 Tree removal : 26.353 26.353 26.353 0.000 V-1: Entering print_timestamp... </pre> </div></div> <p>DNE1 configuration</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> [root@c01 ~]# lfs mkdir -i 0 /scratch1/mdt0 [root@c01 ~]# lfs mkdir -i 1 /scratch1/mdt1 [root@c01 ~]# salloc -N 32 --ntasks-per-node=24 mpirun -np 768 --allow-run-as-root /work/tools/bin/mdtest -n 1000 -u -vv -d /scratch1/mdt0@/scratch1/mdt1 SUMMARY: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 189546.945 189546.945 189546.945 0.000 Directory stat : 688947.817 688947.817 688947.817 0.000 Directory removal : 255838.417 255838.417 255838.417 0.000 File creation : 203077.460 203077.460 203077.460 0.000 File stat : 692292.941 692292.941 692292.941 0.000 File read : 350911.938 350911.938 350911.938 0.000 File removal : 339358.198 339358.198 339358.198 0.000 Tree creation : 38.326 38.326 38.326 0.000 Tree removal : 43.928 43.928 43.928 0.000 V-1: Entering print_timestamp... </pre> </div></div> <p>DNE2 configuration</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> [root@c01 ~]# lfs setdirstripe -c 2 /scratch1/stripedir [root@c01 ~]# lfs setdirstripe -c 2 -D /scratch1/stripedir [root@c01 ~]# salloc -N 32 --ntasks-per-node=24 mpirun -np 768 --allow-run-as-root /work/tools/bin/mdtest -n 1000 -u -vv -d /scratch1/stripedir SUMMARY: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 6585.023 6585.023 6585.023 0.000 &lt;----- Directory stat : 247222.630 247222.630 247222.630 0.000 Directory removal : 3554.469 3554.469 3554.469 0.000 &lt;----- File creation : 250751.232 250751.232 250751.232 0.000 File stat : 566094.009 566094.009 566094.009 0.000 File read : 362051.872 362051.872 362051.872 0.000 File removal : 298873.347 298873.347 298873.347 0.000 Tree creation : 13.632 13.632 13.632 0.000 Tree removal : 4.830 4.830 4.830 0.000 V-1: Entering print_timestamp... </pre> </div></div> 2.12.0 LU-11854

Dir operation on DNE2 are slower than DNE1 or non DNE

Bug Minor Open Unresolved Lai Siyao Shuichi Ihara llnl Mon, 14 Jan 2019 16:17:14 +0000 Mon, 31 Jan 2022 23:24:43 +0000 0 11 <p>Lai</p> <p>Could you please investigate?</p> <p>Ihara</p> <p>Could you please share some more details about the configuration and the test script used?</p> <p>Peter</p> <p>Ihara, setting <tt>lfs setdirstripe -c 2 -D /scratch1/stripedir</tt> causes every subdirectory to also be created with 2 stripes, which triggers distributed transactions and is definitely slower than creating a local 1-stripe directory. That is why we are working on the dynamic restriping, so that the directory can be created with 1 stripe at the start, and only move to DNE2 striping if it is needed.</p> <p>OK, but 100K to 6k dir creations 110K to 3k dir removal, is that level performance drop expected? If this is expected, that's fine when striped dir is enabled by default, but I wonder if we could get a bit better performance for resonable performance on dir operation when stripe dir enabled.</p> <p>It's hard to improve this in a short time, because a striped directory creation will often trigger old transactions (previous creations) commit to make DNE recovery easier, which means it causes sync on all MDTs. Without changing DNE recovery, we can't do much right now.</p> <p>Could this be fixed by <a href="https://jira.whamcloud.com/browse/LU-11999" title="DNE performance improvement" class="issue-link" data-issue-key="LU-11999"><del>LU-11999</del></a>?</p> <p><a href="https://jira.whamcloud.com/browse/LU-11999" title="DNE performance improvement" class="issue-link" data-issue-key="LU-11999"><del>LU-11999</del></a> still doesn't solve problem. Dir creation and removal operations in the striped directory are still much slower than remote Dir.</p> <p>Striped directory creation and removal will start distributed transaction, if the involved MDTs are not located on the same MDS, this may be optimized: if one MDT failed, distributed transactions can be recovered from logs on other MDTs, thus the dependencies between distributed transactions can be removed. This means, only if a distributed transaction depends on a local transaction, it needs to commit the local transaction. But this seems not true on current deployment, there are often more than one MDTs on an MDS.</p> <blockquote><p> if the involved MDTs are not located on the same MDS, this may be optimized ... &lt;redacted&gt; ... But this seems not true on current deployment, there are often more than one MDTs on an MDS.</p></blockquote> <p>Hi Lai,<br/> I'm not sure whose deployment you're referring to, but at least some sites would benefit from that optimization. Our file systems have 16 MDTs each on their own MDS. And if that optimization were implemented and worked well, we would take it into account when designing new file systems.</p> <p>Okay, I'll implement it and add a tunable option for this.</p> <p>Do we have update for this?</p> Related Development Rank 1|i009fr: Rank (Obsolete) 9223372036854775807 Severity