https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11191 Lustre <p>May be dup of <a href="https://jira.whamcloud.com/browse/LU-8118" title="very slow metadata performance with shared striped directory" class="issue-link" data-issue-key="LU-8118">LU-8118</a></p> <p>We have 2 MDSs each with 1 MDT. Running mdtest  on mdt0 or mdt1 directory is fast but a striped (using -D) dir across both is very slow.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ lfs getdirstripe ../MDT0 lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none $ lfs getdirstripe ../MDT0_and_1/ lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: fnv_1a_64 mdtidx FID[seq:oid:ver] 0 [0x200002340:0x3:0x0] 1 [0x240002340:0x3:0x0] </pre> </div></div> <p>STRIPE DIR</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> -- started at 07/29/2018 22:47:01 -- mdtest-1.9.3 was launched with 8 total task(s) on 8 node(s) Command line used: /home/mhanafi/bin/mdtest -z 4 -b 3 -n 500 -i 5 -u -d MDT0_and_1/ Path: /nobackuptest/mhanafi/MDTESTS/MDT0_and_1 FS: 3250.9 TiB Used FS: 0.3% Inodes: 102.4 Mi Used Inodes: 0.0% 8 tasks, 3872 files/directories SUMMARY: (of 5 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 11555.863 11347.736 11457.891 85.542 Directory stat : 27105.495 25926.197 26639.416 389.884 Directory removal : 1101.478 932.399 986.495 65.291 File creation : 26333.527 22289.699 25075.189 1451.802 File stat : 32253.499 28994.953 30613.853 1247.751 File read : 41577.227 37023.290 39056.222 1766.014 File removal : 50925.204 46150.431 47627.634 1694.602 Tree creation : 1215.416 1194.374 1205.364 8.445 Tree removal : 142.877 101.763 130.532 15.266 -- finished at 07/29/2018 22:47:31 -- </pre> </div></div> <p>SINGLE STRIPE DIR</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> -- started at 07/29/2018 22:47:37 -- mdtest-1.9.3 was launched with 8 total task(s) on 8 node(s) Command line used: /home/mhanafi/bin/mdtest -z 4 -b 3 -n 500 -i 5 -u -d MDT0/ Path: /nobackuptest/mhanafi/MDTESTS/MDT0 FS: 3250.9 TiB Used FS: 0.3% Inodes: 102.4 Mi Used Inodes: 0.0% 8 tasks, 3872 files/directories SUMMARY: (of 5 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 57733.445 50424.700 53351.848 2550.582 Directory stat : 79237.751 68950.262 72220.457 3684.412 Directory removal : 60426.946 56294.687 57780.897 1518.306 File creation : 29322.719 28476.241 28922.159 336.506 File stat : 32544.144 29101.692 31061.575 1346.628 File read : 39504.498 36896.728 38229.842 1128.910 File removal : 50177.351 43709.006 46399.363 2125.337 Tree creation : 6880.380 5797.328 6223.052 392.540 Tree removal : 8109.055 7364.743 7683.243 243.272 -- finished at 07/29/2018 22:47:40 -- </pre> </div></div> Client: sles12 SP3 Lustre2.10.3 <br/> servers: cent7.4 Lustre 2.10.3 LU-11191

Striped DIR accross 2 MDTs very slow

Bug Major Resolved Duplicate Andreas Dilger Mahmoud Hanafi Mon, 30 Jul 2018 05:52:59 +0000 Wed, 29 Aug 2018 18:28:59 +0000 Wed, 29 Aug 2018 18:28:50 +0000 Lustre 2.10.3 0 6 <p>I uploaded debug logs to ftp site. <a href="ftp://uploads/dne_debug_LU-11191.tgz" class="external-link" target="_blank" rel="nofollow noopener">ftp://uploads/dne_debug_LU-11191.tgz</a></p> <p>Using the "-D" option means that this will set the default striping for all subdirectories created in this directory tree. Creating a striped directory has more overhead, since this is creating 2 (or N) separate directories, and a distributed transaction to coordinate between the MDTs.</p> <p>Typically, "-D" should be used on the directory above the per-job or per-tinestep output directory when these leaf directories will have millions of files, or be accessed concurrently by hundreds of clients.</p> <p>There is work underway in <a href="https://jira.whamcloud.com/browse/LU-4684" title="DNE3: allow migrating DNE striped directory" class="issue-link" data-issue-key="LU-4684"><del>LU-4684</del></a> to make the transition from single-stripe to multi-stripe directories automatic, but until then the striped directories should be used when they are needed instead of for all directories. </p> <p>@Andreas Dilger, how about using "lfs mkdir -c &lt;count&gt;"? What is the difference than "-D"? As I have not been using "-D" before.</p> <p>The difference is that using <tt>-c N</tt> creates a directory with 'N' stripes, but any subdirectories will be single-striped. Using the <tt>-D</tt> option will set the default directory stripe count for any subdirectories created in that directory, but this doesn't affect the directory on which the default directory layout is set, only new subdirectories. For example, having a directory structure like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>/home/adilger/jobs/bigdirs/d0001/f000001..f999999 /home/adilger/jobs/bigdirs/d0002/f000001..f999999 /home/adilger/jobs/bigdirs/d0003/f000001..f999999 /home/adilger/jobs/bigdirs/d0004/f000001..f999999 </pre> </div></div> <p>It makes sense to use "<tt>lfs mkdir -c 4 jobs/bigdirs/d0001</tt>" to create the output directories with multiple stripes. Alternately, the default directory striping could be set with "<tt>lfs mkdir -D -c 4 jobs/bigdirs</tt>" and all subdirectories created under <tt>bigdirs</tt> will have 4 directory stripes.</p> <p>You can close this case. We will track the work under <a href="https://jira.whamcloud.com/browse/LU-4684" title="DNE3: allow migrating DNE striped directory" class="issue-link" data-issue-key="LU-4684"><del>LU-4684</del></a></p> <p>ok sure</p> Duplicate LU-4684 Development Rank 1|i0000f: Rank (Obsolete) 9223372036854775807 Severity