Details
-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
Lustre 2.10.3
-
None
-
Client: sles12 SP3 Lustre2.10.3
servers: cent7.4 Lustre 2.10.3
-
3
-
9223372036854775807
Description
May be dup of LU-8118
We have 2 MDSs each with 1 MDT. Running mdtest on mdt0 or mdt1 directory is fast but a striped (using -D) dir across both is very slow.
$ lfs getdirstripe ../MDT0
lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none
$ lfs getdirstripe ../MDT0_and_1/
lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: fnv_1a_64
mdtidx FID[seq:oid:ver]
0 [0x200002340:0x3:0x0]
1 [0x240002340:0x3:0x0]
STRIPE DIR
-- started at 07/29/2018 22:47:01 -- mdtest-1.9.3 was launched with 8 total task(s) on 8 node(s) Command line used: /home/mhanafi/bin/mdtest -z 4 -b 3 -n 500 -i 5 -u -d MDT0_and_1/ Path: /nobackuptest/mhanafi/MDTESTS/MDT0_and_1 FS: 3250.9 TiB Used FS: 0.3% Inodes: 102.4 Mi Used Inodes: 0.0% 8 tasks, 3872 files/directories SUMMARY: (of 5 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 11555.863 11347.736 11457.891 85.542 Directory stat : 27105.495 25926.197 26639.416 389.884 Directory removal : 1101.478 932.399 986.495 65.291 File creation : 26333.527 22289.699 25075.189 1451.802 File stat : 32253.499 28994.953 30613.853 1247.751 File read : 41577.227 37023.290 39056.222 1766.014 File removal : 50925.204 46150.431 47627.634 1694.602 Tree creation : 1215.416 1194.374 1205.364 8.445 Tree removal : 142.877 101.763 130.532 15.266 -- finished at 07/29/2018 22:47:31 --
SINGLE STRIPE DIR
-- started at 07/29/2018 22:47:37 -- mdtest-1.9.3 was launched with 8 total task(s) on 8 node(s) Command line used: /home/mhanafi/bin/mdtest -z 4 -b 3 -n 500 -i 5 -u -d MDT0/ Path: /nobackuptest/mhanafi/MDTESTS/MDT0 FS: 3250.9 TiB Used FS: 0.3% Inodes: 102.4 Mi Used Inodes: 0.0% 8 tasks, 3872 files/directories SUMMARY: (of 5 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 57733.445 50424.700 53351.848 2550.582 Directory stat : 79237.751 68950.262 72220.457 3684.412 Directory removal : 60426.946 56294.687 57780.897 1518.306 File creation : 29322.719 28476.241 28922.159 336.506 File stat : 32544.144 29101.692 31061.575 1346.628 File read : 39504.498 36896.728 38229.842 1128.910 File removal : 50177.351 43709.006 46399.363 2125.337 Tree creation : 6880.380 5797.328 6223.052 392.540 Tree removal : 8109.055 7364.743 7683.243 243.272 -- finished at 07/29/2018 22:47:40 --
Attachments
Issue Links
- duplicates
-
LU-4684 DNE3: allow migrating DNE striped directory
-
- Resolved
-
Activity
The difference is that using -c N creates a directory with 'N' stripes, but any subdirectories will be single-striped. Using the -D option will set the default directory stripe count for any subdirectories created in that directory, but this doesn't affect the directory on which the default directory layout is set, only new subdirectories. For example, having a directory structure like:
/home/adilger/jobs/bigdirs/d0001/f000001..f999999 /home/adilger/jobs/bigdirs/d0002/f000001..f999999 /home/adilger/jobs/bigdirs/d0003/f000001..f999999 /home/adilger/jobs/bigdirs/d0004/f000001..f999999
It makes sense to use "lfs mkdir -c 4 jobs/bigdirs/d0001" to create the output directories with multiple stripes. Alternately, the default directory striping could be set with "lfs mkdir -D -c 4 jobs/bigdirs" and all subdirectories created under bigdirs will have 4 directory stripes.
@Andreas Dilger, how about using "lfs mkdir -c <count>"? What is the difference than "-D"? As I have not been using "-D" before.
Using the "-D" option means that this will set the default striping for all subdirectories created in this directory tree. Creating a striped directory has more overhead, since this is creating 2 (or N) separate directories, and a distributed transaction to coordinate between the MDTs.
Typically, "-D" should be used on the directory above the per-job or per-tinestep output directory when these leaf directories will have millions of files, or be accessed concurrently by hundreds of clients.
There is work underway in LU-4684 to make the transition from single-stripe to multi-stripe directories automatic, but until then the striped directories should be used when they are needed instead of for all directories.
I uploaded debug logs to ftp site. ftp://uploads/dne_debug_LU-11191.tgz
ok sure