Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-11191

Striped DIR accross 2 MDTs very slow

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: Lustre 2.10.3
    • Fix Version/s: None
    • Labels:
      None
    • Environment:
      Client: sles12 SP3 Lustre2.10.3
      servers: cent7.4 Lustre 2.10.3
    • Severity:
      3
    • Rank (Obsolete):
      9223372036854775807

      Description

      May be dup of LU-8118

      We have 2 MDSs each with 1 MDT. Running mdtest  on mdt0 or mdt1 directory is fast but a striped (using -D) dir across both is very slow.

       $ lfs getdirstripe ../MDT0
      lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none
      
      $ lfs getdirstripe ../MDT0_and_1/
      lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: fnv_1a_64
      mdtidx		 FID[seq:oid:ver]
           0		 [0x200002340:0x3:0x0]		
           1		 [0x240002340:0x3:0x0]		
      
      
      

      STRIPE DIR

      -- started at 07/29/2018 22:47:01 --
      
      mdtest-1.9.3 was launched with 8 total task(s) on 8 node(s)
      Command line used: /home/mhanafi/bin/mdtest -z 4 -b 3 -n 500 -i 5 -u -d MDT0_and_1/
      Path: /nobackuptest/mhanafi/MDTESTS/MDT0_and_1
      FS: 3250.9 TiB   Used FS: 0.3%   Inodes: 102.4 Mi   Used Inodes: 0.0%
      
      8 tasks, 3872 files/directories
      
      SUMMARY: (of 5 iterations)
         Operation                      Max            Min           Mean        Std Dev
         ---------                      ---            ---           ----        -------
         Directory creation:      11555.863      11347.736      11457.891         85.542
         Directory stat    :      27105.495      25926.197      26639.416        389.884
         Directory removal :       1101.478        932.399        986.495         65.291
         File creation     :      26333.527      22289.699      25075.189       1451.802
         File stat         :      32253.499      28994.953      30613.853       1247.751
         File read         :      41577.227      37023.290      39056.222       1766.014
         File removal      :      50925.204      46150.431      47627.634       1694.602
         Tree creation     :       1215.416       1194.374       1205.364          8.445
         Tree removal      :        142.877        101.763        130.532         15.266
      
      -- finished at 07/29/2018 22:47:31 --
      
      

      SINGLE STRIPE DIR

      -- started at 07/29/2018 22:47:37 --
      
      mdtest-1.9.3 was launched with 8 total task(s) on 8 node(s)
      Command line used: /home/mhanafi/bin/mdtest -z 4 -b 3 -n 500 -i 5 -u -d MDT0/
      Path: /nobackuptest/mhanafi/MDTESTS/MDT0
      FS: 3250.9 TiB   Used FS: 0.3%   Inodes: 102.4 Mi   Used Inodes: 0.0%
      
      8 tasks, 3872 files/directories
      
      SUMMARY: (of 5 iterations)
         Operation                      Max            Min           Mean        Std Dev
         ---------                      ---            ---           ----        -------
         Directory creation:      57733.445      50424.700      53351.848       2550.582
         Directory stat    :      79237.751      68950.262      72220.457       3684.412
         Directory removal :      60426.946      56294.687      57780.897       1518.306
         File creation     :      29322.719      28476.241      28922.159        336.506
         File stat         :      32544.144      29101.692      31061.575       1346.628
         File read         :      39504.498      36896.728      38229.842       1128.910
         File removal      :      50177.351      43709.006      46399.363       2125.337
         Tree creation     :       6880.380       5797.328       6223.052        392.540
         Tree removal      :       8109.055       7364.743       7683.243        243.272
      
      -- finished at 07/29/2018 22:47:40 --
      
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                adilger Andreas Dilger
                Reporter:
                mhanafi Mahmoud Hanafi
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: