Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-14172

DIR Stat performance regression in striped dir

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Lustre 2.12.6
    • Fix Version/s: Lustre 2.14.0, Lustre 2.12.6
    • Labels:
      None
    • Severity:
      2
    • Rank (Obsolete):
      9223372036854775807

      Description

       There is a metadata (DIR Stat) performance regression in 2.12.6 (RC1). It looks like that regression is exist in the part of striped directory and server side.
      Here is a reproducer and test results.

      client: version=2.12.6_RC1_1_g327c8b7
      server: version=2.12.6_RC1_1_g327c8b7 or lustre-2.12.5
      
      # mkdir /ai400x/mdt0
      # lfs setdirstripe -c 4 /ai400x/mdt_stripe
      # lfs setdirstripe -c 4 -D /ai400x/mdt_stripe
      
      #  salloc -p 40n -N 40 --ntasks-per-node=16  mpirun -mca btl_openib_if_include mlx5_1:1 -x UCX_NET_DEVICES=mlx5_1:1 --bind-to core:overload-allowed --allow-run-as-root /work/tools/bin/mdtest -i 3 -p 10 -n 1500 -u -D -d $PATH
      

      Single MDT without DNE
      Server: Lustre-2.12.5

      SUMMARY rate: (of 3 iterations)
         Operation                      Max            Min           Mean        Std Dev
         ---------                      ---            ---           ----        -------
         Directory creation        :      54315.552      50037.128      52618.576       1855.106
         Directory stat            :     186516.109     184354.609     185726.143        972.887
         Directory removal         :      66572.651      64990.546      65627.103        681.777
         Tree creation             :         46.771         24.099         36.301          9.336
         Tree removal              :         16.926         13.890         15.720          1.316
      

      Server: Lustre-2.12.6-RC1

      SUMMARY rate: (of 3 iterations)
         Operation                      Max            Min           Mean        Std Dev
         ---------                      ---            ---           ----        -------
         Directory creation        :      68098.113      59185.349      62208.643       4164.966
         Directory stat            :     193338.869     192650.348     193031.824        285.743
         Directory removal         :      65905.804      64842.618      65212.728        490.440
         Tree creation             :         44.234         33.906         39.452          4.251
         Tree removal              :         17.024         15.068         16.279          0.864
      

      Stripe Directory across four MDTs
      Server: Lustre-2.12.5

      SUMMARY rate: (of 3 iterations)
         Operation                      Max            Min           Mean        Std Dev
         ---------                      ---            ---           ----        -------
         Directory creation        :       6385.748       5929.670       6113.851        196.251
         Directory stat            :     166190.895     162991.180     164733.372       1321.263
         Directory removal         :       4789.518       4294.122       4584.600        211.099
         Tree creation             :         13.200          1.102          6.937          4.948
         Tree removal              :          9.126          8.479          8.810          0.264
      

      Server: Lustre-2.12.6-RC1

      SUMMARY rate: (of 3 iterations)
         Operation                      Max            Min           Mean        Std Dev
         ---------                      ---            ---           ----        -------
         Directory creation        :       6694.539       6505.265       6613.160         79.512
         Directory stat            :      49873.850      48817.530      49260.117        447.881   <--- This is regression.
         Directory removal         :       4768.841       4253.124       4592.927        240.327
         Tree creation             :         13.490          0.705          7.321          5.229
         Tree removal              :          9.051          8.441          8.774          0.252
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              laisiyao Lai Siyao
              Reporter:
              sihara Shuichi Ihara
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: