[LU-14172] DIR Stat performance regression in striped dir

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.14.0, Lustre 2.12.6
• Affects Version/s: Lustre 2.12.6
• Severity: 2
• Assignee: Lai Siyao
• Reporter: Shuichi Ihara

Description

 There is a metadata (DIR Stat) performance regression in 2.12.6 (RC1). It looks like that regression is exist in the part of striped directory and server side.
Here is a reproducer and test results.

client: version=2.12.6_RC1_1_g327c8b7
server: version=2.12.6_RC1_1_g327c8b7 or lustre-2.12.5

# mkdir /ai400x/mdt0
# lfs setdirstripe -c 4 /ai400x/mdt_stripe
# lfs setdirstripe -c 4 -D /ai400x/mdt_stripe

#  salloc -p 40n -N 40 --ntasks-per-node=16  mpirun -mca btl_openib_if_include mlx5_1:1 -x UCX_NET_DEVICES=mlx5_1:1 --bind-to core:overload-allowed --allow-run-as-root /work/tools/bin/mdtest -i 3 -p 10 -n 1500 -u -D -d $PATH

Single MDT without DNE
Server: Lustre-2.12.5

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation        :      54315.552      50037.128      52618.576       1855.106
   Directory stat            :     186516.109     184354.609     185726.143        972.887
   Directory removal         :      66572.651      64990.546      65627.103        681.777
   Tree creation             :         46.771         24.099         36.301          9.336
   Tree removal              :         16.926         13.890         15.720          1.316

Server: Lustre-2.12.6-RC1

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation        :      68098.113      59185.349      62208.643       4164.966
   Directory stat            :     193338.869     192650.348     193031.824        285.743
   Directory removal         :      65905.804      64842.618      65212.728        490.440
   Tree creation             :         44.234         33.906         39.452          4.251
   Tree removal              :         17.024         15.068         16.279          0.864

Stripe Directory across four MDTs
Server: Lustre-2.12.5

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation        :       6385.748       5929.670       6113.851        196.251
   Directory stat            :     166190.895     162991.180     164733.372       1321.263
   Directory removal         :       4789.518       4294.122       4584.600        211.099
   Tree creation             :         13.200          1.102          6.937          4.948
   Tree removal              :          9.126          8.479          8.810          0.264

Server: Lustre-2.12.6-RC1

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation        :       6694.539       6505.265       6613.160         79.512
   Directory stat            :      49873.850      48817.530      49260.117        447.881   <--- This is regression.
   Directory removal         :       4768.841       4253.124       4592.927        240.327
   Tree creation             :         13.490          0.705          7.321          5.229
   Tree removal              :          9.051          8.441          8.774          0.252

Issue Links

• is related to: LU-14146 Massive directory metadata operation performance decrease
• is related to: LU-13437 rename may miss revoking LOOKUP lock to cause stale dentry on client