[LU-11854] Dir operation on DNE2 are slower than DNE1 or non DNE

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Labels: llnl
• Environment: 2.12.0
• Severity: 3
• Assignee: Lai Siyao
• Reporter: Shuichi Ihara

Description

Directory operations (create and removal) on stripe dir with DNE2 has been significantly slower (more than 50x) than non DNE configuration and DNE1. Here is test results.

Non DNE configuration

 [root@c01 ~]# mkdir /scratch1/nodne 
 [root@c01 ~]# salloc -N 32 --ntasks-per-node=24 mpirun -np 768 --allow-run-as-root /work/tools/bin/mdtest -n 1000 -u -vv -d /scratch1/nodne 
SUMMARY: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation:      97259.037      97259.037      97259.037          0.000
   Directory stat    :     347749.775     347749.775     347749.775          0.000
   Directory removal :      96064.306      96064.306      96064.306          0.000
   File creation     :     111509.920     111509.920     111509.920          0.000
   File stat         :     317489.856     317489.856     317489.856          0.000
   File read         :     183132.719     183132.719     183132.719          0.000
   File removal      :     166205.620     166205.620     166205.620          0.000
   Tree creation     :         29.571         29.571         29.571          0.000
   Tree removal      :         26.353         26.353         26.353          0.000
V-1: Entering print_timestamp...

DNE1 configuration

 [root@c01 ~]# lfs mkdir -i 0 /scratch1/mdt0
 [root@c01 ~]# lfs mkdir -i 1 /scratch1/mdt1
 [root@c01 ~]# salloc -N 32 --ntasks-per-node=24 mpirun -np 768 --allow-run-as-root /work/tools/bin/mdtest -n 1000 -u -vv -d /scratch1/mdt0@/scratch1/mdt1
 SUMMARY: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation:     189546.945     189546.945     189546.945          0.000
   Directory stat    :     688947.817     688947.817     688947.817          0.000
   Directory removal :     255838.417     255838.417     255838.417          0.000
   File creation     :     203077.460     203077.460     203077.460          0.000
   File stat         :     692292.941     692292.941     692292.941          0.000
   File read         :     350911.938     350911.938     350911.938          0.000
   File removal      :     339358.198     339358.198     339358.198          0.000
   Tree creation     :         38.326         38.326         38.326          0.000
   Tree removal      :         43.928         43.928         43.928          0.000
V-1: Entering print_timestamp...

DNE2 configuration

 [root@c01 ~]# lfs setdirstripe -c 2 /scratch1/stripedir
 [root@c01 ~]# lfs setdirstripe -c 2 -D /scratch1/stripedir
 [root@c01 ~]# salloc -N 32 --ntasks-per-node=24 mpirun -np 768 --allow-run-as-root /work/tools/bin/mdtest -n 1000 -u -vv -d /scratch1/stripedir
 SUMMARY: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation:       6585.023       6585.023       6585.023          0.000  <----- 
   Directory stat    :     247222.630     247222.630     247222.630          0.000
   Directory removal :       3554.469       3554.469       3554.469          0.000  <-----
   File creation     :     250751.232     250751.232     250751.232          0.000
   File stat         :     566094.009     566094.009     566094.009          0.000
   File read         :     362051.872     362051.872     362051.872          0.000
   File removal      :     298873.347     298873.347     298873.347          0.000
   Tree creation     :         13.632         13.632         13.632          0.000
   Tree removal      :          4.830          4.830          4.830          0.000
V-1: Entering print_timestamp...

Issue Links

• is related to: LU-17821 LMR1c: Improve DNE Distributed Transaction Performance