[LU-6800] Significant performance regression with patch LU-5264

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.8.0
• Labels: patch
• Environment: master
• Severity: 2
• Assignee: Bruno Faccini (Inactive)
• Reporter: Shuichi Ihara (Inactive)

Description

Durding our performance testing, we found siginicant metadata performance regression with LU-5264 on master.

# mpirun -np 128 -ppn 4 -hostfile ./hostfile /work/tools/bin/mdtest -n 1000 -p 10 -i 5 -d /scratch1/mdtest.out

master

SUMMARY: (of 5 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation:      39552.671      33039.129      37024.828       2875.617
   Directory stat    :      33462.417      29340.691      31662.586       1384.330
   Directory removal :      40938.777      40238.677      40571.960        283.701
   File creation     :      17696.663      17209.531      17542.185        171.470
   File stat         :      33892.041      33429.312      33680.603        170.577
   File read         :      11284.121      11012.694      11220.417        104.978
   File removal      :      39718.200      39449.348      39556.254         90.590
   Tree creation     :       4583.939        700.335       3652.356       1487.449
   Tree removal      :        170.563        156.738        162.935          5.172

keep client version, but revert patch 42fdf8355791cb682c6120f7950bb2ecd50f97aa (LU-5264 obdclass: fix race during key quiescency) on servers.

SUMMARY: (of 5 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation:      44937.511      42117.095      43780.402       1335.927
   Directory stat    :     135310.427     129560.951     133625.293       2077.128
   Directory removal :      51525.499      46852.534      49965.297       1611.759
   File creation     :      42978.506      41435.145      42413.409        586.294
   File stat         :     135882.699     133344.886     134466.144        977.577
   File read         :     121788.787     111332.613     116374.190       3351.730
   File removal      :      84827.815      78120.995      80378.741       2522.662
   Tree creation     :       4650.004       3788.893       4268.099        336.241
   Tree removal      :        198.059        129.234        179.980         25.563

Issue Links

• is related to: LU-6823 Performance regression on servers with LU-5264
• is related to: LU-11089 Performance improvements for lu_object locking
• is related to: LU-5264 ASSERTION( info->oti_r_locks == 0 ) at OST umount
• is related to: LU-8130 Migrate from libcfs hash to rhashtable