[LU-12321] Unlink speed needs to be improved in DoM configuration

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.14.0
• Affects Version/s: Lustre 2.13.0
• Severity: 3
• Assignee: Mikhail Pershin
• Reporter: Shuichi Ihara

Description

In DoM configuration, unlink speed is much slower (~35%) than Non-DOM configuration.
Here is test resutls with DNE2 with/without DoM.
Number of MDT and OST are four each. So, in theory, unlink speed in DoM configuration can be same as non-DOM configuration or even faster.
Here is test resutls.

DNE2(4xMDS/4MDT) without DOM, 4 x OSS(4xOST) and 32 clients

[root@c01 ~]# lfs setdirstripe -c 4 /cache1/out
[root@c01 ~]# lfs setdirstripe -c 4 -D /cache1/out
[root@c01 ~]# salloc --nodes=32 --ntasks-per-node=24 mpirun --allow-run-as-root /work/tools/bin/mdtest -F -n 2000 -u -d /cache1/out -vv -C -r -i 3 -p 10
SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :     155882.494     129033.204     140708.430      11235.756
   File stat         :          0.000          0.000          0.000          0.000
   File read         :          0.000          0.000          0.000          0.000
   File removal      :     240765.928     202754.538     226744.942      17032.855
   Tree creation     :          6.561          1.119          3.005          2.516
   Tree removal      :          5.832          5.215          5.600          0.274

DNE2(4xMDS/4MDT) with DOM, 4 x OSS(4xOST) and 32 clients

[root@c01 ~]# lfs setdirstripe -c 4 /cache1/out
[root@c01 ~]# lfs setdirstripe -c 4 -D /cache1/out
[root@c01 ~]# lfs setstripe -L mdt -E 1M /cache1/out
[root@c01 ~]# salloc --nodes=32 --ntasks-per-node=24 mpirun --allow-run-as-root /work/tools/bin/mdtest -F -n 2000 -u -d /cache1/out -vv -C -r -i 3 -p 10
SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :     142402.513     128265.451     135515.522       5739.604
   File stat         :          0.000          0.000          0.000          0.000
   File read         :          0.000          0.000          0.000          0.000
   File removal      :     149165.342     140330.903     144402.349       3595.289
   Tree creation     :         11.520          2.585          5.571          4.206
   Tree removal      :          5.491          4.100          4.606          0.628

Issue Links

• is related to: LU-11276 racer: mdc_dev.c:1346:mdc_req_attr_set()) uncovered page
• is related to: LU-11359 racer test 1 times out with client hung in dir_create.sh, ls, … and MDS in ldlm_completion_ast()
• is related to: LU-12124 slower unlinks with DOM files due to disabled ELC
• is related to: LU-10894 Compilebench results are lower than expected

Comments

[Peter Jones]

Landed for 2.14

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/36442/
Subject: LU-12321 mdc: allow ELC for DOM file unlink
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e07d4a58206756252c28b51048057c87a7cdb3c6

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/36442
Subject: LU-12321 mdc: allow ELC for DOM file unlink
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6fce9bde176337036e8fb256ef1a56439ff87f30

[Patrick Farrell (Inactive)]

Might also try https://review.whamcloud.com/#/c/34736/ , which isn't directly unlink, but improves lock management

[Shuichi Ihara]

Changed test enviroment a bit and re-run with/without DoM and patch.

Baseline No-DOM

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :     159607.234     125369.863     141195.699      14082.427
   File stat         :          0.000          0.000          0.000          0.000
   File read         :          0.000          0.000          0.000          0.000
   File removal      :     294886.243     282798.922     288348.055       4935.056
   Tree creation     :         13.224          1.082          8.957          5.575
   Tree removal      :          6.167          5.097          5.693          0.445

DOM

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :     261609.857     251431.584     256138.911       4045.382
   File stat         :          0.000          0.000          0.000          0.000
   File read         :          0.000          0.000          0.000          0.000
   File removal      :     161622.879     157116.547     159685.163       1885.747
   Tree creation     :         11.129          1.798          6.683          3.822
   Tree removal      :          6.530          3.377          5.251          1.354

DoM with patch 34071 (https://review.whamcloud.com/#/c/34071/)

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :     260525.295     254664.889     258414.166       2447.162
   File stat         :          0.000          0.000          0.000          0.000
   File read         :          0.000          0.000          0.000          0.000
   File removal      :     155803.951     153195.917     154248.451       1112.790
   Tree creation     :         12.622          4.922          7.634          3.531
   Tree removal      :          6.490          5.904          6.106          0.271

I didn't see any improvements from patch 34071 for unlink operatios with DoM.

[Shuichi Ihara]

Alex, yes, DoM files were empty. Mike advised me this is related to LU-12124 and LU-11359. I'm going to test https://review.whamcloud.com/#/c/34071/ if unlink speed improves.

[Alex Zhuravlev]

whether DoM files were empty?