[#LU-12321] Unlink speed needs to be improved in DoM configuration

[LU-12321] Unlink speed needs to be improved in DoM configuration Created: 21/May/19 Updated: 25/Mar/20 Resolved: 14/Feb/20
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 2.13.0
Fix Version/s:	Lustre 2.14.0

Type:

Bug

Priority:

Major

Reporter:

Shuichi Ihara

Assignee:

Mikhail Pershin

Resolution:

Fixed

Votes:

Labels:

None

Issue Links:

Related
is related to	~~LU-11359~~	racer test 1 times out with client hu...	Resolved
is related to	~~LU-12124~~	slower unlinks with DOM files due to ...	Resolved
is related to	~~LU-10894~~	Compilebench results are lower than e...	Resolved
is related to	~~LU-11276~~	racer: mdc_dev.c:1346:mdc_req_attr_se...	Resolved

Severity:

Rank (Obsolete):

9223372036854775807

Description

In DoM configuration, unlink speed is much slower (~35%) than Non-DOM configuration.
Here is test resutls with DNE2 with/without DoM.
Number of MDT and OST are four each. So, in theory, unlink speed in DoM configuration can be same as non-DOM configuration or even faster.
Here is test resutls.

DNE2(4xMDS/4MDT) without DOM, 4 x OSS(4xOST) and 32 clients

[root@c01 ~]# lfs setdirstripe -c 4 /cache1/out
[root@c01 ~]# lfs setdirstripe -c 4 -D /cache1/out
[root@c01 ~]# salloc --nodes=32 --ntasks-per-node=24 mpirun --allow-run-as-root /work/tools/bin/mdtest -F -n 2000 -u -d /cache1/out -vv -C -r -i 3 -p 10
SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :     155882.494     129033.204     140708.430      11235.756
   File stat         :          0.000          0.000          0.000          0.000
   File read         :          0.000          0.000          0.000          0.000
   File removal      :     240765.928     202754.538     226744.942      17032.855
   Tree creation     :          6.561          1.119          3.005          2.516
   Tree removal      :          5.832          5.215          5.600          0.274

DNE2(4xMDS/4MDT) with DOM, 4 x OSS(4xOST) and 32 clients

[root@c01 ~]# lfs setdirstripe -c 4 /cache1/out
[root@c01 ~]# lfs setdirstripe -c 4 -D /cache1/out
[root@c01 ~]# lfs setstripe -L mdt -E 1M /cache1/out
[root@c01 ~]# salloc --nodes=32 --ntasks-per-node=24 mpirun --allow-run-as-root /work/tools/bin/mdtest -F -n 2000 -u -d /cache1/out -vv -C -r -i 3 -p 10
SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :     142402.513     128265.451     135515.522       5739.604
   File stat         :          0.000          0.000          0.000          0.000
   File read         :          0.000          0.000          0.000          0.000
   File removal      :     149165.342     140330.903     144402.349       3595.289
   Tree creation     :         11.520          2.585          5.571          4.206
   Tree removal      :          5.491          4.100          4.606          0.628

Comments

Comment by Alex Zhuravlev [ 21/May/19 ]

whether DoM files were empty?

Comment by Shuichi Ihara [ 21/May/19 ]

Alex, yes, DoM files were empty. Mike advised me this is related to ~~LU-12124~~ and ~~LU-11359~~. I'm going to test https://review.whamcloud.com/#/c/34071/ if unlink speed improves.

Comment by Shuichi Ihara [ 21/May/19 ]

Changed test enviroment a bit and re-run with/without DoM and patch.

Baseline No-DOM

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :     159607.234     125369.863     141195.699      14082.427
   File stat         :          0.000          0.000          0.000          0.000
   File read         :          0.000          0.000          0.000          0.000
   File removal      :     294886.243     282798.922     288348.055       4935.056
   Tree creation     :         13.224          1.082          8.957          5.575
   Tree removal      :          6.167          5.097          5.693          0.445

DOM

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :     261609.857     251431.584     256138.911       4045.382
   File stat         :          0.000          0.000          0.000          0.000
   File read         :          0.000          0.000          0.000          0.000
   File removal      :     161622.879     157116.547     159685.163       1885.747
   Tree creation     :         11.129          1.798          6.683          3.822
   Tree removal      :          6.530          3.377          5.251          1.354

DoM with patch 34071 (https://review.whamcloud.com/#/c/34071/)

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :     260525.295     254664.889     258414.166       2447.162
   File stat         :          0.000          0.000          0.000          0.000
   File read         :          0.000          0.000          0.000          0.000
   File removal      :     155803.951     153195.917     154248.451       1112.790
   Tree creation     :         12.622          4.922          7.634          3.531
   Tree removal      :          6.490          5.904          6.106          0.271

I didn't see any improvements from patch 34071 for unlink operatios with DoM.

Comment by Patrick Farrell (Inactive) [ 21/May/19 ]

Might also try https://review.whamcloud.com/#/c/34736/ , which isn't directly unlink, but improves lock management

Comment by Gerrit Updater [ 14/Oct/19 ]

Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/36442
Subject: ~~LU-12321~~ mdc: allow ELC for DOM file unlink
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6fce9bde176337036e8fb256ef1a56439ff87f30

Comment by Gerrit Updater [ 14/Feb/20 ]

Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36442/
Subject: ~~LU-12321~~ mdc: allow ELC for DOM file unlink
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e07d4a58206756252c28b51048057c87a7cdb3c6

Comment by Peter Jones [ 14/Feb/20 ]

Landed for 2.14

Generated at Sat Feb 10 02:51:33 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.

[LU-12321] Unlink speed needs to be improved in DoM configuration Created: 21/May/19 Updated: 25/Mar/20 Resolved: 14/Feb/20