Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Lustre 2.5.0
-
2
-
11951
Description
lustre-2.5.52 client (and maybe more old client as well) causes metadata performance (unlink files in the single shared directory) regression.
Here is test results on lustre-2.5.52 clients and lustre-2.4.1 clients. lustre-2.5.52 is running on all servers.
1 x MDS, 4 x OSS (32 x OST) and 16 clients(64 processs, 20000 files per process)
lustre-2.4.1 client 4.1-take2.log -- started at 12/09/2013 07:31:29 -- mdtest-1.9.1 was launched with 64 total task(s) on 16 node(s) Command line used: /work/tools/bin/mdtest -d /lustre/dir.0 -n 20000 -F -i 3 Path: /lustre FS: 1141.8 TiB Used FS: 0.0% Inodes: 50.0 Mi Used Inodes: 0.0% 64 tasks, 1280000 files SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 58200.265 56783.559 57589.448 594.589 File stat : 123351.857 109571.584 114223.612 6455.043 File read : 109917.788 83891.903 99965.718 11472.968 File removal : 60825.889 59066.121 59782.774 754.599 Tree creation : 4048.556 1971.934 3082.293 853.878 Tree removal : 21.269 15.069 18.204 2.532 -- finished at 12/09/2013 07:34:53 --
lustre-2.5.5.2 client -- started at 12/09/2013 07:13:42 -- mdtest-1.9.1 was launched with 64 total task(s) on 16 node(s) Command line used: /work/tools/bin/mdtest -d /lustre/dir.0 -n 20000 -F -i 3 Path: /lustre FS: 1141.8 TiB Used FS: 0.0% Inodes: 50.0 Mi Used Inodes: 0.0% 64 tasks, 1280000 files SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 58286.631 56689.423 57298.286 705.112 File stat : 127671.818 116429.261 121610.854 4631.684 File read : 173527.817 158205.242 166676.568 6359.445 File removal : 46818.194 45638.851 46118.111 506.151 Tree creation : 3844.458 2576.354 3393.050 578.560 Tree removal : 21.383 18.329 19.844 1.247 -- finished at 12/09/2013 07:17:07 --
46K ops/sec (lusre-2.5.52) vs 60K ops/sec (lustre-2.4.1). 25% performance drops on Lustre-2.5.52 compared to Lustre-2.4.1.
Attachments
Issue Links
Activity
It appears that the unlink performance has gone up, but the create and stat rate have gone down. Can you please test those two patches separately? If the 10398 patch is fixing the unlink performance without hurting the other performance it could land. It might be that 10426 patch is changing the other performance and needs to be reworked.
Andreas, sorry dely on this... Here is recent our test resutls.
Configuraiton 1 x MDS, 10 x SSD(RAID10) for MDT, 2 x OSS, 10 x OST(100 x NL-SAS) 32 clients, 64 mdtest threads and total 2.56M files creation/stats/removal master branch(47cde804ddc9019ff0793229030211d536d0612f) master branch(47cde804ddc9019ff0793229030211d536d0612f) + patch 10426 + patch 10398
Unique Directory Operation
master branch(47cde804ddc9019ff0793229030211d536d0612f)
mdtest-1.9.3 was launched with 64 total task(s) on 32 node(s) Command line used: ./mdtest -i 3 -n 40000 -u -d /lustre_test/mdtest.out Path: /lustre_test FS: 39.0 TiB Used FS: 0.0% Inodes: 50.0 Mi Used Inodes: 0.0% 64 tasks, 2560000 files/directories SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 48811.145 39252.347 42446.699 4500.354 Directory stat : 299207.829 290254.504 293619.032 3979.199 Directory removal : 89250.695 86672.466 88049.098 1059.809 File creation : 80325.602 71720.354 76539.450 3588.203 File stat : 202533.695 202312.144 202430.663 91.108 File read : 224391.556 222667.559 223733.260 760.494 File removal : 93977.310 81732.593 89128.915 5313.644 Tree creation : 487.540 255.237 408.701 108.529 Tree removal : 7.483 7.376 7.416 0.048
Unique Directory Operation
master branch(47cde804ddc9019ff0793229030211d536d0612f) + patch 10426 + patch 10398
mdtest-1.9.3 was launched with 64 total task(s) on 32 node(s) Command line used: ./mdtest -i 3 -n 40000 -u -d /lustre_test/mdtest.out Path: /lustre_test FS: 39.0 TiB Used FS: 0.0% Inodes: 50.0 Mi Used Inodes: 0.0% 64 tasks, 2560000 files/directories SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 43529.024 38432.682 40492.505 2192.203 Directory stat : 295567.203 248965.236 278082.284 20727.046 Directory removal : 99851.600 97510.819 98692.187 955.746 File creation : 76464.252 61260.049 69836.770 6358.281 File stat : 210322.996 203751.172 206953.520 2685.537 File read : 227658.211 225535.341 226317.238 952.564 File removal : 99144.730 98371.321 98765.310 315.911 Tree creation : 454.766 187.656 357.198 120.339 Tree removal : 7.494 7.383 7.438 0.045
Shared Directory Operation
master branch(47cde804ddc9019ff0793229030211d536d0612f)
mdtest-1.9.3 was launched with 64 total task(s) on 32 node(s) Command line used: ./mdtest -i 3 -n 40000 -d /lustre_test/mdtest.out Path: /lustre_test FS: 39.0 TiB Used FS: 0.0% Inodes: 50.0 Mi Used Inodes: 0.0% 64 tasks, 2560000 files/directories SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 28513.564 27700.587 28038.288 345.860 Directory stat : 142617.694 139431.318 141316.628 1364.858 Directory removal : 60164.271 56562.712 58927.059 1672.450 File creation : 34568.359 34000.466 34304.269 233.536 File stat : 143387.629 140366.792 141459.265 1367.577 File read : 229820.877 222497.139 225426.481 3164.288 File removal : 66583.172 58133.175 61494.514 3659.539 Tree creation : 4132.319 3398.950 3773.387 299.598 Tree removal : 11.422 3.327 7.825 3.365
Shared Directory Operation
master branch(47cde804ddc9019ff0793229030211d536d0612f) + patch 10426 + patch 10398
mdtest-1.9.3 was launched with 64 total task(s) on 32 node(s) Command line used: ./mdtest -i 3 -n 40000 -d /lustre_test/mdtest.out Path: /lustre_test FS: 39.0 TiB Used FS: 0.0% Inodes: 50.0 Mi Used Inodes: 0.0% 64 tasks, 2560000 files/directories SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 28132.040 26630.642 27487.773 631.154 Directory stat : 136965.055 135597.500 136440.164 601.823 Directory removal : 58149.733 55110.750 56638.405 1240.713 File creation : 33170.783 32710.907 32931.837 188.175 File stat : 138870.777 136286.854 137743.643 1080.330 File read : 234861.197 224503.115 228594.555 4499.710 File removal : 77518.626 69571.564 73940.211 3292.142 Tree creation : 4116.098 1102.314 2711.725 1238.885 Tree removal : 9.879 4.938 7.854 2.114
We see performance improvements with patch for unlink operations to unique directories as well as shared directory. I will also want to check with lustre-2.5 to comapre. btw, file/directory creation to shared directory is lower than I expected.. I will check later on other lustre version (e.g. b2_5) as well.
Hi Ihara, is there a chance for you to post the mdtest results for the testing you did on 06-09 for patch http://review.whamcloud.com/10398 ?
Ihara, did you get a chance to test if 10398 fixes the unlink regression? We are ready to land that patch.
Lai should confirm, but I think the most important patch for addressing the unlink regression is http://review.whamcloud.com/10398 so that one should be tested first.
There is also a potential improvement in http://review.whamcloud.com/9696 that is next, but it doesn't affect unlink. I think the http://review.whamcloud.com/9697 is too complex to land for 2.6.0 at this point, but if it gives a significant improvement then it could be landed for 2.7.0 and IEEL.
Hi Lai,
it seems that there are several updates after you posted initial patches. please adivse me which patches should be applied?
Patch to combine close in unlink RPC: http://review.whamcloud.com/#/c/10398/
Ihara, could you apply this only and get results from mdtest?
I've thought of that, but considering the complication of open replay, and possibly SOM, I think it's not a trivial work. I'll think about it more and do some test later (maybe next week).
First, I tried only 10398 patch, but build fails since OBD_CONNECT_UNLINK_CLOSE is defined in 10426 patch. So, I needed both patches at same time to compile.
BTW, here is same mdtest benchmark on same hardware, but lustre version is 2.5.2RC2.
Unique Directory Operation
Shared Directory Operation
Unique directory metadata operation, overall, the result of master + 10398 + 10426 patches are close to 2.5.2RC2 results except directory stats. (stats operation, 2.5 is better than master)
However, metadata operations to a shared directory, most of 2.5.2RC2's numbers are still much higher than master or master + 10398 + 10426 patch's resutls. That's oritinal issue on this ticket, but still big performance gap there. Read operation, master branch much improved against 2.5 branch.