[LU-3305] Quotas affect Metadata performance

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.4.1, Lustre 2.5.0
• Affects Version/s: Lustre 2.4.0
• Environment: Hyperion/LLNL
• Severity: 3
• Assignee: Niu Yawei (Inactive)
• Reporter: Cliff White (Inactive)

Description

We performed a comparison between 2.3.0, 2.1.5 and current Lustre. We say a regression in metadata performance compared to 2.3.0. Spreadsheet attached.

Issue Links

• is blocked by: LU-3396 Disable quota by default for 2.4

Comments

[Andreas Dilger]

This is a previously known problem discussed in LU-2442. The "on-by-default" quota accounting introduced serialization in the quota layer that broke the SMP scaling optimizations done in the 2.3 code. This wasn't fixed until v2_3_63_0-35-g6df197d (http://review.whamcloud.com/5010), so this will hide any regressions in the metadata performance when testing on a faster system, unless the quota feature is disabled on the MDS (tune2fs -O ^quota /dev/mdt).

It may well be that there is still a performance impact from the "on-by-default" quota accounting, which is worthwhile to test before trying to find some other cause for this regression.

[Peter Jones]

Niu

Are you able to advise on the reported drop due to the quotas landing?

Alex/Nasf

Any comments?

Thanks

Peter

[Shuichi Ihara (Inactive)]

Unique direcotry

version	commit	Dir creation	Dir stat	Dir removal	File creation	File stat	File removal
v2_3_50_0-143	9ddf386035767a96b54e21559f3ea0be126dc8cd	22167	194845	32910	50090	132461	36762
v2_3_50_0-142	c61d09e9944ae47f68eb159224af7c5456cc180a	21835	203123	46178	45941	147281	67253
v2_3_50_0-141	48bad5d9db9baa7bca093de5c54294adf1cf8303	20542	202014	48271	46264	142450	66267
2.3.50	04ec54ff56b83a9114f7a25fbd4aa5f65e68ef7a	24727	226128	34064	27859	124521	34591
2.3	ee695bf0762f5dbcb2ac6d96354f8d01ad764903	23565	222482	44421	48333	125765	76709

It seems that there are several regression points between 2.3 and 2.4, but at least on my small client testing(tested mdtest on 16 clients, 32 process), commit 9ddf386035767a96b54e21559f3ea0be126dc8cd might be one of regression point for unlink operation. (I collected more data at commit points, but this is one of point big differnce between commits.
I'm still working to verify wether this is exactly point. will run on large number of client environment.

[Minh Diep]

comparison between 2.3 and 2.4.50 RC1

[Shuichi Ihara (Inactive)]

I also hit same regression on the latest master commit and started to find where retrogression started. "git bisect" would help. I will post soon which commit causes this regressions.

[Liang Zhen (Inactive)]

As we can see from these graphs, the major difference is about directory per process creation performance, but stddev of all creation tests are very high, for example, 2.3 directory per process creation, stddev value is above 50% of average value. One possible reason could be journal size, I think our default journal size is still 1GB? I would suggest to have 2G internal journal at least, and repeat 10 or more times.
Another thing I cannot explain is, why shared directory file stat performance is so much worse than dir per process?

[Liang Zhen (Inactive)]

2.3/2.4 mdtest performance data

[Minh Diep]

Opensfs cluster run

2.3.65
mdtest fpp: https://maloo.whamcloud.com/test_sets/697bfbb2-bca3-11e2-b6d1-52540035b04c
mdtest ssf: https://maloo.whamcloud.com/test_sets/6b79a220-bca3-11e2-8441-52540035b04c

2.3.0
mdtest fpp: https://maloo.whamcloud.com/test_sets/0206a48c-bcb7-11e2-8441-52540035b04c
mdtest ssf: https://maloo.whamcloud.com/test_sets/054d925e-bcb7-11e2-8441-52540035b04c

[Cliff White (Inactive)]

Keith was referring to LU-3281, which was a bug involving IOR.
We also run 5 iterations for mdtest.

[Robert Read]

Do you mean mdtest instead of IOR?

[Cliff White (Inactive)]

Kieth, the runs are an average of 5 iterations for IOR. The storage was fine for all three runs. We've stopped using the problematic nodes for these tests. There is no oprofile data from the MDS.