Details
-
Technical task
-
Resolution: Fixed
-
Major
-
None
-
Lustre 2.4.0, Lustre 2.5.0
-
10570
Description
The quota-replace-dqptr-sem and quota-avoid-dqget-calls patches are applied to the core kernel to improve the locking of quota structures in the VFS. Since quota is enabled by default for 2.4.0 and later kernels, this has a significant performance impact on all systems (see LU-2442).
These patches should be submitted upstream so that we don't need to patch the kernel to get this improvement. Some performance tests should be run to show some benefit to a non-Lustre workload (e.g. multi-threaded IO with quota enabled, maybe dbench or iozone or fio) and these results should be included with the upstream submission.
Attachments
Issue Links
- is related to
-
LU-2442 metadata performance degradation on current master
-
- Resolved
-
Activity
The following patches from Niu Yawei have landed upstream (linus tree):
commit d68aab6b8f572406aa93b45ef6483934dd3b54a6
quota: missing lock in dqcache_shrink_scan()
commit 606cdcca04a609ed4dfbfe788942de9477da556b
quota: protect Q_GETFMT by dqonoff_mutex
commit 1ea06bec78a128adc995ca32bd906a6c9bb9cf91
quota: avoid unnecessary dqget()/dqput() calls
commit 9eb6463f31cf720deaf0e810cacc403d7720b10c
quota: simplify remove_inode_dquot_ref()
commit b9ba6f94b2382ef832f9ck7122976b73004f714714
quota: remove dqptr_sem
I believe all the needed patches are now landed upstream. We can close this ticket.
The patches were pushed upstream http://www.spinics.net/lists/linux-ext4/msg43588.html and were accepted into the quota staging tree:
On Wed 04-06-14 12:19:12, Niu Yawei wrote:
> Subject: [PATCH 1/5] quota: protect Q_GETFMT by dqonoff_mutex
>
> dqptr_sem will go away. Protect Q_GETFMT quotactl by
> dqonoff_mutex instead. This is also enough to make sure
> quota info will not go away while we are looking at it.
>
> Signed-off-by: Lai Siyao <lai.siyao@xxxxxxxxx>
> Signed-off-by: Niu Yawei <yawei.niu@xxxxxxxxx>
Thanks! I've merged all the five patches to my tree. I will give them
some testing. I won't push them in this merge window but it should go in in
the next one.
Honza
I compared the performance in my centos6.5 vm (8 cpus, 4G mem).
mdtest creation:
threads 1 2 4 8 16 disabled quota: 40045 78379 128652 89176 103666 enabled quota: 34939 46725 24095 14321 16510 patched/disabled: 39120 75325 124181 72012 86622 patched/enabled: 34769 67086 111854 85923 87982
mdtest unlink:
threads 1 2 4 8 16 disabled quota: 91587 148808 227496 193661 190477 enabled quota: 72426 48726 14845 12825 15907 patched/disabled: 85246 146369 228514 194053 192407 patched/enabled: 78257 124332 166146 180874 174715
dbench 8 threads:
====== disabled quota: ====== Operation Count AvgLat MaxLat -------------------------------------------------- Deltree 32 3.585 8.437 Flush 82625 3.797 207.561 Close 865785 0.004 1.353 LockX 3840 0.005 0.182 Mkdir 16 0.005 0.007 Rename 49897 0.050 6.085 ReadX 1847719 0.006 6.332 WriteX 588019 0.033 6.968 Unlink 238061 0.050 6.537 UnlockX 3840 0.004 0.302 FIND_FIRST 413054 0.024 2.920 SET_FILE_INFORMATION 95961 0.035 6.998 QUERY_FILE_INFORMATION 187253 0.003 0.478 QUERY_PATH_INFORMATION 1068225 0.010 6.211 QUERY_FS_INFORMATION 195932 0.006 0.541 NTCreateX 1178614 0.021 64.684 Throughput 616.998 MB/sec 8 clients 8 procs max_latency=207.575 ms ====== enabled quota: ====== Operation Count AvgLat MaxLat -------------------------------------------------- Deltree 16 11.240 54.888 Flush 61421 3.627 127.876 Close 643369 0.004 0.924 LockX 2848 0.005 0.253 Mkdir 8 0.005 0.008 Rename 37088 0.116 3.845 ReadX 1372315 0.007 5.024 WriteX 435537 0.106 18.304 Unlink 176928 0.351 29.266 UnlockX 2848 0.004 0.095 FIND_FIRST 306847 0.024 1.689 SET_FILE_INFORMATION 71406 0.040 8.933 QUERY_FILE_INFORMATION 138904 0.003 0.421 QUERY_PATH_INFORMATION 794000 0.011 4.027 QUERY_FS_INFORMATION 145520 0.006 0.473 NTCreateX 875964 0.072 52.923 Throughput 457.433 MB/sec 8 clients 8 procs max_latency=127.902 ms ====== patched/disabled: ====== Operation Count AvgLat MaxLat -------------------------------------------------- Deltree 32 3.332 8.210 Flush 82543 3.790 146.987 Close 865200 0.004 1.289 LockX 3836 0.005 0.142 Mkdir 16 0.008 0.038 Rename 49870 0.052 4.907 ReadX 1846334 0.006 6.107 WriteX 587645 0.033 8.086 Unlink 237737 0.052 6.440 UnlockX 3836 0.004 0.105 FIND_FIRST 412704 0.024 1.597 SET_FILE_INFORMATION 95948 0.034 7.854 QUERY_FILE_INFORMATION 187179 0.003 0.408 QUERY_PATH_INFORMATION 1067460 0.010 5.316 QUERY_FS_INFORMATION 195706 0.006 0.613 NTCreateX 1177689 0.021 6.521 Throughput 616.574 MB/sec 8 clients 8 procs max_latency=147.007 ms ====== patched/enabled: ====== Operation Count AvgLat MaxLat -------------------------------------------------- Deltree 32 3.248 8.430 Flush 80481 3.908 241.537 Close 843781 0.004 0.561 LockX 3746 0.005 0.141 Mkdir 16 0.005 0.007 Rename 48642 0.051 6.466 ReadX 1800754 0.006 87.027 WriteX 573185 0.033 6.750 Unlink 231880 0.058 14.507 UnlockX 3746 0.004 0.103 FIND_FIRST 402463 0.024 1.342 SET_FILE_INFORMATION 93557 0.035 42.348 QUERY_FILE_INFORMATION 182573 0.003 1.305 QUERY_PATH_INFORMATION 1041026 0.010 86.289 QUERY_FS_INFORMATION 190869 0.006 1.240 NTCreateX 1148570 0.022 6.285 Throughput 602.147 MB/sec 8 clients 8 procs max_latency=241.561 ms
We can see the patch helped a lot on performance, will send the patch to upstream soon.
Attached files:
- patch against mainline kernel
- mdtest test result
- dbench test result
Thanks, James.