[LU-3966] Submit quota lock improvement patches upstream

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.0, Lustre 2.5.0
• Type: Technical task
• Priority: Major
• Reporter: Andreas Dilger
• Assignee: Niu Yawei (Inactive)
• Resolution: Fixed
• Labels: kernel

Description

The quota-replace-dqptr-sem and quota-avoid-dqget-calls patches are applied to the core kernel to improve the locking of quota structures in the VFS. Since quota is enabled by default for 2.4.0 and later kernels, this has a significant performance impact on all systems (see LU-2442).

These patches should be submitted upstream so that we don't need to patch the kernel to get this improvement. Some performance tests should be run to show some benefit to a non-Lustre workload (e.g. multi-threaded IO with quota enabled, maybe dbench or iozone or fio) and these results should be included with the upstream submission.

Issue Links

• is related to: LU-2442 metadata performance degradation on c...

Comments

[Andreas Dilger]

Niu, any chance you would be able to push these patches upstream?

[Niu Yawei (Inactive)]

Sure, I'm going to start this work soon.

[Niu Yawei (Inactive)]

Attached files:

• patch against mainline kernel
• mdtest test result
• dbench test result

[Niu Yawei (Inactive)]

I compared the performance in my centos6.5 vm (8 cpus, 4G mem).

mdtest creation:

threads              1       2       4       8       16
disabled quota:   40045    78379   128652   89176   103666
enabled quota:    34939    46725   24095    14321   16510
patched/disabled: 39120    75325   124181   72012   86622
patched/enabled:  34769    67086   111854   85923   87982

mdtest unlink:

threads              1       2       4       8       16
disabled quota:   91587    148808  227496   193661  190477
enabled quota:    72426    48726   14845    12825   15907
patched/disabled: 85246    146369  228514   194053  192407
patched/enabled:  78257    124332  166146   180874  174715

dbench 8 threads:

====== disabled quota: ======

 Operation                Count    AvgLat    MaxLat
 --------------------------------------------------
 Deltree                     32     3.585     8.437
 Flush                    82625     3.797   207.561
 Close                   865785     0.004     1.353
 LockX                     3840     0.005     0.182
 Mkdir                       16     0.005     0.007
 Rename                   49897     0.050     6.085
 ReadX                  1847719     0.006     6.332
 WriteX                  588019     0.033     6.968
 Unlink                  238061     0.050     6.537
 UnlockX                   3840     0.004     0.302
 FIND_FIRST              413054     0.024     2.920
 SET_FILE_INFORMATION     95961     0.035     6.998
 QUERY_FILE_INFORMATION  187253     0.003     0.478
 QUERY_PATH_INFORMATION 1068225     0.010     6.211
 QUERY_FS_INFORMATION    195932     0.006     0.541
 NTCreateX              1178614     0.021    64.684

Throughput 616.998 MB/sec  8 clients  8 procs  max_latency=207.575 ms

====== enabled quota: ======

 Operation                Count    AvgLat    MaxLat
 --------------------------------------------------
 Deltree                     16    11.240    54.888
 Flush                    61421     3.627   127.876
 Close                   643369     0.004     0.924
 LockX                     2848     0.005     0.253
 Mkdir                        8     0.005     0.008
 Rename                   37088     0.116     3.845
 ReadX                  1372315     0.007     5.024
 WriteX                  435537     0.106    18.304
 Unlink                  176928     0.351    29.266
 UnlockX                   2848     0.004     0.095
 FIND_FIRST              306847     0.024     1.689
 SET_FILE_INFORMATION     71406     0.040     8.933
 QUERY_FILE_INFORMATION  138904     0.003     0.421
 QUERY_PATH_INFORMATION  794000     0.011     4.027
 QUERY_FS_INFORMATION    145520     0.006     0.473
 NTCreateX               875964     0.072    52.923

Throughput 457.433 MB/sec  8 clients  8 procs  max_latency=127.902 ms

====== patched/disabled: ======

 Operation                Count    AvgLat    MaxLat
 --------------------------------------------------
 Deltree                     32     3.332     8.210
 Flush                    82543     3.790   146.987
 Close                   865200     0.004     1.289
 LockX                     3836     0.005     0.142
 Mkdir                       16     0.008     0.038
 Rename                   49870     0.052     4.907
 ReadX                  1846334     0.006     6.107
 WriteX                  587645     0.033     8.086
 Unlink                  237737     0.052     6.440
 UnlockX                   3836     0.004     0.105
 FIND_FIRST              412704     0.024     1.597
 SET_FILE_INFORMATION     95948     0.034     7.854
 QUERY_FILE_INFORMATION  187179     0.003     0.408
 QUERY_PATH_INFORMATION 1067460     0.010     5.316
 QUERY_FS_INFORMATION    195706     0.006     0.613
 NTCreateX              1177689     0.021     6.521

Throughput 616.574 MB/sec  8 clients  8 procs  max_latency=147.007 ms

====== patched/enabled: ======

 Operation                Count    AvgLat    MaxLat
 --------------------------------------------------
 Deltree                     32     3.248     8.430
 Flush                    80481     3.908   241.537
 Close                   843781     0.004     0.561
 LockX                     3746     0.005     0.141
 Mkdir                       16     0.005     0.007
 Rename                   48642     0.051     6.466
 ReadX                  1800754     0.006    87.027
 WriteX                  573185     0.033     6.750
 Unlink                  231880     0.058    14.507
 UnlockX                   3746     0.004     0.103
 FIND_FIRST              402463     0.024     1.342
 SET_FILE_INFORMATION     93557     0.035    42.348
 QUERY_FILE_INFORMATION  182573     0.003     1.305
 QUERY_PATH_INFORMATION 1041026     0.010    86.289
 QUERY_FS_INFORMATION    190869     0.006     1.240
 NTCreateX              1148570     0.022     6.285

Throughput 602.147 MB/sec  8 clients  8 procs  max_latency=241.561 ms

We can see the patch helped a lot on performance, will send the patch to upstream soon.

[James A Simmons]

Peter can you link this ticket to LU-20. Thank you.

[Andreas Dilger]

The patches were pushed upstream http://www.spinics.net/lists/linux-ext4/msg43588.html and were accepted into the quota staging tree:

On Wed 04-06-14 12:19:12, Niu Yawei wrote:
> Subject: [PATCH 1/5] quota: protect Q_GETFMT by dqonoff_mutex
>
> dqptr_sem will go away. Protect Q_GETFMT quotactl by
> dqonoff_mutex instead. This is also enough to make sure
> quota info will not go away while we are looking at it.
>
> Signed-off-by: Lai Siyao <lai.siyao@xxxxxxxxx>
> Signed-off-by: Niu Yawei <yawei.niu@xxxxxxxxx>
Thanks! I've merged all the five patches to my tree. I will give them
some testing. I won't push them in this merge window but it should go in in
the next one.
Honza

[James A Simmons]

The following patches from Niu Yawei have landed upstream (linus tree):

commit d68aab6b8f572406aa93b45ef6483934dd3b54a6
quota: missing lock in dqcache_shrink_scan()

commit 606cdcca04a609ed4dfbfe788942de9477da556b
quota: protect Q_GETFMT by dqonoff_mutex

commit 1ea06bec78a128adc995ca32bd906a6c9bb9cf91
quota: avoid unnecessary dqget()/dqput() calls

commit 9eb6463f31cf720deaf0e810cacc403d7720b10c
quota: simplify remove_inode_dquot_ref()

commit b9ba6f94b2382ef832f9ck7122976b73004f714714
quota: remove dqptr_sem

I believe all the needed patches are now landed upstream. We can close this ticket.

[Niu Yawei (Inactive)]

Thanks, James.