[LU-13520] a performance regression on directory creation Created: 06/May/20  Updated: 06/Jun/20  Resolved: 06/Jun/20

Status: Resolved
Project: Lustre
Component/s: None
Affects Version/s: Lustre 2.13.0
Fix Version/s: Lustre 2.14.0

Type: Bug Priority: Minor
Reporter: Shuichi Ihara Assignee: Alex Zhuravlev
Resolution: Fixed Votes: 0
Labels: None

Attachments: File mdt-52b5f4a.svg     File mdt-59a3d72.svg    
Issue Links:
Related
Severity: 3
Rank (Obsolete): 9223372036854775807

 Description   

'git bisect' pointed out a performance regression on directory creation and commit "b7cd65a LU-12988 ldiskfs: mballoc to prefetch groups" caused regression.

mpirun -np 640 mdtest -D -C -r -u -n 1000 -vv -p 10 -i 3 -d /scratch/mdt0

master 52b5f4a LU-10401 tests: fix error from 'tr -d='
SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation        :      44270.444      42144.617      43138.707        873.040
   Directory stat            :          0.000          0.000          0.000          0.000
   Directory removal         :      46058.435      44429.196      45452.138        726.962
   Tree creation             :         39.058         30.573         35.586          3.631
   Tree removal              :         32.583         31.615         32.014          0.413
V-1: Entering PrintTimestamp...

b7cd65a LU-12988 ldiskfs: mballoc to prefetch groups
SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation        :      44626.926      43321.707      43810.517        580.641
   Directory stat            :          0.000          0.000          0.000          0.000
   Directory removal         :      46902.932      45311.241      46091.593        649.716
   Tree creation             :         39.966         25.251         34.092          6.363
   Tree removal              :         32.608         25.252         30.093          3.424
V-1: Entering PrintTimestamp...

59a3d72 LU-12043 llite: move tunable params to sysfs_memparse()
SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation        :      76243.849      62728.925      69264.981       5525.345
   Directory stat            :          0.000          0.000          0.000          0.000
   Directory removal         :      47895.918      47533.099      47709.202        147.805
   Tree creation             :         43.638         38.047         41.193          2.336
   Tree removal              :         33.540         27.030         31.193          2.951
V-1: Entering PrintTimestamp...


 Comments   
Comment by Alex Zhuravlev [ 06/May/20 ]

can you please try with disabled prefetching, on MDS:

echo 0 >/sys/fs/ldiskfs/<mdt-device>mb_prefetch_limit
Comment by Shuichi Ihara [ 06/May/20 ]

attached are two framegraphes on commit 59a3d72(master) 52b5f4a(an commit "b7cd65a LU-12988 ldiskfs: mballoc to prefetch groups") and it pointed out the latest master spent a lot of CPU time on ldiskfs_read_block_bitmap_nowait() with spinlock which doesn't see same CPU time on commit 52b5f4a.

Comment by Alex Zhuravlev [ 06/May/20 ]

which kernel did you use?
(I'll try to cook a quick patch for that kernel)

Comment by Shuichi Ihara [ 06/May/20 ]

looks better after "echo 0 >/sys/fs/ldiskfs/<mdt-device>mb_prefetch_limit"

master 52b5f4a LU-10401 tests: fix error from 'tr -d=' with "echo 0 >/sys/fs/ldiskfs/<mdt-device>mb_prefetch_limit"

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation        :      72279.489      62814.460      67811.718       3881.214
   Directory stat            :          0.000          0.000          0.000          0.000
   Directory removal         :      47547.120      47419.048      47503.194         58.836
   Tree creation             :         53.445         40.462         47.929          5.477
   Tree removal              :         32.093         27.697         30.574          2.036
V-1: Entering PrintTimestamp...
Comment by Shuichi Ihara [ 06/May/20 ]

which kernel did you use?

I'm testing RHEL7.7 kernel.

Comment by Alex Zhuravlev [ 06/May/20 ] 
   Directory creation        :      76243.849      62728.925      69264.981       5525.345
...
   Directory creation        :      72279.489      62814.460      67811.718       3881.214

even with mb_prefetch_limit=0 it's still behind ? can you please try again with fs remount? thanks a lot

Comment by Alex Zhuravlev [ 06/May/20 ]

also, how empty/full MDT was? probably just created?

Comment by Shuichi Ihara [ 06/May/20 ]

I think performance with mb_prefetch_limit=0 is fine. here is resutls after remounted MDT.

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation        :      75747.972      64178.225      70424.704       4767.309
   Directory stat            :          0.000          0.000          0.000          0.000
   Directory removal         :      50648.239      44288.365      47869.443       2657.031
   Tree creation             :         44.080         29.518         37.821          6.118
   Tree removal              :         32.954         28.309         30.811          1.913
V-1: Entering PrintTimestamp...

also, how empty/full MDT was? probably just created?

MDT is emply and yes, it was just created.

Comment by Gerrit Updater [ 06/May/20 ]

Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/38513
Subject: LU-13520 ldiskfs: fastpath in prefetching
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e4267c5f83166bc31475eac7977a1a8ef1070dff

Comment by Alex Zhuravlev [ 06/May/20 ]

sihara can you please try the patch above? unfortunately I can test it with master branch only.

Comment by Shuichi Ihara [ 07/May/20 ]

patch solves problem even with default mb_prefetch_limit parameter.
master (52b5f4a) + 38513

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation        :      83171.845      71796.197      77274.256       4652.121
   Directory stat            :          0.000          0.000          0.000          0.000
   Directory removal         :      55310.503      50699.606      52626.180       1956.720
   Tree creation             :         38.032         26.759         33.917          5.081
   Tree removal              :         32.981         30.756         31.565          1.004
V-1: Entering PrintTimestamp...
Comment by Alex Zhuravlev [ 07/May/20 ]

thanks! making patches for other kernels..

Comment by Gerrit Updater [ 06/Jun/20 ]

Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/38513/
Subject: LU-13520 ldiskfs: fastpath in bitmap prefetching
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 6a15798f51a9dcb29333e8107a59c5551f607fff

Comment by Peter Jones [ 06/Jun/20 ]

Landed for 2.14

Generated at Sat Feb 10 03:01:59 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.