[LU-13520] a performance regression on directory creation

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.14.0
• Affects Version/s: Lustre 2.13.0
• Severity: 3
• Assignee: Alex Zhuravlev
• Reporter: Shuichi Ihara

Description

'git bisect' pointed out a performance regression on directory creation and commit "b7cd65a LU-12988 ldiskfs: mballoc to prefetch groups" caused regression.

mpirun -np 640 mdtest -D -C -r -u -n 1000 -vv -p 10 -i 3 -d /scratch/mdt0

master 52b5f4a LU-10401 tests: fix error from 'tr -d='
SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation        :      44270.444      42144.617      43138.707        873.040
   Directory stat            :          0.000          0.000          0.000          0.000
   Directory removal         :      46058.435      44429.196      45452.138        726.962
   Tree creation             :         39.058         30.573         35.586          3.631
   Tree removal              :         32.583         31.615         32.014          0.413
V-1: Entering PrintTimestamp...

b7cd65a LU-12988 ldiskfs: mballoc to prefetch groups
SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation        :      44626.926      43321.707      43810.517        580.641
   Directory stat            :          0.000          0.000          0.000          0.000
   Directory removal         :      46902.932      45311.241      46091.593        649.716
   Tree creation             :         39.966         25.251         34.092          6.363
   Tree removal              :         32.608         25.252         30.093          3.424
V-1: Entering PrintTimestamp...

59a3d72 LU-12043 llite: move tunable params to sysfs_memparse()
SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation        :      76243.849      62728.925      69264.981       5525.345
   Directory stat            :          0.000          0.000          0.000          0.000
   Directory removal         :      47895.918      47533.099      47709.202        147.805
   Tree creation             :         43.638         38.047         41.193          2.336
   Tree removal              :         33.540         27.030         31.193          2.951
V-1: Entering PrintTimestamp...

Comments

[Peter Jones]

Landed for 2.14

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/38513/
Subject: LU-13520 ldiskfs: fastpath in bitmap prefetching
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 6a15798f51a9dcb29333e8107a59c5551f607fff

[Alex Zhuravlev]

thanks! making patches for other kernels..

[Shuichi Ihara]

patch solves problem even with default mb_prefetch_limit parameter.
master (52b5f4a) + 38513

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation        :      83171.845      71796.197      77274.256       4652.121
   Directory stat            :          0.000          0.000          0.000          0.000
   Directory removal         :      55310.503      50699.606      52626.180       1956.720
   Tree creation             :         38.032         26.759         33.917          5.081
   Tree removal              :         32.981         30.756         31.565          1.004
V-1: Entering PrintTimestamp...

[Alex Zhuravlev]

sihara can you please try the patch above? unfortunately I can test it with master branch only.

[Gerrit Updater]

Alex Zhuravlev ([e-mail]) uploaded a new patch: https://review.whamcloud.com/38513
Subject: LU-13520 ldiskfs: fastpath in prefetching
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e4267c5f83166bc31475eac7977a1a8ef1070dff

[Shuichi Ihara]

I think performance with mb_prefetch_limit=0 is fine. here is resutls after remounted MDT.

SUMMARY rate: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   Directory creation        :      75747.972      64178.225      70424.704       4767.309
   Directory stat            :          0.000          0.000          0.000          0.000
   Directory removal         :      50648.239      44288.365      47869.443       2657.031
   Tree creation             :         44.080         29.518         37.821          6.118
   Tree removal              :         32.954         28.309         30.811          1.913
V-1: Entering PrintTimestamp...

also, how empty/full MDT was? probably just created?

MDT is emply and yes, it was just created.

[Alex Zhuravlev]

also, how empty/full MDT was? probably just created?

[Alex Zhuravlev]

   Directory creation        :      76243.849      62728.925      69264.981       5525.345
...
   Directory creation        :      72279.489      62814.460      67811.718       3881.214

even with mb_prefetch_limit=0 it's still behind ? can you please try again with fs remount? thanks a lot

[Shuichi Ihara]

which kernel did you use?

I'm testing RHEL7.7 kernel.