Details
-
Bug
-
Resolution: Fixed
-
Minor
-
Lustre 2.13.0
-
None
-
3
-
9223372036854775807
Description
'git bisect' pointed out a performance regression on directory creation and commit "b7cd65a LU-12988 ldiskfs: mballoc to prefetch groups" caused regression.
mpirun -np 640 mdtest -D -C -r -u -n 1000 -vv -p 10 -i 3 -d /scratch/mdt0
master 52b5f4a LU-10401 tests: fix error from 'tr -d=' SUMMARY rate: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation : 44270.444 42144.617 43138.707 873.040 Directory stat : 0.000 0.000 0.000 0.000 Directory removal : 46058.435 44429.196 45452.138 726.962 Tree creation : 39.058 30.573 35.586 3.631 Tree removal : 32.583 31.615 32.014 0.413 V-1: Entering PrintTimestamp...
b7cd65a LU-12988 ldiskfs: mballoc to prefetch groups SUMMARY rate: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation : 44626.926 43321.707 43810.517 580.641 Directory stat : 0.000 0.000 0.000 0.000 Directory removal : 46902.932 45311.241 46091.593 649.716 Tree creation : 39.966 25.251 34.092 6.363 Tree removal : 32.608 25.252 30.093 3.424 V-1: Entering PrintTimestamp...
59a3d72 LU-12043 llite: move tunable params to sysfs_memparse() SUMMARY rate: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation : 76243.849 62728.925 69264.981 5525.345 Directory stat : 0.000 0.000 0.000 0.000 Directory removal : 47895.918 47533.099 47709.202 147.805 Tree creation : 43.638 38.047 41.193 2.336 Tree removal : 33.540 27.030 31.193 2.951 V-1: Entering PrintTimestamp...
Attachments
Activity
Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/38513
Subject: LU-13520 ldiskfs: fastpath in prefetching
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e4267c5f83166bc31475eac7977a1a8ef1070dff
I think performance with mb_prefetch_limit=0 is fine. here is resutls after remounted MDT.
SUMMARY rate: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation : 75747.972 64178.225 70424.704 4767.309 Directory stat : 0.000 0.000 0.000 0.000 Directory removal : 50648.239 44288.365 47869.443 2657.031 Tree creation : 44.080 29.518 37.821 6.118 Tree removal : 32.954 28.309 30.811 1.913 V-1: Entering PrintTimestamp...
also, how empty/full MDT was? probably just created?
MDT is emply and yes, it was just created.
Directory creation : 76243.849 62728.925 69264.981 5525.345 ... Directory creation : 72279.489 62814.460 67811.718 3881.214
even with mb_prefetch_limit=0 it's still behind ? can you please try again with fs remount? thanks a lot
looks better after "echo 0 >/sys/fs/ldiskfs/<mdt-device>mb_prefetch_limit"
master 52b5f4a LU-10401 tests: fix error from 'tr -d=' with "echo 0 >/sys/fs/ldiskfs/<mdt-device>mb_prefetch_limit" SUMMARY rate: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation : 72279.489 62814.460 67811.718 3881.214 Directory stat : 0.000 0.000 0.000 0.000 Directory removal : 47547.120 47419.048 47503.194 58.836 Tree creation : 53.445 40.462 47.929 5.477 Tree removal : 32.093 27.697 30.574 2.036 V-1: Entering PrintTimestamp...
which kernel did you use?
(I'll try to cook a quick patch for that kernel)
attached are two framegraphes on commit 59a3d72(master) 52b5f4a(an commit "b7cd65a LU-12988 ldiskfs: mballoc to prefetch groups") and it pointed out the latest master spent a lot of CPU time on ldiskfs_read_block_bitmap_nowait() with spinlock which doesn't see same CPU time on commit 52b5f4a.
can you please try with disabled prefetching, on MDS:
echo 0 >/sys/fs/ldiskfs/<mdt-device>mb_prefetch_limit
sihara can you please try the patch above? unfortunately I can test it with master branch only.