|
if possible it'd be interesting to try with:
1) quota accounting disabled (comment two zap_increment_int() in osd_object_create() in osd-zfs)
2) lma setting disabled (comment osd_init_lma() in tat osd_object_create())
|
|
1) quota accounting disabled (comment two zap_increment_int() in osd_object_create() in osd-zfs)
zfs
[root@mds01 mds01]# dir_count=192 thrlo=192 thrhi=192 file_count=3840000 mds-survey
Tue Jan 15 08:46:01 PST 2013 /usr/bin/mds-survey from mds01
mdt 1 file 3840000 dir 192 thr 192 create 3033.82 [ 0.00,20997.59] lookup 8937.11 [ 0.00,38997.93] md_getattr 2386.50 [ 0.00,19996.00] setxattr 2788.85 [ 0.00,16998.22] destroy 1572.63 [ 0.00,15998.16]
done!
ldiskfs
[root@mds01 mds01]# dir_count=192 thrlo=192 thrhi=192 file_count=3840000 mds-survey
Tue Jan 15 08:27:37 PST 2013 /usr/bin/mds-survey from mds01
mdt 1 file 3840000 dir 192 thr 192 create 12324.83 [ 0.00,191982.34] lookup 2082165.28 [2082165.28,2082165.28] md_getattr 849267.98 [807941.83,807941.83] setxattr 13708.98 [ 0.00,191982.53] destroy 15192.13 [ 0.00,191980.23]
done!
2) lma setting disabled (comment osd_init_lma() in tat osd_object_create()) + quota acct disable
zfs
[root@mds01 mds01]# dir_count=192 thrlo=192 thrhi=192 file_count=3840000 mds-survey
Mon Jan 14 22:17:39 PST 2013 /usr/bin/mds-survey from mds01
mdt 1 file 3840000 dir 192 thr 192 create 6278.19 [ 0.00,20997.61] lookup 60380.25 [ 0.00,128983.49] md_getattr 57846.24 [ 0.00,179987.40] setxattr
3235.05 [ 0.00,57991.13] destroy 2234.80 [ 0.00,14998.41]
done!
ldiskfs
[root@mds01 mds01]# dir_count=192 thrlo=192 thrhi=192 file_count=3840000 mds-survey Mon Jan 14 23:26:56 PST 2013 /usr/bin/mds-survey from mds01
mdt 1 file 3840000 dir 192 thr 192 create 13928.46 [ 0.00,191979.27] lookup 2028421.14 [2028421.14,2028421.14] md_getattr 829627.73 [809944.92,809944.92] setxattr 16770.19 [ 0.00,191985.22] destroy 14236.86 [ 0.00,183984.18]
done!
|
|
please, collect:
1) /proc/spl/kstat/zfs/dmu_tx - just before the run and right after
to see how often txgs are overflowed
2) /proc/spl/kstat/zfs/txgs-*-mdt1 - few times during the run
to see amount of reads/writes, lifetime of txgs
also, on my local setup (4GB RAM) I observed with 16+ threads txgs are overflowed all the time
and only 8 threads let txg to go well. but with 8 threads overall performance isn't that greast.
please try with different number of threads, collect (1) and (2)
|
|
agb5 -MDS/MGS, iwc client, dit29 -OSS
|
|
Alex
Could you please pass comment on this?
Thanks
Peter
|
|
Oprofile data from second mdtest run. MDS/MGS = agb5, OSS = agb14, client = iwc44
|
|
We have done further testing with oprofile on Hyperion. We see ZFS performance about 1/2 to 1/3 of ldiskfs performance on the same hardware.
Sample ZFS run of mdtest 64 clients:
MDTEST RESULTS
0000: SUMMARY: (of 2 iterations)
0000: Operation Max Min Mean Std Dev
0000: --------- — — ---- -------
0000: Directory creation: 6763.114 6344.339 6553.727 209.388
0000: Directory stat : 90793.846 81346.095 86069.971 4723.875
0000: Directory removal : 8081.377 7495.103 7788.240 293.137
0000: File creation : 5954.588 4746.843 5350.716 603.872
0000: File stat : 106938.996 106643.378 106791.187 147.809
0000: File removal : 6058.458 5910.333 5984.395 74.062
0000: Tree creation : 4.561 1.916 3.239 1.323
0000: Tree removal : 4.861 4.851 4.856 0.005
ldiskfs:
0000: SUMMARY: (of 5 iterations)
0000: Operation Max Min Mean Std Dev
0000: --------- — — ---- -------
0000: Directory creation: 23037.696 13215.305 17736.740 4389.854
0000: Directory stat : 109887.449 108488.532 109053.155 461.796
0000: Directory removal : 29968.170 18835.706 26369.453 4035.133
0000: File creation : 25026.952 21532.197 23428.767 1151.335
0000: File stat : 109340.749 107623.272 108451.866 724.519
0000: File removal : 31242.455 19217.523 24616.635 4673.904
0000: Tree creation : 25.988 22.719 24.484 1.050
0000: Tree removal : 16.938 13.962 15.872 1.069
----------
oprofile of MDS, one client and one OSS attached.
|
|
Why are print functions so high?
|
|
in my local testing it's EAs which were pretty expensive. also, I'd expect quota accouting contributes to this as well.
|
|
I would say printf is fine, CDEBUG should contribute most of them.
Actually I cannot get too much information from the oprofile output, everything looks reasonable, which means there could be heavy operations protected by mutex/semaphore.
|
|
Cliff I wonder do you have any iostat data or Lustre /proc stats?
|
|
I remember Brian B. said it's doing OK locally. would you mind to try few createmany in parallel with locally mounted ZFS, please? so we have some basic numbers for pure ZFS? Lustre is doing much more than that (OI, few EAs, etc), but still the numbers would give us some idea.
|
|
We may be able to do this in the next test session. Kieth, there are no brw_stats available under ZFS 
|
|
Yea lets test this the next session.
I will help setup some basic iostat so we can get a little better picture of the data rates to the disk themselves.
|
|
The patch from LU-3671 (http://review.whamcloud.com/7257 "mdd: sync perm for dir and perm reduction only") may help this a little bit, but there are still other issues that need to be worked on.
|
|
Actually, my previous comment is incorrect. That patch may help with some real-world workloads like untar, but would not help mds-survey or similar that are not doing chown/chmod.
In LU-2476 Alex posted a link to http://review.whamcloud.com/7157 "a proto for optimized object accounting" which I think is actually more relevant to this bug. It batches the quota accounting updates, which was part of the change in Minh's first test that doubled the ZFS performance. However it wasn't clear if it was the quota zap or the LMA/LIV xattrs that were the main bottleneck, so it would be good to test those separately.
|
|
Some improvements have been made to ZFS performance, but this is still an ongoing issue so move this to 2.5.1 along with LU-2476.
|
|
I did some benchmarking with Lustre-ZFS vs. Lustre-LDISKFS and ZFS vs. EXT4 with mdtest. The results suggests that the slow metadata performance is probably due to ZFS rather than to Lustre. The following setup is used:
1 MGS/MDT server, formatted with ldiskfs(ext4) or ZFS [build: 2.4.0-RC2-gd3f91c4-PRISTINE-2.6.32-358.6.2.el6_lustre.g230b174.x86_64]
1 OSS/OST server, formatted with ZFS [build: v2_4_92_0-ge089a51-CHANGED-3.6.11-lustre-tstibor-build]
1 Client [build: v2_4_92_0-ge089a51-CHANGED-3.6.11-lustre-tstibor-build]
(lustre mountpoint /mnt)
The benchmark is performed on the client
and gives the following results:
Roughly speaking ldiskfs is nearly twice as fast as ZFS on the
artificial metadata tests except on stat calls and read.
Repeating the experiment, however, this time on plain formated ext4 and ZFS filesystems (no Lustre involved).
The underlying hardware is the original MGS/MDT Server results in:
Of course one can question how well such a metadata benchmark are reflecting true working sets, however, by just observing the plain ZFS vs. ext4 one could conclude that the slow metadata performance is NOT due to Lustre.
Thomas.
|
|
http://review.whamcloud.com/#/c/7157/ was reverted to fix LU-4968 if this is resubmitted please include the changs that happened for LU-4944 (http://review.whamcloud.com/#/c/10064/)
|
|
Just a note that if a patch that uses dsl_sync_task is landed again, we'd need to patch zfs so as not to increase async writes when there's only nowaiter sync tasks pending. See:
https://github.com/zfsonlinux/zfs/pull/2716#issuecomment-58540555
|
|
ZAP prefetching at object creation should improve metadata performance.
|
Generated at Sat Feb 10 01:26:35 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
oprofile.tar.gz