Details
-
Improvement
-
Resolution: Not a Bug
-
Minor
-
None
-
Lustre 2.3.0
-
6208
Description
(I'm not sure about the issue type for this ticket, please adjust as appropriate.)
As discussed with Peter Jones, we are trying to implement a file system where single clients can achieve >900MB/s write throughput over 10GigE connections. Ideally single 10GigE for the clients but 2x10GigE LACP bonding might be an option. The OSSes will initially have 4x 10GigE LACP bonded links, though for some initial testing we might start with fewer links.
The disk backend has now arrived and this is a sample obdfilter-survey result using all one OST and 4 OSSes, without much tuning on the OSS nodes yet. The OSSes are all running Lustre 2.3.0 on RHEL6.
Sat Jan 19 15:49:23 GMT 2013 Obdfilter-survey for case=disk from cs04r-sc-oss05-03.diamond.ac.uk ost 41 sz 687865856K rsz 1024K obj 41 thr 41 write 2975.14 [ 40.00, 105.99] rewrite 2944.84 [ 22.00, 118.99] read 8104.33 [ 40.99, 231.98] ost 41 sz 687865856K rsz 1024K obj 41 thr 82 write 5231.39 [ 49.99, 167.98] rewrite 4984.58 [ 29.98, 171.89] read 13807.08 [ 161.99, 514.92] ost 41 sz 687865856K rsz 1024K obj 41 thr 164 write 9445.93 [ 82.99, 293.98] rewrite 9722.32 [ 149.98, 324.96] read 17851.10 [ 191.97, 869.92] ost 41 sz 687865856K rsz 1024K obj 41 thr 328 write 15872.41 [ 265.96, 533.94] rewrite 16682.58 [ 245.97, 526.97] read 19312.61 [ 184.98, 794.93] ost 41 sz 687865856K rsz 1024K obj 41 thr 656 write 18704.47 [ 222.98, 651.94] rewrite 18733.29 [ 252.90, 634.83] read 21040.28 [ 260.98, 808.92] ost 41 sz 687865856K rsz 1024K obj 41 thr 1312 write 18291.71 [ 161.99, 740.93] rewrite 18443.63 [ 47.00, 704.91] read 20683.56 [ 178.99, 908.91] ost 41 sz 687865856K rsz 1024K obj 41 thr 2624 write 18704.50 [ 19.00, 684.92] rewrite 18583.81 [ 25.00, 729.92] read 20400.08 [ 110.99, 982.88] ost 41 sz 687865856K rsz 1024K obj 82 thr 82 write 5634.08 [ 62.99, 176.98] rewrite 4640.45 [ 55.00, 162.98] read 9459.26 [ 114.98, 320.99] ost 41 sz 687865856K rsz 1024K obj 82 thr 164 write 9615.85 [ 95.99, 308.98] rewrite 8329.19 [ 122.99, 275.99] read 13967.03 [ 150.99, 430.97] ost 41 sz 687865856K rsz 1024K obj 82 thr 328 write 13846.63 [ 229.99, 461.97] rewrite 12576.55 [ 186.98, 390.97] read 18166.27 [ 130.99, 557.94] ost 41 sz 687865856K rsz 1024K obj 82 thr 656 write 18558.35 [ 268.98, 624.93] rewrite 16821.93 [ 246.85, 542.95] read 19645.73 [ 235.85, 676.92] ost 41 sz 687865856K rsz 1024K obj 82 thr 1312 write 18885.19 [ 117.99, 690.92] rewrite 16501.04 [ 115.99, 617.95] read 19255.26 [ 180.97, 832.89] ost 41 sz 687865856K rsz 1024K obj 82 thr 2624 write 18991.31 [ 127.51, 784.92] rewrite 18111.05 [ 31.00, 763.88] read 20333.42 [ 124.48, 997.82] ost 41 sz 687865856K rsz 1024K obj 164 thr 164 write 7513.17 [ 69.99, 236.95] rewrite 5611.77 [ 65.00, 198.96] read 12950.03 [ 80.99, 383.96] ost 41 sz 687865856K rsz 1024K obj 164 thr 328 write 13191.77 [ 216.99, 361.98] rewrite 10104.73 [ 129.99, 313.98] read 18380.92 [ 149.98, 529.97] ost 41 sz 687865856K rsz 1024K obj 164 thr 656 write 16442.83 [ 168.98, 494.91] rewrite 14155.27 [ 213.98, 452.97] read 19564.97 [ 238.85, 616.95] ost 41 sz 687865856K rsz 1024K obj 164 thr 1312 write 18070.58 [ 152.96, 612.91] rewrite 15744.41 [ 62.99, 540.96] read 18846.31 [ 160.99, 660.84] ost 41 sz 687865856K rsz 1024K obj 164 thr 2624 write 18664.83 [ 138.97, 767.93] rewrite 16648.63 [ 81.28, 603.93] read 19319.91 [ 79.97, 864.90] ost 41 sz 687865856K rsz 1024K obj 328 thr 328 write 9028.81 [ 66.00, 277.97] rewrite 6807.19 [ 42.99, 228.98] read 14799.75 [ 123.98, 491.92] ost 41 sz 687865856K rsz 1024K obj 328 thr 656 write 14471.67 [ 155.98, 427.97] rewrite 11632.72 [ 130.99, 375.98] read 19137.29 [ 127.79, 595.92] ost 41 sz 687865856K rsz 1024K obj 328 thr 1312 write 17084.20 [ 179.98, 533.95] rewrite 13810.96 [ 64.00, 449.96] read 18405.80 [ 182.98, 616.95] ost 41 sz 687865856K rsz 1024K obj 328 thr 2624 write 18583.14 [ 24.99, 684.92] rewrite 15588.87 [ 68.99, 579.93] read 18857.33 [ 160.98, 706.96] ost 41 sz 687865856K rsz 1024K obj 656 thr 656 write 9861.09 [ 121.98, 312.96] rewrite 7540.60 [ 70.00, 258.96] read 15160.96 [ 193.96, 483.94] ost 41 sz 687865856K rsz 1024K obj 656 thr 1312 write 15021.83 [ 175.97, 450.95] rewrite 11641.17 [ 97.99, 389.98] read 18470.04 [ 205.99, 597.91] ost 41 sz 687865856K rsz 1024K obj 656 thr 2624 write 17202.58 [ 84.98, 589.90] rewrite 14483.38 [ 143.98, 491.91] read 18475.50 [ 179.98, 631.94]
We have not yet done any tests with clients (in fact the 10GigE network still needs to be configured) but I would like to ask if there is any reason why we should not achieve our goal with this storage hardware.
I will also update the ticket once we've done some tests with clients.
Attachments
Activity
From what I've gathered from other sites, performance over IB seems different from 10GigE. Unfortunately we won't be able to change our infrastructure to IB as part of this project.
Remote access to the test system should be possible, let's discuss details on that over private email, if you don't mind.
Frederik
Is it possible to have remote access to your cluster? Please let me know
it puzzle me that your two oss combine performs slower than a single. Something is not configure correct here. If you still have the system, we should experiment a run on each individual OST.
1. create a ost_pool on first OST on one OSS and setstripe on that pool, run ior
2. create a ost_pool on first OST on both OSS and setstripe 2 on that pool, run ior
3. scale this up to all OST on all OSS to see where we are slow down.
Thanks
No, I am running over IB. My small setup is just 2 SATA drive on each OSS, but I was able to scale linearly up to 4 OSS which reached about 740MB/s. I am trying to reconfigure to try to achieve 1GB/s, either add more oss or more disk per oss.
Minh,
just to confirm, on your test system you are running over 10GigE? (bonded links?) And do you get close to what I get on a single OSS or much less? What is the approximate throughput you get with two OSSes over 10GigE?
Frederik
And another test with all OSTs:
Cut down iostat over 10 second interval in the middle of the test on first OSS, with ost details added. Not that this was fairly constant over the whole test. Note also that I was running iostat 10, so the result is in blocks.
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dm-3 0.00 0.00 0.00 0 0 dm-4 0.00 0.00 0.00 0 0 dm-5 0.00 0.00 0.00 0 0 dm-6 26.90 1.60 53467.20 16 534672 (ost13) dm-7 0.00 0.00 0.00 0 0 dm-8 26.90 1.60 53468.00 16 534680 (ost18) dm-9 26.80 0.80 53467.20 8 534672 (ost11) dm-10 0.00 0.00 0.00 0 0 dm-11 26.80 1.60 53262.40 16 532624 (ost16) dm-12 26.90 1.60 53467.20 16 534672 (ost19) dm-13 26.90 1.60 53468.00 16 534680 (ost17) dm-14 26.60 0.00 53262.40 0 532624 (ost14) dm-15 26.90 1.60 53467.20 16 534672 (ost10) dm-16 26.90 1.60 53467.20 16 534672 (ost15) dm-17 26.80 1.60 53262.40 16 532624 (ost14) dm-18 0.00 0.00 0.00 0 0 dm-19 0.00 0.00 0.00 0 0 dm-20 0.00 0.00 0.00 0 0 dm-21 0.00 0.00 0.00 0 0 dm-22 0.00 0.00 0.00 0 0 dm-23 0.00 0.00 0.00 0 0
Cut down iostat over 10 second interval in the middle of the test on second OSS:
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dm-3 26.60 0.00 53665.60 0 536656 (ost0) dm-4 26.60 0.00 53467.20 0 534672 (ost5) dm-5 26.70 0.00 53467.20 0 534672 (ost6) dm-6 26.80 0.00 53672.00 0 536720 (ost1) dm-7 26.60 0.00 53466.40 0 534664 (ost7) dm-8 26.70 0.00 53467.20 0 534672 (ost8) dm-9 1.10 0.00 9.60 0 96 (mdt) dm-10 26.70 0.00 53467.20 0 534672 (ost9) dm-11 0.00 0.00 0.00 0 0 dm-12 0.00 0.00 0.00 0 0 dm-13 0.00 0.00 0.00 0 0 dm-14 26.80 0.00 53672.00 0 536720 (ost3) dm-15 26.80 0.00 53672.00 0 536720 (ost4) dm-16 0.00 0.00 0.00 0 0 dm-17 0.00 0.00 0.00 0 0 dm-18 26.80 0.00 53672.00 0 536720 (ost2) dm-19 0.00 0.00 0.00 0 0 dm-20 0.00 0.00 0.00 0 0 dm-21 0.00 0.00 0.00 0 0 dm-22 0.00 0.00 0.00 0 0 dm-23 0.00 0.00 0.00 0 0
IOR reported a throughput of about 520MiB/s. Traffic seems fairly balanced to me.
Can I just compare exact Lustre versions? I'm still using lustre 2.3.0 on the servers and 2.5.59 on the clients.
ok, going through the suggestions one step at a time (after playing with lfs setstripe -o a bit, I think I've got it.)
ior on just the second OSS:
[bnh65367@cs04r-sc-serv-68 frederik1]$ lfs setstripe -c 10 -o 0xa single-oss-2/
[bnh65367@cs04r-sc-serv-68 frederik1]$ lfs getstripe single-oss-2/
single-oss-2/
stripe_count: 10 stripe_size: 1048576 stripe_offset: 10
[bnh65367@cs04r-sc-serv-68 frederik1]$ export IORTESTDIR=/mnt/lustre-test/frederik1/single-oss-2/
[bnh65367@cs04r-sc-serv-68 frederik1]$ $MPIRUN ${MPIRUN_OPTS} -np $NSLOTS -machinefile ${TMPDIR}/hostfile /home/bnh65367/code/ior/src/ior -o ${IORTESTDIR}/ior_dat -w -k -t1m -b 20g -i 1 -e
IOR-3.0.0: MPI Coordinated Test of Parallel I/O
Began: Fri Feb 15 17:33:57 2013
Command line used: /home/bnh65367/code/ior/src/ior -o /mnt/lustre-test/frederik1/single-oss-2//ior_dat -w -k -t1m -b 20g -i 1 -e
Machine: Linux cs04r-sc-serv-68.diamond.ac.uk
Test 0 started: Fri Feb 15 17:33:57 2013
Summary:
api = POSIX
test filename = /mnt/lustre-test/frederik1/single-oss-2//ior_dat
access = single-shared-file
ordering in a file = sequential offsets
ordering inter file= no tasks offsets
clients = 1 (1 per node)
repetitions = 1
xfersize = 1 MiB
blocksize = 20 GiB
aggregate filesize = 20 GiB
access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
------ --------- ---------- --------- -------- -------- -------- -------- ----
write 468.96 20971520 1024.00 0.000589 43.67 0.000200 43.67 0
Max Write: 468.96 MiB/sec (491.74 MB/sec)
Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
write 468.96 468.96 468.96 0.00 43.67131 0 1 1 1 0 0 1 0 0 1 21474836480 1048576 21474836480 POSIX 0
Finished: Fri Feb 15 17:34:41 2013
[bnh65367@cs04r-sc-serv-68 frederik1]$ lfs getstripe single-oss-2/ior_dat
single-oss-2/ior_dat
lmm_stripe_count: 10
lmm_stripe_size: 1048576
lmm_layout_gen: 0
lmm_stripe_offset: 10
obdidx objid objid group
10 1381 0x565 0
11 1381 0x565 0
12 1477 0x5c5 0
13 1381 0x565 0
14 1381 0x565 0
15 1381 0x565 0
16 1381 0x565 0
17 1381 0x565 0
18 1381 0x565 0
19 1381 0x565 0
rpc_stats for one OST:
[bnh65367@cs04r-sc-serv-68 frederik1]$ cat /proc/fs/lustre/osc/spfs1-OST0007-osc-ffff880829227800/rpc_stats
snapshot_time: 1360950242.989930 (secs.usecs)
read RPCs in flight: 0
write RPCs in flight: 0
pending write pages: 0
pending read pages: 0
read write
pages per rpc rpcs % cum % | rpcs % cum %
1: 0 0 0 | 0 0 0
2: 0 0 0 | 0 0 0
4: 0 0 0 | 0 0 0
8: 0 0 0 | 0 0 0
16: 0 0 0 | 0 0 0
32: 0 0 0 | 0 0 0
64: 0 0 0 | 0 0 0
128: 0 0 0 | 0 0 0
256: 0 0 0 | 25589 100 100
read write
rpcs in flight rpcs % cum % | rpcs % cum %
0: 0 0 0 | 0 0 0
1: 0 0 0 | 6979 27 27
2: 0 0 0 | 3272 12 40
3: 0 0 0 | 4127 16 56
4: 0 0 0 | 644 2 58
5: 0 0 0 | 515 2 60
6: 0 0 0 | 561 2 62
7: 0 0 0 | 1378 5 68
8: 0 0 0 | 2442 9 77
9: 0 0 0 | 3931 15 93
10: 0 0 0 | 1725 6 99
11: 0 0 0 | 15 0 100
read write
offset rpcs % cum % | rpcs % cum %
0: 0 0 0 | 17 0 0
1: 0 0 0 | 0 0 0
2: 0 0 0 | 0 0 0
4: 0 0 0 | 0 0 0
8: 0 0 0 | 0 0 0
16: 0 0 0 | 0 0 0
32: 0 0 0 | 0 0 0
64: 0 0 0 | 0 0 0
128: 0 0 0 | 0 0 0
256: 0 0 0 | 17 0 0
512: 0 0 0 | 34 0 0
1024: 0 0 0 | 68 0 0
2048: 0 0 0 | 136 0 1
4096: 0 0 0 | 272 1 2
8192: 0 0 0 | 544 2 4
16384: 0 0 0 | 1088 4 8
32768: 0 0 0 | 2125 8 16
65536: 0 0 0 | 4096 16 32
131072: 0 0 0 | 7976 31 63
262144: 0 0 0 | 3072 12 75
524288: 0 0 0 | 4096 16 91
1048576: 0 0 0 | 2048 8 100
[bnh65367@cs04r-sc-serv-68 frederik1]$
just to confirm
for 2) I assume it is -o 1 (and not a)?
for 3) iostat on the OSS or the client?
(And yes it looks like we are using all 20 OSTs:
[bnh65367@cs04r-sc-serv-68 frederik1]$ lfs getstripe /mnt/lustre-test/frederik1/stripe-20-1/ior_dat
/mnt/lustre-test/frederik1/stripe-20-1/ior_dat
lmm_stripe_count: 20
lmm_stripe_size: 1048576
lmm_layout_gen: 0
lmm_stripe_offset: 14
obdidx objid objid group
14 1380 0x564 0
6 1320 0x528 0
15 1380 0x564 0
7 1319 0x527 0
16 1380 0x564 0
8 1319 0x527 0
17 1380 0x564 0
9 1320 0x528 0
18 1380 0x564 0
0 1326 0x52e 0
19 1380 0x564 0
1 1323 0x52b 0
10 1380 0x564 0
2 1320 0x528 0
11 1380 0x564 0
3 1320 0x528 0
12 1476 0x5c4 0
4 1321 0x529 0
13 1380 0x564 0
5 1320 0x528 0
Frederick,
Is there anything else that needs to be done on this ticket?