[LU-2659] single client throughput for 10GigE

• Type: Improvement
• Resolution: Not a Bug
• Priority: Minor
• Affects Version/s: Lustre 2.3.0
• Labels: ptr
• Assignee: Minh Diep
• Reporter: Frederik Ferner (Inactive)

Description

(I'm not sure about the issue type for this ticket, please adjust as appropriate.)

As discussed with Peter Jones, we are trying to implement a file system where single clients can achieve >900MB/s write throughput over 10GigE connections. Ideally single 10GigE for the clients but 2x10GigE LACP bonding might be an option. The OSSes will initially have 4x 10GigE LACP bonded links, though for some initial testing we might start with fewer links.

The disk backend has now arrived and this is a sample obdfilter-survey result using all one OST and 4 OSSes, without much tuning on the OSS nodes yet. The OSSes are all running Lustre 2.3.0 on RHEL6.

Sat Jan 19 15:49:23 GMT 2013 Obdfilter-survey for case=disk from cs04r-sc-oss05-03.diamond.ac.uk
ost 41 sz 687865856K rsz 1024K obj   41 thr   41 write 2975.14 [  40.00, 105.99] rewrite 2944.84 [  22.00, 118.99] read 8104.33 [  40.99, 231.98]
ost 41 sz 687865856K rsz 1024K obj   41 thr   82 write 5231.39 [  49.99, 167.98] rewrite 4984.58 [  29.98, 171.89] read 13807.08 [ 161.99, 514.92]
ost 41 sz 687865856K rsz 1024K obj   41 thr  164 write 9445.93 [  82.99, 293.98] rewrite 9722.32 [ 149.98, 324.96] read 17851.10 [ 191.97, 869.92]
ost 41 sz 687865856K rsz 1024K obj   41 thr  328 write 15872.41 [ 265.96, 533.94] rewrite 16682.58 [ 245.97, 526.97] read 19312.61 [ 184.98, 794.93]
ost 41 sz 687865856K rsz 1024K obj   41 thr  656 write 18704.47 [ 222.98, 651.94] rewrite 18733.29 [ 252.90, 634.83] read 21040.28 [ 260.98, 808.92]
ost 41 sz 687865856K rsz 1024K obj   41 thr 1312 write 18291.71 [ 161.99, 740.93] rewrite 18443.63 [  47.00, 704.91] read 20683.56 [ 178.99, 908.91]
ost 41 sz 687865856K rsz 1024K obj   41 thr 2624 write 18704.50 [  19.00, 684.92] rewrite 18583.81 [  25.00, 729.92] read 20400.08 [ 110.99, 982.88]
ost 41 sz 687865856K rsz 1024K obj   82 thr   82 write 5634.08 [  62.99, 176.98] rewrite 4640.45 [  55.00, 162.98] read 9459.26 [ 114.98, 320.99]
ost 41 sz 687865856K rsz 1024K obj   82 thr  164 write 9615.85 [  95.99, 308.98] rewrite 8329.19 [ 122.99, 275.99] read 13967.03 [ 150.99, 430.97]
ost 41 sz 687865856K rsz 1024K obj   82 thr  328 write 13846.63 [ 229.99, 461.97] rewrite 12576.55 [ 186.98, 390.97] read 18166.27 [ 130.99, 557.94]
ost 41 sz 687865856K rsz 1024K obj   82 thr  656 write 18558.35 [ 268.98, 624.93] rewrite 16821.93 [ 246.85, 542.95] read 19645.73 [ 235.85, 676.92]
ost 41 sz 687865856K rsz 1024K obj   82 thr 1312 write 18885.19 [ 117.99, 690.92] rewrite 16501.04 [ 115.99, 617.95] read 19255.26 [ 180.97, 832.89]
ost 41 sz 687865856K rsz 1024K obj   82 thr 2624 write 18991.31 [ 127.51, 784.92] rewrite 18111.05 [  31.00, 763.88] read 20333.42 [ 124.48, 997.82]
ost 41 sz 687865856K rsz 1024K obj  164 thr  164 write 7513.17 [  69.99, 236.95] rewrite 5611.77 [  65.00, 198.96] read 12950.03 [  80.99, 383.96]
ost 41 sz 687865856K rsz 1024K obj  164 thr  328 write 13191.77 [ 216.99, 361.98] rewrite 10104.73 [ 129.99, 313.98] read 18380.92 [ 149.98, 529.97]
ost 41 sz 687865856K rsz 1024K obj  164 thr  656 write 16442.83 [ 168.98, 494.91] rewrite 14155.27 [ 213.98, 452.97] read 19564.97 [ 238.85, 616.95]
ost 41 sz 687865856K rsz 1024K obj  164 thr 1312 write 18070.58 [ 152.96, 612.91] rewrite 15744.41 [  62.99, 540.96] read 18846.31 [ 160.99, 660.84]
ost 41 sz 687865856K rsz 1024K obj  164 thr 2624 write 18664.83 [ 138.97, 767.93] rewrite 16648.63 [  81.28, 603.93] read 19319.91 [  79.97, 864.90]
ost 41 sz 687865856K rsz 1024K obj  328 thr  328 write 9028.81 [  66.00, 277.97] rewrite 6807.19 [  42.99, 228.98] read 14799.75 [ 123.98, 491.92]
ost 41 sz 687865856K rsz 1024K obj  328 thr  656 write 14471.67 [ 155.98, 427.97] rewrite 11632.72 [ 130.99, 375.98] read 19137.29 [ 127.79, 595.92]
ost 41 sz 687865856K rsz 1024K obj  328 thr 1312 write 17084.20 [ 179.98, 533.95] rewrite 13810.96 [  64.00, 449.96] read 18405.80 [ 182.98, 616.95]
ost 41 sz 687865856K rsz 1024K obj  328 thr 2624 write 18583.14 [  24.99, 684.92] rewrite 15588.87 [  68.99, 579.93] read 18857.33 [ 160.98, 706.96]
ost 41 sz 687865856K rsz 1024K obj  656 thr  656 write 9861.09 [ 121.98, 312.96] rewrite 7540.60 [  70.00, 258.96] read 15160.96 [ 193.96, 483.94]
ost 41 sz 687865856K rsz 1024K obj  656 thr 1312 write 15021.83 [ 175.97, 450.95] rewrite 11641.17 [  97.99, 389.98] read 18470.04 [ 205.99, 597.91]
ost 41 sz 687865856K rsz 1024K obj  656 thr 2624 write 17202.58 [  84.98, 589.90] rewrite 14483.38 [ 143.98, 491.91] read 18475.50 [ 179.98, 631.94]

We have not yet done any tests with clients (in fact the 10GigE network still needs to be configured) but I would like to ask if there is any reason why we should not achieve our goal with this storage hardware.

I will also update the ticket once we've done some tests with clients.