Details
-
Bug
-
Resolution: Fixed
-
Critical
-
None
-
None
-
3
-
9223372036854775807
Description
Lustre 2.10 on the server, single OST, brw_size=16
Lustre 2.10 clients, max_rpcs_in_flight=64, max_dirty_mb=256, max_pages_per_rpc=4096
40 IOR jobs, File-per-process, transfer size 64m, file size 256g, stonewall 120 sec
============================
lctl set_param llite.*.pio=0
============================
DIO write then read
-------------------
read | write
pages per bulk r/w rpcs % cum % | rpcs % cum %
256: 343872 100 100 | 220928 100 100
read | write
discontiguous pages rpcs % cum % | rpcs % cum %
0: 343872 100 100 | 220928 100 100
--------------
POSIX buffered
--------------
read | write
pages per bulk r/w rpcs % cum % | rpcs % cum %
256: 0 0 0 | 44 0 0
512: 0 0 0 | 43 0 0
1K: 0 0 0 | 63 0 0
2K: 0 0 0 | 107 0 0
4K: 40960 100 100 | 68017 99 100
read | write
discontiguous pages rpcs % cum % | rpcs % cum %
0: 40960 100 100 | 65412 95 95
1: 0 0 100 | 2862 4 100
============================
lctl set_param llite.*.pio=1
============================
DIO write then read
-------------------
read | write
pages per bulk r/w rpcs % cum % | rpcs % cum %
256: 352337 79 79 | 259706 80 80
512: 69745 15 94 | 49782 15 95
1K: 20035 4 99 | 12087 3 99
2K: 2357 0 100 | 1283 0 100
read | write
discontiguous pages rpcs % cum % | rpcs % cum %
0: 387211 87 87 | 286802 88 88
1: 50221 11 98 | 32161 9 98
2: 6541 1 99 | 3670 1 99
3: 485 0 99 | 222 0 99
4: 16 0 100 | 3 0 100
--------------
POSIX buffered
--------------
read | write
pages per bulk r/w rpcs % cum % | rpcs % cum %
1: 109334 19 19 | 0 0 0
2: 9844 1 20 | 1 0 0
4: 2482 0 21 | 0 0 0
8: 375 0 21 | 0 0 0
16: 0 0 21 | 1 0 0
32: 0 0 21 | 1 0 0
64: 0 0 21 | 2 0 0
128: 3 0 21 | 10 0 0
256: 398187 70 91 | 153 0 0
512: 34233 6 97 | 244 0 0
1K: 8819 1 99 | 377 0 1
2K: 2322 0 99 | 1004 1 2
4K: 2136 0 100 | 65289 97 100
read | write
discontiguous pages rpcs % cum % | rpcs % cum %
0: 516948 91 91 | 56943 84 84
1: 42268 7 98 | 2582 3 88
2: 6384 1 99 | 2100 3 91
3: 1523 0 99 | 1288 1 93
4: 365 0 99 | 988 1 95
5: 97 0 99 | 774 1 96
Same test setup on the server, IEEL clients
--------------------------
DIO write followed by read
------------------------
read | write
pages per bulk r/w rpcs % cum % | rpcs % cum %
4K: 40960 100 100 | 46224 100 100
read | write
discontiguous pages rpcs % cum % | rpcs % cum %
0: 40960 100 100 | 46224 100 100
--------------
POSIX buffered
--------------
read | write
pages per bulk r/w rpcs % cum % | rpcs % cum %
4K: 40960 100 100 | 66976 100 100
read | write
discontiguous pages rpcs % cum % | rpcs % cum %
0: 40960 100 100 | 66976 100 100