Details
-
Improvement
-
Resolution: Fixed
-
Minor
-
None
-
None
-
ZFS-Lustre: lustre 2.8.57_62_g919224d, zfs-0.7.0
ldiskfs-lustre: lustre 2.7.16.7
-
9223372036854775807
Description
It seems that, when ZFS is used as the backend file system for Lustre OST, there is an additional 128 byte for each IO.
For example, write 1GB write to a ZFS based file system.
# dd if=/dev/zero of=File_1GB count=1024 bs=1M 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 0.869192 s, 1.2 GB/s
[root@kapollo01 ~]# lctl get_param obdfilter.edafs-OST0001.brw_stats
obdfilter.edafs-OST0001.brw_stats=
snapshot_time: 1477281605.706719 (secs.usecs)
read | write
pages per bulk r/w rpcs % cum % | rpcs % cum %
1: 0 0 0 | 1024 50 50
2: 0 0 0 | 0 0 50
4: 0 0 0 | 0 0 50
8: 0 0 0 | 0 0 50
16: 0 0 0 | 0 0 50
32: 0 0 0 | 0 0 50
64: 0 0 0 | 0 0 50
128: 0 0 0 | 0 0 50
256: 0 0 0 | 1024 50 100
read | write
discontiguous pages rpcs % cum % | rpcs % cum %
0: 0 0 0 | 1024 50 50
1: 0 0 0 | 0 0 50
2: 0 0 0 | 0 0 50
3: 0 0 0 | 0 0 50
4: 0 0 0 | 0 0 50
5: 0 0 0 | 0 0 50
6: 0 0 0 | 0 0 50
7: 0 0 0 | 0 0 50
8: 0 0 0 | 0 0 50
9: 0 0 0 | 0 0 50
10: 0 0 0 | 0 0 50
11: 0 0 0 | 0 0 50
12: 0 0 0 | 0 0 50
13: 0 0 0 | 0 0 50
14: 0 0 0 | 0 0 50
15: 0 0 0 | 0 0 50
16: 0 0 0 | 0 0 50
17: 0 0 0 | 0 0 50
18: 0 0 0 | 0 0 50
19: 0 0 0 | 0 0 50
20: 0 0 0 | 0 0 50
21: 0 0 0 | 0 0 50
22: 0 0 0 | 0 0 50
23: 0 0 0 | 0 0 50
24: 0 0 0 | 0 0 50
25: 0 0 0 | 0 0 50
26: 0 0 0 | 0 0 50
27: 0 0 0 | 0 0 50
28: 0 0 0 | 0 0 50
29: 0 0 0 | 0 0 50
30: 0 0 0 | 0 0 50
31: 0 0 0 | 1024 50 100
read | write
disk I/Os in flight ios % cum % | ios % cum %
1: 0 0 0 | 2039 99 99
2: 0 0 0 | 9 0 100
read | write
I/O time (1/1000s) ios % cum % | ios % cum %
read | write
disk I/O size ios % cum % | ios % cum %
128: 0 0 0 | 1024 50 50
256: 0 0 0 | 0 0 50
512: 0 0 0 | 0 0 50
1K: 0 0 0 | 0 0 50
2K: 0 0 0 | 0 0 50
4K: 0 0 0 | 0 0 50
8K: 0 0 0 | 0 0 50
16K: 0 0 0 | 0 0 50
32K: 0 0 0 | 0 0 50
64K: 0 0 0 | 0 0 50
128K: 0 0 0 | 0 0 50
256K: 0 0 0 | 0 0 50
512K: 0 0 0 | 0 0 50
1M: 0 0 0 | 1024 50 100
On the other side, such 128 bytes additional IO does not exist.
[root@kcli01 ost01]# dd if=/dev/zero of=File_1GB count=1024 bs=1M 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 1.79642 s, 598 MB/s
[root@koss02 ~]# lctl get_param obdfilter.krakenfs-OST0001.brw_stats
obdfilter.krakenfs-OST0001.brw_stats=
snapshot_time: 1477297530.705071 (secs.usecs)
read | write
pages per bulk r/w rpcs % cum % | rpcs % cum %
1K: 0 0 0 | 256 100 100
read | write
discontiguous pages rpcs % cum % | rpcs % cum %
0: 0 0 0 | 256 100 100
read | write
discontiguous blocks rpcs % cum % | rpcs % cum %
0: 0 0 0 | 256 100 100
read | write
disk fragmented I/Os ios % cum % | ios % cum %
4: 0 0 0 | 255 99 99
5: 0 0 0 | 0 0 99
6: 0 0 0 | 1 0 100
read | write
disk I/Os in flight ios % cum % | ios % cum %
1: 0 0 0 | 4 0 0
2: 0 0 0 | 5 0 0
3: 0 0 0 | 5 0 1
4: 0 0 0 | 5 0 1
5: 0 0 0 | 2 0 2
6: 0 0 0 | 2 0 2
7: 0 0 0 | 1 0 2
8: 0 0 0 | 1 0 2
9: 0 0 0 | 1 0 2
10: 0 0 0 | 1 0 2
11: 0 0 0 | 1 0 2
12: 0 0 0 | 1 0 2
13: 0 0 0 | 3 0 3
14: 0 0 0 | 11 1 4
15: 0 0 0 | 23 2 6
16: 0 0 0 | 44 4 10
17: 0 0 0 | 81 7 18
18: 0 0 0 | 118 11 30
19: 0 0 0 | 144 14 44
20: 0 0 0 | 133 12 57
21: 0 0 0 | 104 10 67
22: 0 0 0 | 72 7 74
23: 0 0 0 | 45 4 78
24: 0 0 0 | 40 3 82
25: 0 0 0 | 42 4 86
26: 0 0 0 | 42 4 90
27: 0 0 0 | 37 3 94
28: 0 0 0 | 29 2 97
29: 0 0 0 | 16 1 98
30: 0 0 0 | 8 0 99
31: 0 0 0 | 5 0 100
read | write
I/O time (1/1000s) ios % cum % | ios % cum %
4: 0 0 0 | 3 1 1
8: 0 0 0 | 0 0 1
16: 0 0 0 | 13 5 6
32: 0 0 0 | 172 67 73
64: 0 0 0 | 54 21 94
128: 0 0 0 | 14 5 100
read | write
disk I/O size ios % cum % | ios % cum %
512K: 0 0 0 | 2 0 0
1M: 0 0 0 | 1024 99 100