Details
-
Improvement
-
Resolution: Fixed
-
Minor
-
None
-
None
-
23,549
-
4894
Description
The current lustre codes is an limitation that is single client's performance (buffered I/O) when the checksum is turned on. My understanding is that the buffered I/O is handled by ptlrpcd on the client and the lustre checksum is also calculated in this thread if the client read the data from OSSs. ptlrpcd is not multithreaded in current codes, the checksum calculation harms CPU resources and impacts the lustre performance.
The other hand, the latest Intel Nehalem/Westmere CPUs have the hardware crc32c accelerated instruction function and it's implemented in the CPU chip. we can use fast crc32c instruction without any additional costs if server is running with Intel CPUs.
The current lustre supports crc, alder as checksum algorithm. So, I would suggest adding crc32c as one of additional checksum algorithm and enable
See bz#23549 on bugzilla. The initial patch is available and simple testing is done.
The performance was much improved.
e.g)
single client's read performance : 60% improved
1GB/sec (adler) vs 1.6GB/sec (crc32c/w hardware instruction)
single client's write performance : (max) 30% improved
1.3GB/sec (adler) vs 1.7GB/sec (crc32c/w hardware instruction)
see more detail : https://bugzilla.lustre.org/attachment.cgi?id=31604
And I saw this patch can reduce the CPU usages too.
However, see also bz#23771, we saw some some error "checksum protocl errors" sometimes. (not always, and we still don't know when (what's timing) this error shows up.
Done someone have a look at codes in patch and give me some suggestions to fix or how figure out bug 23771?
I wonder if we can land this patch into the lustre mainstream once we can fix 23771.
Attachments
Issue Links
- is related to
-
-
- Resolved
-
- Trackbacks
-
![[Confluence: Engineering]](/images/icons/generic_link_16.png "[Confluence: Engineering]")
Lustre 1.8.x known issues tracker
While testing against Lustre b18 branch, we would hit known bugs which were already reported in Lustre Bugzilla https://bugzilla.lustre.org/. In order to move away from relying on Bugzilla, we would create a JIRA
-
Changelog 2.2
version 2.2.0 Support for networks: o2iblnd OFED 1.5.4 Server support for kernels: 2.6.32220.4.2.el6 (RHEL6) Client support for unpatched kernels: 2.6.18274.18.1.el5 (RHEL5) 2.6.32220.4.2.el6 (RHEL6) 2.6.32.360....
Activity
| Fix Version/s | New: Lustre 2.2.0 [ 10082 ] | |
| Resolution | New: Fixed [ 1 ] | |
| Status | Original: Open [ 1 ] | New: Resolved [ 5 ] |
Landed for 2.2. Unlikely to consider landing this to 1.8.x at this point
Integrated in 
lustre-master » i686,server,el5,ofa #287
LU-241 Support crc32c with hardware accelerated instruction as one of lustre checksums
Oleg Drokin : 0517160dd68ac026513ad1b8e3e6f7abd4acfdef
Files :
- lustre/obdfilter/filter.c
- lustre/utils/wirecheck.c
- lustre/llite/llite_lib.c
- lustre/ptlrpc/wiretest.c
- lustre/utils/wiretest.c
- lustre/mds/mds_lov.c
- lustre/ptlrpc/import.c
- lustre/include/linux/obd_support.h
- lustre/osc/osc_request.c
- lustre/ost/ost_handler.c
- lustre/include/obd_cksum.h
- lustre/include/lustre/lustre_idl.h
- lustre/tests/sanity.sh
Integrated in lustre-master » i686,client,el5,inkernel #287
LU-241 Support crc32c with hardware accelerated instruction as one of lustre checksums
Oleg Drokin : 0517160dd68ac026513ad1b8e3e6f7abd4acfdef
Files :
- lustre/mds/mds_lov.c
- lustre/utils/wirecheck.c
- lustre/utils/wiretest.c
- lustre/obdfilter/filter.c
- lustre/ptlrpc/wiretest.c
- lustre/tests/sanity.sh
- lustre/include/obd_cksum.h
- lustre/ptlrpc/import.c
- lustre/include/linux/obd_support.h
- lustre/ost/ost_handler.c
- lustre/osc/osc_request.c
- lustre/include/lustre/lustre_idl.h
- lustre/llite/llite_lib.c
Integrated in lustre-master » i686,client,el5,ofa #287
LU-241 Support crc32c with hardware accelerated instruction as one of lustre checksums
Oleg Drokin : 0517160dd68ac026513ad1b8e3e6f7abd4acfdef
Files :
- lustre/mds/mds_lov.c
- lustre/include/linux/obd_support.h
- lustre/ptlrpc/import.c
- lustre/utils/wiretest.c
- lustre/ptlrpc/wiretest.c
- lustre/utils/wirecheck.c
- lustre/include/obd_cksum.h
- lustre/tests/sanity.sh
- lustre/ost/ost_handler.c
- lustre/include/lustre/lustre_idl.h
- lustre/llite/llite_lib.c
- lustre/osc/osc_request.c
- lustre/obdfilter/filter.c
Integrated in lustre-master » i686,server,el5,inkernel #287
LU-241 Support crc32c with hardware accelerated instruction as one of lustre checksums
Oleg Drokin : 0517160dd68ac026513ad1b8e3e6f7abd4acfdef
Files :
- lustre/include/lustre/lustre_idl.h
- lustre/utils/wiretest.c
- lustre/ost/ost_handler.c
- lustre/obdfilter/filter.c
- lustre/ptlrpc/wiretest.c
- lustre/llite/llite_lib.c
- lustre/osc/osc_request.c
- lustre/tests/sanity.sh
- lustre/include/linux/obd_support.h
- lustre/ptlrpc/import.c
- lustre/utils/wirecheck.c
- lustre/include/obd_cksum.h
- lustre/mds/mds_lov.c
Integrated in lustre-master » x86_64,server,el6,inkernel #287
LU-241 Support crc32c with hardware accelerated instruction as one of lustre checksums
Oleg Drokin : 0517160dd68ac026513ad1b8e3e6f7abd4acfdef
Files :
- lustre/llite/llite_lib.c
- lustre/osc/osc_request.c
- lustre/ost/ost_handler.c
- lustre/utils/wirecheck.c
- lustre/ptlrpc/wiretest.c
- lustre/utils/wiretest.c
- lustre/include/lustre/lustre_idl.h
- lustre/include/obd_cksum.h
- lustre/mds/mds_lov.c
- lustre/include/linux/obd_support.h
- lustre/obdfilter/filter.c
- lustre/ptlrpc/import.c
- lustre/tests/sanity.sh
Integrated in lustre-master » x86_64,client,ubuntu1004,inkernel #287
LU-241 Support crc32c with hardware accelerated instruction as one of lustre checksums
Oleg Drokin : 0517160dd68ac026513ad1b8e3e6f7abd4acfdef
Files :
- lustre/llite/llite_lib.c
- lustre/tests/sanity.sh
- lustre/ptlrpc/wiretest.c
- lustre/ptlrpc/import.c
- lustre/obdfilter/filter.c
- lustre/mds/mds_lov.c
- lustre/osc/osc_request.c
- lustre/include/linux/obd_support.h
- lustre/ost/ost_handler.c
- lustre/include/obd_cksum.h
- lustre/include/lustre/lustre_idl.h
- lustre/utils/wirecheck.c
- lustre/utils/wiretest.c
LU-1025has a fix for the CRC32C algorithm that was introduced by this change. This will change the wire protocol.Are there any users of this feature in production? Since the CRC32C patch was landed after 2.1.0 was released, we are planning to land the
LU-1025change for 2.2.0 since it will not change the protocol for any released version, but if anyone has pulled this patch into another release their systems will break if they do not upgrade both the client and server at the same time.