[LU-4937] LustreError: 2926:0:(osc_request.c:1608:osc_brw_fini_request()) Protocol error: - Unable to send checksum

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.6.0, Lustre 2.5.3
• Affects Version/s: Lustre 2.4.2
• Labels: llnl mn4
• Environment: Lustre 2.4.2; Red Hat Enterprise Linux Server release 6.5 (Santiago)
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Rajeshwaran Ganesan

Description

We noticed the following error on three of the lustre clients, we like to know why its happening and does it need any fix..

Apr 4 15:34:07 uc1n274 kernel: LustreError: 2926:0:(osc_request.c:1608:osc_brw_fini_request()) Protocol error: server 172.26.8.15@o2ib set the 'checksum' bit, but didn't send a checksum. Not fatal, but please notify on http://bugs.whamcloud.com/
Apr 4 16:38:55 uc1n484 kernel: LustreError: 2981:0:(osc_request.c:1608:osc_brw_fini_request()) Protocol error: server 172.26.8.15@o2ib set the 'checksum' bit, but didn't send a checksum. Not fatal, but please notify on http://bugs.whamcloud.com/
Apr 5 01:14:44 uc1n251 kernel: LustreError: 2914:0:(osc_request.c:1608:osc_brw_fini_request()) Protocol error: server 172.26.8.15@o2ib set the 'checksum' bit, but didn't send a checksum. Not fatal, but please notify on http://bugs.whamcloud.com/

Comments

[Lukasz Flis]

@Andreas:
It seems that crc32 is much better than adler on AMD 6276 cpus:

Using crypto hash: crc32 (crc32-pclmul) speed 3196 MB/s
Using crypto hash: adler32 (adler32-zlib) speed 2335 MB/s

[Andreas Dilger]

Lukaz, you will probably get better performance with adler32 for the checksum on the AMD nodes instead of crc32:

lctl set_param osc.*.checksum_type=adler32

Bobijam, I don't know enough details of the CRC32c checksum to say for sure if "0xffffffff" is a valid value or not. I know that for most checksums "0" is an invalid value because this would mean that the checksum for any length of zero bytes would be the same, which is why they always initialize the starting value to 0xffffffff. The use of ~0 for indicating the checksum is not sent was originally added in 1.4.x when the checksum feature was just new, and is no longer need. A mismatch between the client and server checksum (as is done with your patch) will detect corruption, and there is no reason to special-case the 0xffffffff checksum value.

I've also submitted a version of your patch for master: http://review.whamcloud.com/10354

[Zhenyu Xu]

Andreas,

Is it possible that crc32c has an issue in this case?

[Lukasz Flis]

I have found out the difference which makes AMD nodes not affected by the problem

Intel nodes are using: crc32c
AMD nodes default to: crc32

As a workaround we switched Intel nodes to crc32 (which is a bit slower in that case) and problem is gone

for i in /proc/fs/lustre/osc/*/checksum_type;do echo crc32 > $i; done;

[Lukasz Flis]

this issue affects 2.5.1 clients also so i'd suggest updating affected version list

[Lukasz Flis]

I will ask one of our users to verify sanity of results generated by a patched client.

The most interesting thing is why AMD processor based client with 256GB ram with identical client version and
identical inputs is not affected by ~0 problem.

Assuming that CRC algorithm works in the same way on the client and server only explanation that come to my mind is the different data alignment/content of requests
but i'm just a user here

Is there anything else we can check here?

[Zhenyu Xu]

I'm no expert on the checksum algorithm, if it's possible to get a ~0 checksum from the algorithm, I prefer to think that's not a data corruption here. What's your opinion from your gaussian application, does the data generated correct and valid?

[Lukasz Flis]

Applied http://review.whamcloud.com/10309 to the client.
Our testing shows that I/O issue is gone.

Can we assume this patch as final solution?
Are we sure that the checksum problem is not a result of a data corruption?

Thanks

[Zhenyu Xu]

If client read pops this error, the read could return error. Would you mind trying this patch to see whether this issue persists w/ it? http://review.whamcloud.com/10309

[Lukasz Flis]

It looks like a kind of data corruption. After the second occurence of a log message gaussian application is exiting with I/O error:

[pid 3283] read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1600) = 1600
[pid 3283] read(4, "\307\317\377\377\377\377\377\377\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0ZN phthalocyanine, MP2/6-31G* op"..., 40856) = 40856
[pid 3283] read(4, ">\10]\341\275\240\1@\31-u\221%8\347?*\321\377n]\226\4@\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0C,\303Z\225@\2@\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1787568) = 1787568
[pid 3283] read(4, 0x7fff2f949b68, 8) = -1 EIO (Input/output error)

[Zhenyu Xu]

this line raised my eyebrows

00000010:00008000:1.0:1399928810.277985:0:15665:0:(ost_handler.c:909:ost_brw_read()) checksum at read origin: ffffffff

this shows that the checksum for the bulk data on the OST to be transferred to client is ~0, which makes the client checking code invalid

osc_brw_fini_request

...
                if (server_cksum == ~0 && rc > 0) {
                        CERROR("Protocol error: server %s set the 'checksum' "
                               "bit, but didn't send a checksum.  Not fatal, "
                               "but please notify on http://bugs.whamcloud.com/\n",
                               libcfs_nid2str(peer->nid));
                }
...

I am no expert here, but is it possible that the checksum of a chunk of data to be ~0?