https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4937 Lustre <p>We noticed the following error on three of the lustre clients, we like to know why its happening and does it need any fix..</p> <p>Apr 4 15:34:07 uc1n274 kernel: LustreError: 2926:0:(osc_request.c:1608:osc_brw_fini_request()) Protocol error: server 172.26.8.15@o2ib set the 'checksum' bit, but didn't send a checksum. Not fatal, but please notify on <a href="http://bugs.whamcloud.com/" class="external-link" target="_blank" rel="nofollow noopener">http://bugs.whamcloud.com/</a> <br/> Apr 4 16:38:55 uc1n484 kernel: LustreError: 2981:0:(osc_request.c:1608:osc_brw_fini_request()) Protocol error: server 172.26.8.15@o2ib set the 'checksum' bit, but didn't send a checksum. Not fatal, but please notify on <a href="http://bugs.whamcloud.com/" class="external-link" target="_blank" rel="nofollow noopener">http://bugs.whamcloud.com/</a> <br/> Apr 5 01:14:44 uc1n251 kernel: LustreError: 2914:0:(osc_request.c:1608:osc_brw_fini_request()) Protocol error: server 172.26.8.15@o2ib set the 'checksum' bit, but didn't send a checksum. Not fatal, but please notify on <a href="http://bugs.whamcloud.com/" class="external-link" target="_blank" rel="nofollow noopener">http://bugs.whamcloud.com/</a> </p> Lustre 2.4.2<br/> Red Hat Enterprise Linux Server release 6.5 (Santiago)<br/> LU-4937

LustreError: 2926:0:(osc_request.c:1608:osc_brw_fini_request()) Protocol error: - Unable to send checksum

Bug Minor Resolved Fixed Zhenyu Xu Rajeshwaran Ganesan llnl mn4 Mon, 21 Apr 2014 22:19:43 +0000 Mon, 31 Aug 2015 17:09:39 +0000 Thu, 29 May 2014 15:47:30 +0000 Lustre 2.4.2 Lustre 2.6.0 Lustre 2.5.3 2 9 <p>Bobijam</p> <p>Could you please comment on this one?</p> <p>Thanks</p> <p>Peter</p> <p>what the Lustre version used on client and OST server?</p> <p>Hello,<br/> OST /MDS servers are in 2.4.1<br/> Clients 2.4.2</p> <p>I have uploaded the client logs in the ftp.whamlocud.com</p> <p>Thanks</p> <p>please also upload OST logs.</p> <p>btw, I couldn't find the client log</p> <p>ls -a ftp/uploads/<a href="https://jira.whamcloud.com/browse/LU-4937" title=" LustreError: 2926:0:(osc_request.c:1608:osc_brw_fini_request()) Protocol error: - Unable to send checksum" class="issue-link" data-issue-key="LU-4937"><del>LU-4937</del></a><br/> . ..</p> <p>Please try now..! </p> <p>still need OST logs</p> <p>I have added the OST logs into the FTP site, Please verify. </p> <p>The log collected does not reveal relevant information in it. Please enable "+page" debug mask on OST and try to reproduce the issue and collect OST log again.</p> <p>Since the error on client is from code as follows</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (body-&gt;oa.o_valid &amp; OBD_MD_FLCKSUM) { <span class="code-keyword">static</span> <span class="code-object">int</span> cksum_counter; __u32 server_cksum = body-&gt;oa.o_cksum; ... <span class="code-keyword">if</span> (server_cksum == ~0 &amp;&amp; rc &gt; 0) { CERROR(<span class="code-quote">"Protocol error: server %s set the <span class="code-quote">'checksum'</span> "</span> <span class="code-quote">"bit, but didn't send a checksum. Not fatal, "</span> <span class="code-quote">"but please notify on http:<span class="code-comment">//bugs.whamcloud.com/\n"</span>, </span> libcfs_nid2str(peer-&gt;nid)); } </pre> </div></div> <p>which means the server_cksum is -1 which is unlikely, I've checked OST code, it set server cksum when it set OBD_MD_FLCKSUM flag, I want to check the server side of story</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>ost_brw_read()</b></div><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (body-&gt;oa.o_valid &amp; OBD_MD_FLCKSUM) { cksum_type_t cksum_type = cksum_type_unpack(repbody-&gt;oa.o_valid &amp; OBD_MD_FLFLAGS ? repbody-&gt;oa.o_flags : 0); repbody-&gt;oa.o_flags = cksum_type_pack(cksum_type); repbody-&gt;oa.o_valid = OBD_MD_FLCKSUM | OBD_MD_FLFLAGS; repbody-&gt;oa.o_cksum = ost_checksum_bulk(desc, OST_READ,cksum_type); CDEBUG(D_PAGE, <span class="code-quote">"checksum at read origin: %x\n"</span>, repbody-&gt;oa.o_cksum); } </pre> </div></div> <p>We can confirm the same problem exists with 2.4.1 client and OSS at 2.5.1 and 2.4.x</p> <p>It happens during Gaussian application runs. <br/> Do you need any additional debug info other than +page?</p> <p>please add +info as well.</p> <p>I have uploaded two logs: one for OST and one from Client.<br/> The logs are from different runs but show the same bug. We are able to reproduce it<br/> every time.</p> <p>Files have been uploaded to FTP: uploads/<a href="https://jira.whamcloud.com/browse/LU-4937" title=" LustreError: 2926:0:(osc_request.c:1608:osc_brw_fini_request()) Protocol error: - Unable to send checksum" class="issue-link" data-issue-key="LU-4937"><del>LU-4937</del></a></p> <p>Flags: rpctrace,info,page</p> <p>LU-cksum-client-2.5.1.cyfronet.log.gz<br/> LU-cksum-ost-2.5.1.cyfronet.log.gz</p> <p>Could you please have a look. If something is missing please let me know.<br/> Problem is present in 2.4 as well as in 2.5.1 clients</p> <p>Affected client IP: 172.16.202.160</p> <p>Uploaded another log set:<br/> <a href="https://jira.whamcloud.com/browse/LU-4937" title=" LustreError: 2926:0:(osc_request.c:1608:osc_brw_fini_request()) Protocol error: - Unable to send checksum" class="issue-link" data-issue-key="LU-4937"><del>LU-4937</del></a>-good.tar.gz<br/> this time it's the view from ost and client for the same events</p> <p>00000001:00000040:1.0:1399928810.277979:0:15665:0:(linux-crypto.c:82:cfs_crypto_hash_alloc()) Using crypto hash: crc32c (crc32c-intel) speed 2673 MB/s<br/> 00000010:00000040:1.0:1399928810.277982:0:15665:0:(ost_handler.c:626:ost_checksum_bulk()) Checksum for algo crc32c<br/> 00000010:00008000:1.0:1399928810.277985:0:15665:0:(ost_handler.c:909:ost_brw_read()) checksum at read origin: ffffffff<br/> 00010000:00000040:1.0:1399928810.278234:0:15665:0:(ldlm_resource.c:1223:ldlm_resource_getref()) getref res: ffff88016d6c25c0 count: 2<br/> 00010000:00000040:1.0:1399928810.278251:0:15665:0:(ldlm_resource.c:1262:ldlm_resource_putref()) putref res: ffff88016d6c25c0 count: 1<br/> 00000020:00000040:1.0:1399928810.278255:0:15665:0:(genops.c:814:class_export_put()) PUTting export ffff880528ffd400 : new refcount 7<br/> 00010000:00000040:1.0:1399928810.278258:0:15665:0:(ldlm_lib.c:2377:target_committed_to_req()) last_committed 38665664797, transno 0, xid 1467919094435880<br/> 00000100:00000040:1.0:1399928810.278263:0:15665:0:(connection.c:141:ptlrpc_connection_addref()) conn=ffff880605fb16c0 refcount 4 to 172.16.202.160@o2ib<br/> 00000100:00000040:1.0:1399928810.278266:0:15665:0:(niobuf.c:64:ptl_send_buf()) conn=ffff880605fb16c0 id 12345-172.16.202.160@o2ib<br/> 00000100:00000040:1.0:1399928810.278276:0:15665:0:(connection.c:127:ptlrpc_connection_put()) PUT conn=ffff880605fb16c0 refcount 3 to 172.16.202.160@o2ib<br/> 00000100:00000040:1.0:1399928810.278280:0:15665:0:(lustre_net.h:3283:ptlrpc_rqphase_move()) @@@ move req "Interpret" <del>&gt; "Complete" req@ffff88058e0fb000 x1467919094435880/t0(0) o3</del>&gt;4b54d2df-5ff6-9771-58f4-76c9ae6ad354@172.16.202.160@o2ib:0/0 lens 488/400 e 0 to 0 dl 1399928816 ref 1 fl Interpret:/0/0 rc 4096/4096<br/> 00000100:00000040:1.0:1399928810.278288:0:15665:0:(service.c:1046:ptlrpc_server_finish_active_request()) RPC PUTting export ffff880528ffd400 : new rpc_count 0</p> <p>Additional info:</p> <ul class="alternate" type="square"> <li>problem has been observed on clients with 12 cores and 16-24GB of RAM, easy to reproduce</li> <li>it never occured on clients with 64 cores and 256GB nor we were able to reproduce it on this type of node</li> </ul> <p>this line raised my eyebrows</p> <p>00000010:00008000:1.0:1399928810.277985:0:15665:0:(ost_handler.c:909:ost_brw_read()) checksum at read origin: ffffffff</p> <p>this shows that the checksum for the bulk data on the OST to be transferred to client is ~0, which makes the client checking code invalid</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>osc_brw_fini_request</b></div><div class="codeContent panelContent"> <pre class="code-java">... <span class="code-keyword">if</span> (server_cksum == ~0 &amp;&amp; rc &gt; 0) { CERROR(<span class="code-quote">"Protocol error: server %s set the <span class="code-quote">'checksum'</span> "</span> <span class="code-quote">"bit, but didn't send a checksum. Not fatal, "</span> <span class="code-quote">"but please notify on http:<span class="code-comment">//bugs.whamcloud.com/\n"</span>, </span> libcfs_nid2str(peer-&gt;nid)); } ... </pre> </div></div> <p>I am no expert here, but is it possible that the checksum of a chunk of data to be ~0?</p> <p>It looks like a kind of data corruption. After the second occurence of a log message gaussian application is exiting with I/O error:</p> <p><span class="error">&#91;pid 3283&#93;</span> read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1600) = 1600<br/> <span class="error">&#91;pid 3283&#93;</span> read(4, "\307\317\377\377\377\377\377\377\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0ZN phthalocyanine, MP2/6-31G* op"..., 40856) = 40856<br/> <span class="error">&#91;pid 3283&#93;</span> read(4, "&gt;\10]\341\275\240\1@\31-u\221%8\347?*\321\377n]\226\4@\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0C,\303Z\225@\2@\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1787568) = 1787568<br/> <span class="error">&#91;pid 3283&#93;</span> read(4, 0x7fff2f949b68, 8) = -1 EIO (Input/output error)</p> <p>If client read pops this error, the read could return error. Would you mind trying this patch to see whether this issue persists w/ it? <a href="http://review.whamcloud.com/10309" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10309</a></p> <p>Applied <a href="http://review.whamcloud.com/10309" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10309</a> to the client.<br/> Our testing shows that I/O issue is gone. </p> <p>Can we assume this patch as final solution? <br/> Are we sure that the checksum problem is not a result of a data corruption?</p> <p>Thanks</p> <p>I'm no expert on the checksum algorithm, if it's possible to get a ~0 checksum from the algorithm, I prefer to think that's not a data corruption here. What's your opinion from your gaussian application, does the data generated correct and valid?</p> <p>I will ask one of our users to verify sanity of results generated by a patched client. </p> <p>The most interesting thing is why AMD processor based client with 256GB ram with identical client version and<br/> identical inputs is not affected by ~0 problem.</p> <p>Assuming that CRC algorithm works in the same way on the client and server only explanation that come to my mind is the different data alignment/content of requests<br/> but i'm just a user here </p> <p>Is there anything else we can check here?</p> <p>this issue affects 2.5.1 clients also so i'd suggest updating affected version list </p> <p>I have found out the difference which makes AMD nodes not affected by the problem</p> <p>Intel nodes are using: crc32c<br/> AMD nodes default to: crc32</p> <p>As a workaround we switched Intel nodes to crc32 (which is a bit slower in that case) and problem is gone</p> <p>for i in /proc/fs/lustre/osc/*/checksum_type;do echo crc32 &gt; $i; done;</p> <p>Andreas,</p> <p>Is it possible that crc32c has an issue in this case?</p> <p>Lukaz, you will probably get better performance with adler32 for the checksum on the AMD nodes instead of crc32:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl set_param osc.*.checksum_type=adler32 </pre> </div></div> <p>Bobijam, I don't know enough details of the CRC32c checksum to say for sure if "0xffffffff" is a valid value or not. I know that for most checksums "0" is an invalid value because this would mean that the checksum for any length of zero bytes would be the same, which is why they always initialize the starting value to 0xffffffff. The use of ~0 for indicating the checksum is not sent was originally added in 1.4.x when the checksum feature was just new, and is no longer need. A mismatch between the client and server checksum (as is done with your patch) will detect corruption, and there is no reason to special-case the 0xffffffff checksum value.</p> <p>I've also submitted a version of your patch for master: <a href="http://review.whamcloud.com/10354" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10354</a></p> <p>@Andreas:<br/> It seems that crc32 is much better than adler on AMD 6276 cpus:</p> <p>Using crypto hash: crc32 (crc32-pclmul) speed 3196 MB/s<br/> Using crypto hash: adler32 (adler32-zlib) speed 2335 MB/s</p> <p>Patch landed to Master. Please reopen ticket if more work is needed.</p> <p>@Jodi: Could you please land the patch to b2_5 also? </p> <p>Yes. We are tracking this externally to land in b2_5 as well.<br/> Thank you!</p> Related Development Rank 1|hzwkon: Rank (Obsolete) 13646 Severity