https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9939 Lustre <p>A Lustre client on a KVM hypervisor using SR-IOV for IB has started to generate the following errors:</p> <p>OSS (oak-io1-s1 10.0.2.101@o2ib5):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Aug 31 11:27:04 oak-io1-s1 kernel: LustreError: 168-f: BAD WRITE CHECKSUM: oak-OST001a from 12345-10.0.2.225@o2ib5 inode [0x200002f84:0x6b6a:0x0] object 0x0:4413301 extent [726925312-727973887]: client csum 4ecd330, server csum 5610e5e5 </pre> </div></div> <p>The second OSS in production also has the same errors.</p> <p>SR-IOV based client (oak-gw06 10.0.2.225@o2ib5):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Aug 31 11:27:05 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b6a:0x0] object 0x0:4413301 extent [726925312-727973887] </pre> </div></div> <p>The client also gets some read checksum errors later:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Aug 31 11:37:42 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4413301 extent [1581252608-1582301183] </pre> </div></div> <p>I will attach kernel logs of both.</p> <p>In this particular case, the client is a Globus endpoint, using Lustre a the backend. This is actually the second time we've seen this, indeed the same issue was seen on another VM running rsnapshot jobs. Rebooting the impacted VM does fix the issue.</p> <p>Are you aware of such issues when using SR-IOV? Any idea how we could troubleshoot this?</p> <p>Thanks!<br/> Stephane Thiell</p> Lustre 2.9 including fixes for <a href="https://jira.whamcloud.com/browse/LU-8851" title="nodemap: add flags to limit mapping to UID or GID only" class="issue-link" data-issue-key="LU-8851"><strike>LU-8851</strike></a> (nodemap: add uid/gid only flags to control mapping) and <a href="https://jira.whamcloud.com/browse/LU-9258" title="nodemap: group quota ID not properly mapped " class="issue-link" data-issue-key="LU-9258"><strike>LU-9258</strike></a> (nodemap: group quota ID not properly mapped), kernel 3.10.0-514.16.1.el7_lustre.x86_64 on servers, 3.10.0-514.10.2.el7_lustre.x86_64 on clients LU-9939

Bad checksums from clients using SR-IOV

Bug Major Open Unresolved Bruno Faccini Stephane Thiell Fri, 1 Sep 2017 18:26:30 +0000 Mon, 6 Nov 2017 19:54:03 +0000 Lustre 2.9.0 0 6 <p>Hello Stephane!<br/> Can you check if your Lustre 2.9 version includes patch for <a href="https://jira.whamcloud.com/browse/LU-8376" title="Enhance debugging infos available for Lustre checksum errors" class="issue-link" data-issue-key="LU-8376"><del>LU-8376</del></a> or not?<br/> Because if yes, you should enable pages dump upon cksum error on both Client and OST sides and then may be have more infos to help find the cause of the error.</p> <p>Hey Bruno!</p> <p>That's very good to know! And no, I don't have the patch from <a href="https://jira.whamcloud.com/browse/LU-8376" title="Enhance debugging infos available for Lustre checksum errors" class="issue-link" data-issue-key="LU-8376"><del>LU-8376</del></a> in our 2.9, but we plan to upgrade to 2.10.1 in the near future, so I will just wait for that and then enable this new debugging option.</p> <p>Thanks!</p> <p>Stephane</p> <p>Ok, but on the other hand, I am sorry but I don't have any other option to debug and I did not find any similar report of checksum error running SR-IOV.</p> <p>Hi Bruno!</p> <p>The problem occurred again on a VM used as a Globus endpoint. The good news is that we are now running 2.10.1 on this system (clients and servers), so I did enable this checksum_dump thing and attached some of the resulting files to this ticket. Do you know how to troubleshoot this?</p> <p>Thanks!</p> <p>Stephane</p> <p>Hello Stephane,<br/> Tu nous a manque au dernier LAD!!<br/> Your tarball of dumps only contains files from Client side, so did you also enable dump of pages in a bulk xfer with cksum error on OSSs side? If yes, there should be corresponding dumps available on OSSs that are of interest for comparison.<br/> Concerned Clients and OSSs syslogs would be also helpful along with the striping infos of each affected files/FIDs, and the files content themselves if still available and not modified.</p> <p>Thanks Bruno! I'm still waiting to see a new occurrence to further troubleshoot this issue on the OSS side. I believe our Globus endpoint VMs have been less loaded lately, that might be why I haven't see the problem yet. I'll keep you posted.</p> Development Rank 1|hzzjgf: Rank (Obsolete) 9223372036854775807 Severity