https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11169 Lustre <p>During large IOR testing with network fails introduced, Cray found a data corruption issues.<br/> first issue is related to the 4MB BRW patchset and exist for long time. Bulk will be marked as failed just with real network error, but if one parts of data was lost and request timeout will treat as transfer done.</p> <p>second issue is related with cleanup landed as commit 49d8a7ccd73 where "rc" parameter of obd_commit function was replaced with local data, it horror any errors before it. </p> LU-11169

Data corruption during IOR testing with network error simulation

Bug Critical Resolved Fixed Alexey Lyashkov Alexey Lyashkov Tue, 24 Jul 2018 11:33:41 +0000 Fri, 19 Jul 2019 11:14:11 +0000 Wed, 19 Dec 2018 21:08:40 +0000 Lustre 2.10.0 Lustre 2.11.0 Lustre 2.12.0 0 6 <p>Alexey Lyashkov (c17817@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/32905" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32905</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11169" title="Data corruption during IOR testing with network error simulation" class="issue-link" data-issue-key="LU-11169"><del>LU-11169</del></a> ptlrpc: don't treat bulk is ok<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3c26d98e15582b9fc529e220c47e92be0acec5c8</p> <p>Alexey Lyashkov (c17817@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/32906" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32906</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11169" title="Data corruption during IOR testing with network error simulation" class="issue-link" data-issue-key="LU-11169"><del>LU-11169</del></a> obdclass: fix old return code usage<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 028c5661270d922130d80074727c99b9292bd250</p> <p>In the case of 4MB RPCs that drop some part of the RDMA, doesn't the bulk RPC checksum detect this case and cause the RPC to be resent?</p> <p>Andreas,</p> <p>you are right. OSC checksum can detect client side problem, and "read" part of server side problem.</p> <p>Interesting. How is file corrupted? could you share exmaple of corrupted file?</p> <p>it's depend of what you ask. Server side bug found as reading a zero's from file, after bulk transfer error.</p> <p>Ihara,</p> <p>Basically, it sometimes fails to notice and resend when there's a transfer error.  On reads, this shows up as zeroes read by the client, when the data on disk is correct.  On writes, the result would be whatever data was present before the write (or, I believe, zeroes if the write is to a new region of the file).</p> <p>Patric, </p> <p>In general, random data in file - as pages is unchanged on failed bulk transfer, but client / server think all is OK. zeros is just luck.</p> <blockquote> <p>OSC checksum can detect client side problem, and "read" part of server side problem.</p></blockquote> <p>It should also cause the client to resend a write if one of the RDMAs was missing data (up to the 10x retry limit).</p> <p>NO. server side part is horror any error on server side, so commit write (who make write really) don't know about errors before it.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32906/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32906/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11169" title="Data corruption during IOR testing with network error simulation" class="issue-link" data-issue-key="LU-11169"><del>LU-11169</del></a> obdclass: fix old return code usage<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1db258b57e5669e07934fe848861817f88102475</p> <p>last patch is addressed a theoretical problem, which is impossible now.</p> <p>Alexandr Boyko (c17825@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/35571" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35571</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11169" title="Data corruption during IOR testing with network error simulation" class="issue-link" data-issue-key="LU-11169"><del>LU-11169</del></a> ptlrpc: handle reply and resend reorder<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 88c2a9a0c840d01d50762a71f04319e58c9affef</p> Development Rank 1|hzzznr: Rank (Obsolete) 9223372036854775807 Severity