Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Lustre 2.10.0, Lustre 2.11.0
-
None
-
3
-
9223372036854775807
Description
During large IOR testing with network fails introduced, Cray found a data corruption issues.
first issue is related to the 4MB BRW patchset and exist for long time. Bulk will be marked as failed just with real network error, but if one parts of data was lost and request timeout will treat as transfer done.
second issue is related with cleanup landed as commit 49d8a7ccd73 where "rc" parameter of obd_commit function was replaced with local data, it horror any errors before it.
Attachments
Activity
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/32906/
Subject: LU-11169 obdclass: fix old return code usage
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1db258b57e5669e07934fe848861817f88102475
NO. server side part is horror any error on server side, so commit write (who make write really) don't know about errors before it.
OSC checksum can detect client side problem, and "read" part of server side problem.
It should also cause the client to resend a write if one of the RDMAs was missing data (up to the 10x retry limit).
Patric,
In general, random data in file - as pages is unchanged on failed bulk transfer, but client / server think all is OK. zeros is just luck.
Ihara,
Basically, it sometimes fails to notice and resend when there's a transfer error. On reads, this shows up as zeroes read by the client, when the data on disk is correct. On writes, the result would be whatever data was present before the write (or, I believe, zeroes if the write is to a new region of the file).
it's depend of what you ask. Server side bug found as reading a zero's from file, after bulk transfer error.
Interesting. How is file corrupted? could you share exmaple of corrupted file?
Andreas,
you are right. OSC checksum can detect client side problem, and "read" part of server side problem.
In the case of 4MB RPCs that drop some part of the RDMA, doesn't the bulk RPC checksum detect this case and cause the RPC to be resent?
Alexey Lyashkov (c17817@cray.com) uploaded a new patch: https://review.whamcloud.com/32906
Subject: LU-11169 obdclass: fix old return code usage
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 028c5661270d922130d80074727c99b9292bd250
last patch is addressed a theoretical problem, which is impossible now.