[LU-11697] BAD WRITE CHECKSUM with t10ip4K and t10ip512 checksums

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.12.0
• Affects Version/s: Lustre 2.12.0
• Severity: 3
• Assignee: Li Xi
• Reporter: Alex Zhuravlev

Description

I'm running racer and get the following messages quite often. I think Oleg can confirm this.
if I set checksum_type to adler, then racer runs fine with no single bad checksum.

LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200000402:0x3b89:0x0] object 0x0:1921 extent [326-4194303]: client csum 53f1f04e, server csum 9bf3f057
LustreError: 132-0: lustre-OST0000-osc-ffff8801d25e8800: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200000402:0x3b89:0x0] object 0x0:1921 extent [326-4194303], original client csum 53f1f04e (type 20), server csum 9bf3f057 (type 20), client csum now 53f1f04e

Issue Links

• is related to: LU-11729 ARM: sanity test_810: BAD WRITE CHECKSUM with adler
• is related to: LU-10472 Data Integrity(T10PI) support for Lustre
• is related to: LU-11663 corrupt data after page-unaligned write with zfs backend lustre 2.10

Comments

[Li Xi]

I think multiple problems mixed here: LU-10683 and LU-11663. I am confused how to reproduce the problem.

[Li Xi]

Hi Alex, I am confused. Why reverting LU-10683 helps on reproducingt he problem? I thought the test script of LU-10683 triggers the checksum error, right? RPC checksum should not be related to the OSD level bug.

[Alex Zhuravlev]

lixi_wc, revert 83cb17031913ba2f33a5b67219a03c5605f48f27 (LU-10683) and try that test, please.

[Li Xi]

@Alex, I can not reproduce the problem even with only patch 33727. Would you please let me know what are the exact steps to reproduce?

[Li Xi]

With patch 33752 on top of 33727, I can not reproduce the problem. I am not sure I missed anything or not, since I didn't test the original Lustre without these patches.

# df | grep mnt
/dev/sdb1                102275928     62196   96970852   1% /mnt/mgs
/dev/sdb2                120662808     75284  110101768   1% /mnt/mdt
/dev/sdb3                170773236     61960  162004256   1% /mnt/ost
10.0.0.37@tcp:/server17  170773236     61960  162004256   1% /mnt/lustre
# lfs getstripe file
file
lmm_stripe_count:  1
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 0
        obdidx           objid           objid           group
             0               2            0x2                0

# cat /proc/fs/lustre/osc/server17-OST0000-osc-ffff8805c03ff000/checksum_type 
crc32 adler crc32c t10ip512 [t10ip4K] t10crc512 t10crc4K 
# lctl set_param fail_loc=0x411
fail_loc=0x411
# dd if=/dev/urandom of=file bs=10240 count=2
2+0 records in
2+0 records out
20480 bytes (20 kB) copied, 0.0139395 s, 1.5 MB/s
# md5sum file 
5f67fa4f3b81beaab254a36276f659ce  file
# lctl set_param ldlm.namespaces.*osc*.lru_size=clear
ldlm.namespaces.server17-OST0000-osc-MDT0000.lru_size=clear
ldlm.namespaces.server17-OST0000-osc-ffff8805c03ff000.lru_size=clear
# md5sum file 
5f67fa4f3b81beaab254a36276f659ce  file

[Li Xi]

Please check whether the patch 33752 on top of 33727 works well. I am setuping an environment to test it. And also, setuping a environment with T10PI hardware to check whether everything works.

[Gerrit Updater]

Li Xi ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33752
Subject: LU-11697 obdclass: generate T10PI correctly for unaligned buffer
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 267bc620de222b23ad250f300b8ac0ad796e52f2

[Li Xi]

obd_page_dif_generate_buffer() doesn't handle the non-aligned buffer well. I am checking how T10PI handles the unaligned buffer.

[Alex Zhuravlev]

lixi_wc, can you please explain why it fails depending on page offset? if we put data (2K, iirc) at 2K offset, then the checksums match, but if we put data at 0 offset, then the checksums don't match. probably I missed something, of course..

[Li Xi]

The sanity:810 test that https://review.whamcloud.com/#/c/33726/ adds is still 512 bytes aligned. I guess that is why T10PI512 passes the test, but T10PI4096 doesn't. If the size is not 512 bytes aliged, I guess T10PI512 will fail too.

[Alex Zhuravlev]

adilger, yes it still fails with 4K version, but passes with 512.