[LU-11663] corrupt data after page-unaligned write with zfs backend lustre 2.10

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.12.0, Lustre 2.10.6
• Affects Version/s: Lustre 2.12.0, Lustre 2.10.5, Lustre 2.10.6
• Labels: llnl
• Environment: client catalyst: lustre-2.8.2_5.chaos-1.ch6.x86_64 ; server: porter lustre-2.10.5_2.chaos-3.ch6.x86_64 ; kernel-3.10.0-862.14.4.1chaos.ch6.x86_64 (RHEL 7.5 derivative)
• Severity: 2
• Assignee: Alex Zhuravlev
• Reporter: Olaf Faaland

Description

The apparent contents of a file change after dropping caches:

[root@catalyst110:toss-4371.umm1t]# ./proc6.olaf
+ dd if=/dev/urandom of=testfile20K.in bs=10240 count=2
2+0 records in
2+0 records out
20480 bytes (20 kB) copied, 0.024565 s, 834 kB/s
+ dd if=testfile20K.in of=testfile20K.out bs=10240 count=2
2+0 records in
2+0 records out
20480 bytes (20 kB) copied, 0.0451045 s, 454 kB/s
++ md5sum testfile20K.out
+ original_md5sum='1060a4c01a415d7c38bdd00dcf09dd22  testfile20K.out'
+ echo 3
++ md5sum testfile20K.out
+ echo after drop_caches 1060a4c01a415d7c38bdd00dcf09dd22 testfile20K.out 717122f4dd25f2e75834a8b21c79ce50 testfile20K.out
after drop_caches 1060a4c01a415d7c38bdd00dcf09dd22 testfile20K.out 717122f4dd25f2e75834a8b21c79ce50 testfile20K.out                                                                        

[root@catalyst110:toss-4371.umm1t]# cat proc6.olaf
#!/bin/bash

set -x

dd if=/dev/urandom of=testfile.in bs=10240 count=2
dd if=testfile.in of=testfile.out bs=10240 count=2

#dd if=/dev/urandom of=testfile.in bs=102400 count=2
#dd if=testfile.in of=testfile.out bs=102400 count=2
original_md5sum=$(md5sum testfile.out)
echo 3 >/proc/sys/vm/drop_caches

echo after drop_caches $original_md5sum $(md5sum testfile.out)

Issue Links

• is related to: LU-11697 BAD WRITE CHECKSUM with t10ip4K and t10ip512 checksums
• is related to: LU-11729 ARM: sanity test_810: BAD WRITE CHECKSUM with adler
• is related to: LU-10683 write checksum errors
• is related to: LU-11798 cur_grant goes to 0 and never increases with 2.8 client and 2.10 server

Comments

[Olaf Faaland]

Patrick, I opened a ticket re: grant going to 0, it is https://jira.whamcloud.com/browse/LU-11798

[Peter Jones]

Fix landed for 2.12 and 2.10.6. Checksum issues for master will be covered under LU-11697

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33748/
Subject: LU-11663 osd-zfs: write partial pages with correct offset
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 18d6b8fb2c359431a6da57b178ec0845925ea89c

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33726/
Subject: LU-11663 osd-zfs: write partial pages with correct offset
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c038909fbc2b3b14763dd731e9c877d11d338f04

[Li Xi]

As commented in LU-11697, the correct page offset in lnb_page_offset is the reason why LU-10683 oatch fixed the RPC checksum error. Both osc_checksum_bulk() and tgt_checksum_niobuf() assume the page offsets are properly inited and should be equal to each other.

[Alex Zhuravlev]

well, from filesystem point of view, there is no requirement to use same page offset. moreover, client and server may have different pagesize, which makes it impossible to match offset, right?

[Li Xi]

I feel between the two options that Alex pointed out, reverting the patch of LU-10683 is not a good one. The lnb_page_offset should be the same with the client side page offset in 'struct brw_page', shouldn't it? It doesn't feel right to move the data to the offset 0 of a page when the data has an offset in the page.

[Alex Zhuravlev]

there are two options here: 1) revert LU-10683 (but potentially get bad RPC checksum messages back) 2) apply https://review.whamcloud.com/#/c/33726/ patch which is still under testing.
both options have worked against our reproducer (see in option #2 patch) on b2_10
we are still investigating the root cause for LU-10683 (bad checksums)

[Gerrit Updater]

Oleg Drokin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33748
Subject: LU-11663 osd-zfs: write partial pages with correct offset
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 6f9a0292eacb0d603b14cc03290a574cb7f0c846

[Peter Jones]

Strange. Alex's patch did not get an auto comment - https://review.whamcloud.com/#/c/33726/. As I understand it, this patch seems to be holding up well  against the reproducer but the test cases need some refinement. Are we now at the point when LLNL can use a b2_10 port of this patch on their affected filesystem?