https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11697 Lustre <p>I'm running racer and get the following messages quite often. I think Oleg can confirm this.<br/> if I set checksum_type to adler, then racer runs fine with no single bad checksum.</p> <p>LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-0@lo inode <span class="error">&#91;0x200000402:0x3b89:0x0&#93;</span> object 0x0:1921 extent <span class="error">&#91;326-4194303&#93;</span>: client csum 53f1f04e, server csum 9bf3f057<br/> LustreError: 132-0: lustre-OST0000-osc-ffff8801d25e8800: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode <span class="error">&#91;0x200000402:0x3b89:0x0&#93;</span> object 0x0:1921 extent <span class="error">&#91;326-4194303&#93;</span>, original client csum 53f1f04e (type 20), server csum 9bf3f057 (type 20), client csum now 53f1f04e</p> LU-11697

BAD WRITE CHECKSUM with t10ip4K and t10ip512 checksums

Bug Blocker Resolved Fixed Li Xi Alex Zhuravlev Fri, 23 Nov 2018 18:29:05 +0000 Tue, 4 Dec 2018 23:45:07 +0000 Tue, 4 Dec 2018 23:45:07 +0000 Lustre 2.12.0 Lustre 2.12.0 0 10 <p>Li Xi</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>can't reproduce with crc32<br/> yet another issue is that massive resends (due to bad checksums) lead to evictions which is not correct.<br/> with adler/crc32 I have no evictions.</p> <p>The evictions are likely caused by the resends due to checksum errors. If adler/crc32 do not generate checksum errors then they do not resend and are not evicted. It may be that the RPC resends are not being counted by the OST as "forward progress" for a DLM lock being cancelled, so the client is evicted due to non-responsiveness?</p> <p>In that case, it makes sense to update the DLM lock timer due to the resends so that the client is not evicted if the network is having problems.</p> <p>According to Oleg, there are checksum errors with other RPC types as well when running racer. It isn't clear why this is only affecting Alex with t10ip checksums.</p> <p>Remove the 2.12 tag for now. If this becomes a problem in the field (which is unlikely if it only appears during racer), then a different checksum type can be selected.</p> <p>I have this problem too.</p> <p>I applied this patch and all the RPC checksums went away:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/lustre/include/obd_cksum.h b/lustre/include/obd_cksum.h index a2ce2ec..bcb23e5 100644 --- a/lustre/include/obd_cksum.h +++ b/lustre/include/obd_cksum.h @@ -98,7 +98,7 @@ static inline enum cksum_types obd_cksum_types_supported_clien ret |= OBD_CKSUM_CRC32; /* Client support all kinds of T10 checksum */ - ret |= OBD_CKSUM_T10_ALL; + ret |= 0; //OBD_CKSUM_T10_ALL; return ret; } </pre> </div></div> <p>Also without the patch above I am having sporadic patches in checksum that seems to imply there are problems in this area that are deeper:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[80863.424036] BUG: unable to handle kernel paging request at ffff880229b1a000 [80863.476713] IP: [&lt;ffffffff813ee500&gt;] do_csum+0x70/0x180 [80863.478571] PGD 241b067 PUD 33effc067 PMD 33eeae067 PTE 8000000229b1a060 [80863.478571] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC [80863.483994] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) mdc(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) crc_t10dif crct10dif_generic crct10dif_common pcspkr i2c_piix4 virtio_balloon virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw virtio_blk libata i2c_core floppy [last unloaded: libcfs] [80863.503099] CPU: 8 PID: 24979 Comm: ll_ost_io04_005 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.6-debug #1 [80863.503099] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [80863.503099] task: ffff880303c42900 ti: ffff880089ff4000 task.ti: ffff880089ff4000 [80863.503099] RIP: 0010:[&lt;ffffffff813ee500&gt;] [&lt;ffffffff813ee500&gt;] do_csum+0x70/0x180 [80863.503099] RSP: 0018:ffff880089ff7938 EFLAGS: 00010246 [80863.503099] RAX: 0000000000000000 RBX: ffff880229b1a000 RCX: 0000000000000040 [80863.503099] RDX: ffff880229b1a000 RSI: 0000000000001000 RDI: ffff880229b1a000 [80863.503099] RBP: ffff880089ff7938 R08: 0000000000000000 R09: 0000000000000000 [80863.503099] R10: 0000000000000040 R11: 0000000000000200 R12: 0000000000001000 [80863.503099] R13: 0000000000001000 R14: 0000000000000001 R15: 0000000000001000 [80863.587717] FS: 0000000000000000(0000) GS:ffff88033dc00000(0000) knlGS:0000000000000000 [80863.587717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [80863.587717] CR2: ffff880229b1a000 CR3: 00000000a792e000 CR4: 00000000000006e0 [80863.587717] Call Trace: [80863.599291] [&lt;ffffffff813ee61e&gt;] ip_compute_csum+0xe/0x30 [80863.642807] [&lt;ffffffffa0572e1e&gt;] obd_dif_ip_fn+0xe/0x10 [obdclass] [80863.642807] [&lt;ffffffffa0572ee9&gt;] obd_page_dif_generate_buffer+0xc9/0x190 [obdclass] [80863.642807] [&lt;ffffffffa07a5e3f&gt;] tgt_checksum_niobuf_rw+0x27f/0xea0 [ptlrpc] [80863.642807] [&lt;ffffffffa0572e10&gt;] ? obd_dif_crc_fn+0x20/0x20 [obdclass] [80863.642807] [&lt;ffffffffa0572e10&gt;] ? obd_dif_crc_fn+0x20/0x20 [obdclass] [80863.642807] [&lt;ffffffffa07a842d&gt;] tgt_brw_read+0xc2d/0x1e60 [ptlrpc] [80863.642807] [&lt;ffffffff812127f4&gt;] ? __kmalloc+0x634/0x660 [80863.736021] [&lt;ffffffff813eca64&gt;] ? vsnprintf+0x234/0x6a0 [80863.736021] [&lt;ffffffffa0540119&gt;] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [80863.742878] [&lt;ffffffffa0744fe6&gt;] ? lustre_pack_reply_v2+0x166/0x290 [ptlrpc] [80863.742878] [&lt;ffffffffa074517f&gt;] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [80863.765470] [&lt;ffffffffa0745301&gt;] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [80863.765470] [&lt;ffffffffa07ac365&gt;] tgt_request_handle+0xaf5/0x1590 [ptlrpc] [80863.765470] [&lt;ffffffffa01f9fa7&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] [80863.765470] [&lt;ffffffffa0750436&gt;] ptlrpc_server_handle_request+0x256/0xad0 [ptlrpc] [80863.775960] [&lt;ffffffff810bfbd8&gt;] ? __wake_up_common+0x58/0x90 [80863.775960] [&lt;ffffffff813fb7bb&gt;] ? do_raw_spin_unlock+0x4b/0x90 [80863.775960] [&lt;ffffffffa0754329&gt;] ptlrpc_main+0xa99/0x1f60 [ptlrpc] [80863.775960] [&lt;ffffffff810c32ed&gt;] ? finish_task_switch+0x5d/0x1b0 [80863.775960] [&lt;ffffffffa0753890&gt;] ? ptlrpc_register_service+0xfb0/0xfb0 [ptlrpc] [80863.775960] [&lt;ffffffff810b4ed4&gt;] kthread+0xe4/0xf0 [80863.775960] [&lt;ffffffff810b4df0&gt;] ? kthread_create_on_node+0x140/0x140 [80863.775960] [&lt;ffffffff817c4c77&gt;] ret_from_fork_nospec_begin+0x21/0x21 [80863.775960] [&lt;ffffffff810b4df0&gt;] ? kthread_create_on_node+0x140/0x140 [80863.775960] Code: 00 00 00 40 f6 c7 04 0f 85 de 00 00 00 41 89 d3 c1 ea 04 41 d1 eb 85 d2 41 89 d2 74 48 89 d1 45 31 c0 48 89 fa 66 0f 1f 44 00 00 &lt;48&gt; 03 02 48 13 42 08 48 13 42 10 48 13 42 18 48 13 42 20 48 13 </pre> </div></div> <p>Though I also had evidently a similar crc dump in september (but the t10 patch reowrking this area landed in June I think):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[37976.307908] IP: [&lt;ffffffff813c0ba0&gt;] do_csum+0x70/0x180 [37976.307908] PGD 23e3067 PUD 33edfb067 PMD 33ed49067 PTE 8000000256203060 [37976.307908] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC [37976.307908] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) mdc(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc_t10dif crct10dif_generic crct10dif_common ata_generic pata_acpi ttm drm_kms_helper drm ata_piix i2c_piix4 virtio_console libata serio_raw pcspkr virtio_balloon virtio_blk i2c_core floppy ip_tables rpcsec_gss_krb5 [last unloaded: libcfs] [37976.307908] CPU: 10 PID: 12741 Comm: ll_ost_io05_004 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.5-debug #1 [37976.349349] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [37976.349349] task: ffff8800a77e6600 ti: ffff88026c85c000 task.ti: ffff88026c85c000 [37976.349349] RIP: 0010:[&lt;ffffffff813c0ba0&gt;] [&lt;ffffffff813c0ba0&gt;] do_csum+0x70/0x180 [37976.349349] RSP: 0018:ffff88026c85f940 EFLAGS: 00010246 [37976.349349] RAX: 0000000000000000 RBX: ffff880256203000 RCX: 0000000000000040 [37976.349349] RDX: ffff880256203000 RSI: 0000000000001000 RDI: ffff880256203000 [37976.349349] RBP: ffff88026c85f940 R08: 0000000000000000 R09: 0000000000000000 [37976.349349] R10: 0000000000000040 R11: 0000000000000200 R12: 0000000000001000 [37976.349349] R13: 0000000000001000 R14: 0000000000000001 R15: 0000000000001000 [37976.349349] FS: 0000000000000000(0000) GS:ffff88033dc80000(0000) knlGS:0000000000000000 [37976.349349] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [37976.349349] CR2: ffff880256203000 CR3: 00000000933de000 CR4: 00000000000006e0 [37976.349349] Call Trace: [37976.349349] [&lt;ffffffff813c0cbe&gt;] ip_compute_csum+0xe/0x30 [37976.349349] [&lt;ffffffffa09216fe&gt;] obd_dif_ip_fn+0xe/0x10 [obdclass] [37976.349349] [&lt;ffffffffa09217c9&gt;] obd_page_dif_generate_buffer+0xc9/0x190 [obdclass] [37976.349349] [&lt;ffffffffa09216f0&gt;] ? obd_dif_crc_fn+0x20/0x20 [obdclass] [37976.349349] [&lt;ffffffffa0ba4320&gt;] tgt_checksum_niobuf_rw+0x310/0xd60 [ptlrpc] [37976.349349] [&lt;ffffffffa09216f0&gt;] ? obd_dif_crc_fn+0x20/0x20 [obdclass] [37976.349349] [&lt;ffffffffa0b6947d&gt;] ? __req_capsule_get+0x15d/0x700 [ptlrpc] [37976.349349] [&lt;ffffffffa0ba6b40&gt;] tgt_brw_read+0xc40/0x1e80 [ptlrpc] [37976.349349] [&lt;ffffffff811eae74&gt;] ? __kmalloc+0x634/0x660 [37976.349349] [&lt;ffffffff813bf104&gt;] ? vsnprintf+0x234/0x6a0 [37976.349349] [&lt;ffffffffa08ee9c9&gt;] ? lprocfs_counter_add+0xf9/0x160 [obdclass] [37976.349349] [&lt;ffffffffa0b7b600&gt;] ? null_alloc_rs+0x110/0x330 [ptlrpc] [37976.349349] [&lt;ffffffffa0b44a66&gt;] ? lustre_pack_reply_v2+0x166/0x290 [ptlrpc] [37976.349349] [&lt;ffffffffa0b44bff&gt;] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [37976.349349] [&lt;ffffffffa0b44d81&gt;] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [37976.349349] [&lt;ffffffffa0bab755&gt;] tgt_request_handle+0xaf5/0x1590 [ptlrpc] [37976.349349] [&lt;ffffffffa07b4017&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] [37976.349349] [&lt;ffffffffa0b4fea6&gt;] ptlrpc_server_handle_request+0x256/0xad0 [ptlrpc] [37976.349349] [&lt;ffffffff810b9398&gt;] ? __wake_up_common+0x58/0x90 [37976.349349] [&lt;ffffffff813ccd2b&gt;] ? do_raw_spin_unlock+0x4b/0x90 [37976.349349] [&lt;ffffffffa0b53c9e&gt;] ptlrpc_main+0xabe/0x1f80 [ptlrpc] [37976.349349] [&lt;ffffffffa0b531e0&gt;] ? ptlrpc_register_service+0xeb0/0xeb0 [ptlrpc] [37976.349349] [&lt;ffffffff810ae864&gt;] kthread+0xe4/0xf0 [37976.349349] [&lt;ffffffff810ae780&gt;] ? kthread_create_on_node+0x140/0x140 [37976.349349] [&lt;ffffffff81783777&gt;] ret_from_fork_nospec_begin+0x21/0x21 [37976.349349] [&lt;ffffffff810ae780&gt;] ? kthread_create_on_node+0x140/0x140 [37976.349349] Code: 00 00 00 40 f6 c7 04 0f 85 de 00 00 00 41 89 d3 c1 ea 04 41 d1 eb 85 d2 41 89 d2 74 48 89 d1 45 31 c0 48 89 fa 66 0f 1f 44 00 00 &lt;48&gt; 03 02 48 13 42 08 48 13 42 10 48 13 42 18 48 13 42 20 48 13 [37976.349349] RIP [&lt;ffffffff813c0ba0&gt;] do_csum+0x70/0x180 </pre> </div></div> <p>Most interesting. I remember Oleg running into this problem with patch <a href="https://review.whamcloud.com/#/c/33363/." class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/33363.</a></p> <p>I couldn't see why this error was showing up so easily with that patch. This might be the cause.</p> <p>I am checking whether there is any difference between tgt_checksum_niobuf_t10pi() and osc_checksum_bulk_t10pi(). The extent range is <span class="error">&#91;326-4194303&#93;</span>, which is not 4K aligned. I guess there might be some corner cases (e.g. page not zeroed) which might cause the problem.</p> <p>I saw a similar problem with non-aligned writes where the bulk puts data into a page with non-zero offset. this can't be reproduced with adler algo. probably t10 code doesn't take lnb_page_offset into account (like osd-zfs didn't).</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/33727" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33727</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11697" title="BAD WRITE CHECKSUM with t10ip4K and t10ip512 checksums" class="issue-link" data-issue-key="LU-11697"><del>LU-11697</del></a> osc: wrong page offset for T10PI checksum<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: fcc175d6554adaddb56babf9a46510f86410aeb9</p> <p>I found a problem of osc_checksum_bulk_t10pi(). Alex, would you please check whether it helps?</p> <p>please, try with test in <a href="https://review.whamcloud.com/#/c/33726/2/lustre/tests/sanity.sh" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/33726/2/lustre/tests/sanity.sh</a><br/> (don't forget to remove lctl set_param osc.*.checksum_type=adler in the test)</p> <p>It would be best to rebase 33727 on top of 33726, and then modify the new sanity test to run through all of the checksum types.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=lixi_wc" class="user-hover" rel="lixi_wc">lixi_wc</a>, is page offset used in T10 algo?<br/> it looks like the checksum is different if page offset on the server and the client don't match. other algorithms (e.g. adler) seem to be insensitive to this.</p> <p>Hi Alex, page offset is not used in T10 algorithm.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=lixi_wc" class="user-hover" rel="lixi_wc">lixi_wc</a> then please try to run the test mentioned above. basically the client sends partial non-aligned page, but the server puts data with page offset=0. </p> <p>so that test from <a href="https://jira.whamcloud.com/browse/LU-11663" title="corrupt data after page-unaligned write with zfs backend lustre 2.10" class="issue-link" data-issue-key="LU-11663"><del>LU-11663</del></a> passes with t10ip512, but not with 4K. how 4K version is supposed to work with partial pages?</p> <p>Alex, is that still failing with the 33727 patch applied on top of your 33726 patch?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a>, yes it still fails with 4K version, but passes with 512.</p> <p>The sanity:810 test that <a href="https://review.whamcloud.com/#/c/33726/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/33726/</a> adds is still 512 bytes aligned. I guess that is why T10PI512 passes the test, but T10PI4096 doesn't. If the size is not 512 bytes aliged, I guess T10PI512 will fail too.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=lixi_wc" class="user-hover" rel="lixi_wc">lixi_wc</a>, can you please explain why it fails depending on page offset? if we put data (2K, iirc) at 2K offset, then the checksums match, but if we put data at 0 offset, then the checksums don't match. probably I missed something, of course..</p> <p>obd_page_dif_generate_buffer() doesn't handle the non-aligned buffer well. I am checking how T10PI handles the unaligned buffer.</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/33752" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33752</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11697" title="BAD WRITE CHECKSUM with t10ip4K and t10ip512 checksums" class="issue-link" data-issue-key="LU-11697"><del>LU-11697</del></a> obdclass: generate T10PI correctly for unaligned buffer<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 267bc620de222b23ad250f300b8ac0ad796e52f2</p> <p>Please check whether the patch 33752 on top of 33727 works well. I am setuping an environment to test it. And also, setuping a environment with T10PI hardware to check whether everything works.</p> <p>With patch 33752 on top of 33727, I can not reproduce the problem. I am not sure I missed anything or not, since I didn't test the original Lustre without these patches.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # df | grep mnt /dev/sdb1 102275928 62196 96970852 1% /mnt/mgs /dev/sdb2 120662808 75284 110101768 1% /mnt/mdt /dev/sdb3 170773236 61960 162004256 1% /mnt/ost 10.0.0.37@tcp:/server17 170773236 61960 162004256 1% /mnt/lustre # lfs getstripe file file lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2 0x2 0 # cat /proc/fs/lustre/osc/server17-OST0000-osc-ffff8805c03ff000/checksum_type crc32 adler crc32c t10ip512 [t10ip4K] t10crc512 t10crc4K # lctl set_param fail_loc=0x411 fail_loc=0x411 # dd <span class="code-keyword">if</span>=/dev/urandom of=file bs=10240 count=2 2+0 records in 2+0 records out 20480 bytes (20 kB) copied, 0.0139395 s, 1.5 MB/s # md5sum file 5f67fa4f3b81beaab254a36276f659ce file # lctl set_param ldlm.namespaces.*osc*.lru_size=clear ldlm.namespaces.server17-OST0000-osc-MDT0000.lru_size=clear ldlm.namespaces.server17-OST0000-osc-ffff8805c03ff000.lru_size=clear # md5sum file 5f67fa4f3b81beaab254a36276f659ce file </pre> </div></div> <p>@Alex, I can not reproduce the problem even with only patch 33727. Would you please let me know what are the exact steps to reproduce?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=lixi_wc" class="user-hover" rel="lixi_wc">lixi_wc</a>, revert 83cb17031913ba2f33a5b67219a03c5605f48f27 (<a href="https://jira.whamcloud.com/browse/LU-10683" title="write checksum errors" class="issue-link" data-issue-key="LU-10683"><del>LU-10683</del></a>) and try that test, please.</p> <p>Hi Alex, I am confused. Why reverting <a href="https://jira.whamcloud.com/browse/LU-10683" title="write checksum errors" class="issue-link" data-issue-key="LU-10683"><del>LU-10683</del></a> helps on reproducingt he problem? I thought the test script of <a href="https://jira.whamcloud.com/browse/LU-10683" title="write checksum errors" class="issue-link" data-issue-key="LU-10683"><del>LU-10683</del></a> triggers the checksum error, right? RPC checksum should not be related to the OSD level bug.</p> <p>I think multiple problems mixed here: <a href="https://jira.whamcloud.com/browse/LU-10683" title="write checksum errors" class="issue-link" data-issue-key="LU-10683"><del>LU-10683</del></a> and <a href="https://jira.whamcloud.com/browse/LU-11663" title="corrupt data after page-unaligned write with zfs backend lustre 2.10" class="issue-link" data-issue-key="LU-11663"><del>LU-11663</del></a>. I am confused how to reproduce the problem. </p> <p>I can't reproduce the problem using sanity:810 test even with no patch of this ticket. So, what is the exact process of reproducing the problem?</p> <p>This issue can be reproduced with checksum type "t10ip512", "t10ip4K", "t10crc512" and "t10crc4K", and won't be<br/> reproduced by checksum type "crc32", "adler", and "crc32c", the reason is the data offset in pages (in this case, the offset of<br/> the first page) is taken into account during calculating the "t10*" series checksum.</p> <p>the patch from <a href="https://jira.whamcloud.com/browse/LU-10683" title="write checksum errors" class="issue-link" data-issue-key="LU-10683"><del>LU-10683</del></a> fixed the issue by aligning the data with the client, but it introduced another problem when<br/> committing the data into ZFS backend.</p> <p>Hongchao and Alex, I don't understand why <a href="https://jira.whamcloud.com/browse/LU-10683" title="write checksum errors" class="issue-link" data-issue-key="LU-10683"><del>LU-10683</del></a> should be reverted. If <a href="https://jira.whamcloud.com/browse/LU-10683" title="write checksum errors" class="issue-link" data-issue-key="LU-10683"><del>LU-10683</del></a> is reverted, it will cause inconsistent offset in page between OSD and OSC, i.e. lnb_page_offset != 'struct brw_page'-&gt;off. And that would certainly means checksum error of none-page-aligned data for all checksum types, including crc32, adler and crc32c, because the checksums will be calculated from different start points between OST and OSC. Both osc_checksum_bulk() and tgt_checksum_niobuf() assumes the page offset is properly inited and should be equal to each other.</p> <p>That said, I will try to reproduce the probelm without reverting <a href="https://jira.whamcloud.com/browse/LU-10683" title="write checksum errors" class="issue-link" data-issue-key="LU-10683"><del>LU-10683</del></a>. Reverting <a href="https://jira.whamcloud.com/browse/LU-10683" title="write checksum errors" class="issue-link" data-issue-key="LU-10683"><del>LU-10683</del></a> and reproduce this problem doesn't look correct.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=lixi_wc" class="user-hover" rel="lixi_wc">lixi_wc</a> how current approach (which requires matching offsets) works on setups where page size between client and server doesn't match?</p> <p>Hi Alex, good question. I did assume that page sizes are the same on clients and servers. However, even page sizes are different, it is still the same.</p> <p>In tgt_brw_write(), the server asked to transfer the data into the page with lnb_page_offset. And thus, the server should use this offset of the page to access the data, including writing to disk and calculating the RPC checksum.</p> <p>so there should be no issue whether server put data (and write from) offset=1K or offset=0K, right?</p> <p>Hi Alex, I guess this might be related to the characteristics of checksum algorithms? I am not an expert on this, but I guess not all checksum types generate the same value for the following two process:</p> <p>1) Calculate checksum of 3K byte and them accumulate checksum of 1K byte <br/> 2) Calculate checksum of 4K byte data</p> <p>Even the 3K byte data + 1K byte data = 4 K byte data, the checksum values of 1) and 2) might be different for some checksum types, right? I am not sure about whether T10PI checksum or crc32, crc32c supports this feature. But let's assume there is an algorithm which doesn't support this feature.</p> <p>Then, if we don't apply the patch of <a href="https://jira.whamcloud.com/browse/LU-10683" title="write checksum errors" class="issue-link" data-issue-key="LU-10683"><del>LU-10683</del></a>, two pages with size 1K byte and 3Kbyte on client side might be changed to one page of 4K bytes on server side. Maybe checksum calculation might be different for these two cases?</p> <p>The problem <a href="https://jira.whamcloud.com/browse/LU-10683" title="write checksum errors" class="issue-link" data-issue-key="LU-10683"><del>LU-10683</del></a> shows with crc32c checksum, right? Since the checksum type is OBD_CKSUM_CRC32C(4). We really need to understand whether crc32c and T10PI supports this feature. Otherwise, checksum corruption might happen in the case of 1) non-page aligned I/O 2) page size differences between client and server...</p> <p>I tend to believe 4K T10PI checksum doesn't support this feature, since it should calculate the whole page for each sector. Thus, merging data belongs to two pages on server side will corrupt the checksum for sure...</p> <p>Hmm, I think <a href="https://review.whamcloud.com/#/c/32899" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/32899</a> is still needed, because of the reason that I said. And I think the key reason here is not the line of "lnb<span class="error">&#91;i&#93;</span>.lnb_page_offset = poff;"<br/> It can be changed to "lnb<span class="error">&#91;i&#93;</span>.lnb_page_offset = 0;" and the checksum should still be the same. However, the most important codes are:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> plen = min_t(<span class="code-object">int</span>, poff + sz_in_block, PAGE_SIZE); plen -= poff; </pre> </div></div> <p>This fixes the problem of merging two pages into a single one.</p> <p>There is still a problem with T10-PI checksums for partial pages, and I don't think that <a href="https://review.whamcloud.com/33752" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33752</a> is fixing the problem properly. That patch makes the T10-PI checksums consistent between the client and the server, but runs the risk of corrupting the data in the rest of the page that is being checksummed. Also, the main problem is that a T10-PI checksum on a partial sector is never going to be correct when sent to the underlying storage.</p> <p>What needs to be done is to have two separate T10-PI checksums for the first and last sectors, if they are partially overwritten. One checksum on the partial sector is needed to compute the checksum-of-checksums and match with the RPC checksum. A second checksum needs to be computed on a whole sector after it is modified, in order to submit to disk. If whole sectors are being modified (e.g. in the 10240-byte writes and 512-byte sector size) then the partial-sector handling is not needed.</p> <p>In order to ensure that there is proper end-to-end integrity for the checksums, when first or last partial page has been read from disk, but before the overwrite is done, the partial checksum should be computed for the areas that are not going to be overwritten (start of first partial sector, end of last partial sector). Ideally, the full sector will be checksummed again afterward to verify it still matches the original full-sector checksum, to verify that the partial sector checksum was done on correct data. Then, after the BRW/RDMA has overwritten the partial page, a full-sector checksum needs to be computed from the modified data. Next, the checksum on the first partial-sector checksum should be verified to match the original value computed, and the end partial-sector checksum should be verified against the value sent from the client. That ensures the two (non-overlapping) parts of the sector match the original data, and that the new full-sector checksum was done on the valid data. The full-sector checksum should be used when submitting the bio.</p> <p>For 2.12 it may be enough to read the page, apply the write, compute the checksum on the full sector for bio submission, and then verify that the partial-sector checksum matches the checksum from the client. This ensures the data sent from the network is valid, with an overlap for the full-sector checksum. The risk of corrupting the partial sector in RAM between the read/verify from disk and the overwrite is <b>very</b> small, and if this is happening then it is likely we would see other corruption on the full-sector writes as well. However, doing the extra partial-sector checksum for the unmodified part of the sector (as described above) would catch the case if the partial-sector overwrite is done incorrectly.</p> <p>Hi Andreas, I agree on the long term ideal solution and 2.12 fix. And agreed that the implementation on 2.12 might be enough for most of the cases. The ideal solution is too complex thus error prone. And I doubt the correctness and usefulness of the ideal solution unless we have complex codes for fault injection.</p> <p>Anyway, I will refresh the <a href="https://review.whamcloud.com/33752" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33752</a> patch.</p> <p>the checksum mismatch issue also exists for LDiskFS backend.</p> <p>Hongchao Zhang (hongchao@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33768" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33768</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11697" title="BAD WRITE CHECKSUM with t10ip4K and t10ip512 checksums" class="issue-link" data-issue-key="LU-11697"><del>LU-11697</del></a> osc: calc checksum at right offset<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 425fd09d48a3d564324d36fa52e31191845eda6e</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33727/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33727/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11697" title="BAD WRITE CHECKSUM with t10ip4K and t10ip512 checksums" class="issue-link" data-issue-key="LU-11697"><del>LU-11697</del></a> osc: wrong page offset for T10PI checksum<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c1f0520554460237e835a3ad37e7d3baa0dca937</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33752/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33752/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11697" title="BAD WRITE CHECKSUM with t10ip4K and t10ip512 checksums" class="issue-link" data-issue-key="LU-11697"><del>LU-11697</del></a> ost: do not reuse T10PI guards of unaligned page write<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6d8a7af1cd65409bdb5a307e76090ed460d5a6f6</p> <p>Landed for 2.12</p> Related LU-11663 LU-11729 LU-10472 Development Rank 1|i006tj: Rank (Obsolete) 9223372036854775807 Severity