https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16896 Lustre <p>This issue was created by maloo for bobijam &lt;bobijam@hotmail.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/7e88bd04-8d1f-4f34-96cc-99bca0f7b47a" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/7e88bd04-8d1f-4f34-96cc-99bca0f7b47a</a></p> <p>test_70a failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed </pre> </div></div> <p>Test session details:<br/> clients: <a href="https://build.whamcloud.com/job/lustre-b_es-reviews/11663" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-b_es-reviews/11663</a> - 3.10.0-1160.88.1.el7.x86_64<br/> servers: <a href="https://build.whamcloud.com/job/lustre-b_es-reviews/11663" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-b_es-reviews/11663</a> - 3.10.0-1160.88.1.el7_lustre.ddn17.x86_64</p> <p>&lt;&lt;Please provide additional information about the failure here&gt;&gt;</p> <p>lfs mirror verify: chunk [0, 0x100000) has different checksum value on mirror 1 and mirror 9.<br/> lfs mirror verify: chunk [0, 0x100000) has different checksum value on mirror 1 and mirror 10.<br/> lfs mirror verify: chunk [0, 0x100000) has different checksum value on mirror 1 and mirror 11.<br/> lfs mirror verify: chunk [0, 0x100000) has different checksum value on mirror 1 and mirror 12.<br/> lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22<br/> CRC-32 checksum value for chunk [0, 0x400000):<br/> Mirror 1: 0xe1dd0e93<br/> Mirror 8: 0xe1dd0e93<br/> Mirror 9: 0xe1dd0e93<br/> Mirror 10: 0xe1dd0e93<br/> Mirror 11: 0xe1dd0e93<br/> Mirror 12: 0xe1dd0e93</p> <p>another run: <a href="https://testing.whamcloud.com/test_sets/08a96162-2365-4e4d-b565-a2b82ca3e8f7" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/08a96162-2365-4e4d-b565-a2b82ca3e8f7</a></p> <p>lfs mirror verify: chunk [0, 0x100000) has different checksum value on mirror 1 and mirror 4.<br/> lfs mirror verify: chunk [0, 0x100000) has different checksum value on mirror 1 and mirror 6.<br/> lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 22<br/> CRC-32 checksum value for chunk [0, 0x400000):<br/> Mirror 1: 0xd9cbfb8d<br/> Mirror 2: 0xd9cbfb8d<br/> Mirror 4: 0xd9cbfb8d<br/> Mirror 6: 0xd9cbfb8d</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity-flr test_70a - fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed</p> LU-16896

sanity-flr test_70a: lfs mirror verify: chunk [0, 0x100000) has different checksum value on mirror <n> and mirror <m>

Bug Minor Resolved Fixed Zhenyu Xu Maloo Wed, 14 Jun 2023 11:46:13 +0000 Tue, 9 Jan 2024 22:28:48 +0000 Thu, 9 Nov 2023 00:34:58 +0000 Lustre 2.16.0 0 5 <p>some discovery so far, after mirror resync, the file size has been changed.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>The file size has been set by fallocate, and during mirror resync, the mirror 2 is the uptodated mirror, and it is trying to resync mirror 1,4,6 000388 1686757145.433953 fallocate  0x03473b thru 0x03ffff (0xb8c5 bytes) file size 40000 stat 40000 000397 1686757145.448408 mirror resync 4 mirrors file size 40000 punch mirror 1 to a000 1:truncate mirror 4 to 1000 1:truncate mirror 6 to 1000 punch mirror 1 to 1a000 1:truncate mirror 4 to f000 1:truncate mirror 6 to f000 punch mirror 1 to 2a000 1:truncate mirror 4 to 1f000 1:truncate mirror 6 to 1f000 punch mirror 1 to 40000 1:truncate mirror 4 to 2f000 2:truncate mirror 4 to 40000 1:truncate mirror 6 to 2f000 2:truncate mirror 6 to 40000 so mirror 1, 4, 6 has reached to 0x40000 after that the file size has been changed to 0x2f000 file size 2f000 the data in the mirrors are identical, but file size has been shrinked # hexdump /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr | tail -10 002ebb0 3881 1481 8381 c581 2f81 dd81 aa81 f081 002ebc0 7281 b281 c881 8281 cc81 2e81 e781 eb81 002ebd0 0581 b581 8781 5d81 bb81 4e81 a781 af81 002ebe0 2d81 9481 4981 6881 1081 e581 9a81 5981 002ebf0 6581 e181 a981 8981 e781 f581 3181 3781 002ec00 f181 0881 6681 a081 1b81 9d81 f381 5481 002ec10 b481 f281 b981 6281 0081 0000 0000 0000 002ec20 0000 0000 0000 0000 0000 0000 0000 0000 * 002f000 # hexdump /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood | tail -10 002ebb0 3881 1481 8381 c581 2f81 dd81 aa81 f081 002ebc0 7281 b281 c881 8281 cc81 2e81 e781 eb81 002ebd0 0581 b581 8781 5d81 bb81 4e81 a781 af81 002ebe0 2d81 9481 4981 6881 1081 e581 9a81 5981 002ebf0 6581 e181 a981 8981 e781 f581 3181 3781 002ec00 f181 0881 6681 a081 1b81 9d81 f381 5481 002ec10 b481 f281 b981 6281 0081 0000 0000 0000 002ec20 0000 0000 0000 0000 0000 0000 0000 0000 * 0040000 </pre> </div></div> <p><a href="https://review.whamcloud.com/c/ex/lustre-release/+/50974/8" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/ex/lustre-release/+/50974/8</a> causes this issue, revert it make sanity-flr/test_70a pass, I'm investigating what change makes this file size change.</p> <p>The comment I posted 2 hours before is not correct, that patch just make this bug easy to show up.</p> <p>The file mirror 1 contains 2 component covering [0, 0x100000) and [0x100000, EOF), llapi_mirror_resync_many() read from mirror 2, seeking data from 0x2f000 and find 0x40000 is its end of the file, and since mirror 1 component [0, 0x10000) does not end with EOF, so it calls llapi_mirror_punch() to punch a hole [0x2f000, 0x40000), while it seems that that punch changes the file size of the mirror 1 to 0x2f000, and stat() of the file retrieves this incorrect file size from mirror 1.</p> <p>As a proof of concept, immediately after llapi_mirror_punch() I adds a llapi_mirror_truncate() and the test passes, the file size retrieved is the correct number 0x40000.</p> <p>I'm wondering if the punch end just reaches the end of the file, even though the FALLOC_FL_KEEP_SIZE is set, when glimpse is checking the OST object, would it reports back the original object size 0x40000, or the new one 0x2f000 (since objects between 0x2f000 to 0x40000 has been deleted).</p> <p>This workaround can pass the fsx mirror mode test, but I suspect some issue lies in OSD about this special falloc case.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> diff --git a/lustre/utils/liblustreapi_layout.c b/lustre/utils/liblustreapi_layout.c index 1920110a45..cc4ebf770d 100644 --- a/lustre/utils/liblustreapi_layout.c +++ b/lustre/utils/liblustreapi_layout.c @@ -3262,6 +3262,7 @@ <span class="code-object">int</span> llapi_mirror_resync_many(<span class="code-object">int</span> fd, struct llapi_layout *layout, struct llapi_resync_comp *comp_array, <span class="code-object">int</span> comp_size, uint64_t start, uint64_t end) { + struct stat stbuf; size_t page_size = sysconf(_SC_PAGESIZE); <span class="code-keyword">const</span> size_t buflen = 4 &lt;&lt; 20; <span class="code-comment">/* 4M */</span> void *buf; @@ -3276,6 +3277,9 @@ <span class="code-object">int</span> llapi_mirror_resync_many(<span class="code-object">int</span> fd, struct llapi_layout *layout, <span class="code-keyword">if</span> (rc) <span class="code-keyword">return</span> -rc; + <span class="code-keyword">if</span> (fstat(fd, &amp;stbuf) &lt; 0) + <span class="code-keyword">return</span> -errno; + <span class="code-keyword">while</span> (pos &lt; end) { uint64_t mirror_end; ssize_t bytes_read; @@ -3336,23 +3340,24 @@ <span class="code-object">int</span> llapi_mirror_resync_many(<span class="code-object">int</span> fd, struct llapi_layout *layout, <span class="code-keyword">else</span> to_punch = data_off - cur_pos; - <span class="code-keyword">if</span> (comp_array[i].lrc_end == OBD_OBJECT_EOF) { + <span class="code-keyword">if</span> (comp_array[i].lrc_end == OBD_OBJECT_EOF) /* the last component can be truncated * safely */ rc = llapi_mirror_truncate(fd, mid, cur_pos); - /* hole at the end of file, so just - * truncate up to set size. - */ - <span class="code-keyword">if</span> (!rc &amp;&amp; data_off == data_end) - rc = llapi_mirror_truncate(fd, + <span class="code-keyword">else</span> + rc = llapi_mirror_punch(fd, mid, + cur_pos, to_punch); + /** + * hole at the end of file, so just + * truncate up to set size. + */ + <span class="code-keyword">if</span> (!rc &amp;&amp; data_off == data_end &amp;&amp; + data_end == stbuf.st_size) + rc = llapi_mirror_truncate(fd, mid, data_end); - } <span class="code-keyword">else</span> { - rc = llapi_mirror_punch(fd, - comp_array[i].lrc_mirror_id, - cur_pos, to_punch); - } + <span class="code-comment">/* <span class="code-keyword">if</span> failed then read failed hole range */</span> <span class="code-keyword">if</span> (rc &lt; 0) { rc = 0;  </pre> </div></div> <p>"Zhenyu Xu &lt;bobijam@hotmail.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51344" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51344</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16896" title="sanity-flr test_70a: lfs mirror verify: chunk [0, 0x100000) has different checksum value on mirror &lt;n&gt; and mirror &lt;m&gt;" class="issue-link" data-issue-key="LU-16896"><del>LU-16896</del></a> flr: resync should not change file size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 526cbab26f222966c5ebc2bf4d4aff517086f2e5</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51344/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51344/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16896" title="sanity-flr test_70a: lfs mirror verify: chunk [0, 0x100000) has different checksum value on mirror &lt;n&gt; and mirror &lt;m&gt;" class="issue-link" data-issue-key="LU-16896"><del>LU-16896</del></a> flr: resync should not change file size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b9ce342ee196af48d2d25e2811121fe4471f5fd2</p> <p>Landed for 2.16</p> <p>"Zhenyu Xu &lt;bobijam@hotmail.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52489" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52489</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16896" title="sanity-flr test_70a: lfs mirror verify: chunk [0, 0x100000) has different checksum value on mirror &lt;n&gt; and mirror &lt;m&gt;" class="issue-link" data-issue-key="LU-16896"><del>LU-16896</del></a> flr: resync could messes mirror content<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: add8c61ae6abd5c4a4ac8188683858361628345b</p> <p>found a glitch in previous patch, fix at <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52489" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52489</a></p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52489/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52489/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16896" title="sanity-flr test_70a: lfs mirror verify: chunk [0, 0x100000) has different checksum value on mirror &lt;n&gt; and mirror &lt;m&gt;" class="issue-link" data-issue-key="LU-16896"><del>LU-16896</del></a> flr: resync could mess mirror content<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1b694ad04f099924cb5760cbaf1f09e5843975aa</p> <p>Landed for 2.16</p> Related LU-16842 Development Rank 1|i03o0v: Rank (Obsolete) 9223372036854775807 Severity