[LU-16896] sanity-flr test_70a: lfs mirror verify: chunk [0, 0x100000) has different checksum value on mirror <n> and mirror <m> Created: 14/Jun/23 Updated: 09/Jan/24 Resolved: 09/Nov/23 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | Lustre 2.16.0 |
| Type: | Bug | Priority: | Minor |
| Reporter: | Maloo | Assignee: | Zhenyu Xu |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Issue Links: |
|
||||||||
| Severity: | 3 | ||||||||
| Rank (Obsolete): | 9223372036854775807 | ||||||||
| Description |
|
This issue was created by maloo for bobijam <bobijam@hotmail.com> This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/7e88bd04-8d1f-4f34-96cc-99bca0f7b47a test_70a failed with the following error: fsx FLR file /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr failed Test session details: <<Please provide additional information about the failure here>> lfs mirror verify: chunk [0, 0x100000) has different checksum value on mirror 1 and mirror 9. another run: https://testing.whamcloud.com/test_sets/08a96162-2365-4e4d-b565-a2b82ca3e8f7 lfs mirror verify: chunk [0, 0x100000) has different checksum value on mirror 1 and mirror 4. VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV |
| Comments |
| Comment by Zhenyu Xu [ 14/Jun/23 ] |
|
some discovery so far, after mirror resync, the file size has been changed. The file size has been set by fallocate, and during mirror resync, the mirror 2 is the uptodated mirror, and it is trying to resync mirror 1,4,6 000388 1686757145.433953 fallocate 0x03473b thru 0x03ffff (0xb8c5 bytes) file size 40000 stat 40000 000397 1686757145.448408 mirror resync 4 mirrors file size 40000 punch mirror 1 to a000 1:truncate mirror 4 to 1000 1:truncate mirror 6 to 1000 punch mirror 1 to 1a000 1:truncate mirror 4 to f000 1:truncate mirror 6 to f000 punch mirror 1 to 2a000 1:truncate mirror 4 to 1f000 1:truncate mirror 6 to 1f000 punch mirror 1 to 40000 1:truncate mirror 4 to 2f000 2:truncate mirror 4 to 40000 1:truncate mirror 6 to 2f000 2:truncate mirror 6 to 40000 so mirror 1, 4, 6 has reached to 0x40000 after that the file size has been changed to 0x2f000 file size 2f000 the data in the mirrors are identical, but file size has been shrinked # hexdump /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr | tail -10 002ebb0 3881 1481 8381 c581 2f81 dd81 aa81 f081 002ebc0 7281 b281 c881 8281 cc81 2e81 e781 eb81 002ebd0 0581 b581 8781 5d81 bb81 4e81 a781 af81 002ebe0 2d81 9481 4981 6881 1081 e581 9a81 5981 002ebf0 6581 e181 a981 8981 e781 f581 3181 3781 002ec00 f181 0881 6681 a081 1b81 9d81 f381 5481 002ec10 b481 f281 b981 6281 0081 0000 0000 0000 002ec20 0000 0000 0000 0000 0000 0000 0000 0000 * 002f000 # hexdump /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr.fsxgood | tail -10 002ebb0 3881 1481 8381 c581 2f81 dd81 aa81 f081 002ebc0 7281 b281 c881 8281 cc81 2e81 e781 eb81 002ebd0 0581 b581 8781 5d81 bb81 4e81 a781 af81 002ebe0 2d81 9481 4981 6881 1081 e581 9a81 5981 002ebf0 6581 e181 a981 8981 e781 f581 3181 3781 002ec00 f181 0881 6681 a081 1b81 9d81 f381 5481 002ec10 b481 f281 b981 6281 0081 0000 0000 0000 002ec20 0000 0000 0000 0000 0000 0000 0000 0000 * 0040000 |
| Comment by Zhenyu Xu [ 15/Jun/23 ] |
|
https://review.whamcloud.com/c/ex/lustre-release/+/50974/8 causes this issue, revert it make sanity-flr/test_70a pass, I'm investigating what change makes this file size change. |
| Comment by Zhenyu Xu [ 15/Jun/23 ] |
|
The comment I posted 2 hours before is not correct, that patch just make this bug easy to show up. The file mirror 1 contains 2 component covering [0, 0x100000) and [0x100000, EOF), llapi_mirror_resync_many() read from mirror 2, seeking data from 0x2f000 and find 0x40000 is its end of the file, and since mirror 1 component [0, 0x10000) does not end with EOF, so it calls llapi_mirror_punch() to punch a hole [0x2f000, 0x40000), while it seems that that punch changes the file size of the mirror 1 to 0x2f000, and stat() of the file retrieves this incorrect file size from mirror 1. As a proof of concept, immediately after llapi_mirror_punch() I adds a llapi_mirror_truncate() and the test passes, the file size retrieved is the correct number 0x40000. |
| Comment by Zhenyu Xu [ 15/Jun/23 ] |
|
I'm wondering if the punch end just reaches the end of the file, even though the FALLOC_FL_KEEP_SIZE is set, when glimpse is checking the OST object, would it reports back the original object size 0x40000, or the new one 0x2f000 (since objects between 0x2f000 to 0x40000 has been deleted). |
| Comment by Zhenyu Xu [ 15/Jun/23 ] |
|
This workaround can pass the fsx mirror mode test, but I suspect some issue lies in OSD about this special falloc case. diff --git a/lustre/utils/liblustreapi_layout.c b/lustre/utils/liblustreapi_layout.c index 1920110a45..cc4ebf770d 100644 --- a/lustre/utils/liblustreapi_layout.c +++ b/lustre/utils/liblustreapi_layout.c @@ -3262,6 +3262,7 @@ int llapi_mirror_resync_many(int fd, struct llapi_layout *layout, struct llapi_resync_comp *comp_array, int comp_size, uint64_t start, uint64_t end) { + struct stat stbuf; size_t page_size = sysconf(_SC_PAGESIZE); const size_t buflen = 4 << 20; /* 4M */ void *buf; @@ -3276,6 +3277,9 @@ int llapi_mirror_resync_many(int fd, struct llapi_layout *layout, if (rc) return -rc; + if (fstat(fd, &stbuf) < 0) + return -errno; + while (pos < end) { uint64_t mirror_end; ssize_t bytes_read; @@ -3336,23 +3340,24 @@ int llapi_mirror_resync_many(int fd, struct llapi_layout *layout, else to_punch = data_off - cur_pos; - if (comp_array[i].lrc_end == OBD_OBJECT_EOF) { + if (comp_array[i].lrc_end == OBD_OBJECT_EOF) /* the last component can be truncated * safely */ rc = llapi_mirror_truncate(fd, mid, cur_pos); - /* hole at the end of file, so just - * truncate up to set size. - */ - if (!rc && data_off == data_end) - rc = llapi_mirror_truncate(fd, + else + rc = llapi_mirror_punch(fd, mid, + cur_pos, to_punch); + /** + * hole at the end of file, so just + * truncate up to set size. + */ + if (!rc && data_off == data_end && + data_end == stbuf.st_size) + rc = llapi_mirror_truncate(fd, mid, data_end); - } else { - rc = llapi_mirror_punch(fd, - comp_array[i].lrc_mirror_id, - cur_pos, to_punch); - } + /* if failed then read failed hole range */ if (rc < 0) { rc = 0; |
| Comment by Gerrit Updater [ 17/Jun/23 ] |
|
"Zhenyu Xu <bobijam@hotmail.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51344 |
| Comment by Gerrit Updater [ 31/Aug/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51344/ |
| Comment by Peter Jones [ 31/Aug/23 ] |
|
Landed for 2.16 |
| Comment by Gerrit Updater [ 23/Sep/23 ] |
|
"Zhenyu Xu <bobijam@hotmail.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52489 |
| Comment by Zhenyu Xu [ 23/Sep/23 ] |
|
found a glitch in previous patch, fix at https://review.whamcloud.com/c/fs/lustre-release/+/52489 |
| Comment by Gerrit Updater [ 08/Nov/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52489/ |
| Comment by Peter Jones [ 09/Nov/23 ] |
|
Landed for 2.16 |