[LU-9491] v2.9: silent data corruption with writev() and O_APPEND

Duplicate — Fri, 3 May 2019 17:25:57 +0000

Lustre client version 2.9.0 will silently corrupt data if:

a file is opened with mode O_APPEND, and
is written to using the writev() syscall using two or more iovs, and
the file offset written to by any but the last iov touches a page boundary.

If all of the above hold true, the last iov overwrites some or all data from the previous iovs. It can be easily reproduces with eg.

/usr/lib64/lustre/tests/rwv -w -a -n 2 4096 4096 /lustre/test131.out

On a Lustre client running v2.9.0, the first chunk is lost, and test.out ends up as a 4k-sized file. Earlier versions create an 8k-sized test.out containing both chunks. In both cases, the writev() syscall returns 8192, indicating that the complete data set had been written.

Test 131b from sanity.sh already triggers this bug. It only checks return codes, but doesn't verify the written file, and therefore currently won't catch the error. (Obvious patch to follow.)

A git bisect run revealed that commit 1101120d3258509fa74f952cd8664bfdc17bd97d (~~LU-4257~~ llite: fix up iov_iter implementation) had introduced the error. So far, I've verified that v2.9.0 is affected on both RHEL6 and RHEL7, both of which use the various iov_iter_*() functions from lustre/llite/llite_internal.h introduced by this changeset. There's a chance that more recent kernel versions that implement iov_iter_*() themselves are unaffected, but I haven't been able to check.

I'll attach a brief vfstrace from running the rwv reproducer mentioned above. Note the second call to ll_write_begin() logs "Writing 0 of 0 to 4096 bytes" just like the first invocation whereas it should have advanced a page, ie. "Writing 1 of 0 to 4096 bytes" would be the expected output. This means that ll_write_begin() may be called with an incorrect pos field for all but the first iov in the list, but I haven't been able to pinpoint the spot where it goes wrong.

If O_APPEND is dropped from the open() call (ie. rwv is called without the -a switch), everything works fine, and the second call to ll_write_begin() indeed logs "Writing 1 of 0 to 4096 bytes" as expected.

Whamcloud Community JIRA

[LU-9491] v2.9: silent data corruption with writev() and O_APPEND