https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-17124 Lustre <p>fiemap FIEMAP_FLAG_SYNC can race while client is writing data to disk</p> <p>In such a case fiemap() call returns that the data is not on disk (no data for the range) and cp can just truncates (or sparse fill) based on the size of the file/extent.</p> <p>strace shows that FIEMAP_FLAG_SYNC was sent. Further and the user reports that a 'sync; cp &lt;blah&gt;' does not fail and a newer cp that uses copy_file_range() also does not fail.</p> <p>Looking further FIEMAP_FLAG_SYNC <b>expects</b> the data to be on disk aka filemap_write_and_wait() not just filemap_fdatawrite()</p> LU-17124

fiemap FIEMAP_FLAG_SYNC flag expects filemap_write_and_wait() or similar

Bug Critical Resolved Fixed Shaun Tancheff Shaun Tancheff Mon, 18 Sep 2023 10:40:44 +0000 Wed, 15 Nov 2023 17:09:26 +0000 Thu, 28 Sep 2023 15:21:13 +0000 Lustre 2.16.0 0 5 <p>"Shaun Tancheff &lt;shaun.tancheff@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52402" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52402</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17124" title="fiemap FIEMAP_FLAG_SYNC flag expects filemap_write_and_wait() or similar" class="issue-link" data-issue-key="LU-17124"><del>LU-17124</del></a> llite: Write and wait on FIEMAP_FLAG_SYNC<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 39825b198f06234da2876637660663981d866bf1</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52402/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52402/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17124" title="fiemap FIEMAP_FLAG_SYNC flag expects filemap_write_and_wait() or similar" class="issue-link" data-issue-key="LU-17124"><del>LU-17124</del></a> llite: Write and wait on FIEMAP_FLAG_SYNC<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1af966fe6983f28e9d438b57944c4dccb8941c61</p> <p>Landed for 2.16</p> <p>Cyfronet here,</p> <p>Server: 2.15.3 <br/> Client 2.15.3 +  <a href="https://jira.whamcloud.com/browse/LU-17124" title="fiemap FIEMAP_FLAG_SYNC flag expects filemap_write_and_wait() or similar" class="issue-link" data-issue-key="LU-17124"><del>LU-17124</del></a> llite: Write and wait on FIEMAP_FLAG_SYNC<br/> Cllient OS: Rocky 9, cp is using copy_file_range instead of read/write</p> <p>We have observed the same issue prior applying the patch. When copying 4096 files from one directory to another (on the same fs), in parallel using 32 processes, some copies where corrupted. Some files were truncated more or less, around 10-30 destination files corrupted in total.</p> <p>Having applied the patch we observe less corrupted files and less bytes are missing. </p> <p>It seems that solution doesn't fix the problem completly, or maybe we miss some other patches between 2.15.3 and 2.16.0</p> <p>This is how we copy the files:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> find ./source_dir -name *.exr | xargs -P32 -I<span class="code-quote">'{}'</span> cp -v {} ./dest_dir</pre> </div></div> <p>Parameter -P32 keeps 32 processes running in parallel</p> <p>If you are copying from Lustre to Lustre on the same file system then I do not see this patch would have any obvious effect on your use case. I do not see fiemap() being involved on el9.2 when copy_file_range() can be used.</p> <p>So I think your find | xargs -P32 cp &lt;args&gt; case is may be exposing a separate bug.</p> <p>If you could capture strace(s) of the cp -v and provide / attach a 'good' sample and a 'bad' sample we may be able to see what is happening.</p> <p>Two traces in the attachment, both captured for the same file</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#good ... uname({sysname="Linux", nodename="t0006", ...}) = 0 copy_file_range(3, NULL, 4, NULL, 9223372035781033984, 0) = 62924467 copy_file_range(3, NULL, 4, NULL, 9223372035781033984, 0) = 0 ... #bad ... uname({sysname="Linux", nodename="t0006", ...}) = 0 copy_file_range(3, NULL, 4, NULL, 9223372035781033984, 0) = 19005440 copy_file_range(3, NULL, 4, NULL, 9223372035781033984, 0) = 1441792 copy_file_range(3, NULL, 4, NULL, 9223372035781033984, 0) = 0 &lt;= short read? ... </pre> </div></div> <p> </p> <p>Quick update.<br/> Forcing fsync(fd_out) before copy_file_range() call fixes the problem with truncated output files</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /*  *   gcc -c -Wall -Werror -fpic ./<span class="code-keyword">this</span>.c -o cfr.o  *   gcc -ldl -shared -o ./cfr.so ./cfr.o  *   LD_PRELOAD=/full_path_to/cfr.so cp A B  *  * */ #define _GNU_SOURCE 1 #include &lt;dlfcn.h&gt; #include &lt;unistd.h&gt; #include &lt;stdio.h&gt; #include &lt;sys/types.h&gt; #include &lt;sys/stat.h&gt; #include &lt;fcntl.h&gt; typedef ssize_t *(*cfr_t)(<span class="code-object">int</span>, loff_t*, <span class="code-object">int</span>, loff_t*, size_t, unsigned <span class="code-object">int</span>); <span class="code-keyword">static</span> cfr_t p = NULL; ssize_t copy_file_range(<span class="code-object">int</span> fd_in, loff_t *off_in, <span class="code-object">int</span> fd_out, loff_t *off_out, size_t len, unsigned <span class="code-object">int</span> flags) {     fsync(fd_out);     ssize_t r=-1;     <span class="code-keyword">if</span> (p == NULL) {         p =  dlsym(RTLD_NEXT, <span class="code-quote">"copy_file_range"</span>);         <span class="code-keyword">if</span> (p == NULL) {             <span class="code-comment">/* Error handling */</span>             <span class="code-keyword">return</span> r;         }     }    r = (ssize_t) p(fd_in, off_in, fd_out, off_out, len,flags);     <span class="code-keyword">return</span> r; }</pre> </div></div> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p>"Shaun Tancheff &lt;shaun.tancheff@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53140" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53140</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17124" title="fiemap FIEMAP_FLAG_SYNC flag expects filemap_write_and_wait() or similar" class="issue-link" data-issue-key="LU-17124"><del>LU-17124</del></a> llite: sync on splice write<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c6c5e7c6664fcaeb4b4df51f6e0562f35a91d985</p> <p>Please try the patch and see if the sync() occurs early enough to resolve the corruption you are finding.</p> <p>If not we may need to implement a file-system specific copy_file_range()</p> <p>Thanks!</p> <p>Thank you for the patch. We tested the changes and unfortunately we still see corrupted destnation files. </p> <p> </p> <p>A ticket exist for copy_file_range(). I just never got the cycles to implement for Lustre. Also RHEL7 doesn't support a proper hook for copy_file_range.</p> Related LU-11621 Development Rank 1|i03vyv: Rank (Obsolete) 9223372036854775807 Severity