https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2580 Lustre <p>We are seeing an issue at KIT where cp will occasionally use the FIEMAP extension to create a completely sparse file instead of actually copying the file. It seems to occur under a workload involving creating and deleting many files at once. It only involves a single client though, it's not a parallel workload. </p> <p>Relevant strace from 'bad' cp:<br/> ioctl(3, 0xc020660b, 0x7fff392c0950) = 0<br/> ftruncate(4, 12853) = 0</p> <p>strace from 'good' cp:<br/> read(3, "#!/bin/bash -u\n\n#localisation\nex"..., 2097152) = 12853<br/> write(4, "#!/bin/bash -u\n\n#localisation\nex"..., 12853) = 12853<br/> read(3, "", 2097152) = 0</p> <p>The strace didn't print the stat block information, but I'm assuming the st_blocks == 0 in the bad one. I will ask the customer to get a full strace -v to confirm, but it appears to be something similar to <a href="https://jira.whamcloud.com/browse/LU-417" title="block usage is reported as zero by stat call for tens of seconds after creating a file" class="issue-link" data-issue-key="LU-417"><del>LU-417</del></a>?</p> SLES 11 SP2 (client), Lustre 2.1.2 RHEL6 (server) LU-2580

cp with FIEMAP support creates completely sparse file

Bug Blocker Resolved Fixed Peter Jones Kit Westneat LB Mon, 7 Jan 2013 10:24:38 +0000 Wed, 24 Apr 2013 17:55:19 +0000 Tue, 5 Mar 2013 11:29:27 +0000 Lustre 2.3.0 Lustre 2.4.0 0 10 <p>There is a 41MB (800MB uncompressed) client debug log from the event. Would it be useful to upload it somewhere?</p> <p>Thanks for the ticket Kit.</p> <p>Kit, is this only happening when trying to copy a 1-stripe file that was recently created? Is the file created on the same node or a remote node? What does "stat" report on the file before it is copied?</p> <p>Here is the stat, stracing the cp reports the same:<br/> File: `/work/ws/jl95/simulation/OSC/OSC_traps_temperatur2_kopie/calc_series_OSC_multi_neu'<br/> Size: 12899 Blocks: 1 IO Block: 2097152 regular file<br/> Device: 24ce0ea6h/617483942d Inode: 144115253070874816 Links: 1<br/> Access: (0755/-rwxr-xr-x) Uid: (28181/ jl95) Gid: (11700/ jl00)<br/> Access: 2013-01-09 13:28:34.000000000 +0100<br/> Modify: 2013-01-09 13:28:34.000000000 +0100<br/> Change: 2013-01-09 13:28:34.000000000 +0100<br/> Birth: -</p> <p>Response from customer:</p> <p>&gt; is this only happening when trying to copy a 1-stripe file that was recently created? <br/> Yes, it seems. The file was recently created and has stripe count 1. (The file system <br/> has defaultr stripe count 4 but since the user creates so many files I advised him to <br/> use stripe count 1. </p> <p>&gt; Is the file created on the same node or a remote node? <br/> The file is created on a local disk of the same node. </p> <p>&gt; What does "stat" report on the file before it is copied? <br/> I just asked the user to run his reproducer again and create that data. (It might happen that <br/> with the stat the problem might not appear. With a "sleep 5" the problem does not appear most <br/> of the times. After the copy we check the source and the copied file with md5sum and when the <br/> problem appears the md5sum is different.) </p> <p>stats and md5sums of both good and bad cps. In the bad cp, the file only reports 1 used block. In the good one, the file reports 32 blocks. straces confirm that's what is seen by cp. </p> <p>What version of fileutils is in use here? Was it part of the distro, or upgraded afterward?</p> <p>From KIT: They are using normal fileutils (which are part of coreutils) of the SLES11 SP2 distribution. coreutils version is 8.12 and release is 6.23.1. (Source RPM is coreutils-8.12-6.23.1.src.rpm)</p> <p>I don't think this issue is related to FIEMAP. stat reported st_blocks=1 for the file and a size of 12899 bytes. So cp correctly called the FIEMAP ioctl. </p> <p>The problem seems to be Lustre reporting wrong number of blocks on a recently created/written file. This fix leads stat to report st_blocks=1 instead of 0 - <a href="http://git.whamcloud.com/?p=fs/lustre-release.git;a=commitdiff;h=829845ac9ddbdfd170de215742c033ea1102db3e;hp=fc4b46df111bbf9d2207265d18b3f0d72f49502c" class="external-link" target="_blank" rel="nofollow noopener">http://git.whamcloud.com/?p=fs/lustre-release.git;a=commitdiff;h=829845ac9ddbdfd170de215742c033ea1102db3e;hp=fc4b46df111bbf9d2207265d18b3f0d72f49502c</a></p> <p>Further regarding the ftruncate that we see in the strace (instead of the fseek that I was expecting) - even though Lustre says st_blocks=1, fiemap ioctl says that no blocks are allocated leading to the ftruncate call with the size of the file. </p> <p>On SLES11 SP2 with coreutils-8.12-6.19.1, looks like cp is always setting the FIEMAP_FLAG_SYNC flag as well.</p> <p>Kalpak, AFAIK the st_blocks value is only used to determine whether the file is sparse (st_blocks &lt; st_size / 512) or dense (st_blocks &gt;= st_size / 512). For dense files they are copied via "while (read() &gt; 0) write()", and for sparse files newer "cp" copies only the list of extents returned by FIEMAP. In both cases, my understanding is that st_blocks is not used for determining how much data is copied.</p> <p>The problem, as I see it, is that Lustre FIEMAP (which only returns something useful to "cp" for single-striped files) does not return FIEMAP_EXTENT_DELALLOC extents for pages that are only in the client cache and not on the OST yet. "cp" should be using FIEMAP_FLAG_SYNC and causing all of the cached extents to be flushed, but somehow this isn't happening.</p> <p>Could you please clarify as to what versions of Lustre (and any patches running) that are being used here? You mention that it is Lustre 2.1.2 servers but what version of Lustre is being used on the client? </p> <p>Peter, the clients are running Lustre 2.3 on SLES11 SP2.</p> <p>Thanks Kalpak. With any patches applied?</p> <p>Update from KIT: With Lustre 2.3.0 on the client and patches 4477 and 4659 from <a href="https://jira.whamcloud.com/browse/LU-2367" title="Hang in osc_extent_wait under fsync" class="issue-link" data-issue-key="LU-2367"><del>LU-2367</del></a> and <a href="https://jira.whamcloud.com/browse/LU-2286" title="Test failure on test suite parallel-scale, subtest test_write_append_truncate" class="issue-link" data-issue-key="LU-2286"><del>LU-2286</del></a>, the issue cannot be reproduced.</p> <p><a href="https://jira.whamcloud.com/browse/LU-2367" title="Hang in osc_extent_wait under fsync" class="issue-link" data-issue-key="LU-2367"><del>LU-2367</del></a> fixes a race in unplugging the IO queue which can affect flush and fsync - and "cp" always calls FIEMAP with the SYNC flag set causing the cached extents to be flushed. <a href="https://jira.whamcloud.com/browse/LU-2286" title="Test failure on test suite parallel-scale, subtest test_write_append_truncate" class="issue-link" data-issue-key="LU-2286"><del>LU-2286</del></a> fixes a bug where an extent does not get flushed to disk until the next write to the file occurs. So it does seem logical that the issue is not being reproduced with these 2 patches applied.</p> <p>ok thanks for the update. That explains why we have been unable to reproduce this issue on the latest 2.4 code. I will close out this ticket.</p> <p>This bug is still applicable to 2.1 when using cp built from coreutils 8.12, right? <span class="error">&#91;I can&#39;t confirm that, but I think we&#39;re seeing this on 2.2&#93;</span> If so and since b2_1 is still the current maintenance branch, do we want to land a fix there?</p> Related LU-417 LU-3219 Development Rank 1|hzveon: Rank (Obsolete) 6020 Severity