https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10405 Lustre <p>The FIEMAP ioctl is used by many tools like cp(1) to optimize sparse files, but any error is treated as failure and such tools usually fall back by reading the whole file (so, zeroes if there is no stripe)<br/> lov_object_fiemap returns -ENODATA if it could not get the object's lov_stripe_md. We should just fill the structure with no extent and return success instead.</p> <p>We have many files with no stripe here because our copytool destripes released files, so they get assigned new OSTs on restore (instead of being restored where they originally were created, which might be bad balance) ; this mostly only impact admins copying released data around very carefully so this is somewhat minor for us, but still a bug imo.</p> <p> </p> <p>FWIW we also have actual sparse files with data (e.g. VM images) on lustre with non-trivial striping so fiemap returning ENOTSUPP on call without FIEMAP_FLAG_DEVICE_ORDER, and cp would just read everything. I think we could support this better as well... But that's another issue <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p> </p> <p>It's actually not so easy to create a file without stripe for testing, so I attached a simple program that does.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">$ gcc -o create-nostripe create-nostripe.c $ ./create-nostripe /mnt/lustre/testfile $ strace -e ioctl,read cp -a /mnt/lustre/testfile /tmp |&amp; head -n 20 read(3, <span class="code-quote">"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0&gt;\0\1\0\0\0\300j\0\0\0\0\0\0"</span>..., 832) = 832 read(3, <span class="code-quote">"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0&gt;\0\1\0\0\0\200\37\0\0\0\0\0\0"</span>..., 832) = 832 read(3, <span class="code-quote">"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0&gt;\0\1\0\0\0\320\23\0\0\0\0\0\0"</span>..., 832) = 832 read(3, <span class="code-quote">"\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0&gt;\0\1\0\0\0@\34\2\0\0\0\0\0"</span>..., 832) = 832 read(3, <span class="code-quote">"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0&gt;\0\1\0\0\0\360\25\0\0\0\0\0\0"</span>..., 832) = 832 read(3, <span class="code-quote">"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0&gt;\0\1\0\0\0`\16\0\0\0\0\0\0"</span>..., 832) = 832 read(3, <span class="code-quote">"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0&gt;\0\1\0\0\0\240l\0\0\0\0\0\0"</span>..., 832) = 832 read(3, <span class="code-quote">"nodev\tsysfs\nnodev\trootfs\nnodev\tr"</span>..., 1024) = 360 read(3, "", 1024) = 0 ioctl(3, FS_IOC_FIEMAP, 0x7ffd374e88e0) = -1 ENODATA (No data available) read(3, <span class="code-quote">"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"</span>..., 65536) = 65536 read(3, <span class="code-quote">"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"</span>..., 65536) = 65536 read(3, <span class="code-quote">"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"</span>..., 65536) = 65536 read(3, <span class="code-quote">"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"</span>..., 65536) = 65536 read(3, <span class="code-quote">"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"</span>..., 65536) = 65536 read(3, <span class="code-quote">"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"</span>..., 65536) = 65536 read(3, <span class="code-quote">"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"</span>..., 65536) = 65536 read(3, <span class="code-quote">"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"</span>..., 65536) = 65536 read(3, <span class="code-quote">"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"</span>..., 65536) = 65536 read(3, <span class="code-quote">"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"</span>..., 65536) = 65536 .... </pre> </div></div> <p>I will submit a patch for this right away.</p> LU-10405

FIEMAP on object with no stripe should not return ENODATA

Improvement Minor Resolved Fixed Dominique Martinet CEA cea patch Tue, 19 Dec 2017 08:59:05 +0000 Wed, 5 Aug 2020 13:27:55 +0000 Thu, 25 Jan 2018 05:08:55 +0000 Lustre 2.7.0 Upstream Lustre 2.11.0 0 5 <p>Dominique Martinet (dominique.martinet@cea.fr) uploaded a new patch: <a href="https://review.whamcloud.com/30591" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30591</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10405" title="FIEMAP on object with no stripe should not return ENODATA" class="issue-link" data-issue-key="LU-10405"><del>LU-10405</del></a> lov: fill no-extent fiemap on object with no stripe.<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bbf2fcfe575de341f31074f8190d1a310c18f5cf</p> <p>FYI, the "mcreate" program in the lustre-tests RPM is typically used in test scripts to create a file without any striping.  It uses <tt>mknod(2)</tt> syscall to create a regular file without opening it, since the <tt>mknod(1)</tt> utility does not allow the creation of regular files.</p> <p>This doesn't work for my specific example here, as cp only uses fiemap on files given a certain size – my create-nostripe is misnamed and also ftruncate()s the file after creating it to reflect a bigger size like we have on our filesystems.</p> <p>Thanks for the info though, I wasn't aware of it.</p> <p>Two comments here:</p> <ul class="alternate" type="square"> <li>the “truncate -s” command can change the file size without opening it, so that should work together with “mcreate” without adding another tool. There is also “mulitop” that should allow similar kinds of operations already.</li> <li>you mention “copying (HSM) released files”, but in that case this could result in data loss to return an empty extent map to cp for a released file, since cp would think there is no data in the file, when in fact it is on the archive.</li> </ul> <p>I had tried truncate -s, it seems to open + ftruncate so that assigns a striping. multiop would work, I'll have a look <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p> </p> <p>And yes, admin copying (HSM) released files is tricky... But we're trying to be careful. Migrating from one filesystem to another without reading all the tapes is an interesting operation... Might only work for us with our hybrid lustre/HSM and old CEA hsm monster still crawling around and helping around the edges.</p> <p>I actually hadn't thought of the impact of this for <em>user</em> cp, since lustre/hsm restores on first read it will wrongly think there is nothing to read, and this will cause real problems. We <del>will</del> apparently already have the same problem with tar that just looks at stat.st_blocks and skips anything with 0 with a lstat() without ever even opening the file. That's going to be a tricky problem, makes me think we will want to drop this patch and open a new issue about faking st_blocks for tar...</p> <p>Yes, we have a special check in the client code to return a non-zero blocks count for stat() of the inode if there are dirty pages on the client (<a href="https://jira.whamcloud.com/browse/LU-417" title="block usage is reported as zero by stat call for tens of seconds after creating a file" class="issue-link" data-issue-key="LU-417"><del>LU-417</del></a>, see comment in cl_glimpse_lock() and vvp_object_glimpse()), even if no pages have been written to disk yet, to avoid problems with cp and tar. </p> <p>It makes sense to do the same for HSM released files if that isn't done already. </p> <p>Ok, this does make sense, I see what's done here. The two functions here look insufficient for an HSM released file as far as I can see, because dirty_cnt will also be 0; will likely need to add a second check "if blocks is still 0 after dirty_cnt check for hsm state and set an arbitrary value (size/blocksize?)"</p> <p>I'll see if I can reproduce the problem with tar (cp no longer bases itself on i_blocks) and open a new ticket likely early next year.</p> <p> </p> <p>I'm thinking something similar is needed here, if there is no stripe then also check hsm state and either return NODATA as currently if released or return an empty map. I won't have time to add and test that this year so giving my own patch a -1 to hold it off.</p> <p>In both 2.7 and master, we do return a "minimal FIEMAP" for released files. See <tt>lov_fiemap()</tt> and <tt>lov_object_fiemap()</tt>. And this works (at least on master) but we do not seem to have a test for it.</p> <p>We also return a non zero block count for release files.</p> <p>When you update your patch, please include a test that fails before the code change and passes after.</p> <p>Hi John.</p> <p>As I said in a previous comment, the truncate command does an open followed by ftruncate so it allocates stripes for the file, unlike my test example. I did reproduce it on master (80515fa <a href="https://jira.whamcloud.com/browse/LU-10232" title="kernel BUG at cl_object.c:206!" class="issue-link" data-issue-key="LU-10232"><del>LU-10232</del></a> lov: call cl_object_attr_get under cl_attr lock) so I'm fairly sure of myself.</p> <p>I've also checked multiop and I don't see how to open a file with O_LOV_DELAY_CREATE. I can do mknod but then the file isn't opened so ftruncate doesn't help. I also double-checked the truncate(1) command but can't make it call truncate() with a path instead of ftruncate()... I'll do more checking we can do with normal lustre test programs, but please use my create-nostripe.c for now.</p> <p> </p> <p>I see the code part about lhsm released file now, I hadn't really looked at code past the FIEMAP_FLAG_DEVICE_ORDER check though because cp does not handle that. Your example shows a working fiemap because you likely have a stripecount of 1, which is not the case for our LHSM filesystem... I can't use lsm-&gt;lsm_is_released in my patch though, since lsm is returned as NULL, I'll need another way to check for released state? Will do some more digging, I'm not very familiar with this code.</p> <p>I'll also add new tests on refresh, it'll have to wait for new year though...</p> <p>Dominique,</p> <p>You are correct about the truncate.</p> <p>BTW, the multiop command line to use would be:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>multiop f0 oO_RDWR:O_CREAT:O_LOV_DELAY_CREATE:T10737418240c </pre> </div></div> <p>Unfortunately multiop does not support 64 bit file sizes. (This would be easy to fix however.)</p> <p>Hi and happy new year,</p> <p>I have update the patch with a new test, that checks fiemap on unstripped files with filefrag -e like the other fiemap tests (and your multiop line, thanks! Hadn't noticed 'o' could take a list of flags as arguments, this help output made it sound like 'o' is always just O_CREAT)</p> <p>Regarding hsm-files, I've double-checked this should be fine. I wasn't clear that a released file would always have a `lsm` so now this is cleared up my patch doesn't need to change, this won't cause any new false-positive of not copying data when we should.</p> <p> </p> <p>As a side-note, I do notice that released files have st_blocks set to 1 (presumably to tell tar and friends to look at the content), but I could not find in the code where that is set. cl_glimpse_lock() and vvp_object_glimpse() have a comment about that for dirty_cnt() but this function doesn't do any HSM-related check... Well, I guess this works so I don't really need to look further.</p> <p>Thanks,</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30591/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30591/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10405" title="FIEMAP on object with no stripe should not return ENODATA" class="issue-link" data-issue-key="LU-10405"><del>LU-10405</del></a> lov: fill no-extent fiemap on object with no stripe.<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 113fa79b7dc9b598c615d4cbfa6e3513d2c6d35b</p> <p>Landed for 2.11</p> Development Rank 1|hzzptz: Rank (Obsolete) 9223372036854775807 Severity