https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3864 Lustre <p>HSM release in not preserving block count as reported by stat()</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># cd /mnt/lustre # dd if=/dev/zero of=Antoshka bs=1M count=10 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0740321 s, 142 MB/s # stat Antoshka File: `Antoshka' Size: 10485760 Blocks: 20480 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205255725060 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2013-08-30 10:13:48.000000000 -0500 Modify: 2013-08-30 10:13:48.000000000 -0500 Change: 2013-08-30 10:13:48.000000000 -0500 # lfs hsm_archive Antoshka # lfs hsm_release Antoshka # stat Antoshka File: `Antoshka' Size: 10485760 Blocks: 0 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205255725060 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2013-08-30 10:13:48.000000000 -0500 Modify: 2013-08-30 10:13:48.000000000 -0500 Change: 2013-08-30 10:13:48.000000000 -0500 </pre> </div></div> <p>I had intended to fix this with <a href="https://jira.whamcloud.com/browse/LU-3811" title="non-root users cannot archive files, root cannot archive non-root users&#39; files" class="issue-link" data-issue-key="LU-3811"><del>LU-3811</del></a> but it will require some work in the MD* attr_set patch.</p> <p>If you're thinking (philosophically) hmm, well maybe it should report a block count of 0 here then you're just wrong.</p> LU-3864

stat() on HSM released file returns st_blocks = 0

Technical task LU-3647 Blocker Resolved Fixed Bruno Faccini John Hammond HSM Fri, 30 Aug 2013 15:19:45 +0000 Wed, 2 Oct 2013 20:39:43 +0000 Wed, 2 Oct 2013 20:39:43 +0000 Lustre 2.5.0 Lustre 2.5.0 0 9 <p>Also note that in HSM release the client is already sending the correct LA_BLOCKS to the MDT.</p> <p>Hi Andreas, do you think we need to show original blocks for a released file? I tend to think it's fine to set block as 0 because released files won't consume any blocks in lustre file system.</p> <p>Bruno</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>I think that this behaviour was by design, but I now see that this could be a source of major data corruption for newer versions of "cp" or "tar" trying to copy the file. If they call stat() on the file and see st_blocks == 0 the file they will incorrectly assume that the file is completely sparse and not even try to read the file data. This might easily be tested by something like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> yes | dd of=$DIR/$tfile bs=1M count=1 conv=sync || error <span class="code-quote">"creating $DIR/$tfile"</span> $LFS hsm_archive --archive $HSM_ARCHIVE_NUMBER $DIR/$tfile wait_request_state $(path2fid $DIR/$tfile) ARCHIVE SUCCEED $LFS hsm_release $DIR/$tfile mkdir $DIR/$tdir # only newer versions of cp detect sparse files by stat/FIEMAP (LU-2580) cp --sparse $DIR/$tfile $DIR/$tfile.2 || error <span class="code-quote">"copying $DIR/$tfile"</span> cmp $DIR/$tfile $DIR/$tfile.2 || error <span class="code-quote">"comparing copied $DIR/$tfile"</span> tar cf - --sparse $DIR/$tfile | tar xvf - -C $DIR/$tdir || error <span class="code-quote">"tar failed"</span> cmp $DIR/$tfile $DIR/$tdir/$DIR/$tfile || error <span class="code-quote">"comparing untarred $DIR/$tfile"</span> </pre> </div></div> <p>For us the block count represent the Lustre block count, so for a released file is 0. Later for partial release, it will be the count of blocks in Lustre, not released. File size is ok, a released file is like a sparse file but with data out of Lustre.</p> <p>"This is not a bug, this is a feature" but as Andreas pointed out, we will have to modify this as this could lead to issue when using, too smart, tools.</p> <p>So, where could we store the original block count? There's no place for that in the inode. Should we put this in HSM EA and use this value when replying to GETATTR?</p> <p>Another workaround, to avoid storing the previous block count, we could be to returned a theoretical maximum block count: computing (FILESIZE / BLOCKSIZE) + 1</p> <p>The default of this method will be that st_blocks could be larger than what it was before the file was released. But this is must simpler to patch.</p> <p>I imagine we have to also hack FIEMAP for released file?</p> <p>Hi Andreas, is there any side effect if we set the st_blocks to be nonzero even it doesn't consume any disk blocks in Lustre?</p> <p>To be honest, I'm totally okay to report st_blocks as zero for release files. Backup usually applies to latest files but HSM should move <b>old</b> files to secondary storage so their target are not conflicted. If backup has to restore every files in the lustre this is probably not what administrator wants.</p> <p>I am late in this thread (even if ticket is assigned to me !), but even if I understand the possible issue when using "smart" tools, I wonder what will happen when using "simple" tools like "du" if st_blocks no longer reflects the current number of consumed blocks ?</p> <p>Do we know how behave other HSMs, like DataMigration and others ? I seem to remember DM did not do anything tricky with st_blocks, but may be it changed since.</p> <p>Jinshan, even I thought that reporting a 0 value for st_block was a good idea first, we cannot afford to keep doing this due to cp/tar issue.</p> <p>This is <b>unacceptable</b> that users doing "<tt>cp --sparse</tt>" or "<tt>tar --sparse</tt>" copy WRONG data if this is applied to a released file.<br/> After looking at coreutils code and <a href="https://jira.whamcloud.com/browse/LU-2580" title="cp with FIEMAP support creates completely sparse file" class="issue-link" data-issue-key="LU-2580"><del>LU-2580</del></a> I think the bug here is that FIEMAP output for released file is wrong.<br/> coreutils is just using st_blocks to detect if the file <b>looks</b> sparse.</p> <p>It seems we can return a special FIEMAP for released file, with one region, with FIEMAP_EXTENT_DELALLOC or FIEMAP_EXTENT_UNKNOWN set.</p> <p>I've not checked <tt>tar</tt>.</p> <p>Andreas, do you think this solution is reasonable?</p> <p>Aurelien,<br/> Just to be sure that I fully understand what you mean in your last comment :</p> <p> _ for a released file, we can return st_blocks = 0 with correct st_size as already.<br/> _ this allows cp (and tar?) to presume it is a sparse-file and go thru related special logic.<br/> _ but then we need to change Lustre FIEMAP to return something like FIEMAP_EXTENT_DELALLOC or FIEMAP_EXTENT_UNKNOWN for released files to make cp happy.</p> <p>Did I understand and is this what you mean ?</p> <p>Then, only difference I see already is that we will not use the same cp logic for "real" non-sparse (plain or dense) but released files, if any specifics are made in cp mainly for performance but this could be just nothing regarding time for restore.</p> <p>&gt; Did I understand and is this what you mean ?</p> <p>This is perfectly correct!</p> <p>&gt; Then, only difference I see already is that we will not use the same cp logic for "real" non-sparse (plain or dense) but released files, if any specifics are made in cp mainly for performance but this could be just nothing regarding time for restore.</p> <p>I do not understand what you mean <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>What I see is that cp will read sequentially the just-previously-released-but-now-restored file, instead of doing something more efficient because the file could really be "a little bit" sparse.<br/> So before releasing the file, cp would have copied it efficiently, creating a sparse file.<br/> After restore, it will copy it fully, and the file copy will be no more sparse.<br/> I think this is acceptable.</p> <p>Hi Aurelien and Bruno,</p> <p>If we can fix this issue by fiemap that'll be great. I was afraid that we would fix tar issue but import another one by reporting incorrect block number.</p> <p>Jinshan</p> <p>I think this is very simple and that we're being a bit lazy here.</p> <p>It is a bug for release to change st_blocks. Remember that HSM is supposed to be transparent to the user. We shouldn't be trying to guess if some application will depend on st_blocks. Instead we should assume that some application depends on st_blocks and that there will be data loss because of this bug. It can be a blocker or not, I don't care so much. But it's a bug.</p> <p>The fact that we're focused on cp and tar is troubling. What about bbcp, rsync, zip, and all of those terrible data movers that the open science sites use? Have we checked them too?</p> <p>Moreover, it's very easy to think of situations where regular users will be confused by this. For example, when they run 'du -hs' on their homedir and then tell admin that the 8TB of data they just created must be lost. The admin who uses HSM on the other hand can be told and should know that 'dh -hs' will not always reflect the frontend FS usage.</p> <p>Hi John, by design, st_blocks is the actual disk blocks consumed by the file data. So it should be reasonable to report 0 for st_blocks for released files, this matches the . But I agree that reporting st_blocks to be zero is confusing here because this may cause applications to think this is a totally sparse file so fiemap won't be consulted at all.</p> <p>Any decisions for this problem are reasonable to me <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>John, </p> <p>I agree with you that checking each possible tool is not the right way.</p> <ul> <li>Users will be confused at first by this, but we should teach them. You cannot really consider this will be really fully transparent for them. I'm pretty sure that users will ask admins why their file accesses are random, because they do not realize that files could restored when accessed and this could requires mounting tapes which could be really long.</li> </ul> <ul> <li>DMF, which is the major HSM in HPC environment, and probably the HSM that will be used the most with Lustre/HSM, is exactly doing this. DMF, on their XFS/CXFS frontend, reports st_block == 0 for released(offline) files. DMF is probably doing this for 20 years now. This is the main solution right now on the market and DMF is very interested by Lustre as a frontend because CXFS does not scale (and because Lustre is very well deployed).</li> </ul> <ul> <li>Admins like they can see this difference between theoretical space and real disk usage through stat/du.</li> </ul> <blockquote><p>I think this is very simple and that we're being a bit lazy here.</p></blockquote> <p>What will be your proposal?</p> <p>I still consider this behavior could be changed, but this is not as simple as: this is totally wrong to reports 0 for st_blocks. I think reporting st_block as 0 is a feature, not a bug. Only the potential data corruption for some "too-smart" tools is a concern to me.</p> <p>Why this bug as been put as wontfix?</p> <p>We must, at least, fix FIEMAP.</p> <p>OK, I've checked with GHI (GPFS/HPSS Interface), which is the only concurrent product of Lustre/HSM.<br/> GHI also returns st_blocks == 0 when files are RELEASED.</p> <p>I propose that FIEMAP is modified for RELEASED files, and st_block still return 0. We will see if lot of people complains about that.</p> <p>Hello Aurelien,<br/> Seems everybody "agree" with st_blocks to be Null for released files. This is the reason ticket resolution has been set as "won't fix".</p> <p>Concerning change in FIEMAP, this can be tracked as a new ticket (or still this one), but with lower priority.</p> <p>BTW, do we know what answer to FIEMAP do provide DMF and/or GHI ??</p> <p>Seems to me that it is not a frozen interface, because when googling I found that a FIEMAP_EXTENT_SECONDARY flag exists (to indicate that extent data are on HSM) in some implementations that may fit our needs here, but then is it used/tested by the coreutils and other tools ? </p> <p>But anyway, I wrote a 1st patch attempt that returns a single extent with (FIEMAP_EXTENT_DELALLOC | FIEMAP_EXTENT_UNKNOWN | FIEMAP_EXTENT_LAST). It is <a href="http://review.whamcloud.com/7584" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7584</a>.</p> <p>&gt; BTW, do we know what answer to FIEMAP do provide DMF and/or GHI ??</p> <p>Ok I got the information from SGI.<br/> XFS(for DMF) is returning 1 full extent, with normal flag (not UNKNOWN or DELALLOC, etc)</p> <p>I do not know for GHI.</p> <p>There is still a patch to land for this bug.</p> <p>Humm during 1st patch-set testing I also found (seems not already reported) that doing a "filefrag" (ie FIEMAP syscall!) on a released file just crash with the follwing LBUG :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 13811:0:(lov_obd.c:2488:lov_fiemap()) ASSERTION( fm_local ) failed: LustreError: 13811:0:(lov_obd.c:2488:lov_fiemap()) LBUG Pid: 13811, comm: filefrag Call Trace: [&lt;ffffffffa0206895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa0206e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa084e56c&gt;] lov_get_info+0x10ac/0x1cb0 [lov] [&lt;ffffffff8112fca0&gt;] ? __lru_cache_add+0x40/0x90 [&lt;ffffffffa08697ab&gt;] ? lov_lsm_addref+0x6b/0x130 [lov] [&lt;ffffffffa0dbaab1&gt;] ll_do_fiemap+0x411/0x6b0 [lustre] [&lt;ffffffffa0dc5d97&gt;] ll_fiemap+0x117/0x590 [lustre] [&lt;ffffffff811956e5&gt;] do_vfs_ioctl+0x505/0x580 [&lt;ffffffff811957e1&gt;] sys_ioctl+0x81/0xa0 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>and this with or without my patch, this is due to unconditionally freeing "fm_local" at the end of lov_fiemap() routine, even if it was not allocated because of no-object/ENOMEM and also now when released!!</p> <p>Will fix that in patch-set #7 in addition to answer/address Andreas comments for patch-set #6.</p> <p>Hummm seems that "tar --sparse", instead of using FIEMAP as expected, uses an odd optimization (if st_blocks = 0) that cause a released file to be archived as a fully sparse file. I did not check all/latest versions of tar.</p> <p>Seems that for btrfs they encountered the problem with small files (ie, enough small to have the datas stored with the meta-datas and then report st_blocks = 0), as detailed in RedHat Bugzilla #757557 (at <a href="https://bugzilla.redhat.com/show_bug.cgi?id=757557" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.redhat.com/show_bug.cgi?id=757557</a>). And if I correctly understand, they fixed it by returning st_blocks = 1. Should we do the same ??</p> <p>We already do the same on the client if it only has writeback cache pages - if st_blocks == 0 but there is dirty data in the client cache we return st_blocks = 1. See cl_glimpse_lock():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (cl_isize_read(inode) &gt; 0 &amp;&amp; inode-&gt;i_blocks == 0) { /* * LU-417: Add dirty pages block count * lest i_blocks reports 0, some <span class="code-quote">"cp"</span> or * <span class="code-quote">"tar"</span> may think it's a completely * sparse file and skip it. */ inode-&gt;i_blocks = dirty_cnt(inode); </pre> </div></div> <p>Note that returning st_blocks = 1 can still be justified as "correct" for HSM released files, since this is still consuming one sector on the MDT filesystem for the inode.</p> <p>Most of filesystem does not count the space used by inode (unless there is extra block used by metadata, like for EA) when filling st_blocks, but POSIX does not say if it is only data or not. </p> <p>Anyway, I think we will have to go for the "st_blocks == 1" solution.</p> <p>I checked what XFS/DMF does for that. SGI says "don't use tar".... :/</p> <p>Submitted patch to have st_blocks = 1 returned for released files, at <a href="http://review.whamcloud.com/7776" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7776</a></p> <p><a href="http://review.whamcloud.com/7776" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7776</a> has land and <a href="http://review.whamcloud.com/7584" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7584</a> only misses reviews now.</p> <p>Landed for <a href="https://jira.whamcloud.com/browse/LU-3864" title="stat() on HSM released file returns st_blocks = 0" class="issue-link" data-issue-key="LU-3864"><del>LU-3864</del></a></p> Development Rank 1|hzvzpz: Rank (Obsolete) 10028