https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5896 Lustre <p>After a file has been archived and migrated, it's not possible to release it.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># dd if=/dev/urandom of=rand1 bs=4096 count=1024 ... # lfs hsm_archive rand1 # lfs migrate -o 0 -S 65536 rand1 # lfs hsm_release rand1 Cannot send HSM request (use of rand1): Device or resource busy </pre> </div></div> <p>The file content is still accessible, and its hsm state is correct.</p> LU-5896

lfs: can't release a migrated file

Bug Major Open Unresolved Artem Blagodarenko Frank Zago CEA Mon, 10 Nov 2014 21:11:38 +0000 Thu, 30 Nov 2023 16:44:45 +0000 Upstream 2 17 <p>What happens here is a data version mismatch. The error originally returned by the MDS is EPERM (in mdt_hsm_release) but is translated into EBUSY by ll_close_inode_openhandle.</p> <p>I am unsure how to fix it best. I can think of different approaches:</p> <ul class="alternate" type="square"> <li>A brutal workaround is to mark files as dirty after migration, which sucks since we know that data have not changed... Whoever hits this bug (and is willing to re-archive files) can already issue `lfs set_state --dirty &lt;filename&gt;' to fix things up;</li> <li>Remove the data version check in hsm release? The dirty flag and file lease should allow that but I am concerned there may exist crash/recov cases where doing so is unsafe. A mistake in this area could easily result in data corruption / loss;</li> <li>Add an ioctl to force the hsm data version on archived files.</li> </ul> <p>Henri no longer works on this.</p> <p>I will try to publish a fix, but I am not familiar with the affected code base.<br/> If anyone is willing to help, I would really appreciate it.</p> <p>Currently the approach I am taking is to have <tt>lfs migrate</tt> send both the initial and final dataversions of the migrated file when it commits the migration (with either <tt>llapi_fswap_layouts()</tt> / <tt>llapi_fswap_layouts_grouplock()</tt>).</p> <p>The MDS can then compare the initial dataversion with the one stored in the HSM attributes. If it is the same, it will update the old value with the new one; otherwise (HSM's dirty bit should already be set, but in case it is not) it will set HSM's dirty bit.</p> <p>A little update,</p> <p>I have not gotten around to implement the fix suggested in my previous post. I think I need to add a field to the SWAP_LAYOUT RPC and I was warned this is not a trivial change.</p> <p>In the meantime, I have re-implemented lfs migrate to forge the HSM xattr on the volatile file right before it is swapped. This seems to work.</p> <p>There are several shortcomings to this approach:</p> <ul> <li>forging the HSM xattr is not normally possible upstream (we already have a patch in production to allow just that) ;</li> <li>even with our patch, only root can forge the HSM xattr (as it is a "trusted" xattr) ;</li> <li>I did not re-implement every feature of lfs migrate (only <tt>lfs_migrate()</tt> and only the blocking mode) ;</li> <li>any user still using lfs migrate (instead of my re-implemented version) will hit this LU.</li> </ul> <p>For now though, this is probably what we will use in production. At least until we can produce an upstream-able solution.</p> <p>Ben Evans (bevans@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/35022" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35022</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5896" title="lfs: can&#39;t release a migrated file" class="issue-link" data-issue-key="LU-5896">LU-5896</a> hsm: can't remove a migrated file<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d065772a16d1183cd41f013468ac88d73d4c4047</p> <p>I can confirm that there are not enough fields to hold two <tt>data_versions</tt> in the <tt>MDS_SWAP_LAYOUTS</tt> RPC. So we have to add them somewher. I think <tt>struct mdc_swap_layouts</tt> – being the payload of the RPC – is a good candidate:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /** layout swap request structure * fid1 and fid2 are in mdt_body */ struct mdc_swap_layouts { __u64 msl_flags; #<span class="code-keyword">if</span> TODO __u64 msl_dvs[2]; #endif }; </pre> </div></div> <p>On the userspace side, the struct passed to the <tt>LL_IOC_LOV_SWAP_LAYOUTS</tt> ioctl already has the required fields:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> struct lustre_swap_layouts { __u64 sl_flags; __u32 sl_fd; __u32 sl_fd; __u64 sl_dv1; __u64 sl_dv2; }; </pre> </div></div> <p>There also needs to be a new flag, something like: <tt>SWAP_LAYOUTS_SAME_DATA</tt> to indicate to the MDS that both layouts point at the same data and the HSM xattrs should be updated accordingly. When this flag is used, it should be passed down to the MDS, as well as both <tt>data_versions</tt>.</p> <p>If the flag is not used, the new default should be to swap the HSM xattrs. There currently is a flag for that (which only the MDT code uses), but even without it, there really is no reason not to swap them: the HSM xattr is part of a file's layout.</p> <p>After that, updating <tt>lfs_migrate()</tt> should be straightforward.</p> <p>—</p> <p>This fix is not fully satisfying though. Swapping layouts should really just... <em>swap layouts</em>. If there is information that is tied to the layout (eg. HSM xattr), it should also be swapped, shouldn't it? So why am I proposing something that goes the opposite way? (ie. keeping the HSM xattr in some circumstances and update it in-place)</p> <p>Well, it all comes down to how HSM's security works. The HSM xattr lives under the <tt>trusted</tt> xattr prefix. That means that only root is able to modify it <span class="error">&#91;*&#93;</span>. But any user can run a layout swap (on the condition that they have write access to both files), and by extension a file migration. This feels right, but it is simply incompatible with how HSM works. Ideally, to migrat{{e }}<tt>file.txt</tt><tt>}}, one woud create a volatile file; copy another {{file.txt</tt>'s data into the volatile; <b>set the HSM xattr of the volatile</b>; and swap <tt>file.txt</tt>'s layout with the volatile's. Not possible.</p> <p>Personnally, I do not understand why users are not allowed to tell Lustre "there is a copy of &lt;this&gt; file &lt;here&gt;" (read "&lt;this&gt; file" as "&lt;this#1&gt; version of &lt;this#2&gt; file"). I mean, I get that it would open all sorts of scary doors, like "What if &lt;here&gt; points at data the user is not supposed to have access to?". But that really should be solved by copytools, or some piece of software between Lustre and whatever &lt;here&gt; is. I feel like it was convenient not to have to care about this at first, but now it just comes back to bite us.</p> <p><span class="error">&#91;*&#93;</span> actually, not even root can edit the HSM xattr, Lustre silently ignores setxattr operations on <tt>trusted.hsm</tt>, but that could be changed, so it does not really matter.</p> <p>—</p> <p>As Ben pointed out on gerrit, sites that have already migrated archived files are still going to need to "repair" the "broken" HSM xattrs. I suggest forcibly setting the dirty bit (with <tt>lfs hsm_set --dirty &lt;file&gt;</tt>) and schedule ARCHIVE requests on the concerned files (or let the policy engine schedule those if it can).</p> <p>There are probably more efficient ways to do this though. Ways that would not involve re-archiving <em>a lot of data</em>.</p> <p>Artem Blagodarenko (c17828@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/36548" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36548</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5896" title="lfs: can&#39;t release a migrated file" class="issue-link" data-issue-key="LU-5896">LU-5896</a> tests: archive-migrate-release hsm test<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f93b23ec03335fb45d2adc6db684d56f9fbb92e3</p> <p>Artem Blagodarenko (c17828@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/36549" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36549</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5896" title="lfs: can&#39;t release a migrated file" class="issue-link" data-issue-key="LU-5896">LU-5896</a> hsm: update data version if layout is changed<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 75135d80e76454724b08b0b38d989845f4065e8a</p> <p>When<br/> swap_layouts o1 o2，<br/> we must consider the following cases:<br/> 1. migrate<br/> we know that the data content in o1 and o2 are same,<br/> then it needs to indicate it via a flags SWAP_LAYOUTS_DATA_MIGRATE，<br/> then when MDT know the swapping layouts is for data migration, it will do as following:</p> <ul class="alternate" type="square"> <li>if o1 has been archived, check whether whether the archive is dirty (HS_DIRTY).<br/> if so, do nothing to the HSM attrs for o1;<br/> otherwise, update the data version in HSM attrs of o1 with the data version of o2.</li> <li>if o2 has been archived, check whether whether the archive is dirty (HS_DIRTY).<br/> if so, do nothing to the HSM attrs for o2;<br/> otherwise, update the data version in HSM attrs of o2 with the data version of o1.</li> </ul> <p>2. normal layout swap:</p> <ol> <li>lfs swap_layouts o1 o2<br/> As we don't know whether the content of o1 and o2 are same, thus on MDT:</li> </ol> <ul class="alternate" type="square"> <li>if o1 has been archived, check whether whether the archive is dirty (HS_DIRTY).<br/> if so, do nothing to the HSM attrs for o1;<br/> otherwise, update the state with HS_DIRTY for HSM attrs of o1.</li> <li>if o2 has been archived, check whether whether the archive is dirty (HS_DIRTY).<br/> if so, do nothing to the HSM attrs for o2;<br/> otherwise, update the state with HS_DIRTY for HSM attrs of o1.</li> </ul> <p>3. data restore<br/> MDT will swap the layouts after finished HSM data restore.<br/> SWAP_LAYOUTS_MDS_HSM<br/> do as it what does in Lustre currently.</p> <p>All these modification must be in MDD layout in a transaction. Otherwise, it may result in an inconsistence. </p> <p>But I am worried that an extra flag SWAP_LAYOUTS_DATA_MIGRATE will be used maliciously by a user...</p> <p>if the content of o1 and o2 are not same, then after SWAP_LAYOUTS_DATA_MIGRATE, it would wrongly think the data in Lustre after swapped is same as in HSM archive...</p> <p>To avoid malicious using of SWAP_LAYOUTS_DATA_MIFRATE flag when migrate a file, it would better to move the dat copy phase migrate_copy_data() in migrate_block() to the kernel which can borrow the code pcc_data_copy() in PCC, thus we can ensure the content of two files are exactly same.</p> <p>"<br/> If the flag is not used, the new default should be to swap the HSM xattrs. There currently is a flag for that (which only the MDT code uses), but even without it, there really is no reason not to swap them: the HSM xattr is part of a file's layout.<br/> "<br/> -------------------<br/> We can not simply swap the HSM xattrs, as the HSM copy is naming and locating according to FID.<br/> Only swapping the HSM attrs, will lead the file points to the wrong HSM copy...<br/> Maybe we need to store the correct FID in HSM xattrs also which may not same as the FID of the file, but the one contain the consistent data in HSM archive with the data in Lustre.</p> <p>So if the extra SWAP_LAYOUTS_DATA_MIGRATE flag is Problematic, maybe storing the FID (via it we can locate the HSM copy in archive) into HSM attrs is a solution.<br/> But this may need to make lots of change for the HSM code...</p> <p>HSM solves this by calling ll_data_version to get the current data_version of the fid (see ll_ioc_copy_start)</p> <p>There's no reason we couldn't do a similar thing in the kernel layer for migrate.  Get the currrent dv, compare it to the version stored in the HSM xattr.  If they're the same, perform the layout swap and update the xattr.  If they're different, perform the layout swap, and make sure to mark the xattr with the DIRTY flag, if it's not already.</p> <blockquote><p>But I am worried that an extra flag SWAP_LAYOUTS_DATA_MIGRATE will be used maliciously by a user...</p></blockquote> <p>It cannot really be that malicious. To be able to do a <tt>swap_layout</tt>, one has to have write-access to the file. At that point, you can just truncate the file to 0 or overwrite it.</p> <blockquote><p>We can not simply swap the HSM xattrs, as the HSM copy is naming and locating according to FID. (...)</p></blockquote> <p>The archived copy does not need to be named using the file's FID in Lustre. That is just what the lustre copytool does.<br/> I also feel like you think migrating a file changes its FID. It does not.</p> <p>Hi,</p> <p>Is there anyone maintaining the <a href="https://review.whamcloud.com/36549" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36549</a> or actively developing something for this issue (or for the non-blocking case <a href="https://jira.whamcloud.com/browse/LU-13048" title="Non-blocking migrate: Unable to release file from disk (Cannot send HSM request)" class="issue-link" data-issue-key="LU-13048">LU-13048</a>) ?<br/> If not, the CEA, would like to continue the development for the blocking and the non-blocking version.</p> <p>This become a blocking issue for the CEA. Without a workaround, this forces to re-archive every migrated files (not acceptable on filesystem mixing SSD and rotational pools with HSM).</p> <p>"Guillaume Courrier &lt;guillaume.courrier@cea.fr&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49377" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49377</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5896" title="lfs: can&#39;t release a migrated file" class="issue-link" data-issue-key="LU-5896">LU-5896</a>: allow release after blocking migrations<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 041adca0986b27cb8ce905343f9bf95a068ddbd0</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53296" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53296</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5896" title="lfs: can&#39;t release a migrated file" class="issue-link" data-issue-key="LU-5896">LU-5896</a> mdt: allow release after blocking migrations<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 2efedbd708390c745ef0a61dee8943a293fde429</p> Related LU-13048 Development Rank 1|hzx0mn: Rank (Obsolete) 16477 Severity