https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12031 Lustre <p> There is an issue when releasing a file striped with DoM after an hsm_restore.</p> <p>To reproduce:</p> <p>1) create a file with a 1st component on MDT:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lfs setstripe -E 1M -L mdt -E -1 -S 4M -c -1 /mnt/lustre/domfile</pre> </div></div> <p>2) archive and release the file (requires HSM set up)<br/>  </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lfs hsm_archive /mnt/lustre/domfile # (wait for archive to complete) lfs hsm_release</pre> </div></div> <p>3) restore the file</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lfs hsm_restore /mnt/lustre/domfile # or cat /mnt/lustre/domfile</pre> </div></div> <p>4) release the file =&gt; FAILS </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lfs hsm_release /mnt/lustre/domfile Cannot send HSM request (use of /mnt/lustre/domfile): Device or resource busy</pre> </div></div> <p> <br/> It may be something wrong with the data version stored in hsm EA.</p> LU-12031

DoM/HSM: hsm_release fails after hsm_restore

Bug Minor Resolved Fixed Mikhail Pershin CEA CEA DoM HSM Thu, 28 Feb 2019 13:03:30 +0000 Wed, 7 Feb 2024 23:28:55 +0000 Wed, 19 Jul 2023 13:10:59 +0000 Lustre 2.12.0 Lustre 2.16.0 0 18 <p>Mike</p> <p>Can you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>yes, there is problem with data version mismatch. I will investigate</p> <p>This issue is bigger than it seems at first sign. The initial problem was about DOM data version mismatch after restore. When file with DoM stripe is being restored, the HSM XATTR stores data version but the same <tt>setxattr</tt> operation changes inode version which is used as data version for MDT stripe as well. So it is just impossible to store current data version for DOM file because of that. <br/> The proposed solution is:<br/> 1. if there is next component after DoM one then don't restore DoM stripe but delete it. The data will go to the next component. This makes sense because if the next component is used already then DoM stripe lost most of benefits and there is no big sense in having it.<br/> 2. if DoM stripe is the only stripe being used then it is worth to keep it. In that case such file can be considered as non-released always. In general that is OK because they consume not much space. The question is - would that be handled nicely by HSM software? I suppose that should be OK at least because we can set <tt>norelease</tt> flag on selected files, technically that is the same.</p> <p>Meanwhile, working on that issue I have found another one related to VOLATILE file handling. Usually such files are used as temporary file to copy data and then swap layout with original file, e.g. during HSM release and restore. The problem is that such files they can be created with DoM layout which cannot be swapped because we swap layouts but not inodes data. Such VOLATILE files can get layout from striping saved during archive operation or from default layout. In any case, it is unsafe to create VOLATILE file with DoM layout and that should be prohibited. <a href="https://jira.whamcloud.com/browse/LU-13515" title="prohibit DOM layout for VOLATILE file" class="issue-link" data-issue-key="LU-13515">LU-13515</a> was created for that</p> <p>Would adding <a href="https://jira.whamcloud.com/browse/LU-13384" title="HSM copytool API for external coordinator" class="issue-link" data-issue-key="LU-13384">LU-13384</a> and performing Restore-&gt;Archive (without datacopy)-&gt;Release fix most of this all in Userspace?  We would regenerate the correct version during the fake archive step and be able to release cleanly.</p> <p>This probably won't address the DoM stripe issue, changing HSM to use file layouts would probably be the best way to handle that.</p> <p>Ben, yes, version could be corrected by userspace  tools, though I wouldn't call that as a fix. </p> <p>While I proposed possible solution above with non-released DoM component and have patch for it, I am not confident in it. It looks 'hacky' in all that layout manipulations just to avoid wrong data version. The correct solution would be separated <tt>'data_version'</tt> maintained for DoM file in addition to <tt>inode_version</tt>. I am figuring out how difficult that could be</p> <p>well, it looks like having separate <tt>data_version</tt> is not the solution here. The main problem is a restore process which is using volatile file for data transfer. When transfer is completed the file data version is calculated including DoM stripe version and is stored in HSM extended attribute. The next step is layout swap between volatile file and original one which cannot keep the same DoM stripe because it is an inode data. So after layout swap  data versions of DoM stripe and file itself are always different from the one stored in HSM xattr which is just copied to the original file from volatile one. I can't find the way how this could be done correctly without compromising whole process. We can't just allow DoM stripe data version be different because that would hide possible real problem by that. From that moment any further attempts to avoid that looks not less 'hacky' than the solution proposed initially. </p> <p>The another idea could be different approach to calculating data_version for DoM stripe, e.g. make it not transaction-based but content-based, like checksum of DoM data, considering it is not big in size and that <tt>data_version</tt> is being used by HSM mostly, so quite rare operation to affect performance. Also that means we don't need to store it anywhere which is also pros, since separate 'data_version' can be stored as new XATTR only</p> <p>Mike, in case it is helpful to you, newer ext4 code has a "swap data" operation that is meant to allow swapping a "volatile" file into the boot loader inode. This could be used to swap data between two DoM files if needed. </p> <p>That said, your recent comments indicate that it isn't the DoM data swap that is the main obstacle, but the ordering problem of the data version. IMHO, a content-based hash is probably still too expensive if the data version is used regularly. That would make inode operations that need 1KB/inode into data operations that need (possibly) 1MB/inode, or at least 64KB/inode. There was some discussion recently on whether the data version should be used for NFS file modification tracking, so doing a DoM checksum on every file access would be punishing. Storing a separate xattr would be <b>much</b> more efficient. </p> <p>Maybe I'm missing something, but is it not possible to store the "original" object version in the swapped MDT inode? This might mess with recovery, but if the volatile file is gone it would be pretty clear that the layout swap could not be replayed in any case. We could also special-case the replay operation for layout swap to take this into consideration. </p> <p>I think it's much more sinister than that, in a non-DoM case, the data_version is calculated on each portion of the file (all on OSTs) then combined into a single data version and written to an XATTR on the MDT.  For DoM, the act of writing the HSM data_version to an XATTR would cause the data_version on the MDT to change.  Unless we can predict what the "next" DoM data_version is, so that the HSM XATTR agrees with the calculated data_version after the XATTR is written.  So for a restore-&gt;release case it will <b>always</b> be wrong.</p> <p>Ben, unlike <tt>inode_version</tt> the <tt>data_version</tt> is not changed by xattr set, that is why I was trying to introduce it. Like on OST it would be changed only on data change - write, truncate and fallocate. So that solves problems when metadata operations affects <tt>data_version</tt> though required separated xattr to store it.</p> <p>but what is actually bad about DoM release/restore - it is the fact that DoM stripe is not actually archived and is not restored after all though it 'looks' so. On archive operation it is read and stored in atchive but unlike OST object the data in inode is not truncated and stay untouched. Upon restore DoM data is read from archive and is written to volatile file inode. But on swap layout it is gone along with volatile file actually and original data in original inode become just visible as layout says it exists. So all that time DoM data stays in inode and its copy in archive is just lost along with volatile file. That means there is no any sense to archive what is always kept in inode on disk. Therefore I tend to return back to first solution when DoM stripe is either non-released or just removed in favor of first ost stripe if exists</p> <p>It would be nice if instead of all the playing around with temp files we could just restore to a stripe, and once completed mark it as primary.  We should also be able to restore all the other layout information as well and mark them as secondary.</p> <p>Ben - see <a href="https://jira.whamcloud.com/browse/LU-9961" title="FLR2: Relocating objects to a new OST" class="issue-link" data-issue-key="LU-9961">LU-9961</a></p> <p>"Mike Pershin &lt;mpershin@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47139" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47139</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12031" title="DoM/HSM: hsm_release fails after hsm_restore" class="issue-link" data-issue-key="LU-12031"><del>LU-12031</del></a> mdt: explicit data version of DoM files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5dbeabb034654f258162f05722477645e5f2fffe</p> <p>Rather than storing yet another xattr on the DoM inode in this case (which might have issues with backup/restore, etc.), what about just <b>not</b> updating i_version on setxattr from HSM restore (or resetting it to the pre-update i_version)?</p> <p>"Sergey Cheremencev &lt;sergey.cheremencev@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47497" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47497</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12031" title="DoM/HSM: hsm_release fails after hsm_restore" class="issue-link" data-issue-key="LU-12031"><del>LU-12031</del></a> mdt: proof of concept<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b05ecdbcc682c1d2ef290110bd052bc3e0f2e61a</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/47139/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/47139/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12031" title="DoM/HSM: hsm_release fails after hsm_restore" class="issue-link" data-issue-key="LU-12031"><del>LU-12031</del></a> mdt: explicit data version of DoM files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: aae3289adb2bbc192870f195b78044484f717e16</p> <p>Peter, is there any remaining work for this issue or can it be closed?</p> <p>There is still an unhanded patch tracked under this ticket - <a href="https://review.whamcloud.com/#/c/fs/lustre-release/+/47497/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/fs/lustre-release/+/47497/</a> - but I'll defer to <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=scherementsev" class="user-hover" rel="scherementsev">scherementsev</a> as to whether that is needed...</p> <p>I've abandoned <a href="https://review.whamcloud.com/#/c/fs/lustre-release/+/47497/." class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/fs/lustre-release/+/47497/</a>. Agree to close if there is no known issues after landing 47139.</p> <p>there is nothing left on my side as well</p> Duplicate Related LU-13515 Development Rank 1|i00chz: Rank (Obsolete) 9223372036854775807 Severity