https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6866 Lustre <p>Migrating a file between MDTs changes the FID of the file. The file FID is the used to identify the file in the HSM archive (unfortunately). So, for example, if a released file is migrated from one MDT to another then it cannot be restored.</p> LU-6866

MDT file migration is incompatible with HSM

Bug Major Resolved Fixed WC Triage John Hammond HSM migration Fri, 17 Jul 2015 15:10:37 +0000 Wed, 17 Apr 2019 17:11:33 +0000 Fri, 18 Dec 2015 14:10:44 +0000 Lustre 2.8.0 0 18 <p>I think the path forward here is to generate a ChangeLog record for each migrated file (per <a href="https://jira.whamcloud.com/browse/LU-6868" title="MDT migration does not generate changelog records" class="issue-link" data-issue-key="LU-6868"><del>LU-6868</del></a>) and let the HSM policy engine and copy tool handle the rename from the old FID to the new FID. AFAIK, the current archives in use (at least POSIX and HPSS) can handle renames of the files in the archive.</p> <p>Some archives, such as S3, cannot handle rename of files in the archive since the S3 objects are immutable. However, the use of FIDs in the immutable archive also causes problems with HSM import (unrelated to DNE migrate), which current requires the ability to change the file FIDs in the archive after import, so there have been discussions for long-term changes to Lustre HSM to store an archive-supplied UUID to persistently identify objects in Lustre rather than using the Lustre FID to track files in the archive. That would also solve this problem, but is a rather significant change, and would likely make sense to implement when HSM is integrated with PFL to store the archive UUID in a new "archive layout" type.</p> <p>Just watching the <a href="http://www.eofs.eu/fileadmin/lad2015/slides/05_Herve_Toureille_cines_lustre_hsm.pdf" class="external-link" target="_blank" rel="nofollow noopener">CINES HSM</a> and <a href="http://www.eofs.eu/fileadmin/lad2015/slides/06_Maria_Perez_Gutierrez_LustreHSMfor_LAD_v1_4.pdf" class="external-link" target="_blank" rel="nofollow noopener">DDN WOS</a> HSM presentations at LAD, and it seems to me that both of these implementations could benefit from storing the UUID into an HSM file layout. Storing the WOS object UUID in the Lustre inode layout could be used to identify objects in the archive, instead of (or in addition to) using the Lustre FID to identify the object in the archive. This could also solve the problem that CINES reported with not being able to undelete files directly from the archive back into the Lustre filesystem, because the deleted files are identified by the old FID in the archive and there is no way to recreate that FID in Lustre today..</p> <p>As Robert suggested previously, storing the archive UUID into the Lustre HSM layout, it will allow creating a new stub file or migrating the inode (along with the archive UUID stored in the layout xattr) across MDTs, and then nothing needs to be done in the archive since the UUID stays the same. The HSM archive probably doesn't even need to care about what the Lustre FID is, since it may change over time and that shouldn't cause the file to be re-archived.</p> <p>Developing the PFL composite layouts is starting, but as yet there isn't a plan to implement the HSM layout (i.e. LOV_MAGIC_HSM containing essentially struct hsm_attrs + struct uuid, assuming the standard 128-bit/16-byte UUID is enough for S3/WOS and other archive identifiers). I don't think there is anyone at Intel that will be working on the HSM-in-layout currently.</p> <p>That's right, HSM archives should never see the FID, and should only use the UUID or what identifier makes sense for that particular archive. Currently I store the UUID as an xattr in to achieve the same result. </p> <p>Di</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>If I understand correctly, there are two sub-tasks here.</p> <p>1. add an change log for migration, which should be easy, and can be done under <a href="https://jira.whamcloud.com/browse/LU-6868" title="MDT migration does not generate changelog records" class="issue-link" data-issue-key="LU-6868"><del>LU-6868</del></a>.<br/> 2. Add HSM layout (lsm_attrs + uuid) into the MDS file. So release and restores will based on UUID or FID? (Sorry, not so familiar with the process). If it will be based on UUID, does that mean we need lookup FID by UUID? who will maintain the lookup table, lustre, the HSM tool or some others? </p> <p>HSM actions contain FIDs, and the hsm tool retrieves the UUID using getxattr(). I don't see where we would need to lookup FID by UUID, so we don't need any lookup table for that. A copytool/policy engine might maintain a mapping between UUID and pathname(s), but that would be done outside of Lustre.</p> <p>Actually, the delete action needs the UUID, since at that point the FID and its xattrs have already been deleted. Another option might be to include the UUID in the unlink changelog record, and the policy engine can pass that on the backends associated with that file.</p> <p>Hmm, I thought the problem is that migration changes the FID, so the FIDs in archives is not valid anymore? so we instead use UUID to present the object, which will never be changed? I must miss sth here.</p> <p>The actual bug here is that FIDs are used in archive. The archive needs to use UUIDs (or anything that is unique), and not FIDs. </p> <p>Ah, it is copytool to take care the mapping from this unique identifier to the achieve file. I had wrong ideas before. Then we should not use FID as the identifier indeed.</p> <p>Ok, I will add migration record for change log. But it seems a temporary solution, i.e. once we have UUID, this can be removed. So should I add a migration record or some one will work on this HSM layout thing (Sorry, I am not familiar with the HSM release/restore thing)? </p> <p>If my undestanding is correct, archive UUID means we can define unique ID that is stored into EA?<br/> If so, that would be really helpful. Robert and Andreas mentioned above, one of big limitation with WOS and Lustre today, that is delete operation. we are storing OID (WOS object ID) into EA after migration, but if file is removed, it's hard to search objects in WOS becouse OIDs is gone when file removed.<br/> So, we need extra database to map FID and OID for now. If the changelog keeps UUID ratehr than FID, that would be easy to do asynchronous remove objects in WOS without external database after unlink files on the Lustre.</p> <p>I believe currently MDT migration triggers unlink and create records in the source MDT and target MDT changelogs respectively, so one option is to just add a MIGRATE flag to those records so we can differentiate the migrate unlink from a real unlink. Or is do yo have a better solution for this already?</p> <p>Would it be possible to standardize the EA field we use for the UUID instead of creating a hsm layout? As Ihara mentioned, we eed to support the ability to assign UUIDs to a file, so it is useful to use the normal EA interface for this instead of adding a new API. Also, some backends might use different kinds of identifiers, so it might not always be an actual UUID. If we have a standardize this, then we can add this extra data to the delete changelog record, and that solves the main problem we have today with using external IDs. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>I believe currently MDT migration triggers unlink and create records in the source MDT and target MDT changelogs respectively, so one option is to just add a MIGRATE flag to those records so we can differentiate the migrate unlink from a real unlink. Or is do yo have a better solution for this already? </pre> </div></div> <p>The whole migration is implemented in MDD layer, where changelogs are generated, and there are no changelog created for migration. So I can just add one for this ticket. I will push the patch to <a href="https://jira.whamcloud.com/browse/LU-6868" title="MDT migration does not generate changelog records" class="issue-link" data-issue-key="LU-6868"><del>LU-6868</del></a>.</p> <p>As long is there is no unlink changelog generated, then that's fine. A migration changelog is useful for the PE to know that file has changed MDTs, but this doesn't really impact the archive. I'll open a new ticket for adding UUID to delete changelog, which is much more important for HSM. </p> <p>Please see <a href="https://jira.whamcloud.com/browse/LU-7207" title="HSM: Add Archive UUID to delete changelog records" class="issue-link" data-issue-key="LU-7207">LU-7207</a> for the proposed format for the external ID EA and including it in the delete changelog. </p> <p>Thanks Robert. Then I will close this one, and push the migration record patch to <a href="https://jira.whamcloud.com/browse/LU-6868" title="MDT migration does not generate changelog records" class="issue-link" data-issue-key="LU-6868"><del>LU-6868</del></a></p> <p>Reopening since <tt>lhsmtool_posix</tt> does not implement the discussed functionality.</p> <p>John L. Hammond (john.hammond@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17511" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17511</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6866" title="MDT file migration is incompatible with HSM" class="issue-link" data-issue-key="LU-6866"><del>LU-6866</del></a> hsm: prevent migration of HSM archived files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3f194332d27db47f0ff1ec6ef947454f8b51c61d</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17511/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17511/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6866" title="MDT file migration is incompatible with HSM" class="issue-link" data-issue-key="LU-6866"><del>LU-6866</del></a> hsm: prevent migration of HSM archived files<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: fdddeb25a54324cbcc2c99daf38b9f778fdbbec3</p> <p>Landed for 2.8</p> Related LU-2430 LU-6868 LU-7207 LU-7586 LU-7607 Development Rank 1|hzxifz: Rank (Obsolete) 9223372036854775807 Severity