https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10318 Lustre <p>HSM functionality for DoM files to be supported and tested</p> LU-10318

DoM: HSM support

Technical task LU-10176 Major Resolved Fixed Mikhail Pershin Mikhail Pershin DoM HSM Mon, 4 Dec 2017 14:27:58 +0000 Mon, 17 Sep 2018 21:19:35 +0000 Sat, 3 Mar 2018 13:57:43 +0000 Lustre 2.11.0 0 5 <p>What does this mean exactly? Surely we don’t want to archive the small files from the MDT to tape, but it does make sense to allow PFL files with DoM components to be archived to tape and restored. IMHO, it makes sense to drop the DoM component when a larger PFL file is restored or migrated by <tt>lfs migrate</tt>, since the DoM component isn’t very useful at that point. </p> <p>&gt; Surely we don’t want to migrate the small files from the MDT to tape,</p> <p>We do want the ability to archive every kind of file. Right?</p> <p>Also, migrate is not an HSM action.</p> <p>I agree we want to be able to archive DoM files, probably for backup purposes and for consistency, but it won't really save much space if they are released since the inode currently still needs to be kept on the MDT. For archiving many small files, and trees of directories with HSM stub inodes, I think it would make the most sense to archive the files via an ldiskfs filesystem image, which is more efficient for both the Lustre and archive. Upon access, the filesystem image could be restored from the archive to an OST object in one shot, and mounted internally on the MDT via osd-ldiskfs and directly exported to clients, rather than having thousands of archive restore actions.</p> <p>On the restore side, it doesn't really make sense to restore a large PFL file with a DoM first component vs. dropping the DoM component and just extending the first OST component to start at offset zero. For that matter, restoring any file should potentially drop the first component(s) and use the layout of the last instantiated component to determine the stripe count/size, since a PFL layout is most useful when the file size is not known in advance.</p> <p>So let's summarize that a bit. <br/> 1) DOM files should be archived. <br/> 2) DOM-only files could be not released technically, by setting HS_NORELEASE flag if that is needed.<br/> 3) DOM+OST files are better to convert to only OST layout, and the best moment for this is release time, we may create new layout without DOM component and them swap layouts. So file will be restored right to OSTs after all</p> <p>There are tasks to support all above:<br/> 1) return data_version for DOM component (I have patch for this) so archive will work correctly<br/> 2) consider to add HS_RELEASE flag on archiving action if file has only DOM component<br/> 3) special case for release of DOM+OST files and support for MDT inode data blocks truncate upon release action (this is also needed for migration cases anyway)</p> <p>Mike Pershin (mike.pershin@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30449" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30449</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10318" title="DoM: HSM support" class="issue-link" data-issue-key="LU-10318"><del>LU-10318</del></a> dom: support DATA_VERSION IO type<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e4ccbac16e4f601815fc1dc0701fac99772370e9</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30449/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30449/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10318" title="DoM: HSM support" class="issue-link" data-issue-key="LU-10318"><del>LU-10318</del></a> dom: support DATA_VERSION IO type<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1e7fc14bbf48f7e89876cbaa609972981e343944</p> <p>Landed for 2.11</p> Development Rank 1|hzzomv: Rank (Obsolete) 9223372036854775807