[LU-10318] DoM: HSM support

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.11.0
• Type: Technical task
• Priority: Major
• Reporter: Mikhail Pershin
• Assignee: Mikhail Pershin
• Resolution: Fixed
• Labels: DoM, HSM

Description

HSM functionality for DoM files to be supported and tested

Comments

[Andreas Dilger]

What does this mean exactly? Surely we don’t want to archive the small files from the MDT to tape, but it does make sense to allow PFL files with DoM components to be archived to tape and restored. IMHO, it makes sense to drop the DoM component when a larger PFL file is restored or migrated by lfs migrate, since the DoM component isn’t very useful at that point.

[John Hammond]

> Surely we don’t want to migrate the small files from the MDT to tape,

We do want the ability to archive every kind of file. Right?

[John Hammond]

Also, migrate is not an HSM action.

[Andreas Dilger]

I agree we want to be able to archive DoM files, probably for backup purposes and for consistency, but it won't really save much space if they are released since the inode currently still needs to be kept on the MDT. For archiving many small files, and trees of directories with HSM stub inodes, I think it would make the most sense to archive the files via an ldiskfs filesystem image, which is more efficient for both the Lustre and archive. Upon access, the filesystem image could be restored from the archive to an OST object in one shot, and mounted internally on the MDT via osd-ldiskfs and directly exported to clients, rather than having thousands of archive restore actions.

On the restore side, it doesn't really make sense to restore a large PFL file with a DoM first component vs. dropping the DoM component and just extending the first OST component to start at offset zero. For that matter, restoring any file should potentially drop the first component(s) and use the layout of the last instantiated component to determine the stripe count/size, since a PFL layout is most useful when the file size is not known in advance.

[Mikhail Pershin]

So let's summarize that a bit.
1) DOM files should be archived.
2) DOM-only files could be not released technically, by setting HS_NORELEASE flag if that is needed.
3) DOM+OST files are better to convert to only OST layout, and the best moment for this is release time, we may create new layout without DOM component and them swap layouts. So file will be restored right to OSTs after all

There are tasks to support all above:
1) return data_version for DOM component (I have patch for this) so archive will work correctly
2) consider to add HS_RELEASE flag on archiving action if file has only DOM component
3) special case for release of DOM+OST files and support for MDT inode data blocks truncate upon release action (this is also needed for migration cases anyway)

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/30449
Subject: LU-10318 dom: support DATA_VERSION IO type
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e4ccbac16e4f601815fc1dc0701fac99772370e9

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/30449/
Subject: LU-10318 dom: support DATA_VERSION IO type
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1e7fc14bbf48f7e89876cbaa609972981e343944

[Peter Jones]

Landed for 2.11