It would be useful to add an attribute to a file component that prevents it from being automatically migrated to another storage pool or removed. This is useful, for example, to keep files in a flash pool because it is known that the file is accessed frequently with a poor IO pattern, because it is a VIP user, or whatever. This could be set on file components on an arbitrary basis, rather than having to describe a very complex policy to exclude specific files by pathname or FID.
It may also be useful to similarly force a file to be kept in a disk pool because it is known to always be accessed via large streaming IOs, and even if it is frequently accessed, the aggregate bandwidth of the spinning storage is higher than that of the flash, or the file is so large that it would consume too much of the flash pool, or maybe because the admin really doesn't like some user. 
I think there are several possible mechanisms for this:
- per file attribute stored in LUSTRE_*_FL attribute flags of the inode. There are common "chattr" and "lsattr" flags for changing and showing these attributes. However, there is not currently anything similar to this functionality in the upstream kernel, and the remaining unused attributes are very limited, and may conflict with other upstream users, so we would have an uphill battle to get something accepted into the kernel.
- per file attribute stored in LMAC_* or LMAI_* flags in the LMA xattr of the inode. This has the drawback that we don't currently have a user interface to set/change these flags, and they are often core to the functionality of the inode and we would (essentially) need per-bit authentication to avoid users doing bad things to their files.
- per comoponent/mirror flag stored in LCME_FL_PIN flag, similar to LCME_FL_PREFER and LCME_FL_NOSYNC. This has the benefit that there is already an interface for setting/showing these attributes, even at create time, and they could be set on a per-component basis (e.g. either on the flash or disk components, or both) instead of a per-file basis. We would need to decide on semantics for what these flags mean if set on only one component of a file mirrored on two different pools.
I think the third option is the preferred one, since it also ties in nicely with the whole concept of mirroring/pools, etc. Some open questions on behavior for different cases:
- do we need to limit LCME_FL_PIN to be root-only, or controlled by a tunable like mdt.*.enable_pin_gid=<0,gid,-1> to limit/allow root, admin group, or any user to set the flag? Otherwise, malicious users could create files in a flash pool and set LCME_FL_PIN on all their files and prevent them from being moved to a cold pool.
- do we need separate LCME_FL_PIN and LCME_FL_NOMIRROR flags, or is one enough to cover expected uses and two just add confusion for developers and users?
- would the low-level tools like "lfs mirror split/delete" and "lfs migrate" check and honor these flags (possibly with an --force-migrate option) to mirror/migrate the files despite the presence of the flag(s), or would it be policy engines that use the flag as advice for how to handle the file, but the user/engine could still call the low-level tools to mirror/migrate the file if they really want?
- what happens if LCME_FL_PIN is on a disk mirror, but not on a flash mirror? Can the flash mirror be dropped if needed? Can a new flash mirror be created on such a file, or does it need a separate LCME_FL_NOMIRROR flag to be prevent this?
- what happens if only some of the components in a mirror have an LCME_FL_PIN flag, but not all components? Does that prevent the whole mirror from being migrated, or just those components (in the future when it is possible to have partial mirrors)?