[LU-12648] FLR2: File versioning with FLR and "nosync" files

Related — Thu, 8 Aug 2019 20:40:43 +0000

The FLR implementation and struct lcme_entry support functionality to mark an FLR file mirror with "lcme_flags |= LCME_FL_NOSYNC" flag using the command:

lfs setstripe --comp-set --comp-id=N --comp-flags=nosync <file>

This causes the MDS to store the current timestamp into the component entry's lcme_timestamp field (should it get a timestamp in the RPC from the client?), and "lfs mirror resync <file>" will no longer resync that mirror by default. New mirrors can still be created for the file, and any mirrors not marked NOSYNC will continue to be updated by "lfs mirror resync" when it is run.

This allows a basic form of file versioning to be enabled for a file, where that mirror will keep the original file contents if the file is later modified, and "lfs mirror read --mirror-id=M <file>" can be used to read from a specific mirror of the file.

In order to make this functionality into a more useful feature, a number of improvements should be made:

add a "lfs mirror flags --mirror-id=M --comp-flags=<flags> <file> ..." command so that the "nosync" flag (and others) will consistently be set on all mirrors of a file, rather than just a single component, and puts this functionality in the same place as other mirror commands where it is more easily found
add a "lfs mirror open [--mirror-in=M] [--mirror-out=N]" command to allow opening file descriptors on specific mirrors for stdin and/or stdout of a process (similar to "tee" or "script") so that they can work more easily with file versions. This could work with any utility that can use stdin or stdout for the file, either with a specific "-" filename option, or /dev/stdin and /dev/stdout as the filenames.
add the ability to "auto-version" a file on the MDS, so that a new (STALE) mirror component is added to a file when it is modified and a NOSYNC mirror would otherwise become STALE. The new mirror component would be resync'd by an external tool as needed.
allow imposing a maximum limit on the number of file versions created for a single file. This should be stored in the file layout along with the "auto-version" setting, and also be storable in a default directory layout that can be inherited by new files created in the directory. When a new mirror is created for a file, the oldest version (by lcme_timestamp) will be removed from the file as in "lfs mirror split --delete".
add the ability to preserve (pin) specific mirror versions so they aren't auto deleted?
add a "lfs mirror restore --mirror-id=M" command to restore version M to be the current version of the file (essentially "lfs mirror resync" with the specified mirror as the source and updating all STALE and valid mirrors (except one to be kept as a new file version)
add handling to "lfs mirror split" to allow cleanup of versions by count and/or age to prune files that are consuming space for mirrors that are no longer needed.
add versioning support to the filesystem namespace, so that specific file versions can be accessed with a "magic" pathname. According to File versioning and deleting on OpenVMS the file versions were accessed with a ";version" separator at the end of the file, but using ";" may be problematic since this is a command separator in bash. Perhaps "," would be better in this case, even if it may already be commonly used, by restricting it to the last bytes of the filename and matching a mirror number.
wildcards should be added to match VMS behavior filename lookups in the VFS by stripping the version number from the end of the filename, doing a lookup on the basename, and then looking in the layout to find the version numer, such as:

",*" to mean all versions
",0" to mean oldest version
",-0" to mean most recent version
",-1" to mean next most recent version
",-n" to mean n-th previous version

Whamcloud Community JIRA

[LU-12648] FLR2: File versioning with FLR and "nosync" files