[LU-19548] FLR-EC: support for other lfs mirror commands

• Type: Technical task
• Resolution: Unresolved
• Priority: Medium
• Severity: 3
• Assignee: Marc Vef
• Reporter: Patrick Farrell

Description

The lfs utility has a fairly extensive set of mirror commands. This ticket discusses the changes that are required for each command to support parity mirrors.

lfs mirror commands

lfs mirror create

Create a new file with a specified number of mirrors. If --ec is passed, a parity mirror is created as well (see SETSTRIPE_OPTIONS).
Notable options:

• --flags prefer: No changes - this can still only apply for a data mirror.

Example:

# Create a file with 2 data mirrors and 2 parity mirrors
lfs mirror create -N2 --ec 8:2 /mnt/lustre/file1

lfs mirror resync

Writes replicas (other data mirrors) and parity mirrors that are stale.
Notable options:

• --only: This can target both data and parity mirrors. The question here is how it should behave for data-parity mirror pairs and parity mirrors only.
  1. Consider the case where both mirrors are stale and --only is passed for the parity mirror. This would then calculate the parities for the stale data mirror. This can be useful for old file versions (LCME_FL_NOSYNC flag set). This way, the EC can be maintained for an old version of the file without causing the data mirror to resync. 
  2. In other cases, --only applies to both data-parity mirror pairs, rather than to them individually. Therefore, regardless of which mirror type is selected for a pair, both mirrors are updated. To avoid reading the primary data mirror twice, once to copy to the stale data mirror, and once to compute the parity, both mirrors can be updated concurrently. This can be done by reading some data into a buffer, writing to the stale mirror, then computing the EC for the buffer and writing it to the EC mirror. Note, llapi and "lfs mirror extend" allow multiple parity mirrors for a data mirror. So, for case 2 above, the computed EC can be written to all corresponding parity mirrors. If a data mirror is targeted, and the unusual case is needed that the parity mirror should not be updated.

Example:

# File 1 has three mirrors: data 1, data 2 (stale), parity 3 (for 2, stale). Here, data 2 is updated from data 1, and parity 3 is computed.
lfs mirror resync --only 3 /mnt/lustre/file1

lfs mirror read

Reads the specified mirrors' content, outputting it to a (specified) file.
Notable options:

• N/-mirror-id can refer to a parity mirror that would dump the parity data to a file. Allowing this could be useful for disaster recovery/testing, but there is also the option to block it outright. There is a patch to test that reads from the parity mirror is possible: https://review.whamcloud.com/62731. This is also related to lfs mirror split (see below). A warning message could be included if a parity mirror is read.

# Read raw parity data from parity mirror 2 (with warning)
lfs mirror read --mirror-id 2 -o /tmp/parity /mnt/lustre/file1

lfs mirror write

Writes content to a file's mirror.
Notable options:

• N/mirror-id can refer to a parity mirror (similar to read) for writing. This can be useful for data management tools that want to write to it or recovery scenarios. To avoid targeting a parity mirror by accident, it must be explicitly set via a new option, e.g., -force-parity, otherwise writing to parity mirrors should be denied.

Note, that functionality prevents the MDS from selecting a parity mirror as the normal write destination. lfs mirror write overrides that already, so it should work and we want to allow it to write the parity mirror.

# Write to parity mirror 2 (requires explicit --force-parity flag)
lfs mirror write --mirror-id 2 --force-parity -i /tmp/parity /mnt/lustre/file1

lfs mirror verify

Verifies that each SYNC mirror of a mirror file contains the same data. For EC, only non-stale parity mirrors are checked. This requires re-computing the parities and comparing them to the parity mirror. If -s is passed, the parity mirror is set as "stale". Corresponding patch: https://review.whamcloud.com/62643

Example:

# Verify all mirrors (including parity) and mark mismatched parity mirrors as stale
lfs mirror verify -s /mnt/lustre/file1

lfs mirror copy

Copies a mirrors content to other mirror(s) of a mirrored file. The target mirror must already exist since it is addressed by its mirror ID. For EC, the command could behave as follows:

1. It should not be possible to copy between data and parity mirrors, and across parity mirrors that have a different component/stripe layout (geometry).
2. (not in first version) Copying parity to parity mirror is allowed if the source parity mirror is non-stale (since the target will be marked as non-stale) and the geometry is the same. If the geometry is different, an error should be returned. This is a potentially low-value improvement so keeping it here for future versions if needed.
   • > Copying a source parity mirror to -o -1 should be denied in general.
3. With points 1. and 2., this means parity-to-parity mirror copying is not supported for now.
4. Copying a source data mirror to another mirror, which has parity mirrors, should be default also updated the target parity mirrors. This works analogously for -o -1. 
5. (expert option) Add a new option --force-no-ec which disables the implicit update of parity mirrors. In this case, the parity mirrors become stale.

Example:

# Copy data mirror 1 and its parity mirror 2 to data mirror 3 and its parity mirror 4
lfs mirror copy -i 1 -o 3 /mnt/lustre/file1

lfs mirror split

Splits a specified mirror from an existing mirrored file. This is related to how lfs mirror read/write will behave. In the following, I assume that reading from and writing to parity mirrors is possible. For EC, the command could behave as follows:

1. A parity mirror can be selected to be split to a (specified) file. This reuses the existing mechanism to split data mirrors from a file for parity mirrors, splitting the components into a separate layout, and then adding the parity mirror to the stub file.
   • This works with any other existing options.
   • A warning message can be included if a parity mirror is split.
2. If -d is specified, the parity mirror is removed: https://review.whamcloud.com/c/fs/lustre-release/+/62316

Similar to the other commands, we need to handle the case where a data mirror is selected to split while it has parity mirrors. In this case, the parity mirror would remain behind which is typically not useful. To be consistent with the other commands, this case should split off the parity mirrors as well. This affects -d in the same way.

The argument --force-no-ec can be used to avoid implicitly splitting off the parity mirrors if a data mirror was selected.

Example:

# Split parity mirror 2 to file (removes EC protection from data mirror 1, shows warning)
lfs mirror split --mirror-id 2 -f /tmp/parity /mnt/lustre/file1

# Delete parity mirror 2 without saving (removes EC protection from data mirror 1)
lfs mirror split --mirror-id 2 -d /mnt/lustre/file1

lfs mirror extend

Adds mirror(s) to an existing file. For EC, new parity mirrors can be added that refer to a specific data mirror. One possible use case is adding parity mirrors to files in bulk. For data mirrors, this command does not need to be changed to accommodate EC. To support EC, new options {}ec{-} and -mirror-id are added.

1. A specific data mirror can be targeted to extend a parity mirror to it, e.g., lfs mirror extend --mirror-id x --ec <D+P>.
2. A new data and parity mirror can be added to a (mirrored) file, e.g., lfs mirror extend [-N <K>] --ec <D+P> (K defaults to 1).

Created mirrors are always stale and never writable. 

Notable options:

• f/-no-verify: (from Andreas) The same reason for allowing this for regular mirrors would appear to exist for parity mirrors. Namely, it should be possible to change the layout of a file to include a parity mirror and then defer the actual data read + EC computation to a service node instead of doing it inline with the lfs mirror extend command that is being run. That allows users or "lfs find" commands to add EC to existing files en masse and then have data mover nodes (which may be located closer to or on the storage nodes) do the bulk IO and computation.

Example:

# Add parity mirror to existing data mirror 1
lfs mirror extend --mirror-id 1 --ec 8+2 /mnt/lustre/file1

# Add 2 new data+parity mirror pairs (creates 2 data mirrors + 2 parity mirrors)
lfs mirror extend -N2 --ec 4+2 /mnt/lustre/file1