Details
-
New Feature
-
Resolution: Unresolved
-
Minor
-
None
-
None
-
3
-
9223372036854775807
Description
Overview
Erasure coding provides a more space-efficient method for adding data redundancy than mirroring, at a somewhat higher computational cost. This would typically be used for adding redundancy for large and longer-lived files to minimize space overhead. For example, RAID-6 10+2 adds only 20% space overhead while allowing two OST failures, compared to mirroring which adds 100% overhead for single-failure redundancy or 200% overhead for double-failure redundancy. Erasure coding can add redundancy for an arbitrary number of drive failures (e.g. any 3 drives in a group of 16) with a fraction of the overhead.
It would be possible to implement delayed erasure coding on striped files in a similar manner to Phase 1 mirrored files, by storing the parity stripes in a separate component in the file, having a layout that indicates the erasure coding algorithm, number of data and parity stripes, stripe_size (should probably match file stripe size), etc. The encoding would be similar to RAID-4, with specific "data" stripes (the traditional Lustre RAID-0 file layout) in the primary component, and one or more "parity" stripes stored in a separate parity component, unlike RAID-5/6 that have the parity interleaved. For widely-striped files, there could be separate parity stripes for different sets of file stripes (e.g. 10x 12+3 for a 120-stripe file), so that data+parity would be able to use all of the OSTs in the filesystem without having double failures within a single parity group. For very large files, it would be possible to split the parity component into smaller extents to reduce the parity reconstruction overhead for sub-file overwrites. Erasure coding could also be added after-the-fact to existing RAID-0 striped files, after the initial file write, or when migrating a file from an active storage tier to an archive tier.
Reads from an erasure-coded file would normally use only the primary RAID-0 component (unless data verification on read was also desired), as with non-redundant files. If a stripe in the primary component for the file fails, the client would read the data stripes and one or more parity stripes component and reconstruct the data from parity on the fly, and/or depend on the resync tool to reconstruct the failed stripe from parity.
Writes to an erasure-coded file would mark the parity component stale matching the extent of the data component that was modified, as with a regular mirrored file, and writes would continue on the primary RAID-0 striped file. The main difference from an FLR mirrored file is that the writes would always need to go to the primary data component, and the parity component would always be marked stale. It would not be possible to write to an erasure-coded file that has a failure in a primary stripe without first reconstructing it from parity.
Space Efficient Data Redundancy
Erasure coding will add the ability to add full redundancy of large files or whole filesystems, rather than using full mirroring. This will allow striped Lustre files to store redundancy in parity components that allow recovery from a specified number of OST failures (e.g. 3 OST failures per 12 stripes, or 4 OST failures per 24 stripes) in a manner similar to RAID-4 with fixed parity stripes.
Required Lustre Functionality
Erasure Coded File Read
The actual parity generation will be done with the lfs mirror resync tool in userspace. The Lustre client will do normal reads from the RAID-0 data component, unless there is an OST failure or other error reading from a data stripe. Add support for data reconstruction from the data and parity components, leveraging existing functionality for reading mirrored files.
Erasure Coded File Write
To avoid losing redundancy on erasure-coded files that are modified, the Mirrored File Writes functionality could be used during writes to such files. Changes would be merged into the erasure coded component after the file is closed, using the Phase 1 ChangeLog consumer, and then the mirror component can be dropped.
External Components
Erasure Coded Resync Tool
The lfs mirror resync tool needs to be updated to generate the erasure code for the file striped file, storing the parity in a separate component from the main RAID-0 striped file. There are CPU-optimized implementations of the erasure coding algorithms available, so the majority of the work would be integrating these optimized routines into the Lustre kernel modules and userspace tools, rather than actually developing the encoding algorithms.
Attachments
Issue Links
- is related to
-
LU-12649
Tracker for ongoing FLR improvements
-
- Open
-
-
-
- Open
-
-
LUDOC-463 Add feature documentation for Erasure Coding
-
- Open
-
-
-
- Open
-
-
-
- Resolved
-
- is related to
-
-
- Open
-
Activity
I just did a rebase to the latest master and I get a build error with the latest code due to the landing of LU-12142. For lov_io_lru_reserve() we use
lov_foreach_io_layout() and lov_io_fault_store() uses lov_io_layout_at(). Both functions have changed to handle both LCT_DATA and LCT_CODE types. The question is it safe to just pass LCT_DATA in both cases or do we need to examine every component to see what type LCT_* we have?
Just an update. We have moved the flr branch to the latest master and having been running normal sanity tests. Currently we are fixing various bugs we are encountering.
Can someone provide a better HLD than attached? This document just about some userspace tools, and some common changes for structures. But this document don't describe anything with parity calculation - a specially in case REwrite don't covered a whole data stripes and old data need to be read to calculate a parity. No fail scenario in document, no recovery handling but it looks recovery is very complex in this case. No describing how it have plan avoid a parity rewrite with old data in case two parity updates in flight (CR lock permit this). It have bad describing a lock protection for parity between nodes, in case two nodes have a parallel write for half data stripes.
No description about compatibility with old client.
Can design document updated to solve these questions ?
Andreas, i'm confused. You are OK with landing untested / buggy code?
yes, great insight, ISAL_ENABLE could be used to protect pre-EC file behavior and smooth the transition.
Bobijam,
since we are very close to the end of the 2.14 feature landing window, that it makes sense to submit the patches initially so that they are conditionally compiled under #ifdef ISAL_ENABLED, so that they can be landed and tested not to cause any problems with the current master code (i.e. the code is mostly a no-op initially). Then, patches can be landed to enable ISAL_ENABLED during the build, and tests should be conditional on this support (so there needs to be some way to detect it in userspace).
That will ensure that the EC code is included as part of the 2.14 release, and gives us more time to improve the build system, fix EC bugs, etc. We would want to have the #ifdef ISAL_ENABLE checks for this code anyway, so that Lustre can still build if ISA-L is not available/usable for some systems. We shouldn't leave it like disabled for a long time, because untested code is going to break quickly, but the 2.14 feature landing window is supposed to close on April 30 (already 2 weeks late), and I think there are still changes that need to be finished to the before this feature is ready. Those can still be worked on after the feature is landed to master, before the 2.14 final release.
Andreas - can be right Gerrit links provided in tickets ?
https://review.whamcloud.com/34678 isn't valid anymore.
same for any other.
The patches are in Gerrit under the sub-tasks linked above. LU-12186 thru LU-12189 and LU-12668 and LU-12669 (I think these last two are still finishing development).
I think it's ok to just pass LCT_DATA in both cases, parity code pages won't be cached after EC IO since they are ephemeral and later EC IO could use other parity components.