Details
-
New Feature
-
Resolution: Unresolved
-
Minor
-
None
-
None
-
3
-
9223372036854775807
Description
Overview
Erasure coding provides a more space-efficient method for adding data redundancy than mirroring, at a somewhat higher computational cost. This would typically be used for adding redundancy for large and longer-lived files to minimize space overhead. For example, RAID-6 10+2 adds only 20% space overhead while allowing two OST failures, compared to mirroring which adds 100% overhead for single-failure redundancy or 200% overhead for double-failure redundancy. Erasure coding can add redundancy for an arbitrary number of drive failures (e.g. any 3 drives in a group of 16) with a fraction of the overhead.
It would be possible to implement delayed erasure coding on striped files in a similar manner to Phase 1 mirrored files, by storing the parity stripes in a separate component in the file, having a layout that indicates the erasure coding algorithm, number of data and parity stripes, stripe_size (should probably match file stripe size), etc. The encoding would be similar to RAID-4, with specific "data" stripes (the traditional Lustre RAID-0 file layout) in the primary component, and one or more "parity" stripes stored in a separate parity component, unlike RAID-5/6 that have the parity interleaved. For widely-striped files, there could be separate parity stripes for different sets of file stripes (e.g. 10x 12+3 for a 120-stripe file), so that data+parity would be able to use all of the OSTs in the filesystem without having double failures within a single parity group. For very large files, it would be possible to split the parity component into smaller extents to reduce the parity reconstruction overhead for sub-file overwrites. Erasure coding could also be added after-the-fact to existing RAID-0 striped files, after the initial file write, or when migrating a file from an active storage tier to an archive tier.
Reads from an erasure-coded file would normally use only the primary RAID-0 component (unless data verification on read was also desired), as with non-redundant files. If a stripe in the primary component for the file fails, the client would read the data stripes and one or more parity stripes component and reconstruct the data from parity on the fly, and/or depend on the resync tool to reconstruct the failed stripe from parity.
Writes to an erasure-coded file would mark the parity component stale matching the extent of the data component that was modified, as with a regular mirrored file, and writes would continue on the primary RAID-0 striped file. The main difference from an FLR mirrored file is that the writes would always need to go to the primary data component, and the parity component would always be marked stale. It would not be possible to write to an erasure-coded file that has a failure in a primary stripe without first reconstructing it from parity.
Space Efficient Data Redundancy
Erasure coding will add the ability to add full redundancy of large files or whole filesystems, rather than using full mirroring. This will allow striped Lustre files to store redundancy in parity components that allow recovery from a specified number of OST failures (e.g. 3 OST failures per 12 stripes, or 4 OST failures per 24 stripes) in a manner similar to RAID-4 with fixed parity stripes.
Required Lustre Functionality
Erasure Coded File Read
The actual parity generation will be done with the lfs mirror resync tool in userspace. The Lustre client will do normal reads from the RAID-0 data component, unless there is an OST failure or other error reading from a data stripe. Add support for data reconstruction from the data and parity components, leveraging existing functionality for reading mirrored files.
Erasure Coded File Write
To avoid losing redundancy on erasure-coded files that are modified, the Mirrored File Writes functionality could be used during writes to such files. Changes would be merged into the erasure coded component after the file is closed, using the Phase 1 ChangeLog consumer, and then the mirror component can be dropped.
External Components
Erasure Coded Resync Tool
The lfs mirror resync tool needs to be updated to generate the erasure code for the file striped file, storing the parity in a separate component from the main RAID-0 striped file. There are CPU-optimized implementations of the erasure coding algorithms available, so the majority of the work would be integrating these optimized routines into the Lustre kernel modules and userspace tools, rather than actually developing the encoding algorithms.
Attachments
Issue Links
Activity
shadow in the initial implementation there will not be any "live" writes to an EC file. The FLR EC feature will follow the same development model as FLR mirroring:
- stage 1: is essentially "read-only EC" where clients write a RAID-0 striped file (or component) as today, and an external tool ("lfs ec create" or similar) does a delayed update of the file to add layout components for redundancy (e.g. mirror(s) for 1-stripe components, 8+2 or 16+3 for all of the striped components). any writes to the file would mark the relevant component(s) as STALE and they would need to be resync'd by "lfs ec resync" or similar).
- stage 2a: where it might be possible to start with the EC components marked STALE, and immediately write the EC components at the same time as the regular data write, and only mark them "un-STALE" if all of the writes have completed successfully (i.e. no RAID write holes to recover, etc.) for a new file write (can be implemented independently of 2b)
- stage 2b: is essentially "read-only EC with sparse mirrored write", where a new mirror component is added to the file to handle the (sparse) write, then this mirror is "flattened" into the EC file in a second step (e.g. "lfs ec resync" again) (can be implemented independently of 2a)
I don't think a "stage 3 - immediate EC writes with live parity" is reasonable to implement any time soon. Any distributed parity solution I've seen ends up being "two phase writes" in the end anyway (e.g. mirrored writes and then EC in the background), so not fundamentally different than stage 1/2b. Otherwise, there is always the issue of "3 of 10 stripes were not written" or "overwrite 3 of 10 stripes and fail in the middle" and recovery is impossible.
James, can you drop some comments about recovery with FLR2 ? how it planed to be find which stripe is good and which is outdated and needs to be reconstructed.
An outside party has contacted our group at ORNL so we pushed the current prototype for early review with them. This project is at the beta code stage.
kernel: Lustre: 42446:0:(osd_handler.c:1938:osd_trans_start()) lustre-MDT0000: credits 19393 > trans_max 9984
That is probably introduced by patches from LU-14134, possibly combined with large write RPCs. It isn't really fatal, but annoying and should be fixed.
There is a prototype patch in LU-14641 that would be useful to test if you can reproduce this easily.
In my testing I'm seeing:
kernel: Lustre: DEBUG MARKER: == sanity test 130g: FIEMAP (overstripe file) ================================================
======== 14:15:49 (1620152149)
kernel: Lustre: 42446:0:(osd_handler.c:1938:osd_trans_start()) lustre-MDT0000: credits 19393 > trans_max 9984
kernel: Lustre: 42446:0:(osd_handler.c:1867:osd_trans_dump_creds()) create: 300/1200/0, destroy: 1/4/0
kernel: Lustre: 42446:0:(osd_handler.c:1867:osd_trans_dump_creds()) Skipped 4001 previous similar messages
kernel: Lustre: 42446:0:(osd_handler.c:1874:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 304/148/0
kernel: Lustre: 42446:0:(osd_handler.c:1874:osd_trans_dump_creds()) Skipped 4001 previous similar messages
kernel: Lustre: 42446:0:(osd_handler.c:1884:osd_trans_dump_creds()) write: 1501/12910/0, punch: 0/0/0, quota 4/4/0
kernel: Lustre: 42446:0:(osd_handler.c:1884:osd_trans_dump_creds()) Skipped 4001 previous similar messages
kernel: Lustre: 42446:0:(osd_handler.c:1891:osd_trans_dump_creds()) insert: 301/5116/0, delete: 2/5/0
kernel: Lustre: 42446:0:(osd_handler.c:1891:osd_trans_dump_creds()) Skipped 4001 previous similar messages
kernel: Lustre: 42446:0:(osd_handler.c:1898:osd_trans_dump_creds()) Skipped 4001 previous similar messages
kernel: Pid: 42446, comm: mdt03_001 3.10.0-1160.15.2.el7.x86_64 #1 SMP Thu Jan 21 16:15:07 EST 2021
kernel: Call Trace:
kernel: [<0>] libcfs_call_trace+0x90/0xf0 [libcfs]
kernel: [<0>] osd_trans_start+0x4bb/0x4e0 [osd_ldiskfs]
I think it's ok to just pass LCT_DATA in both cases, parity code pages won't be cached after EC IO since they are ephemeral and later EC IO could use other parity components.
I just did a rebase to the latest master and I get a build error with the latest code due to the landing of LU-12142. For lov_io_lru_reserve() we use
lov_foreach_io_layout() and lov_io_fault_store() uses lov_io_layout_at(). Both functions have changed to handle both LCT_DATA and LCT_CODE types. The question is it safe to just pass LCT_DATA in both cases or do we need to examine every component to see what type LCT_* we have?
Just an update. We have moved the flr branch to the latest master and having been running normal sanity tests. Currently we are fixing various bugs we are encountering.
Can someone provide a better HLD than attached? This document just about some userspace tools, and some common changes for structures. But this document don't describe anything with parity calculation - a specially in case REwrite don't covered a whole data stripes and old data need to be read to calculate a parity. No fail scenario in document, no recovery handling but it looks recovery is very complex in this case. No describing how it have plan avoid a parity rewrite with old data in case two parity updates in flight (CR lock permit this). It have bad describing a lock protection for parity between nodes, in case two nodes have a parallel write for half data stripes.
No description about compatibility with old client.
Can design document updated to solve these questions ?
"James Simmons <jsimmons@infradead.org>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/58843
Subject: LU-10911 idl: add OBD_CONNECT2_PARITY flag
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 292e85a2f5b5220abe726fc3b4e714b2becc06a8