[LU-10070] PFL self-extending file layout

• Type: New Feature
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.13.0
• Affects Version/s: Lustre 2.10.0
• Labels: FLR2
• Assignee: Vitaly Fertman
• Reporter: Andreas Dilger

Description

One interesting idea discussed at LAD was to have a PFL layout that is "self extending".

For several use cases, such as HSM partial-file release/restore, partial-file migration to/from burst buffers, partial-file FLR resync it is advantageous to avoid the need to restore/migrate/resync the entire file at once, but rather only to process the required chunks of the file.

Essentially, a PFL file would have the normal few components that define the start of the file (e.g. [0-32MB), [32MB-1GB), [1GB-16GB)) and they would be instantiated as with normal PFL today. What is new for "self-extending layouts" is that the last component becomes the template for additional components (as needed) rather than having a component to "EOF" that freezes the layout of the rest of the file.

This avoids the overhead of explicitly specifying many identical components for the file, only in order to limit the size of the components that need to be processed.

Issue Links

• is related to: LU-9096 sanity test_253: File creation failed after rm
• is related to: LU-12681 Data corruption - due incorrect KMS with SEL files
• is related to: LU-12712 sanity-pfl tests triggering “not SEL magic on SEL file”
• is related to: LU-13395 unable to set "--comp-flags=prefer" on a new file
• is related to: LU-13589 PFL "lfs setstripe -E 1M -S 65536" incorrectly parses stripe_size units
• is related to: LU-9846 Overstriping - more than stripe per OST per component
• is related to: LU-10169 Spillover space
• is related to: LU-13058 Intermediate component removal (PFL/SEL)
• is related to: LU-9479 sanity test 184d 244: don't instantiate PFL component when taking group lock
• is related to: LU-9809 RTDS(Real-Time Dynamic Striping): A policy based striping framework
• is related to: LUDOC-436 Self Extending Layout Documentation
• is related to: LU-8998 Progressive File Layout (PFL)
• is related to: LU-9771 FLR1: Landing tickets for File Level Redundancy Phase 1
• is related to: LU-10808 DoM: component end should align with dom_stripesize
• is related to: LU-7880 add performance statistics to obd_statfs