[LU-10070] PFL self-extending file layout

• Type: New Feature
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.13.0
• Affects Version/s: Lustre 2.10.0
• Labels: FLR2
• Assignee: Vitaly Fertman
• Reporter: Andreas Dilger

Description

One interesting idea discussed at LAD was to have a PFL layout that is "self extending".

For several use cases, such as HSM partial-file release/restore, partial-file migration to/from burst buffers, partial-file FLR resync it is advantageous to avoid the need to restore/migrate/resync the entire file at once, but rather only to process the required chunks of the file.

Essentially, a PFL file would have the normal few components that define the start of the file (e.g. [0-32MB), [32MB-1GB), [1GB-16GB)) and they would be instantiated as with normal PFL today. What is new for "self-extending layouts" is that the last component becomes the template for additional components (as needed) rather than having a component to "EOF" that freezes the layout of the rest of the file.

This avoids the overhead of explicitly specifying many identical components for the file, only in order to limit the size of the components that need to be processed.

Issue Links

• is related to: LU-9096 sanity test_253: File creation failed after rm
• is related to: LU-12681 Data corruption - due incorrect KMS with SEL files
• is related to: LU-12712 sanity-pfl tests triggering “not SEL magic on SEL file”
• is related to: LU-13395 unable to set "--comp-flags=prefer" on a new file
• is related to: LU-13589 PFL "lfs setstripe -E 1M -S 65536" incorrectly parses stripe_size units
• is related to: LU-9846 Overstriping - more than stripe per OST per component
• is related to: LU-10169 Spillover space
• is related to: LU-13058 Intermediate component removal (PFL/SEL)
• is related to: LU-9479 sanity test 184d 244: don't instantiate PFL component when taking group lock
• is related to: LU-9809 RTDS(Real-Time Dynamic Striping): A policy based striping framework
• is related to: LUDOC-436 Self Extending Layout Documentation
• is related to: LU-8998 Progressive File Layout (PFL)
• is related to: LU-9771 FLR1: Landing tickets for File Level Redundancy Phase 1
• is related to: LU-10808 DoM: component end should align with dom_stripesize
• is related to: LU-7880 add OST/MDT performance statistics to obd_statfs

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/36554/
Subject: LU-10070 utils: move new SEL find_param fields to end
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 5222ba07e3bd087ddb0812e2185610b725cb9d1a

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/36554
Subject: LU-10070 utils: move new SEL find_param fields to end
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ba7fe211563652856c481a694d0fdaddad3d8dfe

[Peter Jones]

Landed for 2.13

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35704/
Subject: LU-10070 lod: SEL inheritance fix
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: dee41b90e74c0c2021207c0431d6c34642c7019c

[Gerrit Updater]

Vitaly Fertman ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35704
Subject: LU-10070 lod: SEL inheritance fix
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 44e6e8608de68b61b5ce4b904936456dd59003cf

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33779/
Subject: LU-10070 test: llapi_layout_test enhancements
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4a68bcfd8b0d206d5d5e5c18d5fcc8e55d1732b5

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35414/
Subject: LU-10070 lod: SEL cleanup
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 63e90bdb42a9fedd368726877fd4edfaf8e328c7

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35314/
Subject: LU-10070 utils: setstripe component-add support for SEL
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e768624788b7b8797c97ca27f2e6f9f63124dba0

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35144/
Subject: LU-10070 lod: SEL: interoperability support
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4c7fe0c7cdae7170e9ec1a6a48423dd20046500e

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34909/
Subject: LU-10070 utils: SEL: lfs find & getstripe support
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c7cf7a5076440f68dee8f7798f46f50c94404a7e