[LU-10070] PFL self-extending file layout

• Type: New Feature
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.13.0
• Affects Version/s: Lustre 2.10.0
• Labels: FLR2
• Assignee: Vitaly Fertman
• Reporter: Andreas Dilger

Description

One interesting idea discussed at LAD was to have a PFL layout that is "self extending".

For several use cases, such as HSM partial-file release/restore, partial-file migration to/from burst buffers, partial-file FLR resync it is advantageous to avoid the need to restore/migrate/resync the entire file at once, but rather only to process the required chunks of the file.

Essentially, a PFL file would have the normal few components that define the start of the file (e.g. [0-32MB), [32MB-1GB), [1GB-16GB)) and they would be instantiated as with normal PFL today. What is new for "self-extending layouts" is that the last component becomes the template for additional components (as needed) rather than having a component to "EOF" that freezes the layout of the rest of the file.

This avoids the overhead of explicitly specifying many identical components for the file, only in order to limit the size of the components that need to be processed.

Issue Links

• is related to: LU-9096 sanity test_253: File creation failed after rm
• is related to: LU-12681 Data corruption - due incorrect KMS with SEL files
• is related to: LU-12712 sanity-pfl tests triggering “not SEL magic on SEL file”
• is related to: LU-13395 unable to set "--comp-flags=prefer" on a new file
• is related to: LU-13589 PFL "lfs setstripe -E 1M -S 65536" incorrectly parses stripe_size units
• is related to: LU-9846 Overstriping - more than stripe per OST per component
• is related to: LU-10169 Spillover space
• is related to: LU-13058 Intermediate component removal (PFL/SEL)
• is related to: LU-9479 sanity test 184d 244: don't instantiate PFL component when taking group lock
• is related to: LU-9809 RTDS(Real-Time Dynamic Striping): A policy based striping framework
• is related to: LUDOC-436 Self Extending Layout Documentation
• is related to: LU-8998 Progressive File Layout (PFL)
• is related to: LU-9771 FLR1: Landing tickets for File Level Redundancy Phase 1
• is related to: LU-10808 DoM: component end should align with dom_stripesize
• is related to: LU-7880 add OST/MDT performance statistics to obd_statfs

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33784/
Subject: LU-10070 lod: SEL: Layout sanity checking
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4eca26ddab3186a68888862218fa8904f812e5a1

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33783/
Subject: LU-10070 lod: SEL: Implement basic spillover space
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: ff5eb304fa371d879da38621fac3aec7d4548a5e

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33782/
Subject: LU-10070 lod: SEL: Add flag & setstripe support
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: fed241911f61b1d76aa7d80bfd370c822a3926ef

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33780/
Subject: LU-10070 lod: SEL: split lod_del_layout
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 96bfcd13b6cc3fce12f1e6f5abe4971cc8a59e1f

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33778/
Subject: LU-10070 tests: New test-framework functionality
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c1aaa3e55090c7a5e067ec52cf74b2e6406133d2

[Gerrit Updater]

Vitaly Fertman ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35414
Subject: LU-10070 lod: SEL cleanup
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4f7c529cd73b87e093ffcf539e7e6bf2bbec31d1

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35270/
Subject: LU-10070 lod: layout_del memleak
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b0ce7701d1e9dd1269b99b1c660a140fe85b9592

[Gerrit Updater]

Vitaly Fertman ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35314
Subject: LU-10070 utils: setstripe component-add support for SEL
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8a87eab09f17c8efacde106f4c74543dad8669df

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35232/
Subject: LU-10070 ldlm: layout lock fixes
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 51f23ffa4dae3015da627203fb6f160db4911bee

[Gerrit Updater]

Vitaly Fertman ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35270
Subject: LU-10070 lod: layout_del memleak
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: d97c1343529fd4c38be5226772d0f2d9543db854

[Gerrit Updater]

Vitaly Fertman ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35232
Subject: LU-10070 ldlm: layout lock fixes
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9e58fe77666aac33bf78d2f6fb67dee2532e3ae7