[LU-8998] Progressive File Layout (PFL)

• Type: New Feature
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.10.0
• Affects Version/s: Lustre 2.10.0
• Assignee: Zhenyu Xu
• Reporter: Joseph Gmitter (Inactive)

Description

The work tracked under this ticket will implement Progressive File Layouts (PFL).

This feature simplifies Lustre usage of large files, avoiding the need for users to predetermine the required layout for each file. The goal is that a single file layout specified at file creation time can balance the following conflicting needs:

• minimize the number of stripes for small files to avoid the overhead of creating and accessing multiple OST objects
• maximize the number of stripes for large files to utilize the available bandwidth as well as distribute space usage to avoid consuming all of the space on a single OST
• flexibility to specify a layout that is suitable for a majority of uses, while allowing for variability and flexibility between users, applications, and sites

The administrator will be able to set a specific progressive layout for new files created in the filesystem, and users will be able to specify progressive layouts on their own files for their specific needs. It should be possible to specify a default progressive layout template for a directory that is inherited by new files and directories created in that directory, as well as a filesystem-wide global default layout that is used in the absence of more specific layouts.

Issue Links

• is related to: LU-9479 sanity test 184d 244: don't instantiate PFL component when taking group lock
• is related to: LU-9300 sanity-hsm test_1b: request on 0x200000401:0x8:0x0 is not SUCCEED on mds1
• is related to: LU-9334 LBUG lu_device_fini()) ASSERTION( atomic_read(&d->ld_ref) == 0 ) failed: Refcount is 1
• is related to: LU-9340 PFL fails performance testsSpirit
• is related to: LU-9342 It is possible that user defined LOVEA contains uninstantiated flag
• is related to: LU-9351 'lfs setstripe' failed to set composite layout to an existing file
• is related to: LU-9522 sanity test 27z is failing with 'test_27z failed with 6'
• is related to: LU-9341 PFL: append should not instantiate full layout
• is related to: LU-9357 PFL llapi_layout_comp_add() should inherit pool from previous component layout
• is related to: LU-9268 Add PFL test to check striping distribution
• is related to: LU-9315 New static analysis issues in v2_9_55_0-35-gaa32cc5
• is related to: LU-9366 Interop 2.9.0<->master sanity test_27z: FF stripe 0 stripe_size=0 stripe_count=0 != 0
• is related to: LU-9393 Enhance setstripe options for components
• is related to: LU-9513 New static analysis issues in v2_9_57_0-66-gb6d5b5b
• is related to: LU-10070 PFL self-extending file layout
• is related to: LU-3480 Layout Enhancement
• is related to: LU-9344 sanity test_244: sendfile_grouplock test12() test hung
• is related to: LU-9359 sanity test_27p: /mnt/lustre/d27p.sanity/f27p.sanity has size 4, not 80000004
• is related to: LU-9362 PFL: Files with multiple components should not return multiple values for ‘lfs getstripe’
• is related to: LU-9380 sanity test_27y: lod_comp_ost_in_use() ASSERTION(inuse->op_count < inuse->op_size) failed
• is related to: LU-9324 sanity-pfl test 10 needs to reset the file system default layout
• is related to: LU-9007 Improved object allocator for FLR composite files
• is related to: LU-9378 extract lfs-getstripe.1 from lfs.1
• is related to: LU-9349 PFL known issues tracking ticket
• is related to: LUDOC-361 Documentation for Progressive File Layouts (PFL) feature

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/26926/
Subject: LU-8998 test: Test 'lfs find' composite file options
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b468a3f4da3232d17040972e33f55a2647273131

[Peter Jones]

Landed for 2.10

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/26491/
Subject: LU-8998 tools: parse optional argument properly
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: cbeffd6a981b8c6a79cd83a6b063bf55d4e9bb2b

[Joseph Gmitter (Inactive)]

Last patch to land under this ticket is: https://review.whamcloud.com/#/c/26491/

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/26569/
Subject: LU-8998 tests: improve sanity.sh::check_seq_oid()
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e79e711e8f61dab7978102c5be04b94c5557b3dc

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/26490/
Subject: LU-8998 tools: support negative flags
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 7a1c0f950f8733c3f44e28a3b96d579797e46e80

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/26484/
Subject: LU-8998 llapi: add LLAPI_LAYOUT_COMP_USE_PREV
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 6746484df44e456ca3df47708e7d1bc14e4404e8

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/26483/
Subject: LU-8998 llapi: rename llapi_layout_comp_move -> *use
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1390c68c28c3f260fb96658d9559e31969f0e98a

[Gerrit Updater]

James Nunez ([e-mail]) uploaded a new patch: https://review.whamcloud.com/26926
Subject: LU-8998 test: Test 'lfs find' composite file options
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: aeb8eacb8b440845115826c2e9cdff7f02f568fb

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/26699/
Subject: LU-8998 utils: allow "-component*" to be abbreviated
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 23cc21d6f46e79900cf9b7b1bc32c2dda65121be