[#LU-8998] Progressive File Layout (PFL)

[LU-8998] Progressive File Layout (PFL) Created: 10/Jan/17 Updated: 22/Nov/17 Resolved: 29/May/17
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 2.10.0
Fix Version/s:	Lustre 2.10.0

Type:

New Feature

Priority:

Critical

Reporter:

Joseph Gmitter (Inactive)

Assignee:

Zhenyu Xu

Resolution:

Fixed

Votes:

Labels:

None

Attachments:

TestPlan-PFL.pdf

Issue Links:

Related
is related to	~~LU-9349~~	PFL known issues tracking ticket	Resolved
is related to	~~LUDOC-361~~	Documentation for Progressive File La...	Resolved
is related to	LU-9479	sanity test 184d 244: don't instantia...	Open
is related to	LU-9300	sanity-hsm test_1b: request on 0x200...	Open
is related to	~~LU-9334~~	LBUG lu_device_fini()) ASSERTION( ato...	Resolved
is related to	~~LU-9340~~	PFL fails performance testsSpirit	Resolved
is related to	~~LU-9342~~	It is possible that user defined LOVE...	Resolved
is related to	~~LU-9351~~	'lfs setstripe' failed to set composi...	Resolved
is related to	~~LU-9522~~	sanity test 27z is failing with 'test...	Resolved
is related to	~~LU-9341~~	PFL: append should not instantiate fu...	Resolved
is related to	~~LU-9357~~	PFL llapi_layout_comp_add() should in...	Resolved
is related to	~~LU-9268~~	Add PFL test to check striping distri...	Resolved
is related to	~~LU-9315~~	New static analysis issues in v2_9_55...	Resolved
is related to	~~LU-9366~~	Interop 2.9.0<->master sanity test_27...	Resolved
is related to	~~LU-9393~~	Enhance setstripe options for components	Resolved
is related to	~~LU-9513~~	New static analysis issues in v2_9_57...	Resolved
is related to	~~LU-10070~~	PFL self-extending file layout	Resolved
is related to	~~LU-3480~~	Layout Enhancement	Closed
is related to	~~LU-9344~~	sanity test_244: sendfile_grouplock t...	Resolved
is related to	~~LU-9359~~	sanity test_27p: /mnt/lustre/d27p.san...	Resolved
is related to	~~LU-9362~~	PFL: Files with multiple components s...	Resolved
is related to	~~LU-9380~~	sanity test_27y: lod_comp_ost_in_use(...	Resolved
is related to	~~LU-9324~~	sanity-pfl test 10 needs to reset the...	Resolved
is related to	~~LU-9007~~	Improved object allocator for FLR com...	Resolved
is related to	~~LU-9378~~	extract lfs-getstripe.1 from lfs.1	Resolved

Sub-Tasks:

Key

Summary

Type

Status

Assignee

LU-9004

Merge in patches from development branch

Technical task

Resolved

Niu Yawei

LU-9005

Layout inheritance from parent direct...

Technical task

Resolved

Zhenyu Xu

LU-9006

Layout modification from userspace of...

Technical task

Resolved

Zhenyu Xu

LU-9008

Dynamic layout modification during wr...

Technical task

Resolved

Zhenyu Xu

LU-9009

Server LFSCK support for Compound Lay...

Technical task

Resolved

nasf

LU-9165

MDS handling of PFL layout initializa...

Technical task

Resolved

Zhenyu Xu

LU-9185

flawed layout add replay

Technical task

Resolved

Zhenyu Xu

LU-9198

LFSCK needs to skip verifying layout ...

Technical task

Resolved

nasf

LU-9256

lfs setstripe/getstripe support negat...

Technical task

Resolved

Niu Yawei

Rank (Obsolete):

9223372036854775807

Description

The work tracked under this ticket will implement Progressive File Layouts (PFL).

This feature simplifies Lustre usage of large files, avoiding the need for users to predetermine the required layout for each file. The goal is that a single file layout specified at file creation time can balance the following conflicting needs:

minimize the number of stripes for small files to avoid the overhead of creating and accessing multiple OST objects
maximize the number of stripes for large files to utilize the available bandwidth as well as distribute space usage to avoid consuming all of the space on a single OST
flexibility to specify a layout that is suitable for a majority of uses, while allowing for variability and flexibility between users, applications, and sites

The administrator will be able to set a specific progressive layout for new files created in the filesystem, and users will be able to specify progressive layouts on their own files for their specific needs. It should be possible to specify a default progressive layout template for a directory that is inherited by new files and directories created in that directory, as well as a filesystem-wide global default layout that is used in the absence of more specific layouts.

Comments

Comment by Gerrit Updater [ 11/Jan/17 ]

Niu Yawei (yawei.niu@intel.com) uploaded a new patch: https://review.whamcloud.com/24822
Subject: ~~LU-8998~~ pfl: Basic data structures for PFL
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: adfb0ab1f6a8c908291e4032b3160add5c360307

Comment by Gerrit Updater [ 11/Jan/17 ]

Niu Yawei (yawei.niu@intel.com) uploaded a new patch: https://review.whamcloud.com/24823
Subject: ~~LU-8998~~ lod: accomodate to composite layout
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: f5c7f2fedac6fd7826fe5801fb5112e1ed9661ee

Comment by Gerrit Updater [ 12/Jan/17 ]

Niu Yawei (yawei.niu@intel.com) uploaded a new patch: https://review.whamcloud.com/24849
Subject: ~~LU-8998~~ lov: add composite layout unpacking
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: 7446c7383962bb41eb2ef127bcd42453d8ac5de2

Comment by Gerrit Updater [ 12/Jan/17 ]

Niu Yawei (yawei.niu@intel.com) uploaded a new patch: https://review.whamcloud.com/24850
Subject: ~~LU-8998~~ clio: Client side implementation for PFL
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: 67985895abc35e92708a37b4a19eef8624ce0711

Comment by Gerrit Updater [ 12/Jan/17 ]

Niu Yawei (yawei.niu@intel.com) uploaded a new patch: https://review.whamcloud.com/24851
Subject: ~~LU-8998~~ clio: getstripe support comp layout
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: ceb8f8906ecf02bda152a0cd976301acd0905a5f

Comment by Gerrit Updater [ 13/Jan/17 ]

Niu Yawei (yawei.niu@intel.com) uploaded a new patch: https://review.whamcloud.com/24872
Subject: ~~LU-8998~~ tests: test scripts for PFL
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: 3956a7fe3a535cb9db55f6326ec125347dae4b04

Comment by Gerrit Updater [ 13/Jan/17 ]

Niu Yawei (yawei.niu@intel.com) uploaded a new patch: https://review.whamcloud.com/24873
Subject: ~~LU-8998~~ utils: enlarge default inode size
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: bab212e8afd362e7d0b4352056c35c760d231a77

Comment by Niu Yawei (Inactive) [ 13/Jan/17 ]

Niu Yawei (yawei.niu@intel.com) uploaded a new patch: https://review.whamcloud.com/24876
Subject: ~~LU-8050~~ osp: not use obdo::o_lcookie for non-onwire RPC data
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: c79ef5ad1e51d55fc85e7f2fdf91e10bf76029a0

Comment by Gerrit Updater [ 13/Jan/17 ]

Fan Yong (fan.yong@intel.com) uploaded a new patch: https://review.whamcloud.com/24883
Subject: ~~LU-8998~~ pfl: layout LFSCK handles PFL file
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: 2c95bbd8e94947589dbf30a464583ed1bf1d7129

Comment by Gerrit Updater [ 13/Jan/17 ]

Fan Yong (fan.yong@intel.com) uploaded a new patch: https://review.whamcloud.com/24884
Subject: ~~LU-8998~~ pfl: test cases for lfsck on PFL
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: dcab7a5b1ff70e610e12a692c3995ca853fd4381

Comment by Gerrit Updater [ 13/Jan/17 ]

Fan Yong (fan.yong@intel.com) uploaded a new patch: https://review.whamcloud.com/24882
Subject: ~~LU-8998~~ pfl: enhance PFID EA for PFL
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: d184d9df7d94d6d35a2b9626f61635084e890e99

Comment by Gerrit Updater [ 03/Feb/17 ]

Fan Yong (fan.yong@intel.com) uploaded a new patch: https://review.whamcloud.com/25238
Subject: ~~LU-8998~~ pfl: layout LFSCK handles PFL file
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4c515381de9b494e8adcb7f8743956538c344c7b

Comment by Gerrit Updater [ 03/Feb/17 ]

Fan Yong (fan.yong@intel.com) uploaded a new patch: https://review.whamcloud.com/25239
Subject: ~~LU-8998~~ pfl: test cases for lfsck on PFL
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b4162d8b82f3385bce7315f80f03e3d23b4bfbfa

Comment by Gerrit Updater [ 03/Feb/17 ]

Fan Yong (fan.yong@intel.com) uploaded a new patch: https://review.whamcloud.com/25237
Subject: ~~LU-8998~~ pfl: enhance PFID EA for PFL
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8ef648657c437da37aa3f873883a7d93cbe129ae

Comment by Gerrit Updater [ 09/Feb/17 ]

Jinshan Xiong (jinshan.xiong@intel.com) merged in patch https://review.whamcloud.com/24822/
Subject: ~~LU-8998~~ pfl: Basic data structures for composite layout
Project: fs/lustre-release
Branch: pfl
Current Patch Set:
Commit: 25397a808489671116c8732e8781df8e9b87b03d

Comment by Gerrit Updater [ 09/Feb/17 ]

Jinshan Xiong (jinshan.xiong@intel.com) merged in patch https://review.whamcloud.com/24823/
Subject: ~~LU-8998~~ lod: accomodate to composite layout
Project: fs/lustre-release
Branch: pfl
Current Patch Set:
Commit: a11a1e4c1484cb93f99de73d5679eb059021eb3d

Comment by Gerrit Updater [ 16/Feb/17 ]

Andreas Dilger (andreas.dilger@intel.com) merged in patch https://review.whamcloud.com/24849/
Subject: ~~LU-8998~~ lov: add composite layout unpacking
Project: fs/lustre-release
Branch: pfl
Current Patch Set:
Commit: 5850d1f9a6dec27471d33d67fe59a467249c73ab

Comment by Gerrit Updater [ 03/Mar/17 ]

Niu Yawei (yawei.niu@intel.com) uploaded a new patch: https://review.whamcloud.com/25740
Subject: ~~LU-8998~~ docs: man pages for composite layout APIs
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: a13ff92153161ec6cebf542f61951eb13ec97e81

Comment by Gerrit Updater [ 03/Mar/17 ]

Niu Yawei (yawei.niu@intel.com) uploaded a new patch: https://review.whamcloud.com/25741
Subject: ~~LU-8998~~ docs: man pages for tools of PFL
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: 1ddbe787aa6e663ccfa2f53468529b190a0a7118

Comment by Gerrit Updater [ 07/Mar/17 ]

Jinshan Xiong (jinshan.xiong@intel.com) merged in patch https://review.whamcloud.com/24850/
Subject: ~~LU-8998~~ clio: Client side implementation for PFL
Project: fs/lustre-release
Branch: pfl
Current Patch Set:
Commit: 1fd3ec455a8e17d8f647ad43a5ff7cc2e7b71da3

Comment by Gerrit Updater [ 07/Mar/17 ]

Jinshan Xiong (jinshan.xiong@intel.com) merged in patch https://review.whamcloud.com/24851/
Subject: ~~LU-8998~~ clio: getstripe support comp layout
Project: fs/lustre-release
Branch: pfl
Current Patch Set:
Commit: 3c8ba6bc531f2ce27d755e9fff1c32b70939745e

Comment by Gerrit Updater [ 08/Mar/17 ]

Jinshan Xiong (jinshan.xiong@intel.com) merged in patch https://review.whamcloud.com/24866/
Subject: ~~LU-8998~~ lustreapi: composite llapi_layout
Project: fs/lustre-release
Branch: pfl
Current Patch Set:
Commit: c2b3dba3930dcc65a4c85fcd6a9d295843d5030c

Comment by Gerrit Updater [ 11/Mar/17 ]

Jinshan Xiong (jinshan.xiong@intel.com) merged in patch https://review.whamcloud.com/24867/
Subject: ~~LU-8998~~ lfs: user space tools for PFL
Project: fs/lustre-release
Branch: pfl
Current Patch Set:
Commit: 926b96176abce16805c4d2777c0dd951a49d53e6

Comment by Gerrit Updater [ 11/Mar/17 ]

Jinshan Xiong (jinshan.xiong@intel.com) merged in patch https://review.whamcloud.com/25741/
Subject: ~~LU-8998~~ docs: man pages for tools of PFL
Project: fs/lustre-release
Branch: pfl
Current Patch Set:
Commit: a1e0fd6f6c2e27bc565e20b2c7c21c78d473ebc2

Comment by Gerrit Updater [ 13/Mar/17 ]

Jinshan Xiong (jinshan.xiong@intel.com) merged in patch https://review.whamcloud.com/24872/
Subject: ~~LU-8998~~ tests: test scripts for PFL
Project: fs/lustre-release
Branch: pfl
Current Patch Set:
Commit: ec6584fbe21b057cee7aa64384904a845909e677

Comment by Gerrit Updater [ 13/Mar/17 ]

Jinshan Xiong (jinshan.xiong@intel.com) merged in patch https://review.whamcloud.com/24873/
Subject: ~~LU-8998~~ utils: enlarge default inode size
Project: fs/lustre-release
Branch: pfl
Current Patch Set:
Commit: 6f29288eabae47647ec9762237ee3da0e7f15068

Comment by Gerrit Updater [ 27/Mar/17 ]

Niu Yawei (yawei.niu@intel.com) uploaded a new patch: https://review.whamcloud.com/26198
Subject: ~~LU-8998~~ tools: support negative flags
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: 76d556c794cb6d757e1e87bfa33548770e73abec

Comment by Gerrit Updater [ 27/Mar/17 ]

Jinshan Xiong (jinshan.xiong@intel.com) merged in patch https://review.whamcloud.com/25740/
Subject: ~~LU-8998~~ docs: man pages for composite layout APIs
Project: fs/lustre-release
Branch: pfl
Current Patch Set:
Commit: 17b04017b5f27226a7a921457f049e281137683f

Comment by Gerrit Updater [ 27/Mar/17 ]

Jinshan Xiong (jinshan.xiong@intel.com) merged in patch https://review.whamcloud.com/24882/
Subject: ~~LU-8998~~ pfl: enhance PFID EA for PFL
Project: fs/lustre-release
Branch: pfl
Current Patch Set:
Commit: ceff6cb807bad07161c64381f660605e44bc6896

Comment by Gerrit Updater [ 29/Mar/17 ]

Jinshan Xiong (jinshan.xiong@intel.com) merged in patch https://review.whamcloud.com/24883/
Subject: ~~LU-8998~~ pfl: layout LFSCK handles PFL file
Project: fs/lustre-release
Branch: pfl
Current Patch Set:
Commit: e1142e06d190d6ae5b966b32ae2921b9b6db1c7a

Comment by Gerrit Updater [ 30/Mar/17 ]

Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: https://review.whamcloud.com/26268
Subject: ~~LU-8998~~ docs: add lfs-setripe manpage to dist
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: e874b20b078926845fda8454ca0fafbd2585774c

Comment by Gerrit Updater [ 30/Mar/17 ]

Jinshan Xiong (jinshan.xiong@intel.com) merged in patch https://review.whamcloud.com/26268/
Subject: ~~LU-8998~~ docs: add lfs-setripe manpage to dist
Project: fs/lustre-release
Branch: pfl
Current Patch Set:
Commit: 97dab8458adde6bb1ece58359483b91deca960f1

Comment by Gerrit Updater [ 30/Mar/17 ]

-Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: https://review.whamcloud.com/26271-
~~Subject: ~~LU-8998~~ docs: add layout component API manpages to dist~~
~~Project: fs/lustre-release~~
~~Branch: pfl~~
~~Current Patch Set: 1~~
~~Commit: 7b455b37277f619654112e028f55c1cf10d1ab18~~

Comment by Gerrit Updater [ 30/Mar/17 ]

Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: https://review.whamcloud.com/26275
Subject: ~~LU-8998~~ docs: add llapi_ man pages to Makefile.am
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: 41d538fb393c642398c88aefc68ea0c1f93eaa96

Comment by Gerrit Updater [ 31/Mar/17 ]

Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: https://review.whamcloud.com/26276
Subject: ~~LU-8998~~ llapi: rename llapi_layout_comp_move -> *use
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: 42f0259a1e2a2bda4ddd26c73e2236fbe1ea3869

Comment by Gerrit Updater [ 31/Mar/17 ]

Jinshan Xiong (jinshan.xiong@intel.com) merged in patch https://review.whamcloud.com/26275/
Subject: ~~LU-8998~~ docs: add llapi_ man pages to Makefile.am
Project: fs/lustre-release
Branch: pfl
Current Patch Set:
Commit: ecd39edc714c98fe49da11bf61a38931a001f912

Comment by Gerrit Updater [ 31/Mar/17 ]

Jinshan Xiong (jinshan.xiong@intel.com) merged in patch https://review.whamcloud.com/24884/
Subject: ~~LU-8998~~ pfl: test cases for lfsck on PFL
Project: fs/lustre-release
Branch: pfl
Current Patch Set:
Commit: 20933bd62fbb170d8f34c4e976c52bce6dc901ac

Comment by Gerrit Updater [ 31/Mar/17 ]

Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: https://review.whamcloud.com/26290
Subject: ~~LU-8998~~ llapi: remove llapi_layout_comp_use_at()
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: 03f384b77244566c16e1fb43dae02c5223a6c1ab

Comment by Gerrit Updater [ 04/Apr/17 ]

Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: https://review.whamcloud.com/26333
Subject: ~~LU-8998~~ tools: support negative flags
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: 771a9b430a3a75c697821d22758722c6292b614f