[LU-15011] implement lod pool spilling

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.15.0
• Type: Improvement
• Priority: Minor
• Reporter: Andreas Dilger
• Assignee: Alex Zhuravlev
• Resolution: Fixed

Description

If an OST pool becomes full, allow configuring a global default spill target to hold files that could not be written due to the pool running out of space. This is similar to SEL, but implemented at the pool selection level, rather than on a per-write basis. While SEL is probably flexible (essentially it implements pool spill on a per-component basis), very few users know enough about SEL to use it effectively, and any changes to the SEL configuration would require updating every file that is using a particualr SEL layout. In contrast, pool spill is a tunable on the MDS (for each source pool) and affects all existing layouts using that pool if there is a spill target pool and used space threshold specified.

Issue Links

• is related to: LUDOC-517 add pool_spill parameters to manual
• is related to: LU-15072 Pool spill is activated just by chang...
• is related to: LU-15043 OST spill pools should not allow spil...
• is related to: LU-13058 Intermediate component removal (PFL/SEL)

Comments

[Andreas Dilger]

Originally submitted as LU-14825, but that issue was lost due to a database error:

"Alex Zhuravlev <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/43989
Subject: LU-14825 lod: pool spilling
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0a998f4723f58e68d0fe75934df5c282eb529337

[Gerrit Updater]

"James Nunez <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/45056
Subject: LU-15011 tests: pool spill test modifications
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a097f6ea7e5a9514c08995d6ed721cba05bb1fb0

[Gerrit Updater]

"Alex Zhuravlev <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/45120
Subject: LU-15011 tests: ost-pools to remove big files
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ad32cbc4f0654c3ae3c96b4bd1d1526445d94913

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/45056/
Subject: LU-15011 tests: pool spill test modifications
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: d8ff73097be78cc3e603ab49b6aaa03a1c766fcb

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/45120/
Subject: LU-15011 tests: ost-pools to remove big files
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: ebed0531abed9512682574e44dd17f465fa9b04c

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/44947/
Subject: LU-15011 lod: count all spilling events
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4bd9b524537bce27fb7f53cf481b78290cd55a1d

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/45074/
Subject: LU-15011 tests: additional checks for pool spilling
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 26e765e0d23ba37577775331ac77ec857affd845

[Cory Spitz]

Just curious, but was a LUDOC ever opened to track this Improvement?

[Andreas Dilger]

spitzcor, I don't see it in the manual. However, I found your patch to add this documentation https://review.whamcloud.com/49725 "LUDOC-512 manual: add append_pool and append_stripe_count" and will land it now.

[Andreas Dilger]

I filed LUDOC-517 for the "pool_spill" documentation.