[LU-11022] FLR1.5: "lfs mirror" usability for Burst Buffer

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.13.0
• Labels: FLR2
• Assignee: Andreas Dilger
• Reporter: Andreas Dilger

Description

I've been going through a simple exercise for how to use FLR to mirror/unmirror files for a burst-buffer application. The workflow would be something like:

1. job startup script has directives that specify the list of existing files that are used for input/output, and the directory used for output
2. job scheduler processes job startup script at appropriate time before job launch to mirror input/output files into burst buffer (flash-based OSTs in a special OST pool[*]) and mark them "prefer" so that the clients will use those replicas
3. job scheduler specifies default directory layout for output directory, using DoM and/or FLR to keep new files on flash storage
4. job runs and does IO entirely to flash storage
5. optionally a background task does file resync/migration to copy modified files to HDD-based OSTs
6. job scheduler runs post-job script to resync/migrate all of the new and modified files to HDD-based OSTs, drops flash mirror copy on input/output files if they are not used by another job in the queue. This avoids the need to scan the OSTs and manage OST space on demand.

[*] We don't have any way to prevent users from using a pool if they want to. We need some kind of OST/pool quota to limit the amount of space a user can consume on a given OST/pool. It might be desirable to allow privileged users (e.g. job scheduler) to still create files on an OST/pool, even if it exceeds the user's quota, so that they can stage files there.

The #1 item is not immediately in my control.

I was trying out what commands would be used for #2. The obvious choice is lfs mirror extend -N<copies> /path/to/file, but one problem I see with this is that "-N<copies>" means add <copies> mirrors to the file, rather than make the number of mirrors = <copies>. This is problematic, since lfs mirror extend will keep on adding mirrors to the file, even if it already has mirrors >= <copies>, but not an insurmountable problem (the caller needs to use "lfs getstripe -N" to get the current number of mirror copies, then call "lfs mirror extend -N$((copies - current))" in most cases).

• There should be an option in lfs mirror extend to specify something like -N=2 to indicate that only 2 mirrors should be created with the given parameters (though "=2" won't work because getopt_long() will just treat that the same as -N 2). I don't think it would be acceptable to change the meaning of "-N" in 2.12 after it has been available in 2.11.

For #3 and #4 we would set a default layout on the output directory to create files with DoM + PFL layouts to keep the output files entirely on flash.

For #5 we could use a ChangeLog user to follow files from each JobID to do resync (to HDD-based OSTs) in the background as they are closed, but it would make sense to apply a policy for this (e.g. migrate only 1/4 of incremental checkpoints out of the BB).

For #6 it would use lfs mirror extend or lfs mirror resync to migrate files specified in the job submission script from the flash OSTs to HDD OSTs. What is difficult is to remove the flash OST replicas afterward. The lfs mirror split command requires specifying an explicit mirror ID, but lfs getstripe has no option to extract the mirror ID for a component. This raises the need for several new options:

• lfs mirror split should allows specifying lfs getstripe options like --comp-flags=prefer or --pool=flash to remove replicas.
• lfs mirror split should accept --component-id=<comp_ID> from lfs getstripe as removing the entire mirror.
• lfs getstripe should allow printing the mirror ID for matching components.

Issue Links

• is related to: LU-12649 Tracker for ongoing FLR improvements
• is related to: LU-12648 FLR2: File versioning with FLR and "nosync" files
• is related to: LU-11023 OST Pool Quotas
• is related to: LU-13484 inode access time tables
• is related to: LU-11467 FLR1.5: add "lfs mirror delete" command
• is related to: LU-13720 "lfs mirror delete" should resync file if needed
• is related to: LU-10191 FLR2: Server Local Client (SLC)
• is related to: LU-16446 'lfs mirror extend' should allow specifying total mirror count

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/36194/
Subject: LU-11022 lfs: remove mirror by pool name
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 3a8d96a1ebff995cbdad359208ea0ce99482606a

[Gerrit Updater]

Jian Yu ([e-mail]) uploaded a new patch: https://review.whamcloud.com/36194
Subject: LU-11022 lfs: remove mirror by pool name
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 36841f23dc32e73f41c0bfae9877bb9a5bcbcf72

[Andreas Dilger]

Reopen this issue since it is a high-level tracker for multiple different issues related to tiered storage in Lustre.

[Peter Jones]

All work in 2.13 landed

[Joseph Gmitter (Inactive)]

Perhaps we should break out LU-11467 to its own ticket (not a subtask) so that this can be marked resolved for 2.13. Otherwise, this ticket won't show up in the change log for the release since it will be open and need to remove the fix version.

[Andreas Dilger]

There are no remaining patches left for this ticket - is this task now complete?

It won't be possible to close this issue until the sub-task LU-11467 is finished. I think that is relatively easily implemented, but is not necessarily as important as the "local client" functionality (LU-10191).

[Peter Jones]

There are no remaining patches left for this ticket - is this task now complete?

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35329/
Subject: LU-11022 lfs: remove mirror by pool name
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 0c710a46cfb43366dc57ff6e83e414086b1d0e6c

[Gerrit Updater]

Alex Zhuravlev ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35329
Subject: LU-11022 lfs: remove mirror by pool name
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: f86542c007c652ac5758eb3dcf51c1175807182e

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/32455/
Subject: LU-11022 lfs: accept specifing comp_id in mirror split
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 82298b9cf8d6a4e4a8529366536815bcf80c6bd6