[LU-11022] FLR1.5: "lfs mirror" usability for Burst Buffer

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.13.0
• Labels: FLR2
• Assignee: Andreas Dilger
• Reporter: Andreas Dilger

Description

I've been going through a simple exercise for how to use FLR to mirror/unmirror files for a burst-buffer application. The workflow would be something like:

1. job startup script has directives that specify the list of existing files that are used for input/output, and the directory used for output
2. job scheduler processes job startup script at appropriate time before job launch to mirror input/output files into burst buffer (flash-based OSTs in a special OST pool[*]) and mark them "prefer" so that the clients will use those replicas
3. job scheduler specifies default directory layout for output directory, using DoM and/or FLR to keep new files on flash storage
4. job runs and does IO entirely to flash storage
5. optionally a background task does file resync/migration to copy modified files to HDD-based OSTs
6. job scheduler runs post-job script to resync/migrate all of the new and modified files to HDD-based OSTs, drops flash mirror copy on input/output files if they are not used by another job in the queue. This avoids the need to scan the OSTs and manage OST space on demand.

[*] We don't have any way to prevent users from using a pool if they want to. We need some kind of OST/pool quota to limit the amount of space a user can consume on a given OST/pool. It might be desirable to allow privileged users (e.g. job scheduler) to still create files on an OST/pool, even if it exceeds the user's quota, so that they can stage files there.

The #1 item is not immediately in my control.

I was trying out what commands would be used for #2. The obvious choice is lfs mirror extend -N<copies> /path/to/file, but one problem I see with this is that "-N<copies>" means add <copies> mirrors to the file, rather than make the number of mirrors = <copies>. This is problematic, since lfs mirror extend will keep on adding mirrors to the file, even if it already has mirrors >= <copies>, but not an insurmountable problem (the caller needs to use "lfs getstripe -N" to get the current number of mirror copies, then call "lfs mirror extend -N$((copies - current))" in most cases).

• There should be an option in lfs mirror extend to specify something like -N=2 to indicate that only 2 mirrors should be created with the given parameters (though "=2" won't work because getopt_long() will just treat that the same as -N 2). I don't think it would be acceptable to change the meaning of "-N" in 2.12 after it has been available in 2.11.

For #3 and #4 we would set a default layout on the output directory to create files with DoM + PFL layouts to keep the output files entirely on flash.

For #5 we could use a ChangeLog user to follow files from each JobID to do resync (to HDD-based OSTs) in the background as they are closed, but it would make sense to apply a policy for this (e.g. migrate only 1/4 of incremental checkpoints out of the BB).

For #6 it would use lfs mirror extend or lfs mirror resync to migrate files specified in the job submission script from the flash OSTs to HDD OSTs. What is difficult is to remove the flash OST replicas afterward. The lfs mirror split command requires specifying an explicit mirror ID, but lfs getstripe has no option to extract the mirror ID for a component. This raises the need for several new options:

• lfs mirror split should allows specifying lfs getstripe options like --comp-flags=prefer or --pool=flash to remove replicas.
• lfs mirror split should accept --component-id=<comp_ID> from lfs getstripe as removing the entire mirror.
• lfs getstripe should allow printing the mirror ID for matching components.

Issue Links

• is related to: LU-12649 Tracker for ongoing FLR improvements
• is related to: LU-12648 FLR2: File versioning with FLR and "nosync" files
• is related to: LU-11023 OST Pool Quotas
• is related to: LU-13484 inode access time tables
• is related to: LU-11467 FLR1.5: add "lfs mirror delete" command
• is related to: LU-13720 "lfs mirror delete" should resync file if needed
• is related to: LU-10191 FLR2: Server Local Client (SLC)
• is related to: LU-16446 'lfs mirror extend' should allow specifying total mirror count

Comments

[Andreas Dilger]

Reopen this issue since it is a high-level tracker for multiple different issues related to tiered storage in Lustre.

[Peter Jones]

All work in 2.13 landed

[Joseph Gmitter (Inactive)]

Perhaps we should break out LU-11467 to its own ticket (not a subtask) so that this can be marked resolved for 2.13. Otherwise, this ticket won't show up in the change log for the release since it will be open and need to remove the fix version.

[Andreas Dilger]

There are no remaining patches left for this ticket - is this task now complete?

It won't be possible to close this issue until the sub-task LU-11467 is finished. I think that is relatively easily implemented, but is not necessarily as important as the "local client" functionality (LU-10191).

[Peter Jones]

There are no remaining patches left for this ticket - is this task now complete?

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35329/
Subject: LU-11022 lfs: remove mirror by pool name
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 0c710a46cfb43366dc57ff6e83e414086b1d0e6c

[Gerrit Updater]

Alex Zhuravlev ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35329
Subject: LU-11022 lfs: remove mirror by pool name
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: f86542c007c652ac5758eb3dcf51c1175807182e

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/32455/
Subject: LU-11022 lfs: accept specifing comp_id in mirror split
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 82298b9cf8d6a4e4a8529366536815bcf80c6bd6

[Gerrit Updater]

Bobi Jam ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32455
Subject: LU-11022 lfs: accept specifing comp_id in mirror split
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8aa29517a49f583e89646c406843601fbc1147a7

[Jinshan Xiong]

For #2, I think it should check if the corresponding files have already had mirrors on flash OSTs, before it tries to add new mirrors. Checking mirrors should be lightweight in general so I don't think this would be an issue. Anyway, extending file to have a specified number of mirrors is probably less useful in this case, because what we want is to have a mirror on flash OSTs, no matter how many mirrors it already has.

For #6, yes if it drops mirrors at the end of job, it won't need database to keep track of files on burst buffer; but I'm not sure if this is the best approach to use burst buffer. If for any reason the jobs need to restart from the last checkpoint, we always want(and most likely) the required data are already inside burst buffer to avoid data transfer from slow HDD OSTs. I agree that file management database will add complexity but it's totally worth it.

• lfs mirror split should allows specifying lfs getstripe options like -comp-flags=prefer or -pool=flash to remove replicas.

The issue with deleting mirrors by flags is that sometimes those flags can't identify a mirror uniquely. lfs utilities are fundamental features, which could be used many features, so we don't want anything special for burst buffer at this level.

Yes, there is no options to print mirror id only in lfs getstripe, so it will need a grep to print it out, like:

[root@centos tests]# ../utils/lfs getstripe /mnt/lustre/tm | grep lcme_mirror_id
    *lcme_mirror_id*:      1
    *lcme_mirror_id*:      2

I think there should be utilities developed for burst buffer to iterate and check mirror ID by components through liblustreapi, but I don't think this should be a feature that belongs to lfs itself. Only burst buffer knows which OSTs are being identified as burst buffer.

I agree on the rest.