[LU-8324] HSM: prioritize HSM requests

• Type: Improvement
• Resolution: Unresolved
• Priority: Minor
• Severity: 3
• Assignee: Quentin Bouget (Inactive)
• Reporter: CEA

Description

Most of the time (unless the filesystem is full), RESTORE and REMOVE requests should be processed first as they have the highest priority from a user's point of view ; ARCHIVE requests should have a lower priority.

Issue Links

• is duplicated by: LU-14363 Prioritize HSM cancel request
• is related to: LU-8382 HSM: reorder coordinator's cleanup functions
• is related to: LU-10968 add coordinator bypass upcalls for HSM archive and remove

Comments

[Gerrit Updater]

Quentin Bouget ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34749
Subject: LU-8324 hsm: prioritize one RESTORE once in a while
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 3fa2b1682755eeb988d10af53797a8d5e1a3679d

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/31723/
Subject: LU-8324 hsm: prioritize one RESTORE once in a while
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 0dce1ddefc673a3f39b4964d6b669e2a11aaf903

[Gerrit Updater]

Quentin Bouget ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33239
Subject: LU-8324 hsm: prioritize one RESTORE once in a while
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 93364e9f3b0c9694904d2c1e2a687af61a980c1f

[Quentin Bouget (Inactive)]

The patch above is the shortest/simplest hack I could come up with to help bear with LU-8324 until a more definitive fix is developed (it is more of a band-aid than anything else).

The idea is to use the times when the coordinator traverses its whole llog to "force-schedule" at least one RESTORE request. In practice, this means that you should see at least one RESTORE request scheduled every "loop_period" (the value in /proc/<fsname>/mdt/<mdt-name>/hsm/loop_period) seconds.

[Gerrit Updater]

Quentin Bouget ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31723
Subject: LU-8324 hsm: prioritize one RESTORE once in a while
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b30b036607a2bc4928e13e06462701bf5ba62d3d

[Nathan Rutman]

This thread looks kind of dead, but we have a desire to see some prioritization mechanism as well.
Some options:
1. FIFO (today)
2. Restore-first. All restore requests are prioritized over archive requests. (Except in-progress archives.)
3. Archive-first. All archives are prioritized.
4. Interleaved. Archive and Restore requests are alternated, as long as some of each are waiting.
5. Tunable. Adjustable ratio of archive:restore processing. Maybe this covers the above 2-4 as well.
6. Batched. Archives and Restores are grouped into separate batches, potentially resulting in fewer tape swaps.
7. Time-boxed. A variant of batched; batch ends after a fixed time period.
Many other options I'm sure...

Ultimately I'm in agreement with Robert Read's comment above that the prioritization should really be done outside of Lustre, but if the patch here implements #5 that might cover enough of the use cases to make most people happy...

[Gerrit Updater]

Quentin Bouget ([e-mail]) uploaded a new patch: https://review.whamcloud.com/27800
Subject: LU-8324 hsm: ease the development of a different coordinator
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0ee6e5d71c6f549b48be59607d5f55f28e950f47

[Gerrit Updater]

Quentin Bouget ([e-mail]) uploaded a new patch: https://review.whamcloud.com/27394
Subject: LU-8324 hsm: prioritize HSM requests
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: f8d7f289866c2219d19a51693ae44cc6c3fdf867

[Peter Jones]

I think that 2.10.1 is more likely option at this stage. It seems like there will be some discussions about this area at LUG next week.

[Cory Spitz]

If not 2.10, it seems that 2.10.1 would be possible.