[LU-8324] HSM: prioritize HSM requests

• Type: Improvement
• Resolution: Unresolved
• Priority: Minor
• Severity: 3
• Assignee: Quentin Bouget (Inactive)
• Reporter: CEA

Description

Most of the time (unless the filesystem is full), RESTORE and REMOVE requests should be processed first as they have the highest priority from a user's point of view ; ARCHIVE requests should have a lower priority.

Issue Links

• is duplicated by: LU-14363 Prioritize HSM cancel request
• is related to: LU-8382 HSM: reorder coordinator's cleanup functions
• is related to: LU-10968 add coordinator bypass upcalls for HSM archive and remove

Comments

[Peter Jones]

I think that 2.10.1 is more likely option at this stage. It seems like there will be some discussions about this area at LUG next week.

[Cory Spitz]

If not 2.10, it seems that 2.10.1 would be possible.

[Henri Doreau (Inactive)]

Any chance for this patch to make it into 2.10? It is a very useful feature for HSM users and we believe that the patch is mature.

[Quentin Bouget (Inactive)]

Hello Matt,

I think the patch is mature enough for you to test it if you are still interested in it.

[Matt Rásó-Barnett (Inactive)]

Hello,
I would just like to say that we would be very keen on this kind of feature at Cambridge - I've just run into this issue today where a single file restore operation is at the back of the queue behind ~10TB of archive jobs.

I'd be interested in testing this patch against one of our test filesystems, but I just wanted to add a comment that we would really appreciate having more ability to control the coordinator queue - whether it's in it's current state or some future tool as Robert suggests.

Kind regards,
Matt Raso-Barnett
University of Cambridge

[Gerrit Updater]

Quentin Bouget ([e-mail]) uploaded a new patch: http://review.whamcloud.com/21494
Subject: LU-8324 hsm: prioritize HSM requests
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0655c8faf7cb7dcdb3b19dd761aad6c06fcda159

[Henri Doreau (Inactive)]

We would happily consider a more resilient and distributed mecanism for the coordinator. Nevertheless, I see it as a non-trivial project that should not block improvements of HSM, if it targets mid-term future (I have neither seen any design document nor heard any discussion about it).

The patch has not been pushed yet but the solution that Quentin proposes is leightweight and elegant and I believe that it significantly improves the experience of using HSM in production.
It is more subjective, but I also find that it improves code quality and makes it easier to reason about the logic of the CDT, which would be helpful for future replacement work.

[Robert Read]

I agree we need to prioritize these operations, however I don’t believe adding prioritization to the coordinator is right answer here. We are currently on a path that will turn the coordinator into a general purpose request queue, and this is not something that belongs in Lustre code and certainly not in the kernel.

Instead, we should move the HSM request processing out of the kernel and into user space. Although Lustre will still need to keep track of the implicit restore requests triggered by file access, all other operations could be done without using a coordinator. Lustre should provide the mechanisms needed for a correct HSM system, and allow the user space tools manage all of the policies around what and when is copied and their priorities.

I’m still thinking about exactly what this should look like, but at a minimum an Archive operation begins with setting the EXISTS flag, and completes with setting ARCHIVE flag. If the file is modified after EXISTS is set, then the MDT will set the DIRTY flag and reject the ARCHIVE flag when mover attempts to set it later.

A Restore operation is primarily a layout swap, though it may need to be a special case to ensure the RELEASED flag is cleared atomically with the swap.

A Remove operation is done by clearing the EXISTS and ARCHIVE flags.

The existing coordinator should remain in place for some time to continue to support current set of tools, but I would like to discourage adding further complexity, and solve issues like this in a different way.

[Peter Jones]

ok Quentin. Let us know how you progress

[Quentin Bouget (Inactive)]

I am working on addind a dynamic policy to the coordinator that would define which requests are to be sent first to copytools.

The policy I am currently implementing defines two levels of priority and allows administrators to set which kind of request gets which priority (default will be low_priority = [ ARCHIVE ], high_priority = [ RESTORE, CANCEL, REMOVE, ... ]). One could also set a ratio that the coordinator tries to follow to batch requests and send them to copytools (X% high_priority and (100 -X) % low_priority), this prevents starvation. The ratio is a soft limit (if there is too little of one priority level of request to fill the buffers the ratio is not used), this prevents from wasting time.