https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8324 Lustre <p>Most of the time (unless the filesystem is full), RESTORE and REMOVE requests should be processed first as they have the highest priority from a user's point of view ; ARCHIVE requests should have a lower priority.</p> LU-8324

HSM: prioritize HSM requests

Improvement Minor Open Unresolved Quentin Bouget CEA Fri, 24 Jun 2016 12:30:27 +0000 Mon, 25 Jan 2021 10:21:08 +0000 2 17 <p>I am working on addind a dynamic policy to the coordinator that would define which requests are to be sent first to copytools.</p> <p>The policy I am currently implementing defines two levels of priority and allows administrators to set which kind of request gets which priority (default will be low_priority = [ ARCHIVE ], high_priority = [ RESTORE, CANCEL, REMOVE, ... ]). One could also set a ratio that the coordinator tries to follow to batch requests and send them to copytools (X% high_priority and (100 -X) % low_priority), this prevents starvation. The ratio is a soft limit (if there is too little of one priority level of request to fill the buffers the ratio is not used), this prevents from wasting time.</p> <p>ok Quentin. Let us know how you progress</p> <p>I agree we need to prioritize these operations, however I don’t believe adding prioritization to the coordinator is right answer here. We are currently on a path that will turn the coordinator into a general purpose request queue, and this is not something that belongs in Lustre code and certainly not in the kernel. </p> <p>Instead, we should move the HSM request processing out of the kernel and into user space. Although Lustre will still need to keep track of the implicit restore requests triggered by file access, all other operations could be done without using a coordinator. Lustre should provide the mechanisms needed for a correct HSM system, and allow the user space tools manage all of the policies around what and when is copied and their priorities. <br/> 
I’m still thinking about exactly what this should look like, but at a minimum an Archive operation begins with setting the EXISTS flag, and completes with setting ARCHIVE flag. If the file is modified after EXISTS is set, then the MDT will set the DIRTY flag and reject the ARCHIVE flag when mover attempts to set it later.</p> <p>A Restore operation is primarily a layout swap, though it may need to be a special case to ensure the RELEASED flag is cleared atomically with the swap. </p> <p>A Remove operation is done by clearing the EXISTS and ARCHIVE flags.</p> <p>The existing coordinator should remain in place for some time to continue to support current set of tools, but I would like to discourage adding further complexity, and solve issues like this in a different way. </p> <p>We would happily consider a more resilient and distributed mecanism for the coordinator. Nevertheless, I see it as a non-trivial project that should not block improvements of HSM, if it targets mid-term future (I have neither seen any design document nor heard any discussion about it).</p> <p>The patch has not been pushed yet but the solution that Quentin proposes is leightweight and elegant and I believe that it significantly improves the experience of using HSM in production.<br/> It is more subjective, but I also find that it improves code quality and makes it easier to reason about the logic of the CDT, which would be helpful for future replacement work.</p> <p>Quentin Bouget (quentin.bouget.ocre@cea.fr) uploaded a new patch: <a href="http://review.whamcloud.com/21494" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21494</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8324" title="HSM: prioritize HSM requests" class="issue-link" data-issue-key="LU-8324">LU-8324</a> hsm: prioritize HSM requests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0655c8faf7cb7dcdb3b19dd761aad6c06fcda159</p> <p>Hello,<br/> I would just like to say that we would be very keen on this kind of feature at Cambridge - I've just run into this issue today where a single file restore operation is at the back of the queue behind ~10TB of archive jobs.</p> <p>I'd be interested in testing this patch against one of our test filesystems, but I just wanted to add a comment that we would really appreciate having more ability to control the coordinator queue - whether it's in it's current state or some future tool as Robert suggests.</p> <p>Kind regards,<br/> Matt Raso-Barnett<br/> University of Cambridge</p> <p>Hello Matt,</p> <p>I think the patch is mature enough for you to test it if you are still interested in it.</p> <p>Any chance for this patch to make it into 2.10? It is a <em>very</em> useful feature for HSM users and we believe that the patch is mature.</p> <p>If not 2.10, it seems that 2.10.1 would be possible.</p> <p>I think that 2.10.1 is more likely option at this stage. It seems like there will be some discussions about this area at LUG next week.</p> <p>Quentin Bouget (quentin.bouget@cea.fr) uploaded a new patch: <a href="https://review.whamcloud.com/27394" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27394</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8324" title="HSM: prioritize HSM requests" class="issue-link" data-issue-key="LU-8324">LU-8324</a> hsm: prioritize HSM requests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f8d7f289866c2219d19a51693ae44cc6c3fdf867</p> <p>Quentin Bouget (quentin.bouget@cea.fr) uploaded a new patch: <a href="https://review.whamcloud.com/27800" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27800</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8324" title="HSM: prioritize HSM requests" class="issue-link" data-issue-key="LU-8324">LU-8324</a> hsm: ease the development of a different coordinator<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0ee6e5d71c6f549b48be59607d5f55f28e950f47</p> <p>This thread looks kind of dead, but we have a desire to see some prioritization mechanism as well.<br/> Some options:<br/> 1. FIFO (today)<br/> 2. Restore-first. All restore requests are prioritized over archive requests. (Except in-progress archives.)<br/> 3. Archive-first. All archives are prioritized.<br/> 4. Interleaved. Archive and Restore requests are alternated, as long as some of each are waiting.<br/> 5. Tunable. Adjustable ratio of archive:restore processing. Maybe this covers the above 2-4 as well.<br/> 6. Batched. Archives and Restores are grouped into separate batches, potentially resulting in fewer tape swaps.<br/> 7. Time-boxed. A variant of batched; batch ends after a fixed time period.<br/> Many other options I'm sure...</p> <p>Ultimately I'm in agreement with Robert Read's comment above that the prioritization should really be done outside of Lustre, but if the patch here implements #5 that might cover enough of the use cases to make most people happy...</p> <p>Quentin Bouget (quentin.bouget@cea.fr) uploaded a new patch: <a href="https://review.whamcloud.com/31723" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31723</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8324" title="HSM: prioritize HSM requests" class="issue-link" data-issue-key="LU-8324">LU-8324</a> hsm: prioritize one RESTORE once in a while<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b30b036607a2bc4928e13e06462701bf5ba62d3d</p> <p>The patch above is the shortest/simplest hack I could come up with to help bear with <a href="https://jira.whamcloud.com/browse/LU-8324" title="HSM: prioritize HSM requests" class="issue-link" data-issue-key="LU-8324">LU-8324</a> until a more definitive fix is developed (it is more of a band-aid than anything else).</p> <p>The idea is to use the times when the coordinator traverses its whole llog to "force-schedule" at least one RESTORE request. In practice, this means that you should see at least one RESTORE request scheduled every "loop_period" (the value in <em>/proc/&lt;fsname&gt;/mdt/&lt;mdt-name&gt;/hsm/loop_period</em>) seconds.</p> <p>Quentin Bouget (quentin.bouget@cea.fr) uploaded a new patch: <a href="https://review.whamcloud.com/33239" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33239</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8324" title="HSM: prioritize HSM requests" class="issue-link" data-issue-key="LU-8324">LU-8324</a> hsm: prioritize one RESTORE once in a while<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 93364e9f3b0c9694904d2c1e2a687af61a980c1f</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/31723/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31723/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8324" title="HSM: prioritize HSM requests" class="issue-link" data-issue-key="LU-8324">LU-8324</a> hsm: prioritize one RESTORE once in a while<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0dce1ddefc673a3f39b4964d6b669e2a11aaf903</p> <p>Quentin Bouget (quentin.bouget@cea.fr) uploaded a new patch: <a href="https://review.whamcloud.com/34749" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34749</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8324" title="HSM: prioritize HSM requests" class="issue-link" data-issue-key="LU-8324">LU-8324</a> hsm: prioritize one RESTORE once in a while<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 3fa2b1682755eeb988d10af53797a8d5e1a3679d</p> Duplicate LU-14363 Related LU-10968 LU-8382 Development Rank 1|hzyfp3: Rank (Obsolete) 9223372036854775807 Severity