https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8626 Lustre <p>Several presentations at RUG'16 mentioned that Lustre has poor performance when there are very large numbers of HSM actions outstanding on the coordinator.</p> <p>Firstly, having a /proc file that exposes the number of entries currently in the HSM action list would allow RBH and monitoring scripts to easily monitor the number of enties.</p> LU-8626

limit number of items in HSM action queue

Improvement Major Reopened Unresolved Quentin Bouget Andreas Dilger hsm Mon, 19 Sep 2016 15:37:47 +0000 Wed, 4 Apr 2018 07:24:30 +0000 2 16 <p>I don't know enough about the details to write a good description of what needs to be done here. There is a <tt>max_requests</tt> tunable, but that appears to control the number of requests outstanding from the coordinator to a single copytool (<tt>cdt_max_requests</tt>).</p> <p>Stephane, could you please add some info here about what variable(s) should be exposed via /proc, and what should be used to limit the size of the queue? What error should be returned by the coordinator if the action queue size limit is exceeded, -EFBIG?</p> <p>Andreas, you sure about the fact that <span class="error">&#91;cdt_&#93;</span>max_requests only concerns requests from the coordinator to a single copytool? I thought (and seems confirmed with my related code reading) it is used to account and limit the whole set of "active" requests been handled by a CDT and regardless to the concerned Agent(s). </p> <p>About the performance to access the hsm_actions LLOG content when there is a huge back-log, I think that there have been already some work done, and not only in <a href="https://jira.whamcloud.com/browse/LU-7988" title="HSM: high lock contention for cdt_llog_lock" class="issue-link" data-issue-key="LU-7988"><del>LU-7988</del></a>. I will try to gather these infos and add to this ticket.</p> <p>And also, as a first step, could it be acceptable to implement a simple/basic limit of a maximum number of (active and not) requests to be handled by a single CDT and simply return an error (-EFBIG ?) when reached ?</p> <p>Hopefully Stéphane or Patrick can answer your question here. I was just trying to record the issue raised during RUG. It appears this is related to <a href="https://jira.whamcloud.com/browse/LU-7988" title="HSM: high lock contention for cdt_llog_lock" class="issue-link" data-issue-key="LU-7988"><del>LU-7988</del></a> also caused by having a large number of items in the action list.</p> <p>max_requests is the maximum number of active requests at the same time per coordinator, so it has nothing to do with the HSM action queue. By the way, the Lustre documentation for max_requests is correct.</p> <p>actions is used to dump the action queue<br/> (Note: it is also accessible through "lctl get_param mdt.lustre-MDT0000.hsm.actions")</p> <p>Typical entries in actions look like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lrh=[type=10680000 len=136 idx=342/13652] fid=[0x200034e87:0x8700:0x0] dfid=[0x200034e87:0x8700:0x0] compound/cookie=0x57d8a6d9/0x57d89f3e action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=342/13655] fid=[0x20002a3ec:0xf909:0x0] dfid=[0x20002a3ec:0xf909:0x0] compound/cookie=0x57d8a6dc/0x57d89f41 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] </pre> </div></div> <p>To retrieve statistics from this file to put them into Graphite/Grafana, I use the following script that extracts the number of lines by grouping them by "action" and also "status":</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">prefix=<span class="code-quote">"$CARBON_PREFIX.$MDT.hsm.actions"</span> lctl get_param mdt.$MDT.hsm.actions | awk -v prefix=$prefix <span class="code-quote">'BEGIN { now = systime() } / status=/ { action=gensub(/action=(.+)/, <span class="code-quote">"\\1"</span>, <span class="code-quote">"g"</span>, $7); status=gensub(/status=(.+)/, <span class="code-quote">"\\1"</span>, <span class="code-quote">"g"</span>, $(NF-1)); arr[action<span class="code-quote">"."</span>status] += 1 } END { <span class="code-keyword">for</span> (s in arr) { printf <span class="code-quote">"%s.%s %s %d\n"</span>, prefix, s, arr[s], now} }'</span> </pre> </div></div> <p>So I get lines that are valid for Graphite that look like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">srcc.sherlock.lustre.mdt.regal-MDT0000.hsm.actions.ARCHIVE.SUCCEED 3 1474463707 </pre> </div></div> <p>I am sure you do something similar for IML as you have a graph for HSM, but it is also broken when too many hsm actions are present.</p> <p>The problem is that when the actions file reaches 100K's entries, it takes so long it is not parsable in a timely manner anymore...</p> <p>I think having counters per "action" and "status" could be very useful, perhaps something like:</p> <p>actions_stats:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">status=archive status=STARTED count=23 status=archive status=SUCCEED count=1234 status=restore status=STARTED count=0 status=restore status=SUCCEED count=1 </pre> </div></div> <p>About the way to limit the size of the queue, a tunable like max_actions and -EFBIG sounds good, but still there might be a problem if restore actions are triggered when the action queue is full. Maybe the best would be a max_actions per action (archive, restore, remove)?...</p> <p>I think we should stick with a single value per file, since this is required when moving stats into /sys/fs/lustre, so something like <tt>action_archive_started_count</tt>, <tt>action_archive_succeed_count</tt>, <tt>action_restore_started_count</tt>, <tt>action_restore_succeed_count</tt>.</p> <p>Quentin Bouget (quentin.bouget@cea.fr) uploaded a new patch: <a href="https://review.whamcloud.com/28677" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28677</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8626" title="limit number of items in HSM action queue" class="issue-link" data-issue-key="LU-8626">LU-8626</a> hsm: count the number of started requests of each type<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4e84ba8aa669554b2d1b77459ebe79770aa4ad37</p> <p>Quentin Bouget (quentin.bouget@cea.fr) uploaded a new patch: <a href="https://review.whamcloud.com/30336" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30336</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8626" title="limit number of items in HSM action queue" class="issue-link" data-issue-key="LU-8626">LU-8626</a> hsm: expose the number of active hsm requests per type<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 294a0e4ff77fb09ac643b5e15af224027ded4aee</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/28677/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28677/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8626" title="limit number of items in HSM action queue" class="issue-link" data-issue-key="LU-8626">LU-8626</a> hsm: count the number of started requests of each type<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 973759d1ff3bbcb217754bd9942fdf670dec2d96</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30336/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30336/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8626" title="limit number of items in HSM action queue" class="issue-link" data-issue-key="LU-8626">LU-8626</a> hsm: expose the number of active hsm requests per type<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 42e40555f250b83730d233dc5e22fd1f9396ccfe</p> <p>Landed for 2.11</p> <p>Hi Peter,</p> <p>I am not sure this issue should be marked as resolved yet.<br/> The patches that landed only provide information about how many requests the coordinator is currently handling, there are no built-in limitations yet.</p> <p>ok</p> <p>About already landed <a href="https://review.whamcloud.com/30336/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30336/</a>:</p> <p>Given the implemented counters refers to the contents of "active_requests" list, they should rather be named "active_archive_count", "active_restore_count", ... instead of "archive_count", etc... to be more explicit and avoid any confusion with the contents of hsm/actions that contains all requested actions.</p> <p>This change should be done before releasing 2.11 to avoid changing names in /proc later after the feature is released.</p> <p>2.11 has already been released. </p> Related LU-7988 Development Rank 1|hzyown: Rank (Obsolete) 9223372036854775807