https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5134 Lustre <p>lctl set_param should have an option to set a parameter across multiple matched files in parallel. For instance, if you execute this lctl set_param command:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl set_param [parallel-option] ldlm.namespaces.*osc*.lru_size=clear </pre> </div></div> <p>it should write "clear" to the files matching the given parameter pattern in parallel.</p> <p>This enhancement is required to speed up clearing of Lustre caches. When there are many OSTs, executing</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl set_param ldlm.namespaces.*.lru_size=clear </pre> </div></div> <p>takes a long time, and there is no reason that the lru_size files can't be written to in parallel. Then the work can be done on each OST in parallel.</p> <p>For example, with 16 OSTs, it takes 5.4 seconds to clear caches across all namespaces. This could be sped up by parallelizing the write to lru_size across the namespaces.</p> <p>If this enhancement is added, then <a href="https://jira.whamcloud.com/browse/LU-3970" title="Add procfs interface for clearing lustre caches in parallel" class="issue-link" data-issue-key="LU-3970"><del>LU-3970</del></a> can also be resolved.</p> LU-5134

Add option to lctl set_param for setting parameters in parallel

Improvement Minor Resolved Fixed Ryan Haasken Ryan Haasken patch Mon, 2 Jun 2014 16:15:33 +0000 Wed, 25 Oct 2023 19:50:16 +0000 Wed, 25 Oct 2023 19:50:16 +0000 Lustre 2.16.0 0 6 <p>I was just discussing this issue on <a href="https://jira.whamcloud.com/browse/LU-5030" title="&quot;lctl {get,set}_param&quot; should also check in /sys/fs/{lnet,lustre}" class="issue-link" data-issue-key="LU-5030"><del>LU-5030</del></a>. Besides lru_size we have the same issue with reading all imports. See sanityn test 35 for a example. This problem is blocking us from moving the test suite completely to using lctl <span class="error">&#91;g/s&#93;</span>et_param. This would be a most useful piece of work. I recommend that you would work off of patch <a href="http://review.whamcloud.com/#/c/10300" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10300</a>. I still need to fix the patch up but it at a good start.</p> <p>Here is a potential implementation of a parallel lctl set_param: <a href="http://review.whamcloud.com/#/c/10555/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10555/</a></p> <p>I am looking for feedback on that patch. Specifically, I am wondering if it's really necessary to check for HAVE_LIBPTHREAD everywhere since it makes the code a little messy. I tried to minimize the number of places where I checked "#ifdef HAVE_LIBPTHREAD", but it's still a little ugly. Is portability to platforms without libpthread still a concern?</p> <p>Second of all, I am wondering if it is really necessary to have an option to enable the parallel set_param, or it should just be parallel by default. Does anybody have any input?</p> <p>James, I didn't see your first comment before posting my patch. I'll take a look.</p> <p>James, how does <a href="http://review.whamcloud.com/#/c/10300" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10300</a> relate to my change? Those changes are in get_param functions in liblustreapi.c, which, as far as I can tell, are not used by lctl. Is the idea to change lctl to start using the liblustreapi functions to do getting and setting of parameters?</p> <p>I've figured out the answer to the question in my previous comment. That does appear to be the case based on Andreas' comment on <a href="https://jira.whamcloud.com/browse/LU-5030" title="&quot;lctl {get,set}_param&quot; should also check in /sys/fs/{lnet,lustre}" class="issue-link" data-issue-key="LU-5030"><del>LU-5030</del></a>:</p> <blockquote><p> Ideally, this would eventually result in usable llapi_get_param() and llapi_set_param() functions which can be used by lctl, lfs, and other applications that hide the details of the interface and the location of the files in /proc or /sys or /debugfs or whatever </p></blockquote> <p>It looks like there is still a lot of work to be done there, and I'm not sure how to approach it yet.</p> <p>James, can you please explain the problem in sanityn.sh test_35? I don't see the problem there and how it relates to this enhancement.</p> <p>Yes much work is left to be done. I spent yesterday updating the patch for <a href="https://jira.whamcloud.com/browse/LU-5030" title="&quot;lctl {get,set}_param&quot; should also check in /sys/fs/{lnet,lustre}" class="issue-link" data-issue-key="LU-5030"><del>LU-5030</del></a>. I think I know what you want (get_param and set_param) so I'm going to work that in.</p> <p>Test sanityn.sh 35 is another loop through the proc file system to gather import data much like the sanity 900 test. Thinking about it a really nice feature would to get_param with filters. So if you only get results back for a specific value.</p> <p>I've attached a quick test script which demonstrates the performance difference between parallel and serial set_param when canceling unused locks across many namespaces by writing to lru_size. It also includes other functional tests of <tt>lctl set_param -p</tt> that I used as I was developing. I'm hoping there already exist enough tests in sanity.sh and others that will verify the {set,get,list}_param functionality.</p> <p>I've also attached sample output from the test script showing the results of running it on a VM. In that sample output, a serial set_param took 4.175 seconds, while a parallel set_param took 0.401 seconds.</p> <p>Per John's comment on the gerrit change, I wrote a bash function to do accomplish a set_param in parallel:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">function bash_cancel_locks() { declare -a pids local pid_idx=0 local rc=0 local wait_rc=0 local namespaces=$(lctl list_param ldlm.namespaces.*.lru_size) <span class="code-keyword">for</span> namespace in $namespaces; <span class="code-keyword">do</span> lctl set_param $namespace=clear &amp; pids[$pid_idx]=$! ((pid_idx++)) done <span class="code-keyword">for</span> i in $(seq $pid_idx); <span class="code-keyword">do</span> wait ${pids[$i]} wait_rc=$? <span class="code-keyword">if</span> [[ $wait_rc -ne 0 ]]; then rc=$wait_rc fi done <span class="code-keyword">return</span> $rc } </pre> </div></div> <p>In my testing on a VM with many mounts, this function achieves about 50-100% of the performance of the <tt>lctl set_param -p</tt>, but performance varies a lot. This is pretty good, but we would still like a single interface to set general Lustre parameters (including lru_size) in parallel. <tt>lctl</tt> seems like the most appropriate place to do this, although the implementation in C is more complicated than the above implementation in bash.</p> <p>I think the closeness in performance is due to the fact that the bash implementation has no artifical limit to the number of subprocesses it spawns while the <tt>lctl set_param -p</tt> implementation limits itself to 8 threads per core. I'm going to bump that up to 32 threads per core and see how it performs. I'd also like to do a performance comparison on real hardware with a file system with many more OSTs.</p> <p>The performance of the <tt>lctl set_param -p</tt> seems to improve when I increase <tt>LCTL_THREADS_PER_CPU</tt> from 8 to 32. The times are not that consistent for any of the methods between runs of the tests though.</p> <p>Perhaps you should the number of threads as a option for the user, -p 32 or something along those lines.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/10555/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/10555/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5134" title="Add option to lctl set_param for setting parameters in parallel" class="issue-link" data-issue-key="LU-5134"><del>LU-5134</del></a> utils: Add parallel option to lctl set_param<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 345a2497d08f6b9afd74ed0188a70489f7a43e5d</p> <p>Landed for 2.16</p> Related LU-3970 Development Epic/Theme Performance Rank 1|hzwnkf: Rank (Obsolete) 14163