https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-947 Lustre <p>It should be possible to dynamically tune the number of ptlrpc threads at runtime for testing purposes. Currently it is possible to increase the maximum thread count, but it is not possible to stop threads that are already running.</p> <p>This was being worked on in bug 22417:<br/> <a href="https://bugzilla.lustre.org/attachment.cgi?id=32351" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/attachment.cgi?id=32351</a></p> <p>and later enhanced in bug 22516:<br/> <a href="https://bugzilla.lustre.org/attachment.cgi?id=32510" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/attachment.cgi?id=32510</a></p> <p>The later patch includes code to dynamically tune the thread counts based on the threads in use over the past several minutes.</p> LU-947

ptlrpc dynamic service thread count handling

Improvement Minor Resolved Fixed Andreas Dilger Andreas Dilger Tue, 20 Dec 2011 12:50:33 +0000 Fri, 24 Nov 2023 07:35:45 +0000 Tue, 17 Mar 2020 03:34:37 +0000 Lustre 2.13.0 0 4 <p>We also want to reduce the default maximum number of service threads, as this is typically too high for most systems.</p> <p>Chris, can you please provide details about which threads should be reduced, and what the preferred thread count is.</p> <p>For systems with large number of clients (ie. ~1000) we find the max number of OSS &amp; MDS threads too high. This causes high system load &amp; lost network connections. Too many service threads particularly impacts Ethernet &amp; OPA since they have higer CPU utilization/system load.</p> <p>AFAIK the current default is max 512 threads. We usually set fixed values <em>mds_num_threads=256</em> and <em>oss_num_threads=256</em> via module options (ie. half default). To set kernel module options we have to stop lustre &amp; reload lustre modules. For virtual machines with limited CPU cores we often use smaller values.</p> <p>We also tested tuneable <em>ost.OSS.ost_io.threads_max</em>. I believe we also have to reload lustre modules to set this parameters.</p> <p> </p> <p>Could you comment on the core count for 256 threads vs. smaller systems? I'm wondering if that could be made automatic? </p> <p>The <tt>threads_max</tt> parameter can currently be increased to allow more threads to be started, if needed, but decreasing it does not stop the threads. I'm just lookin my at the code to determine if this is practical to change (it was previously not with the "obdfilter" code, but it seems the "ofd" code does not suffer the same limitations. </p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34400" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34400</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-947" title="ptlrpc dynamic service thread count handling" class="issue-link" data-issue-key="LU-947"><del>LU-947</del></a> ptlrpc: stop threads if more than threads_max<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 039e8c72f5ace8b7d9b6b1d3ec543f1dbf95335b</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=chunteraa" class="user-hover" rel="chunteraa">chunteraa</a> it looks like <tt>oss&#95;max&#95;threads=512</tt> and <tt>MDS&#95;NTHRS&#95;MAX=1024</tt> in the code. The <tt>oss&#95;max&#95;threads</tt> upper limit is tunable since commit <tt>v2&#95;8&#95;50&#95;0&#45;44&#45;gaa84d18864</tt>, but <tt>MDS&#95;NTHRS&#95;MAX=1024</tt> is fixed. It seems that rather than setting <tt>oss&#95;<b>num</b>&#95;threads</tt> and <tt>mds&#95;<b>num</b>&#95;threads</tt> (which sets the minimum, maximum, and number of threads started) it might be better to set <tt>oss&#95;<b>max</b>&#95;threads=256</tt> which sets the upper limit of threads (and add a tunable <tt>mds&#95;<b>max</b>&#95;threads</tt> also), but this allows a system to start fewer threads if more are not needed (e.g. only few clients).</p> <p>It looks like <tt>LDLM&#95;NTHRS&#95;MAX</tt> is already somewhat dependent on the number of cores (<tt>num&#95;online&#95;cpus() == 1 ? 64 : 128</tt>), but this is probably a holdover from days gone by, or maybe single-core VMs?  It does show that it is possible to auto-tune based on the core count, however.</p> <blockquote><p>Could you comment on the core count for 256 threads vs. smaller systems? I'm wondering if that could be made automatic?</p></blockquote> <p>example VM environments:</p> <ul class="alternate" type="square"> <li>6 cpu core 32G memory: oss_num_threads=192 or mds_num_threads=128; we also used ost.OSS.ost_io.threads_max=150 when there are many disks installed.</li> </ul> <p> </p> <ul class="alternate" type="square"> <li>16 cpu core 90G memory: oss_num_threads=256 or mds_num_threads=192</li> </ul> <p> </p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34418" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34418</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-947" title="ptlrpc dynamic service thread count handling" class="issue-link" data-issue-key="LU-947"><del>LU-947</del></a> ptlrpc: reduce default MDS/OSS thread count<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 52ebfd503a17fac1a02cb63a8807d3d289a1c853</p> <p>A general rule of mdt_threads_max=16*num_cpus has been good starting point, with caveats if processor hyperthreading is enabled.</p> <p>For ost &amp; ost_io threads , the <a href="http://doc.lustre.org/lustre_manual.xhtml#dbdoclet.50438272_55226" class="external-link" target="_blank" rel="nofollow noopener">LOM states</a> "You may want to start with a number of OST threads equal to the number of actual disk spindles on the node. " That makes sense for spinning media but perhaps not relevant for flash storage. With flash storage, I suspect ost_io_threads_max=N*num_cpus, with N in range 10-20 is good starting point. However not sure if storage blk_mq support means more or less ost_io threads.</p> <p>Of course ability to reduce active thread count would help tuning.</p> <p>I agree reducing number of threads might be help if large number of clients send messages simultaneously, but need to keep maximum performance with small number of client with large network bandwdith. e.g. 8 clients with EDR needs to get 80GB/sec too.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>For systems with large number of clients (ie. ~1000) we find the max number of OSS &amp; MDS threads too high. This causes high system load &amp; lost network connections. Too many service threads particularly impacts Ethernet &amp; OPA since they have higer CPU utilization/system load. </pre> </div></div> <p>did you disable 16MB rpc here? most of large instllastion, memory presure comes from large rpc and we have been disabling rpc size to 8M or 4MB.</p> <blockquote><p>did you disable 16MB rpc here? most of large instllastion, memory presure comes from large rpc and we have been disabling rpc size to 8M or 4MB.</p></blockquote> <p>Reducing <tt>brw_size</tt> doesn't help much with many clients.</p> <blockquote><p>but need to keep maximum performance with small number of client with large network bandwdith. e.g. 8 clients with EDR needs to get 80GB/sec too.</p></blockquote> <p>I suspect the main factors:<br/> 1. number of clients (ie. number of simultaneous messages to OST target)<br/> 2. number of OSTs per server<br/> 3. performance difference between flash and spinning disk storage</p> <p>Ideally we would have ability to set <tt>max_threads</tt> based on different environments and change value if workload changes.</p> <p> </p> <blockquote> <p>Ideally we would have ability to set <tt>max_threads</tt> based on different environments and change value if workload changes.</p></blockquote> <p>That ability <em>already</em> exists, it is a module parameter, but can also be increased at runtime if needed. What is new in the first patch is the ability to reduce it at runtime. That said, changing it based on workload seems impractical since there may be many different jobs running at the same time. What I'd like is to have a reasonable out-of-the box value, possibly a function of some well-known parameters (RAM, core count, possibly OST count, <em>maybe</em> client count though I'm not sure that is right).</p> <blockquote> <p> What is new in the first patch is the ability to reduce it at runtime. </p></blockquote> <p>This would help for managing larger systems.</p> <blockquote> <p>reasonable out-of-the box value, possibly a function of some well-known parameters (RAM, core count, possibly OST count, maybe client count though I'm not sure that is right).</p></blockquote> <p>Perhaps we could provide guidelines in the documentation ?</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34400/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34400/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-947" title="ptlrpc dynamic service thread count handling" class="issue-link" data-issue-key="LU-947"><del>LU-947</del></a> ptlrpc: allow stopping threads above threads_max<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 183cb1e3cdd2de93aca5dff79b3d56bbadc00178</p> Related LU-17312 Bugzilla ID 22516.0 Development Rank 1|hzw41j: Rank (Obsolete) 10749