https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11454 Lustre <p>It is not always advantageous to bind the various server service threads to specific CPTs.  If all or most traffic is coming in from one node (likely a router), then activity on the server ends up limited to the CPT associated with that router.  This is advantageous if NUMA latencies are relatively high, but can be disadvantageous if CPTs are small and latencies are low.</p> <p>Specifically, the work is limited to the CPUs in the CPT, which means that some workloads can end up needing more CPU, but are unable to get it.</p> <p>In essence, the default behavior of strict binding is fine but is not always preferable.  So, add an option to disable this strict binding.</p> LU-11454

Allow switching off CPT binding for PTLRPC threads

Bug Minor Resolved Fixed Patrick Farrell Patrick Farrell Mon, 1 Oct 2018 19:25:16 +0000 Sun, 13 Jun 2021 06:55:39 +0000 Sat, 13 Oct 2018 05:00:37 +0000 Lustre 2.12.0 0 12 <p>Patrick Farrell (paf@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/33262" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33262</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11454" title="Allow switching off CPT binding for PTLRPC threads" class="issue-link" data-issue-key="LU-11454"><del>LU-11454</del></a> ptlrpc: Make binding switchable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: fea65e1c476f634196cf87b3597f2accda7f69a6</p> <p>By the way, I'd like to sound people out - Olaf and Andreas in particular - on what we think about changing the default to "CPT aware but no binding", at least for the MDT. I think it is going to be better for almost all real world server configs. (Server, not client)</p> <p>The "bind to just this CPT" behavior seems like something designed for high NUMA distances/latencies, like seen on the SGI/HPE big iron - but my understanding is they're not used as servers, just clients. Servers are usually much smaller systems.</p> <p>We've seen significant performance improvements in routed configs (where there is effectively a router-CPT binding) by disabling the binding of worker threads, and no performance loss in other configs (even a slight gain). We still want the CPT awareness so there are multiple queues for sleeping &amp; getting work for the ptlrpc threads, we just don't want to limit the CPUs they can use.</p> <p>The patch today does <b>not</b> change the default behavior - but I think we should consider it. Currently we've only got solid numbers for the MDT - We don't see any major performance problems on our OSTs related to this, so I'm not (yet) suggesting changing the default there.</p> <p>If there's interest in changing the default, I can work to get those numbers.</p> <p>Patrick, how about this approach:</p> <p>If the configuration is binding an NI to one or more CPTs, we continue to keep the worker threads bound to those CPTs.  This should continue to support the SGI big iron.  If there is no binding of the NIs, then we are free to use all cores and can have a single pool of worker threads which don't have any CPT bindings.</p> <p>Locks and other "limited" resources continue to be CPT based to alleviate contention.  If we properly round-robin over the worker threads, we should get a good distribution of work over the CPT resources.</p> <p>Thoughts?</p> <p>There are definitely reason to have CPT bindings on clients as well - avoiding Lustre contention/jitter with application threads comes to mind. </p> <p>I've added Ihara to the ticket, as he is better positioned to report if this change improves performance (due to more efficient CPU usage), or hurts it (due to cross-core contention).</p> <p>I saw huge IOPS drops if oss_cpu_bind and oss_create_cpu_bind are disabled conjunctions with oss_max_threads, oss_num_threads, oss_num_create_threads. Something like this.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>options libcfs cpu_npartitions=16 cpu_pattern="" options ost oss_max_threads=128 oss_num_threads=128 oss_num_create_threads=128 oss_cpu_bind=0 oss_create_cpu_bind=0 </pre> </div></div> <p>Here is test results on my test box.</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <th class='confluenceTh'>params</th> <th class='confluenceTh'>IOPS (4K ranodm read)</th> </tr> <tr> <td class='confluenceTd'>oss_max_threads=128,<br/> oss_num_threads=128,<br/> oss_num_create_threads=128</td> <td class='confluenceTd'>833K</td> </tr> <tr> <td class='confluenceTd'>oss_max_threads=128,<br/> oss_num_threads=128,<br/> oss_num_create_threads=128,<br/> oss_cpu_bind=0,<br/> oss_create_cpu_bind=0</td> <td class='confluenceTd'>621K</td> </tr> </tbody></table> </div> <p>The parameters are still tunable and disabled by default, so, it's no impacts at all. Howerver, any particular reasons this performance drop? or did we expect? btw, OSS has single numa domain.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33262/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33262/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11454" title="Allow switching off CPT binding for PTLRPC threads" class="issue-link" data-issue-key="LU-11454"><del>LU-11454</del></a> ptlrpc: Make CPU binding switchable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3eb7a1dfc3e7401ebcc45ccb116ed673607fd27f</p> <p>Landed for 2.12</p> Related LU-14676 Development Rank 1|i003d3: Rank (Obsolete) 9223372036854775807 Severity