https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8395 Lustre <p>On dense cored systems with hyperthreading active, you end up with a lot of execution threads. An example would be a system with 256 total threads.</p> <p>The libcfs partition code divides those up into 16 partitions of 16 threads each (based on a math algorithm...not logical grouping based on hyperthreading). </p> <p>Lnet-selftest then creates a group of scheduler threads in each partition equal to the number of execution threads minus 1 (so 15 per partition for our example). That means it creates 240 scheduler threads. This actually reduces performance rather than improves it.</p> <p>We need to limit the number of lnet-selftest scheduler threads to some reasonable number per partition.</p> <p>Note: another bug will be created to look at how we group the execution threads into partitions. It needs to make more sense than it does.</p> LU-8395

Limit lnet-selftest Threads

Bug Major Resolved Incomplete Amir Shehata Doug Oucharek Wed, 13 Jul 2016 22:26:01 +0000 Wed, 11 Dec 2019 19:52:57 +0000 Wed, 11 Dec 2019 19:52:57 +0000 0 5 <p>Doug Oucharek (doug.s.oucharek@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/21299" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21299</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8395" title="Limit lnet-selftest Threads" class="issue-link" data-issue-key="LU-8395"><del>LU-8395</del></a> lnet: Limit total number of lnet-selftest threads<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9a3bc67c22e9c52b64a1a2eb3fad220ace85d434</p> <p>Unfortunately this patch makes performance on KNL worse.</p> <p>Doug, Dmitry, could you please describe the parameters of your performance testing so that you can agree on what is being measured. </p> <p>Doug, was the 64-thread limit based on anything concrete, or just a guess? It might be useful to run some testing with different thread counts to see where the threshold is hit. It might also be possible to determine at compile time (based on CPP constant for KNL) or at runtime (not sure how) to limit the threads differently for Xeon Phi vs. other systems with many cores.</p> <p>Dmitry, can you run your testing similarly, to see where the threshold is for reducing thread counts. Do we need to have one LNet thread per core, or do we have some benefit to reserve a few cores for other tasks?</p> <p>Doug, as for the CPT partition calculations, this is discussed in <a href="https://jira.whamcloud.com/browse/LU-5050" title="cpu partitioning oddities" class="issue-link" data-issue-key="LU-5050"><del>LU-5050</del></a> and <a href="https://jira.whamcloud.com/browse/LU-7553" title="Lustre cpu_npartitions default value breaks memory allocation on clients" class="issue-link" data-issue-key="LU-7553"><del>LU-7553</del></a>, so this ticket should be focused on the LST thread count.</p> <p>There was one patch <a href="http://review.whamcloud.com/17824" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17824</a> "<a href="https://jira.whamcloud.com/browse/LU-5050" title="cpu partitioning oddities" class="issue-link" data-issue-key="LU-5050"><del>LU-5050</del></a> libcfs: default CPT matches NUMA topology" that tried to change this, but it had to be reverted for a relatively minor reason, and should probably be revived.</p> <p>This is results with different partitions count. The current settings looks much better. </p> <p>This configuration is not a development target today, and there is no real information in this ticket. Maybe revisit this in the future.</p> Related LU-7553 LU-5050 Development Rank 1|hzyhfr: Rank (Obsolete) 9223372036854775807 Severity