https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11415 Lustre <p>Recently I'm benchmarking a newly setup lustre servers with 40Gbps ethernet network connection. Jumbo frame is enabled and MTU is set to 9000 for both NICs on the client and server side. The connection between client and server is really simple, they are under the same TOR switch, no routers in between.</p> <p>Firstly I used iperf3 to verify the throughput between client and server, and the throughput is stable at 30~32gib/s from either direction. However, when I launched lnet selftest, I usually see less throughput than perf3, which is about ~2500MiB/s.</p> <p>After speaking with Amir and Doug, I monitored ksocklnd threads on both client and server side, the problem we're seeing is that when lnet selftest is performing reading test, there is only one ksocklnd thread consuming 100% CPU time, while the other threads don't take any workload; write test is similar but only one server ksocklnd thread is busy doing work. The workload doesn't seem to spread out to all threads in the pool.</p> <p> </p> <p>It would be possible that the only thread is enough to handle all the traffic so there is no need to launch the workload to the other threads, but it's also possible that there are some scheduling problems in the implementation of ksocklnd. Doug mentioned that o2iblnd could spread the workload well.</p> <p> </p> <p> </p> LU-11415

ksocklnd performance improvement on 40Gbps ethernet

Improvement Minor Resolved Fixed Amir Shehata Jinshan Xiong Fri, 21 Sep 2018 20:27:22 +0000 Mon, 8 Apr 2019 14:10:59 +0000 Fri, 4 Jan 2019 05:28:13 +0000 Lustre 2.13.0 Lustre 2.12.1 0 8 <p>Typically this is what I have seen from the server side when doing write test:</p> <p>{{<tt>top - 21:45:16 up 21:57, 1 user, load average: 11.02, 7.19, 5.00</tt>}}<br/> {{ <tt>Tasks: 1175 total, 3 running, 1172 sleeping, 0 stopped, 0 zombie</tt>}}<br/> {{ <tt>%Cpu(s): 0.0 us, 11.3 sy, 0.0 ni, 80.5 id, 6.9 wa, 0.0 hi, 1.3 si, 0.0 st</tt>}}<br/> {{ <tt>KiB Mem : 13174288+total, 56613536 free, 68320632 used, 6808716 buff/cache</tt>}}<br/> {{ <tt>KiB Swap: 4194300 total, 4194300 free, 0 used. 62550556 avail Mem</tt><tt>PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND</tt>}}<br/> {{ <tt>21319 root 20 0 0 0 0 R 100.0 0.0 15:11.80 socknal_sd00_02</tt>}}<br/> {{ 2 root 20 0 0 0 0 S 9.2 0.0 4:09.04 kthreadd}}<br/> {{ 1734 root 0 -20 0 0 0 R 5.6 0.0 2:25.93 spl_dynamic_tas}}</p> <p>I looked at the socklnd scheduling code and it's very similar to the o2iblnd code. There is a scheduler created per CPT. A thread is created for each cpu in the CPT (if the number of threads is not explicitly configured). When creating a connection, the CPT is derived from the peer nid using: lnet_cpt_of_nid() hashing function. The CPT is used to grab the appropriate scheduler and assign it to the connection. All operations on the connection uses the assigned scheduler. Which means if we're running a 1-1 testing, then the same scheduler will always be used. If there are multiple threads in the scheduler then we should round robin over them.</p> <p>In the MLX o2iblnd case most of the work is offloaded to the HW. In case of OPA I believe the HFI driver has its own set of threads which do the work. But in socklnd all the work is done in the socklnd scheduler thread, which causes the thread to consume a lot of the CPU resources if it's the only thread in the scheduler.</p> <p>In the test above how many CPUs are in each CPT?</p> <p>Everything is default there is no settings in `lnet.conf` other than specifying NIC for Lustre.</p> <p>This node has 2 NUMA node with 24 cores, what would be the recommended configuration for CPT?</p> <p>How many socklnd threads were started?</p> <p>These are all the threads:</p> <p><tt># ps ax | grep sock</tt><br/> {{ 43583 pts/0 S+ 0:00 grep --color=auto sock}}<br/> {{ 44194 ? S 0:00 <span class="error">&#91;socknal_cd00&#93;</span>}}<br/> {{ 44195 ? S 0:00 <span class="error">&#91;socknal_cd01&#93;</span>}}<br/> {{ 44196 ? S 0:00 <span class="error">&#91;socknal_cd02&#93;</span>}}<br/> {{ 44197 ? S 0:00 <span class="error">&#91;socknal_cd03&#93;</span>}}<br/> {{ 44198 ? S 0:03 <span class="error">&#91;socknal_reaper&#93;</span>}}<br/> {{ 44199 ? S 42:59 <span class="error">&#91;socknal_sd00_00&#93;</span>}}<br/> {{ 44200 ? S 0:00 <span class="error">&#91;socknal_sd00_01&#93;</span>}}<br/> {{ 44201 ? S 584:14 <span class="error">&#91;socknal_sd00_02&#93;</span>}}<br/> {{ 44202 ? S 0:00 <span class="error">&#91;socknal_sd00_03&#93;</span>}}<br/> {{ 44203 ? S 0:00 <span class="error">&#91;socknal_sd00_04&#93;</span>}}<br/> {{ 44204 ? S 0:00 <span class="error">&#91;socknal_sd00_05&#93;</span>}}<br/> {{ 44205 ? S 0:27 <span class="error">&#91;socknal_sd01_00&#93;</span>}}<br/> {{ 44206 ? S 0:00 <span class="error">&#91;socknal_sd01_01&#93;</span>}}<br/> {{ 44207 ? S 0:00 <span class="error">&#91;socknal_sd01_02&#93;</span>}}<br/> {{ 44208 ? S 0:26 <span class="error">&#91;socknal_sd01_03&#93;</span>}}<br/> {{ 44209 ? S 0:00 <span class="error">&#91;socknal_sd01_04&#93;</span>}}<br/> {{ 44210 ? S 0:00 <span class="error">&#91;socknal_sd01_05&#93;</span>}}</p> <p>I think I found the issue in the code. In slocklnd.c:ksocknal_choose_scheduler_locked():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 683 select_sched: 684 »·······sched = &amp;info-&gt;ksi_scheds[0]; 685 »·······/* 686 »······· * NB: it's safe so far, but info-&gt;ksi_nthreads could be changed 687 »······· * at runtime when we have dynamic LNet configuration, then we 688 »······· * need to take care of <span class="code-keyword">this</span>. 689 »······· */ 690 »·······CDEBUG(D_NET, <span class="code-quote">"info-&gt;ksi_nthreads = %d \n"</span>, info-&gt;ksi_nthreads); 691 »·······<span class="code-keyword">for</span> (i = 1; i &lt; info-&gt;ksi_nthreads; i++) { 692 »·······»·······CDEBUG(D_NET, <span class="code-quote">"sched-&gt;kss_nconns = %d info-&gt;ksi_scheds[%d].kss_nconns = %d\n"</span>, 693 »·······»······· sched-&gt;kss_nconns, i, info-&gt;ksi_scheds[i].kss_nconns); 694 »·······»·······<span class="code-keyword">if</span> (sched-&gt;kss_nconns &gt; info-&gt;ksi_scheds[i].kss_nconns) 695 »·······»·······»·······sched = &amp;info-&gt;ksi_scheds[i]; 696 »·······} 697 698 »·······<span class="code-keyword">return</span> sched; </pre> </div></div> <p>When running an lnet_selftest script the connection is created once and is used for the duration of the run. This will result in using the same scheduler continuously for the same connection. This algorithm works if you have multiple connections, each connection will use a different scheduler thread. But if most of the traffic is between the same two peers we will not change the scheduler thread and we will run into this bottle neck.</p> <p>Seems like we ought to be balancing the traffic not in connection creation time only, but when receiving a new message. I'm gonna create a patch within the next day or so. Would you be able to test it and see if it performs better?</p> <p>I will be happy to try that out. Thanks.</p> <p>Summarized the issue and the proposed solution here:<br/> <a href="https://wiki.whamcloud.com/display/LNet/Socklnd+Scheduler+Improvements" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.whamcloud.com/display/LNet/Socklnd+Scheduler+Improvements</a></p> <p>Will try and get a patch in soon.</p> <p>Amir Shehata (ashehata@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33740" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33740</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11415" title="ksocklnd performance improvement on 40Gbps ethernet" class="issue-link" data-issue-key="LU-11415"><del>LU-11415</del></a> socklnd: improve scheduling algorithm<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 69fbcfc89f165ba4238286afcbcd3a2059615b4f</p> <p>Hey Jinshan, can you try this patch? run a 1-1 selftest and monitor the socknal_sd_* threads' CPU usage.<br/> thanks</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33740/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33740/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11415" title="ksocklnd performance improvement on 40Gbps ethernet" class="issue-link" data-issue-key="LU-11415"><del>LU-11415</del></a> socklnd: improve scheduling algorithm<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 89df5e712ffd40064f1d4ce2f00f9156f68a2262</p> <p>Landed for 2.13</p> <p> </p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34299" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34299</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11415" title="ksocklnd performance improvement on 40Gbps ethernet" class="issue-link" data-issue-key="LU-11415"><del>LU-11415</del></a> socklnd: improve scheduling algorithm<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 35864dd4370c4ece26198033beced450ed6443d0</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34299/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34299/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11415" title="ksocklnd performance improvement on 40Gbps ethernet" class="issue-link" data-issue-key="LU-11415"><del>LU-11415</del></a> socklnd: improve scheduling algorithm<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: ec964395b249087c28e82b1afa1db4a7c9322196</p> Duplicate Development Rank 1|i002vj: Rank (Obsolete) 9223372036854775807