https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12815 Lustre <p>For high-bandwidth Ethernet interfaces (e.g. 100GigE), it would be useful to create multiple TCP connections per interface for bulk transfers in order to maximize performance (i.e. <tt>conns_per_peer=4</tt> for socklnd in addition to o2iblnd). We already have three separate TCP connections per LND - read, write, and small message.</p> <p>For large clusters this may be problematic because of the number of TCP connections to a server, but for smaller configurations this could be very useful.</p> LU-12815

Create multiple TCP sockets per SockLND

Technical task LU-14064 Minor Resolved Fixed Amir Shehata Andreas Dilger performance Fri, 27 Sep 2019 13:52:46 +0000 Tue, 20 Sep 2022 12:26:39 +0000 Wed, 18 Aug 2021 12:44:10 +0000 Lustre 2.15.0 Lustre 2.12.10 2 17 <p>yes the conns_per_peer would be a good parameter to use.</p> <p>I looked at the ksocklnd, and currently there can exist only one unique route between two peers. This in effect translates to one tcp connection between the peers. I don't see a reason though why we can't create multiple tcp connections per peer, and when we select which connection to send from, we can iterate over these connections.</p> <p>However, come to think about it, there is a way to do it, albeit a bit more configuration. What they can do is create multiple virtual interfaces which use the same physical interface, then they can use Multi-Rail to group all these connections. The result will be that socklnd will create multiple connections, one to each of the virtual interfaces.</p> <p>Ex:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ifconfig eth0:0 &lt;ip&gt; ifconfig eth0:1 &lt;ip&gt; ifconfig eth0:2 &lt;ip&gt; ifconfig eth0:3 &lt;ip&gt; lnetctl net add --net tcp --<span class="code-keyword">if</span> eth0:0,eth0:1,eth0:2,eth0:3</pre> </div></div> <p>The rest will be taken care of by the Multi-Rail algorithm.</p> <p>Would that be a sufficient solution?</p> <p>Yes, that workaround is exact what I did and confirmed bump up performance <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> but setting up four logical interfaces on all clients were very annoying and not good idea to create logical interfaces only for lustre. cons_per_peer is able to simplify configuration and improves performance.</p> <p>Serguei is currently looking at this.</p> <p>Some ethernet drivers allow alternate hashing methods to better utilize adapter receive queues for small number of incoming TCP streams</p> <p><a href="https://docs.mellanox.com/display/MLNXOFEDv473290/RSS+Support" class="external-link" target="_blank" rel="nofollow noopener">https://docs.mellanox.com/display/MLNXOFEDv473290/RSS+Support</a></p> <p>Any update on this issue ?</p> <p>It's currently under development</p> <p>Serguei Smirnov (ssmirnov@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41056" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41056</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12815" title="Create multiple TCP sockets per SockLND" class="issue-link" data-issue-key="LU-12815"><del>LU-12815</del></a> socklnd: add conns_per_peer parameter<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d69314d930b84927bd96351c185b7cef42073d3b</p> <p>James Simmons (jsimmons@infradead.org) uploaded a new patch: <a href="https://review.whamcloud.com/41181" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41181</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12815" title="Create multiple TCP sockets per SockLND" class="issue-link" data-issue-key="LU-12815"><del>LU-12815</del></a> socklnd: add conns_per_peer parameter<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 88dd982736c6e681d395c8d5509f030aec1a3289</p> <p>Amir or Serguei, can you please send an email to lustre-discuss (CC lustre-devel) asking if anyone there (or their users) is using the <tt>use_tcp_bonding</tt> option in production?</p> <p>Does it make sense (in a later patch) to dynamically tune the <tt>conns_per_peer</tt> value depending on the network performance? It is always better to avoid the need for tuning if possible.</p> <p>Is it possible to detect from socklnd what the underlying Ethernet device is (e.g. 100GigE) and set conns_per_peer automatically (either at startup or at runtime) unless it is otherwise specified?</p> <p>Running <tt>ethtool</tt> on the Ethernet device reports the available and current interface speed, so it seems at least possible that we could get this same information in <tt>ksocklnd.c</tt> to set the default value of <tt>conns_per_peer</tt> based on the link speed:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># ethtool enp0s3 Settings for enp0s3: Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Speed: 1000Mb/s </pre> </div></div> <p>Running <tt>strace ethtool</tt> shows it is calling <tt>ioctl(SIOCETHTOOL)</tt>, which also is accessible internally via <tt>dev_ioctl()</tt>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">static</span> <span class="code-object">int</span> ethtool_ioctl(struct net *net, struct compat_ifreq __user *ifr32) { : ret = dev_ioctl(net, SIOCETHTOOL, &amp;ifr, NULL); <span class="code-object">int</span> dev_ioctl(struct net *net, unsigned <span class="code-object">int</span> cmd, struct ifreq *ifr, bool *need_copyout) { : <span class="code-keyword">case</span> SIOCETHTOOL: dev_load(net, ifr-&gt;ifr_name); rtnl_lock(); ret = dev_ethtool(net, ifr); rtnl_unlock(); </pre> </div></div> <p>but there may be some more fine-grained method in the kernel to determine the current speed of the interface.</p> <p>I'm thinking, based on the stats from Shuici above and <a href="https://jira.whamcloud.com/browse/LU-14293" title="Poor lnet/ksocklnd(?) performance on 2x100G bonded ethernet" class="issue-link" data-issue-key="LU-14293"><del>LU-14293</del></a>, we want about <tt>conns_per_peer=4-6</tt> for 100GbE. The following functions provide a reasonable default value for <tt>conns_per_peer</tt> (calculations done manually):</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <th class='confluenceTh'>Speed</th> <th class='confluenceTh'><tt>ilog2(Gbps)</tt></th> <th class='confluenceTh'><tt>ilog2(Gbps) / 2</tt></th> <th class='confluenceTh'><tt>(ilog2(Gbps) + 1) / 2</tt></th> <th class='confluenceTh'><tt>ilog2(Gbps / 2)</tt></th> <th class='confluenceTh'><tt>ilog2(Gbps) / 2 + 1</tt></th> </tr> <tr> <td class='confluenceTd'><b>1Gbps</b></td> <td class='confluenceTd'>0</td> <td class='confluenceTd'>0</td> <td class='confluenceTd'>0</td> <td class='confluenceTd'>0</td> <td class='confluenceTd'>1</td> </tr> <tr> <td class='confluenceTd'>2Gbps</td> <td class='confluenceTd'>1</td> <td class='confluenceTd'>0</td> <td class='confluenceTd'>1</td> <td class='confluenceTd'>0</td> <td class='confluenceTd'>1</td> </tr> <tr> <td class='confluenceTd'>4Gbps</td> <td class='confluenceTd'>2</td> <td class='confluenceTd'>1</td> <td class='confluenceTd'>1</td> <td class='confluenceTd'>1</td> <td class='confluenceTd'>2</td> </tr> <tr> <td class='confluenceTd'>8Gbps</td> <td class='confluenceTd'>3</td> <td class='confluenceTd'>1</td> <td class='confluenceTd'>2</td> <td class='confluenceTd'>2</td> <td class='confluenceTd'>2</td> </tr> <tr> <td class='confluenceTd'><b>10Gbps</b></td> <td class='confluenceTd'>3</td> <td class='confluenceTd'>1</td> <td class='confluenceTd'>2</td> <td class='confluenceTd'>2</td> <td class='confluenceTd'>2</td> </tr> <tr> <td class='confluenceTd'>16Gbps</td> <td class='confluenceTd'>4</td> <td class='confluenceTd'>2</td> <td class='confluenceTd'>2</td> <td class='confluenceTd'>3</td> <td class='confluenceTd'>3</td> </tr> <tr> <td class='confluenceTd'>32Gbps</td> <td class='confluenceTd'>5</td> <td class='confluenceTd'>2</td> <td class='confluenceTd'>3</td> <td class='confluenceTd'>4</td> <td class='confluenceTd'>3</td> </tr> <tr> <td class='confluenceTd'><b>50Gbps</b></td> <td class='confluenceTd'>5</td> <td class='confluenceTd'>2</td> <td class='confluenceTd'>3</td> <td class='confluenceTd'>4</td> <td class='confluenceTd'>3</td> </tr> <tr> <td class='confluenceTd'>64Gbps</td> <td class='confluenceTd'>6</td> <td class='confluenceTd'>3</td> <td class='confluenceTd'>4</td> <td class='confluenceTd'>5</td> <td class='confluenceTd'>4</td> </tr> <tr> <td class='confluenceTd'><b>100Gbps</b></td> <td class='confluenceTd'>6</td> <td class='confluenceTd'>3</td> <td class='confluenceTd'>4</td> <td class='confluenceTd'>5</td> <td class='confluenceTd'>4</td> </tr> <tr> <td class='confluenceTd'>128Gbps</td> <td class='confluenceTd'>7</td> <td class='confluenceTd'>3</td> <td class='confluenceTd'>4</td> <td class='confluenceTd'>6</td> <td class='confluenceTd'>4</td> </tr> <tr> <td class='confluenceTd'><b>200Gbps</b></td> <td class='confluenceTd'>7</td> <td class='confluenceTd'>3</td> <td class='confluenceTd'>4</td> <td class='confluenceTd'>6</td> <td class='confluenceTd'>4</td> </tr> <tr> <td class='confluenceTd'>256Gbps</td> <td class='confluenceTd'>8</td> <td class='confluenceTd'>4</td> <td class='confluenceTd'>5</td> <td class='confluenceTd'>7</td> <td class='confluenceTd'>5</td> </tr> </tbody></table> </div> <p>I believe <tt>conns_per_peer=0</tt> and <tt>conns_per_peer=1</tt> is functionally equivalent. In any case, I think either the third or fourth ("log2(speed in multiples of 2Gbps)") or fifth formulas ("log4(Gbps)+1") provide a good starting point. That would give us <tt>conns_per_peer=4/5</tt> at 100Gbps. We should probably prefer the lower value (<tt>(ilog2(Gbps) + 1) / 2</tt> or (<tt>ilog2(Gbps) / 2 + 1</tt>), since we have to balance single-client performance against the number of sockets created to a server. Users can always specify a better value if they have a preference, but typically they will never touch it so better to have <em>something</em> useful (even if not perfect for every situation) rather than repeated complaints about 100GbE being slow.</p> <p>The linux kernel implements features to distribute network packets/TCP fragments over multiple CPU cores. Usually distribution is decided via "hash function" based on incoming TCP port and sender IP address.</p> <ul> <li><a href="https://www.kernel.org/doc/Documentation/networking/scaling.txt" class="external-link" target="_blank" rel="nofollow noopener">https://www.kernel.org/doc/Documentation/networking/scaling.txt</a></li> </ul> <p>In theory a network interface should achieve line rate on a single TCP port with streams from multiple IP addresses.</p> <p>For ORNL we found conns_per_peer=8 gave the best results for 100Gbps.</p> <p>Chris, the problem here is that without <tt>conns_per_peer</tt> there is only a single port for socklnd on a single IP address. Creating multiple sockets on the client avoids that issue, and is much less complex than creating multiple virtual interfaces. With multiple clients a server is less likely to have a problem, but if only a single client is reading/writing (which some workloads do), then it would again be limited performance without multiple sockets. </p> <p>James, looking at <a href="https://jira.whamcloud.com/browse/LU-14293" title="Poor lnet/ksocklnd(?) performance on 2x100G bonded ethernet" class="issue-link" data-issue-key="LU-14293"><del>LU-14293</del></a> it seems like the peak performance could be hit with 6 connections, but no results were shown between 4 and 8. Also note that socklnd creates 3x TCP connections per peer (read, write, small message), to allow different tunings and avoid congestion.</p> <p>It might be worthwhile to see if multiple small message connections is useful or not, so that the total connections would be (2x conns_per_peer +1), but that optimization is probably only needed if we start running out of ports (2500 clients with 8x3 connections). I'm hoping nobody is building a giant cluster with that many Ethernet cards and not using RoCE or similar).</p> <p>You mean like our next machine <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Andreas, </p> <p>To clarify, the socklnd conns_per_peer currently results in 2xconns_per_peer + 1 tcp connections per peer as only bulk_in and bulk_out conn types are multiplied. </p> <blockquote><p>Chris, the problem here is that without conns_per_peer there is only a single port for socklnd on a single IP address. Creating multiple sockets on the client avoids that issue, and is much less complex than creating multiple virtual interfaces. <br/> With multiple clients a server is less likely to have a problem, but if only a single client is reading/writing (which some workloads do), then it would again be limited performance without multiple sockets.</p></blockquote> <p>Thanks Andreas, as stated in the description this feature is intended for systems with small number of clients. It does not appear to benefit systems at scale.</p> <p>Is possible to know ahead of time which TCP ports <tt>conns_per_peer</tt> will use (ie. to adjust firewalls) ?</p> <p>The problem described, insufficient incoming TCP streams to achieve network line rate, is not unique to Lustre.</p> <p>The target port for connections will always be the same, 988, which it is for all new connections, and the actually-assigned port is mostly irrelevant. This is not any different from multiple clients connecting separately.</p> <blockquote><p>which it is for all new connections, and the actually-assigned port is mostly irrelevant. This is not any different from multiple clients connecting separately.</p></blockquote> <p>If I remember correctly this is not totally true. There is still this rare behavior where a TCP socket is broken for some reasons and the first node to need it is the server, trying to send a ldlm callback. When the server detects the tcp connection for the reverse import is broken, it re-establish it itself, creating a server-&gt;client socket.</p> <p>(If the client needs this connection before, (ie: obd_ping), it will re-establish it normally and the server will use this connection as usual). This likely impact the metadata socket, not the bulk I/O sockets.</p> <p>Serguei Smirnov (ssmirnov@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41463" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41463</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12815" title="Create multiple TCP sockets per SockLND" class="issue-link" data-issue-key="LU-12815"><del>LU-12815</del></a> socklnd: allow dynamic setting of conns_per_peer<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: fe3db0979f34afc5139fdc1b6b9ab6eace5cfde4</p> <blockquote> <p>There is still this rare behavior where a TCP socket is broken for some reasons and the first node to need it is the server, trying to send a ldlm callback. When the server detects the tcp connection for the reverse import is broken, it re-establish it itself, creating a server-&gt;client socket.</p></blockquote> <p>That is true, but in this case I still believe that the server will use target port 988 on the client (not sure of source port), and the client will need to allow new connections on port 988 for most reliable behavior. In many cases, it is possible for the client to function properly without allowing <b>any</b> incoming connections, but as you write there may be rare cases that the server needs to initiate a connection, and without that the client may occasionally be evicted. For some sites that may be preferable to having an open port in the firewall. IIRC, there may even be a parameter to disable server-&gt;client connections, but I don't recall the details.</p> <p>That's correct. I wanted to warn about this often unknown use case (I was really surprised when I discovered it), with any potential impact of this feature.</p> <p>Agreed that some sites could prefer risking evictions than changing firewall rules. However, this is surprising behavior and sites tends to reduce as much as possible evictions and having a "normal case" where evictions will happen but are expected, just because of unknown traffic rules is not the best option. This could cause additional JIRA tickets. But this problem already exists and is independent of this ticket. I just wanted to avoid increasing the issue. Does multiple TCP sockets featurer will only apply to Bulk I/O sockets?</p> <p> </p> <blockquote> <p> Does multiple TCP sockets featurer will only apply to Bulk I/O sockets?</p></blockquote> <p>Correct, this is only for bulk sockets. </p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/41056/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41056/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12815" title="Create multiple TCP sockets per SockLND" class="issue-link" data-issue-key="LU-12815"><del>LU-12815</del></a> socklnd: add conns_per_peer parameter<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 71b2476e4ddb95aa42f4a0ea3f23b1826017bfa5</p> <p>There may still be work needed to distribute RPCs from a single client to multiple CPTs on the server, in order to get the best performance for real IO workloads. </p> <p>Otherwise, a client with a single interface (NID) will have all of its RPCs handled by cores in a single CPT, which is not quite the same as having multiple real interfaces on the client. Discussion is ongoing in <a href="https://jira.whamcloud.com/browse/LU-14676" title="Better hash distribution to different CPTs when LNET router is exist" class="issue-link" data-issue-key="LU-14676"><del>LU-14676</del></a>.</p> <p>Serguei Smirnov (ssmirnov@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/44417" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44417</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12815" title="Create multiple TCP sockets per SockLND" class="issue-link" data-issue-key="LU-12815"><del>LU-12815</del></a> socklnd: set conns_per_peer based on link speed<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ca2f4fed6d85d2e4506958fcc2e1c6c98eb2d020</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/41463/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41463/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12815" title="Create multiple TCP sockets per SockLND" class="issue-link" data-issue-key="LU-12815"><del>LU-12815</del></a> socklnd: allow dynamic setting of conns_per_peer<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a5cbe7883db6d77b82fbd83ad4c662499421d229</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/44417/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44417/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12815" title="Create multiple TCP sockets per SockLND" class="issue-link" data-issue-key="LU-12815"><del>LU-12815</del></a> socklnd: set conns_per_peer based on link speed<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c44afcfb72a1c2fd8392bfab3143c3835b146be6</p> <p>Looks like everything has landed for 2.15</p> <p>"Cyril Bordage &lt;cbordage@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47252" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47252</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12815" title="Create multiple TCP sockets per SockLND" class="issue-link" data-issue-key="LU-12815"><del>LU-12815</del></a> socklnd: add conns_per_peer parameter<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: b370ffba7b778f9b5fee325a8c67228ca2454137</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/47252/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47252/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12815" title="Create multiple TCP sockets per SockLND" class="issue-link" data-issue-key="LU-12815"><del>LU-12815</del></a> socklnd: add conns_per_peer parameter<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: be2c4bb928b5cf6b428d7974e8fd89ea177fa2df</p> Duplicate LU-14293 Related LU-14676 Development Rank 1|i00nfr: Rank (Obsolete) 9223372036854775807