https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8943 Lustre <p>OPA driver optimizations are based on the MPI model where it is expected to have multiple endpoints between two given nodes. To enable this optimization for Lustre, we need to make it possible, via an LND-specific tuneable, to create multiple endpoints and to balance the traffic over them.</p> <p>I have already created an experimental patch to test this theory out. I was able to push OPA performance to 12.4GB/s by just having 2 QPs between the nodes and round robin messages between them.</p> <p>This Jira ticket is for productizing my patch and testing it out thoroughly for OPA and IB. Test results will be posted to this ticket.</p> LU-8943

Enable Multiple IB/OPA Endpoints Between Nodes

Improvement Critical Resolved Fixed Doug Oucharek Doug Oucharek lnet Thu, 15 Dec 2016 19:02:29 +0000 Tue, 5 Dec 2017 08:40:39 +0000 Fri, 12 May 2017 12:22:45 +0000 Lustre 2.10.0 0 21 <p>Doug Oucharek (doug.s.oucharek@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/25168" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25168</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8943" title="Enable Multiple IB/OPA Endpoints Between Nodes" class="issue-link" data-issue-key="LU-8943"><del>LU-8943</del></a> lnd: Enable Multiple OPA Endpoints between Nodes<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c10acb41f474365f2b3026bdb04890a065cb9849</p> <p>To activate this patch, you need to use the following option:</p> <p>options ko2iblnd conns_per_peer=&lt;n&gt;</p> <p>Where &lt;n&gt; is the number QPs you want per peer connection.  At the moment, both sides of the connection must have the same setting (I need to fix this in the patch...only the client side should need this).</p> <p>I found that setting &lt;n&gt; to 6 gave me amazing performance.  Note: I have not tried this patch yet with the recommended hfi tunings.  They "will" interfere with this patch and should initially be avoided.</p> <p>Another note: I believe there is a race condition in the hfi driver we trigger when there is too much parallelism.  A couple of times running this patch, I found the hfi driver "missed" an event.  I am talking to the OPA developers about this.</p> <p>The patch for this ticket is showing a lot of promise.  To productize it so we can land it to master, I need to do the following:</p> <ul> <li>Work the code so only the active side of a connection needs to have conns_per_peer set.  The passive side should just adapt automatically.</li> <li>Make sure backwards compatibility is not broken when this feature is turned on in either the active side or passive side.</li> <li>The code which implements the round-robin behaviour has a potential infinite loop when things go wrong.  I need to add protection against that happening.</li> <li>There is no code to recover a downed connection to get us back to the conns_per_peer level.  I'm not sure I will add that but need to evaluate the situation more.</li> <li>Right now, there is no easy way to see if this feature is active and how well it is working.  I need to add some connection-based stats to be queried by lnetctl so we have a way to validate this feature and monitor it.</li> </ul> <p>In addition, testing needs to be done to see how much more CPU this feature consumes when it is activated.  We need to measure the costs as well as the benefits.  This needs to all be done with MLX hardware as well as OPA just to see what happens if this is activated on MLX-based networks.</p> <p>I have attached an Excel spreadsheet showing the performance changes with different conns_per_peer settings for both OPA and MLX-QDR.  For OPA, there is a tab showing the change without any HFI1 tunings (i.e. just the defaults) and with the recommended HFI1 tunings.</p> <p>Summary: Using this patch with conns_per_peer of 3 and the recommended HFI1 tunings provides good and consistent performance.</p> <p>Still to be done: Testing this patch for backwards compatibility.  </p> <p>Backwards compatibility testing looks good.  An upgraded node who initiates connections will create conns_per_peer connections and the non-upgraded receiver node will allow that many connections to be created.  However, the non-upgraded node will not "use" all the connections to send messages, only the first one.  So performance will not improve.</p> <p>If things are reversed (non-upgraded initiator to upgraded receiver) will work as if neither side is upgraded because it is the initiator who decides how many connections to have and in this case, it will just be one.</p> <p>So, to get the performance benefit, both sides of a connection need to be upgraded with this patch and the initiator needs to have conns_per_peer set &gt; 1.</p> <p>Based on the attached spreadsheet, I recommend OPA systems with many cores use conns_per_peer = 3 and these HFI1 parameters:</p> <p>options hfi1 krcvqs=8 piothreshold=0 sge_copy_mode=2 wss_threshold=70</p> <p>However, if you are on a VM or have a limited number of cores, change conns_per_peer = 4 and krcvqs = 4 in the HFI1 parameters.</p> <blockquote> <p>I recommend OPA systems with many cores use <tt>conns_per_peer = 3</tt> and these HFI1 parameters:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>options hfi1 krcvqs=8 piothreshold=0 sge_copy_mode=2 wss_threshold=70 </pre> </div></div></blockquote> <p>Are you going to add these to the <tt>/usr/sbin/ko2iblnd-probe</tt> script, or be set by default in some other manner, or will this be up to the user to discover and set? At a very minimum there should be an update to the Lustre User Manual (see <a href="https://wiki.hpdd.intel.com/display/PUB/Making+changes+to+the+Lustre+Manual" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/PUB/Making+changes+to+the+Lustre+Manual</a>), but providing good performance out of the box is preferred.</p> <p>I did update the OPA defaults to set conns_per_peer to 4 when OPA is detected.  I'll also update the manual under <a href="https://jira.whamcloud.com/browse/LUDOC-374" title="Add notes about conns_per_peer ko2iblnd parameter" class="issue-link" data-issue-key="LUDOC-374"><del>LUDOC-374</del></a>.</p> <p>I bumped the conns_per_peer to 4 from 3 because OPA team is going to start recommending a krcvqs default of 4 especially for a low number of cores (i.e. VMs).  Having a conns_per_peer of 4 helps to compensate for the lower krcvqs number so we should work well out of the box whether krcvqs is 4 or 8.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/25168/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25168/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8943" title="Enable Multiple IB/OPA Endpoints Between Nodes" class="issue-link" data-issue-key="LU-8943"><del>LU-8943</del></a> lnd: Enable Multiple OPA Endpoints between Nodes<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7241e68f37962991ef43a6c01b3a83ff67282d88</p> <p>Landed for 2.10</p> <p>I observed strange behavior. It looks after this commit I cannot unload ko2iblnd module. LNet is busy even all unmounted successfully. Only reboot helps.</p> <p> </p> <p> </p> <p>Does "<tt>lctl network down</tt>" or "<tt>lnetctl lnet unconfigure</tt>" help?</p> <p>No, as I mentoined before only reboot helps.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lustre_rmmod rmmod: ERROR: Module ko2iblnd is in use # lsmod|less Module Size Used by ko2iblnd 233790 1 ptlrpc 1343928 0 obdclass 1744518 1 ptlrpc lnet 483843 3 ko2iblnd,obdclass,ptlrpc libcfs 416336 4 lnet,ko2iblnd,obdclass,ptlrpc [...] # lctl network down LNET busy lnetctl &gt; lnet unconfigure unconfigure: - lnet: errno: -16 descr: "LNet unconfigure error: Device or resource busy" lnetctl &gt; lnet unconfigure --all unconfigure: - lnet: errno: -16 descr: "LNet unconfigure error: Device or resource busy" # lustre_rmmod rmmod: ERROR: Module ko2iblnd is in use </pre> </div></div> <p>When I created the performance spreadsheet, I needed to keep changing conns_per_peer.  I had no problems taking down and brining up LNet using these commands:</p> <p>Up:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>modprobe lnet lctl network configure modprobe lnet-selftest </pre> </div></div> <p>Down:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rmmod lnet-selftest lctl network down rmmod ko2iblnd rmmod lnet </pre> </div></div> <p>There must be something different about what you are doing which is triggering ref counters to not be reduced. Are you using DLC? What is your environment?  Are both nodes running the latest code with this patch?</p> <p>I'm using new Lustre client with this patch and old Lustre servers without this patch. So, I just mount lustre FS then use it and then try to unload after umount. I don't use DLC. I have CentOS 7.3 in both sites.</p> <p> </p> <p>That might be the reason.  The client will create multiple connections, but the server will only have one they are all talking to.  When one connection on the client is closed, the connection on the server will be closed.  I suspect the remaining connections on the client can't be closed.  I'll have to look at the code to see what I can do in this situation.</p> <p>I suspect if the server has the patch, you would not have a problem.</p> <p>I just tried to reproduce with the passive node being unpatched.  Was not able to reproduce your issue.  The "lctl network down" takes a long time, but does succeed.  There must be something else here.  Do you know if your parameters like map_on_demand are different?  Is a reconnection happening to renegotiate the parameters?  This is something I have not tried.</p> <p>Dmitry, when you get the file system mounted, can you issue the following sequence on both nodes to ensure we are creating 4 connections on each:</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl &gt; network o2ib &gt; conn_list </pre> </div></div> <p>You should see 4 connections to the peer if the initiator (usually the client) has the MultiQP patch, and 1 connection to the peer if it doesn't.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl lctl &gt; network o2ib lctl &gt; conn_list 192.168.213.125@o2ib mtu -1 192.168.213.125@o2ib mtu -1 192.168.213.125@o2ib mtu -1 192.168.213.125@o2ib mtu -1 192.168.213.125@o2ib mtu -1 192.168.213.125@o2ib mtu -1 192.168.213.125@o2ib mtu -1 192.168.213.125@o2ib mtu -1 192.168.213.231@o2ib mtu -1 192.168.213.231@o2ib mtu -1 192.168.213.231@o2ib mtu -1 192.168.213.231@o2ib mtu -1 192.168.213.232@o2ib mtu -1 192.168.213.232@o2ib mtu -1 192.168.213.232@o2ib mtu -1 192.168.213.232@o2ib mtu -1 192.168.213.233@o2ib mtu -1 192.168.213.233@o2ib mtu -1 192.168.213.233@o2ib mtu -1 192.168.213.233@o2ib mtu -1 192.168.213.234@o2ib mtu -1 192.168.213.234@o2ib mtu -1 192.168.213.234@o2ib mtu -1 192.168.213.234@o2ib mtu -1 192.168.213.235@o2ib mtu -1 192.168.213.235@o2ib mtu -1 192.168.213.235@o2ib mtu -1 192.168.213.235@o2ib mtu -1 192.168.213.236@o2ib mtu -1 192.168.213.236@o2ib mtu -1 192.168.213.236@o2ib mtu -1 192.168.213.236@o2ib mtu -1 # lnetctl lnet unconfigure --all unconfigure: - lnet: errno: -16 descr: "LNet unconfigure error: Device or resource busy" </pre> </div></div> <p> Client: 2.9.57_48_g0386263<br/> Servers:<br/> lustre: 2.7.19.10<br/> kernel: patchless_client<br/> build: 2.7.19.10--PRISTINE-3.10.0-514.10.2.el7_lustre.x86_64</p> <p>192.168.213.125@o2ib - client</p> <p>From server:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl lctl &gt; network o2ib lctl &gt; conn_list 192.168.213.125@o2ib mtu -1 192.168.213.125@o2ib mtu -1 192.168.213.125@o2ib mtu -1 192.168.213.125@o2ib mtu -1 ... </pre> </div></div> <p>Cliff is seeing this same problem on the soak cluster but there is no OPA, only MLX IB. I'm beginning to wonder if this is a problem with the Mutli-Rail drop rather than this change.</p> <p>Would it be a good idea to track all this under a new ticket instead of tacking onto an already closed one?</p> <p>Cliff created a ticket for this already: <a href="https://jira.whamcloud.com/browse/LU-9524" title="LNET Fails to unload. " class="issue-link" data-issue-key="LU-9524"><del>LU-9524</del></a>.</p> <p>Summary: this appears to have been introduced in patch: <a href="https://review.whamcloud.com/#/c/26959/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/26959/</a> and not the change under this ticket. ptlrpc is not longer being unloaded with lustre_rmmod so lnet won't unload.</p> Duplicate Gantt End to End LUDOC-374 Related Development Rank 1|hzyyj3: Rank (Obsolete) 9223372036854775807