https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8937 Lustre <p>For increase bandwidth and failover we would like to connect 2 IB ports on each side of a router</p> <p>o2ib ---- | HCA1 - ROUTER - HCA3|-- o2ib313</p> <p>o2ib ---- | HCA2 - ROUTER - HCA4|-- o2ib313</p> <p>I know we can doing if we had 2 different nids on each side. Can it be done using the same nid on 2 interfaces?</p> LU-8937

Dual IB port connect Router

Question/Request Minor Closed Not a Bug Amir Shehata Mahmoud Hanafi Wed, 14 Dec 2016 02:55:23 +0000 Thu, 13 Sep 2018 20:45:33 +0000 Thu, 13 Sep 2018 20:45:33 +0000 Lustre 2.7.0 0 5 <p>Currently LNet doesn't support having two ports with the same NIDs, or even on the same LNet. The only way you can do this is by configuring each IB port on a separate LNet network. so the configuration would look something like this</p> <p>o2ib1 &#8212; | HCA1 - ROUTER - HCA3 | &#8211; o2ib313<br/> o2ib2 ----| HCA2 - ROUTER - HCA4 | &#8211; o2ib314</p> <p>However, there is a feature that we have now called Multi-rail. It's not in master, but on its own branch, which allows LNet to use multiple interfaces on the same LNet Network, which will allow you to do what you want there, even without the presence of the router, if it's not needed for other purposes. So one possible configuration for what you have is:</p> <p>o2ib(hca1, hca2) and o2ib(hca3, hca4). This will allow all interfaces to live on the same network. LNet will do weighted round robin over the interfaces, and you should see an aggregate of the performance of both interfaces.</p> <p>If you need the routers you'd do something like</p> <p>o2ib(hca1, hca2) on the nodes on the left side of the routers<br/> o2ib(hca3, hca4) on the routers<br/> o2ib313(hca5, hca6) on the routers</p> <ol> <li> <ol> <li>This assumes that the router has 2 interfaces on each side of the network<br/> o2ib313(hca7, hca8) on the right side of the routers.</li> </ol> </li> </ol> <p>This will increase performance, but the routers will have an impact on this performance gain, because there is still a hop for the messages to go over.</p> <p>Let me know if you need more details</p> <p>What if the node/servers on each of the routers only has 1 interface</p> <p> </p> <p>servers(o2ib)--- o2ib(hca1,hca2)<span class="error">&#91;router&#93;</span>o2ib313(hca3,hca4)--- nodes(o2ib313)</p> <p> </p> <p>I'm assuming you're asking about MR.</p> <p>What you have outlined would work.</p> <p>You would configure the servers and the nodes with the routers interfaces as shown below:</p> <p>on the servers:<br/> lnetctl peer add --nids router@hca1-nid,router@hca2-nid</p> <p>on the nodes<br/> lnetctl peer add --nids router@hca3-nid,router@hca4-nid</p> <p>this will enable both the nodes and the servers to use both interfaces on the routers, which should increase the performance of the router.</p> <p>Without MR you would need to add each of the routers interfaces on a different LNet network (o2ib<span class="error">&#91;1,2,3,4&#93;</span>) and then configure some of the servers to use one LNet network and other servers to configure the other LNet network. With MR the configuration is simplified quiet a bit.</p> <p>We can close this case. We have dual-rail routers working in production.</p> <p>Question was answered.</p> Development Rank 1|hzyydz: Rank (Obsolete) 9223372036854775807