https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6531 Lustre <p>Work on Fujitsu's o2iblnd channel bonding solution.</p> LU-6531

Fujitsu's o2iblnd Channel Bonding Solution

New Feature Major Resolved Won't Do Amir Shehata Amir Shehata Mon, 27 Apr 2015 22:39:54 +0000 Sat, 16 Sep 2017 07:54:00 +0000 Sat, 16 Sep 2017 07:54:00 +0000 Tue, 30 Jun 2015 00:00:00 +0000 0 20 <p>Duplicate of <a href="https://jira.whamcloud.com/browse/LU-6495" title="push Fujitsu IB multi-rail patch to Gerrit" class="issue-link" data-issue-key="LU-6495"><del>LU-6495</del></a></p> <p>Amir Shehata (amir.shehata@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/14625" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14625</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6531" title="Fujitsu&#39;s o2iblnd Channel Bonding Solution" class="issue-link" data-issue-key="LU-6531"><del>LU-6531</del></a> lnet: Fujitsu's Channel Bonding Solution<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2528a0d791fe8c6f2b046905725d12ec56c9bf6a</p> <p>Sorry I made a mistake. Please forget about the above patch.</p> <p>Amir Shehata (amir.shehata@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15170" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15170</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6531" title="Fujitsu&#39;s o2iblnd Channel Bonding Solution" class="issue-link" data-issue-key="LU-6531"><del>LU-6531</del></a> lnet: DLC interface for o2iblnd Channel Bonding<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: db91f5a2fb066a316e383807ad8fec1633237a55</p> <p>I've experimented the lnet channel bonding solution patch using lnet-selftest with following configuration:</p> <ul class="alternate" type="square"> <li>4 clients having a single IB interface</li> <li>1 server having two IB interfaces</li> <li>all IB interfaces are connected to same switch<br/> Here are the test results: <blockquote><p>size=1M duration=10 check= concurrency=16<br/> Lnet data bandwidth of all the servers (MB/s)</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <th class='confluenceTh'>#clients</th> <th class='confluenceTh'>write</th> <th class='confluenceTh'>read</th> </tr> <tr> <td class='confluenceTd'>1</td> <td class='confluenceTd'>5610</td> <td class='confluenceTd'>5816</td> </tr> <tr> <td class='confluenceTd'>2</td> <td class='confluenceTd'>10773</td> <td class='confluenceTd'>10155</td> </tr> <tr> <td class='confluenceTd'>3</td> <td class='confluenceTd'>12069</td> <td class='confluenceTd'>6358</td> </tr> <tr> <td class='confluenceTd'>4</td> <td class='confluenceTd'>12044</td> <td class='confluenceTd'>6332</td> </tr> </tbody></table> </div> </blockquote> <p>Write figures are good with respect with hardware capabilities, but I'm puzzled by figures for read.</p></li> </ul> <p>hi Frederic,</p> <p>Is it possible to share your full configuration?</p> <p>thanks<br/> amir</p> <p>On server's lnet.conf is:<br/> options lnet networks=o2ib0(ib0,ib1)<br/> And each client has the following lnet.conf<br/> options lnet networks=o2ib0(ib0)<br/> All nodes were using the same lustre_o2ibs_config input file:<br/> 10.1.0.41@o2ib0 10.1.0.41 10.1.0.101<br/> 10.1.0.31@o2ib0 10.1.0.31<br/> 10.1.0.32@o2ib0 10.1.0.32<br/> 10.1.0.35@o2ib0 10.1.0.35<br/> I also tried changing ko2iblnd parameters on all nodes:<br/> options ko2iblnd credits=2048 peer_credits=126 concurrent_sends=63 peer_buffer_credits=128</p> <p>This is a high-level design document I wrote based on the Fujitsu Channel Bonding solution. It also describes the new DLC interface I added to configure the Channel bonding solution.</p> <p>The design document is very useful, thanks.</p> <p>I do have one concern: the code looks through lists of routes while holding a spinlock and with interrupts disabled on the CPU (spin_lock_irqsave() and friends). This will definitely be a problem if these lists become large, because a system becomes unstable if one or more of the CPU cores runs for a long time with interrupts disabled.</p> <p>Trying to figure how large these lists can become, if we have a cluster with N clients, M MDS, O OSS, and ignoring routers, assuming just one interface for each system I get something like this:</p> <ul class="alternate" type="square"> <li>on a client: M + O</li> <li>on an MDS: N + M + O</li> <li>on an OSS: N + M</li> </ul> <p>This shouldn't be much of a problem in a small cluster, but in a large cluster it would be the MDS and OSS in particular that have large lists. So my concern is that there is a scaling problem that will render MDS and OSS unstable in large clusters, but will be invisible in the small clusters typically used for testing.</p> <p>To take a step back for a moment, I think we need to have a good answer to the following question:</p> <p>Why is implementing channel bonding at the LND level the right thing to do rather than implementing channel bonding at the LNet level?</p> <p>It is not clear to me that the current configuration approach when done at the LND level is very robust or system administration friendly, and ways to fix that don't seem terribly easy to do since this implementation is all hacked into a single LND component. I think that I can envision a system at the LNet level that would be much easier for system administrators to work with (because NIDs are already shared between nodes). I am also concerned about how credits at the LNet layer are going to interact with multiple invisible peer connections in the LND layer.</p> <p>Hi Chris,</p> <p>I agree with the validity of your question/concerns. I would add one other potential issue with this solution: it does not allow the bonded HCAs to be on different networks. That means it cannot protect you from a switch failure. An LNet layer solution could be developed to allow HCAs on different networks to be bonded.</p> <p>We we first started looking at channel bonding, the solution was based in the LNet layer. That was put on hold when the Fujitsu solution came to light as something already running in production and being offered to the community tree. The philosophy: we are better off with a working known than putting in the effort re-doing a new solution (an unknown). Also, having two solutions would muddy the Lustre waters so with the Fujitsu solution coming to the community tree, we backed off our own approach.</p> <p>Now, seeing the patch comments and your concerns, I feel we should re-review which solution is favoured for the community tree.</p> <p>For now, Intel is going to back off pushing this patch (and the DLC adaptation patch) given 1- it won't make 2.8 feature freeze, and 2- a need to reconsider what is the proper solution. The patches are left in Gerrit for community consideration and guidance as to one way to approach Channel Bonding.</p> <p>Closing this ticket, as LNet multi-rail support landed in 2.10.</p> Duplicate LU-9480 Related LU-6480 Development Rank 1|hzxbw7: Rank (Obsolete) 9223372036854775807