Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-14806] o2iblnd: IB HCA failover with o2ib bonding is broken https://jira.whamcloud.com/browse/LU-14806 Lustre <p>It has been observed that "ko2iblnd dev_failover=1" option used on a node with o2ib bonding doesn't behave properly.</p> LU-14806 o2iblnd: IB HCA failover with o2ib bonding is broken Bug Minor Resolved Fixed Serguei Smirnov Serguei Smirnov Fri, 2 Jul 2021 01:08:40 +0000 Fri, 17 Sep 2021 20:26:42 +0000 Sat, 31 Jul 2021 13:25:40 +0000 Lustre 2.15.0 0 3 <p>Here are the steps to reproduce (2 ib-enabled nodes are required, one with two ib interfaces):</p> <ul> <li>Configure IB bonding. Here's the example of "ip a" output:</li> </ul> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 3: ib0: &lt;BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP&gt; mtu 2044 qdisc mq master bond0 state UP group <span class="code-keyword">default</span> qlen 256 link/infiniband a0:00:02:10:fe:80:00:00:00:00:00:00:00:02:c9:03:00:5a:63:2b brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff 4: ib1: &lt;BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP&gt; mtu 65520 qdisc mq master bond0 state UP group <span class="code-keyword">default</span> qlen 256 link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:00:02:c9:03:00:5a:63:2c brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff 5: bond0: &lt;BROADCAST,MULTICAST,MASTER,UP,LOWER_UP&gt; mtu 2044 qdisc noqueue state UP group <span class="code-keyword">default</span> qlen 1000 link/infiniband a0:00:02:10:fe:80:00:00:00:00:00:00:00:02:c9:03:00:5a:63:2b brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff inet 10.1.0.9/24 brd 10.1.0.255 scope global noprefixroute bond0 valid_lft forever preferred_lft forever inet6 fe80::202:c903:5a:632b/64 scope link valid_lft forever preferred_lft forever</pre> </div></div> <ul> <li>Add "option ko2iblnd dev_failover=1" to lnet.conf</li> <li>Reload LNet modules and configure o2ib net on the bonded interface:</li> </ul> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lnetctl lnet configure lnetctl net add --net o2ib --<span class="code-keyword">if</span> bond0</pre> </div></div> <ul> <li>Discover the other node:</li> </ul> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lnetctl discover 10.1.0.10@o2ib </pre> </div></div> <ul> <li>Make sure lnetctl ping is reliable:</li> </ul> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lnetctl ping 10.1.0.10@o2ib (repeat multiple times)</pre> </div></div> <ul> <li>Find out which ib link is active:</li> </ul> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)Bonding Mode: fault-tolerance (active-backup) (fail_over_mac active) Primary Slave: ib0 (primary_reselect always) Currently Active Slave: ib0 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 100 Down Delay (ms): 100 Peer Notification Delay (ms): 0Slave Interface: ib0 MII Status: up Speed: 20000 Mbps Duplex: full Link Failure Count: 18 Permanent HW addr: a0:00:02:10:fe:80:00:00:00:00:00:00:00:02:c9:03:00:5a:63:2b Slave queue ID: 0Slave Interface: ib1 MII Status: up Speed: 20000 Mbps Duplex: full Link Failure Count: 7 Permanent HW addr: a0:00:02:20:fe:80:00:00:00:00:00:00:00:02:c9:03:00:5a:63:2c Slave queue ID: 0</pre> </div></div> <ul> <li>Pull the cable corresponding to the active link</li> <li>Try pinging the other node (this will work in a second or so once failover completes):</li> </ul> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ping 10.1.0.10</pre> </div></div> <ul> <li>Try lnetctl pinging the other node like before (this won't succeed anymore)</li> </ul> <p>Serguei Smirnov (ssmirnov@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/44139" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44139</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14806" title="o2iblnd: IB HCA failover with o2ib bonding is broken " class="issue-link" data-issue-key="LU-14806"><del>LU-14806</del></a> o2iblnd: clear fatal error on successful failover<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: eb174ff461244ee6aabb68d39dce9e73ad7a085e</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/44139/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44139/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14806" title="o2iblnd: IB HCA failover with o2ib bonding is broken " class="issue-link" data-issue-key="LU-14806"><del>LU-14806</del></a> o2iblnd: clear fatal error on successful failover<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4668283cd13079dd6d86482704aef593f5c01dff</p> <p>Landed for 2.15</p> Related LU-15018 Development Rank 1|i01ybr: Rank (Obsolete) 9223372036854775807 Severity