[LU-14806] o2iblnd: IB HCA failover with o2ib bonding is broken

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.15.0
• Type: Bug
• Priority: Minor
• Reporter: Serguei Smirnov
• Assignee: Serguei Smirnov
• Resolution: Fixed
• Labels: bonding, failover, lnet, o2iblnd
• Severity: 3

Description

It has been observed that "ko2iblnd dev_failover=1" option used on a node with o2ib bonding doesn't behave properly.

Issue Links

• is related to: LU-15018 o2iblnd: kiblnd_dev_failover should t...

Comments

[Serguei Smirnov]

Here are the steps to reproduce (2 ib-enabled nodes are required, one with two ib interfaces):

• Configure IB bonding. Here's the example of "ip a" output:

 3: ib0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 2044 qdisc mq master bond0 state UP group default qlen 256
    link/infiniband a0:00:02:10:fe:80:00:00:00:00:00:00:00:02:c9:03:00:5a:63:2b brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
4: ib1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 65520 qdisc mq master bond0 state UP group default qlen 256
    link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:00:02:c9:03:00:5a:63:2c brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 2044 qdisc noqueue state UP group default qlen 1000
    link/infiniband a0:00:02:10:fe:80:00:00:00:00:00:00:00:02:c9:03:00:5a:63:2b brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 10.1.0.9/24 brd 10.1.0.255 scope global noprefixroute bond0
       valid_lft forever preferred_lft forever
    inet6 fe80::202:c903:5a:632b/64 scope link 
       valid_lft forever preferred_lft forever

• Add "option ko2iblnd dev_failover=1" to lnet.conf
• Reload LNet modules and configure o2ib net on the bonded interface:

lnetctl lnet configure
lnetctl net add --net o2ib --if bond0

• Discover the other node:

lnetctl discover 10.1.0.10@o2ib  

• Make sure lnetctl ping is reliable:

 lnetctl ping 10.1.0.10@o2ib                  (repeat multiple times)

• Find out which ib link is active:

cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)Bonding Mode: fault-tolerance (active-backup) (fail_over_mac active)
Primary Slave: ib0 (primary_reselect always)
Currently Active Slave: ib0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 100
Down Delay (ms): 100
Peer Notification Delay (ms): 0Slave Interface: ib0
MII Status: up
Speed: 20000 Mbps
Duplex: full
Link Failure Count: 18
Permanent HW addr: a0:00:02:10:fe:80:00:00:00:00:00:00:00:02:c9:03:00:5a:63:2b
Slave queue ID: 0Slave Interface: ib1
MII Status: up
Speed: 20000 Mbps
Duplex: full
Link Failure Count: 7
Permanent HW addr: a0:00:02:20:fe:80:00:00:00:00:00:00:00:02:c9:03:00:5a:63:2c
Slave queue ID: 0

• Pull the cable corresponding to the active link
• Try pinging the other node (this will work in a second or so once failover completes):

 ping 10.1.0.10

• Try lnetctl pinging the other node like before (this won't succeed anymore)

[Gerrit Updater]

Serguei Smirnov ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44139
Subject: LU-14806 o2iblnd: clear fatal error on successful failover
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: eb174ff461244ee6aabb68d39dce9e73ad7a085e

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/44139/
Subject: LU-14806 o2iblnd: clear fatal error on successful failover
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4668283cd13079dd6d86482704aef593f5c01dff

[Peter Jones]

Landed for 2.15