Does anyone know what OFED the bug was fixed, if ever? Diego, did you or anyone report this bug to OFA?

As per Bull, they have a workaround for this OFED bug

I think we agree this is an bug in OFED stack. We can close this bug and up to OFED resolution we'll use an ib alias device on each client.

Thanks,

sorry I didn't see the last comment, I agree that it's not something we can fix inside o2iblnd and I don't have a better suggestion to help you make it easier,

Liang

Hi,

Finally I could make what you wrote in your previous comment:

In fact, we are having problems with the arp table. When we run 'arping' from client to server we get not only one arp address but 2, so the client is not able to have a proper arp table.

15:04:36.258634 ARP, Request who-has 60.64.0.31 (00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff) tell 0.0.0.0, length 56
15:04:36.258751 ARP, Reply 60.64.0.31 is-at 80:00:00:48:fe:80:00:00:00:00:00:00:00:02:c9:03:00:04:89:c5, length 56
15:04:36.258773 ARP, Reply 60.64.0.31 is-at 80:00:00:48:fe:80:00:00:00:00:00:00:00:02:c9:03:00:0a:ba:57, length 56

If we set 'arp_ignore=1' then one of the interfaces don't reply to standard ping requests (it's not always the same interface), so we still have the problem. It's like if only one interface can reply.

Also, as a consequence, routing is not well done from server to client as the client is connecting to a bad interface and not to the good one. It's definitively an OFED bug, do you agree?

So, by the moment, the only available solution is to have an alias ip on client and 2 different subnets, but this cannot be easy to do in a cluster with more than 4000 nodes...

Sebatien,

It's very weird if it can't work even server only has one ipoib interface, have tried both interfaces on server (i.e: only with ib0 and only with ib1)?
I checked our code again, we don't have any special tricky while using rdma_resolve_addr & rdma_connect in o2iblnd, so I still suspect it's because ARP flux, would it be possible that ARP cache is not refreshed so you still can't make it work with previous solution?

we can see this from release note of OFED:

2. Known Issues
===============================================================================
1. If a host has multiple interfaces and (a) each interface belongs to a
different IP subnet, (b) they all use the same InfiniBand Partition, and (c)
they are connected to the same IB Switch, then the host violates the IP rule
requiring different broadcast domains. Consequently, the host may build an
incorrect ARP table.

The correct setting of a multi-homed IPoIB host is achieved by using a
different PKEY for each IP subnet. If a host has multiple interfaces on the
same IP subnet, then to prevent a peer from building an incorrect ARP entry
(neighbor) set the net.ipv4.conf.X.arp_ignore value to 1 or 2, where X
stands for the IPoIB (non-child) interfaces (e.g., ib0, ib1, etc). This
causes the network stack to send ARP replies only on the interface with the
IP address specified in the ARP request:

sysctl -w net.ipv4.conf.ib0.arp_ignore=1
sysctl -w net.ipv4.conf.ib1.arp_ignore=1

Or, globally,

sysctl -w net.ipv4.conf.all.arp_ignore=1

To learn more about the arp_ignore parameter, see
Documentation/networking/ip-sysctl.txt.
Note that distributions have the means to make kernel parameters persistent.

Could you please try a few more things:
0. ifconfig output from both client and server
1. regular ping both interfaces of server from client, and collect "arp -na" on client
2. please collect "arp -na" and "ip neighbor show" from the client
3. assume the server has ib0(10.50.0.7), ib1(10.50.1.7), please run this and collect output:

client $ tcpdump -c 3 -nni ib0 arp
server $ tcpdump -c 3 -nni ib0 arp
client $ arping -c 3 -D -I ib0 10.50.0.7

4. collect output of:

client $ tcpdump -c 3 -nni ib0 arp
server $ tcpdump -c 3 -nni ib1 arp
client $ arping -c 3 -D -I ib0 10.50.1.7

5. please clear arp cache on client by "arp -d servername 10.50.0.7" and "arp -d servername 10.50.1.7" (or you have a better way)

6. ping both interfaces of server and collect "arp -na" again on client

7. reload o2iblnd on both client and server to see whether this can work

8. if it still can't help, please add those sysctl to /etc/sysctl.conf and reboot client and server to see how it works

Anyway, if these things can help, I don't really want previous logs/information, but if they still can't, we have to get those information for the next step survey.

Thanks
Liang

Hi,

I tested the two proposals from Liang.

My first tests consisted in tuning the ARP parameters, but it had no effect.

My second test was to configure one IPoIB on the client (60.64.x.x/16), only one IPoIB on the server (61.64.1.x/16), add a route in the IP routing table of the client so that the TCP connection is working. In this configuration, the 'lctl ping' was not working.

So, unless the client has an IPoIB address in the same subnet as the OST to reach, the connection seems to be blocked by a bug.

Sebastien.

from the log, the client got IB_CM_REJ_INVALID_SERVICE_ID from server, which means it thought there is no listener on server, I suspect it's because ARP flux, could you please try to set this and reload o2iblnd?

sysctl -w net.ipv4.conf.ib0.arp_ignore=1
sysctl -w net.ipv4.conf.ib1.arp_ignore=1

If this can't resolve the problem, could you try to see if the client can reach server if server has only one NI (i.e: only start LNet with o2ib0(ib0), then shutdown and try with o2ib1(ib1))

I don't know whether it's safe to use IP aliases for o2iblnd, but I can try to find out if none of previous ways can give us more hints.

Thanks
Liang

Hi,

I tried two alternatives to the client IP alias (servers still have their 2 IB interfaces on 2 different subnets):

• create a route on the client node, telling to use its ib0 interface to reach the servers' second IB interface network;
• create a route on the client node, telling to use a gateway represented by the ib0 IPoIB address of one of the servers to reach the servers' second IB interface network.

Both work at the IP level, but fail at the LNET level (ie 'ping' is OK, but 'lctl ping' is not).

So it seems to work only when the client has an IPoIB address in the same subnet as the OSTs it wants to reach.
Is this particular issue due to an OFED problem again? or could it be something fixable in Lustre?

Moreover, can you confirm it is safe to run Lustre clients with IP aliases?

TIA,
Sebastien.

Debug traces when lctl ping doesn't work in a multirail context with both interfaces in the same subnet.