https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-153 Lustre <p>Clients are not able to connect to server interfaces when there are two IB cards (and two lnets) configured on servers. We have a workaround consisting on "lctl ping" from servers to both lnets on every client. After that clients are able to connect to servers.</p> <p>Once clients are mounted we see the problem if we run the "df -h /lustre" command on clients (obvious cause running this command client needs to contact OSSs).</p> <p>At first we try to ping every interface on server:</p> <p>client&gt; lctl ping 10.50.0.7@o2ib0 =&gt; No response<br/> client&gt; lctl ping 10.50.1.7@o2ib1 =&gt; No response</p> <p>client&gt;dmesg<br/> 00000400:00000100:3.0F:1297255885.268873:0:2998:0:(lib-move.c:1028:lnet_post_send_locked()) Dropping message for<br/> 12345-10.50.0.7@o2ib1: peer not alive<br/> 00000400:00020000:3.0:1297255885.279758:0:2998:0:(lib-move.c:2628:LNetGet()) error sending GET to 12345-10.50.0.7@o2ib1:<br/> -113<br/> 00000800:00000100:0.0F:1297255885.284181:0:2435:0:(o2iblnd_cb.c:462:kiblnd_rx_complete()) Rx from 10.50.0.7@o2ib1 failed:<br/> 5</p> <p>Then we ping client's interface (client has only one if) on both lnets:</p> <p>server&gt; lctl ping 10.50.0.50@o2ib0 =&gt; OK<br/> server&gt; lctl ping 10.50.0.50@o2ib1 =&gt; OK</p> <p>And problem is solved, "df -h /lustre" will run correctly and all "lctl ping" from client to server's interface will work fine.</p> <p>IPoIB ping command is working fine, we don't have DDR infiniband drivers running on our machines and we already tried a network configuration using ip2nets.</p> <p>Here you have our ip2nets config (note that all machines in the <span class="error">&#91;7-10&#93;</span> range are servers, with two IB cards, one lnet on every one, and all the rest of the machines are clients with only one IB interface and two lnets on every one):</p> <p><span class="error">&#91;root@berlin5 ~&#93;</span># cat /sys/module/lnet/parameters/ip2nets <br/> o2ib0(ib0) 10.50.0.<span class="error">&#91;7-10&#93;</span> ; o2ib1(ib1) 10.50.1.<span class="error">&#91;7-10&#93;</span> ; o2ib0(ib0) 10.50.<b>.</b> ; o2ib1(ib0) 10.50.<b>.</b></p> <p>So, it seems like clients are not able to choose between one of the interfaces on servers but once server has 'pinged' clients, these ones are now able to choose the right interface. </p> <p>Do you think this could be an OFED bug? Maybe an lnet bug?</p> RHEL 6.0 GA, ofed1.5.2, Lustre 2.0.0.1, Mellanox QDR Ib cards LU-153

Clients cannot connect to servers with 2 IB cards until "lctl ping" is done from server to clients

Bug Minor Resolved Won't Fix Liang Zhen Diego Moreno Thu, 24 Mar 2011 06:39:02 +0000 Mon, 1 Apr 2019 19:02:40 +0000 Thu, 21 Jul 2011 10:33:11 +0000 Lustre 2.0.0 0 5 <p>it looks like a LNet bug to me, I will look into it very soon</p> <p>Hi,</p> <p>Bad news on this, because CEA is now experiencing this issue on one of their cluster, but the problem is the workaround that consists in 'lctl pinging' all interfaces does not scale. So they are blocked, and this is why I would like to change this ticket's priority to Blocker.</p> <p>Could you please have a look at this very soon?</p> <p>TIA,<br/> Sebastien.</p> <p>Could you please provide ifconfig output and routing table of the server? Also, which OFED version and kernel version are you using? </p> <p>Thanks<br/> Liang</p> <p>Hi Liang,</p> <p>On server's side:</p> <ul class="alternate" type="square"> <li>OFED: 1.5.2 (kernel-ib-1.5-2.6.32_71.14.1.el6.Bull.20.x86_64.ofed1.5.2.Bull.4.el6.x86_64)</li> <li>lnet config:<br/> export LNET_MULTIRAIL_OPTIONS="networks=o2ib0(ib0),o2ib1(ib1)"</li> <li>kernel version: kernel-2.6.32-71.14.1.el6.Bull.20.x86_64</li> <li>ifconfig:<br/> ib0 Link encap:InfiniBand HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 <br/> inet addr:60.64.0.32 Bcast:60.64.255.255 Mask:255.255.0.0<br/> inet6 addr: fe80::202:c903:a:b73f/64 Scope:Link<br/> UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1<br/> RX packets:659 errors:0 dropped:0 overruns:0 frame:0<br/> TX packets:105 errors:0 dropped:5 overruns:0 carrier:0<br/> collisions:0 txqueuelen:256 <br/> RX bytes:48041 (46.9 KiB) TX bytes:18021 (17.5 KiB)<br/> ib1 Link encap:InfiniBand HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 <br/> inet addr:60.64.1.32 Bcast:60.64.255.255 Mask:255.255.0.0<br/> inet6 addr: fe80::202:c903:4:89b1/64 Scope:Link<br/> UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1<br/> RX packets:557 errors:0 dropped:0 overruns:0 frame:0<br/> TX packets:0 errors:0 dropped:5 overruns:0 carrier:0<br/> collisions:0 txqueuelen:256 <br/> RX bytes:31192 (30.4 KiB) TX bytes:0 (0.0 b)</li> </ul> <ul class="alternate" type="square"> <li>Routing:<br/> <span class="error">&#91;root@berlin8 ~&#93;</span># route<br/> Kernel IP routing table<br/> Destination Gateway Genmask Flags Metric Ref Use Iface<br/> 60.64.0.0 * 255.255.0.0 U 0 0 0 ib0<br/> 60.64.0.0 * 255.255.0.0 U 0 0 0 ib1<br/> 60.0.0.0 * 255.248.0.0 U 0 0 0 eth0<br/> default berlin32.echi.l 0.0.0.0 UG 0 0 0 eth0</li> </ul> <p>On client's side:</p> <ul class="alternate" type="square"> <li>OFED: 1.5.2 (kernel-ib-1.5-2.6.32_71.14.1.el6.Bull.20.x86_64.ofed1.5.2.Bull.4.el6.x86_64)</li> <li>lnet config:<br/> export LNET_MULTIRAIL_OPTIONS="networks=o2ib0(ib0),o2ib1(ib0)" </li> <li>kernel version: kernel-2.6.32-71.14.1.el6.Bull.20.x86_64</li> <li>ifconfig:<br/> ib0 Link encap:InfiniBand HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 <br/> inet addr:60.64.2.57 Bcast:60.64.255.255 Mask:255.255.0.0<br/> inet6 addr: fe80::230:48ff:fff4:ca15/64 Scope:Link<br/> UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1<br/> RX packets:1188 errors:0 dropped:0 overruns:0 frame:0<br/> TX packets:86 errors:0 dropped:5 overruns:0 carrier:0<br/> collisions:0 txqueuelen:256 <br/> RX bytes:83356 (81.4 KiB) TX bytes:14580 (14.2 KiB)</li> <li>Routing:<br/> <span class="error">&#91;root@berlin71 ~&#93;</span># route<br/> Kernel IP routing table<br/> Destination Gateway Genmask Flags Metric Ref Use Iface<br/> 60.64.0.0 * 255.255.0.0 U 0 0 0 ib0<br/> 60.0.0.0 * 255.248.0.0 U 0 0 0 eth2<br/> default berlin32.echi.l 0.0.0.0 UG 0 0 0 eth2</li> </ul> <p>Do you need any other information?</p> <p>Sorry, but, while this is undoubtedly an important issue for CEA, this is not a general enough issue to be a 2.1 blocker.</p> <p>It would be a little helpful if you can "echo +neterror &gt; &gt; /proc/sys/lnet/printk" on both server and the client, and reproduce it to get o2iblnd error message on both sides (very likely, you will see nothing on server side).</p> <p>Actually, I remember other people got into similar trouble while having multiple interfaces with same netmask and in the same subnet, and the problem disappeared after change them to different subnets. Would it be possible to change one of those address and make two interfaces in separated subnets and with different netmask? If this can help, it could be an issue of OFED although I'm not sure.</p> <p>Hi Liang,</p> <p>I'll try everything you propose on your last comment. However, having two interfaces in separated subnets won't be possible on our configuration as our clients, which have only one interface, need to access both interfaces on servers, that's why they are in the same subnet.</p> <p>One possibility could be to configure an alias interface on every client with the new subnet but I don't know if this works on lustre...</p> <p>Hi,</p> <p>Putting both interfaces in different subnets made the trick. If I want to have my clients connecting to both interfaces then I need to create an IP alias on every client because my clients have only one IB interface, is this a problem in lustre?</p> <p>I also get the traces you asked me when both cards are in the same subnet. In dmesg I only obtained the next two lines:</p> <p>Lustre: 40750:0:(lib-move.c:1028:lnet_post_send_locked()) Dropping message for 12345-60.64.1.32@o2ib1: peer not alive<br/> LustreError: 40750:0:(lib-move.c:2628:LNetGet()) error sending GET to 12345-60.64.1.32@o2ib1: -113</p> <p>So I also obtained debug daemon lines. The activated traces are: info, neterror, net, warning, nettrace, error and emerg. See lctl_ping_debug attachment.</p> <p>Then do you think this is an issue with OFED and not with lnet?</p> <p>Thanks,</p> <p>Debug traces when lctl ping doesn't work in a multirail context with both interfaces in the same subnet.</p> <p>Hi,</p> <p>I tried two alternatives to the client IP alias (servers still have their 2 IB interfaces on 2 different subnets):</p> <ul class="alternate" type="square"> <li>create a route on the client node, telling to use its ib0 interface to reach the servers' second IB interface network;</li> <li>create a route on the client node, telling to use a gateway represented by the ib0 IPoIB address of one of the servers to reach the servers' second IB interface network.</li> </ul> <p>Both work at the IP level, but fail at the LNET level (ie 'ping' is OK, but 'lctl ping' is not).</p> <p>So it seems to work only when the client has an IPoIB address in the same subnet as the OSTs it wants to reach.<br/> Is this particular issue due to an OFED problem again? or could it be something fixable in Lustre?</p> <p>Moreover, can you confirm it is safe to run Lustre clients with IP aliases?</p> <p>TIA,<br/> Sebastien.</p> <p>from the log, the client got IB_CM_REJ_INVALID_SERVICE_ID from server, which means it thought there is no listener on server, I suspect it's because ARP flux, could you please try to set this and reload o2iblnd? </p> <p> sysctl -w net.ipv4.conf.ib0.arp_ignore=1<br/> sysctl -w net.ipv4.conf.ib1.arp_ignore=1</p> <p>If this can't resolve the problem, could you try to see if the client can reach server if server has only one NI (i.e: only start LNet with o2ib0(ib0), then shutdown and try with o2ib1(ib1))</p> <p>I don't know whether it's safe to use IP aliases for o2iblnd, but I can try to find out if none of previous ways can give us more hints.</p> <p>Thanks<br/> Liang</p> <p>Hi,</p> <p>I tested the two proposals from Liang.</p> <p>My first tests consisted in tuning the ARP parameters, but it had no effect.</p> <p>My second test was to configure one IPoIB on the client (60.64.x.x/16), only one IPoIB on the server (61.64.1.x/16), add a route in the IP routing table of the client so that the TCP connection is working. In this configuration, the 'lctl ping' was not working.</p> <p>So, unless the client has an IPoIB address in the same subnet as the OST to reach, the connection seems to be blocked by a bug.</p> <p>Sebastien.</p> <p>Sebatien, </p> <p>It's very weird if it can't work even server only has one ipoib interface, have tried both interfaces on server (i.e: only with ib0 and only with ib1)? <br/> I checked our code again, we don't have any special tricky while using rdma_resolve_addr &amp; rdma_connect in o2iblnd, so I still suspect it's because ARP flux, would it be possible that ARP cache is not refreshed so you still can't make it work with previous solution? </p> <p>we can see this from release note of OFED:</p> <p>2. Known Issues<br/> ===============================================================================<br/> 1. If a host has multiple interfaces and (a) each interface belongs to a<br/> different IP subnet, (b) they all use the same InfiniBand Partition, and (c)<br/> they are connected to the same IB Switch, then the host violates the IP rule<br/> requiring different broadcast domains. Consequently, the host may build an<br/> incorrect ARP table.</p> <p> The correct setting of a multi-homed IPoIB host is achieved by using a<br/> different PKEY for each IP subnet. If a host has multiple interfaces on the<br/> same IP subnet, then to prevent a peer from building an incorrect ARP entry<br/> (neighbor) set the net.ipv4.conf.X.arp_ignore value to 1 or 2, where X<br/> stands for the IPoIB (non-child) interfaces (e.g., ib0, ib1, etc). This<br/> causes the network stack to send ARP replies only on the interface with the<br/> IP address specified in the ARP request:</p> <p> sysctl -w net.ipv4.conf.ib0.arp_ignore=1<br/> sysctl -w net.ipv4.conf.ib1.arp_ignore=1</p> <p> Or, globally,</p> <p> sysctl -w net.ipv4.conf.all.arp_ignore=1</p> <p> To learn more about the arp_ignore parameter, see<br/> Documentation/networking/ip-sysctl.txt.<br/> Note that distributions have the means to make kernel parameters persistent.</p> <p>Could you please try a few more things:<br/> 0. ifconfig output from both client and server<br/> 1. regular ping both interfaces of server from client, and collect "arp -na" on client<br/> 2. please collect "arp -na" and "ip neighbor show" from the client<br/> 3. assume the server has ib0(10.50.0.7), ib1(10.50.1.7), please run this and collect output:</p> <p> client $ tcpdump -c 3 -nni ib0 arp<br/> server $ tcpdump -c 3 -nni ib0 arp<br/> client $ arping -c 3 -D -I ib0 10.50.0.7</p> <p>4. collect output of:</p> <p> client $ tcpdump -c 3 -nni ib0 arp<br/> server $ tcpdump -c 3 -nni ib1 arp<br/> client $ arping -c 3 -D -I ib0 10.50.1.7</p> <p>5. please clear arp cache on client by "arp -d servername 10.50.0.7" and "arp -d servername 10.50.1.7" (or you have a better way)</p> <p>6. ping both interfaces of server and collect "arp -na" again on client</p> <p>7. reload o2iblnd on both client and server to see whether this can work</p> <p>8. if it still can't help, please add those sysctl to /etc/sysctl.conf and reboot client and server to see how it works</p> <p>Anyway, if these things can help, I don't really want previous logs/information, but if they still can't, we have to get those information for the next step survey.</p> <p>Thanks<br/> Liang</p> <p>Hi,</p> <p>Finally I could make what you wrote in your previous comment:</p> <p>In fact, we are having problems with the arp table. When we run 'arping' from client to server we get not only one arp address but 2, so the client is not able to have a proper arp table.</p> <p>15:04:36.258634 ARP, Request who-has 60.64.0.31 (00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff) tell 0.0.0.0, length 56<br/> 15:04:36.258751 ARP, Reply 60.64.0.31 is-at 80:00:00:48:fe:80:00:00:00:00:00:00:00:02:c9:03:00:04:89:c5, length 56<br/> 15:04:36.258773 ARP, Reply 60.64.0.31 is-at 80:00:00:48:fe:80:00:00:00:00:00:00:00:02:c9:03:00:0a:ba:57, length 56</p> <p>If we set 'arp_ignore=1' then one of the interfaces don't reply to standard ping requests (it's not always the same interface), so we still have the problem. It's like if only one interface can reply.</p> <p>Also, as a consequence, routing is not well done from server to client as the client is connecting to a bad interface and not to the good one. It's definitively an OFED bug, do you agree?</p> <p>So, by the moment, the only available solution is to have an alias ip on client and 2 different subnets, but this cannot be easy to do in a cluster with more than 4000 nodes...</p> <p>sorry I didn't see the last comment, I agree that it's not something we can fix inside o2iblnd and I don't have a better suggestion to help you make it easier, <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Liang</p> <p>I think we agree this is an bug in OFED stack. We can close this bug and up to OFED resolution we'll use an ib alias device on each client.</p> <p>Thanks,</p> <p>As per Bull, they have a workaround for this OFED bug</p> <p>Does anyone know what OFED the bug was fixed, if ever? Diego, did you or anyone report this bug to OFA?</p> Related LU-12132 Development Epic lnet lustre-2_0 multirail ping Rank 1|hzvsm7: Rank (Obsolete) 8540 Severity