https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14454 Lustre <p>I built 2 new LNET routers and added them to our LNET env. The version of software of OS/LNET/MLNX OFED is exactly the same as 2 other existing lnet routers in this location. I added lnet routes on the 2 Lustre filesystem we have in this physical location to point to the 2 new lnet routers. I tested one client in another data center we have by adding the 2 lnet routes on the client to point to the new lnet routers. The client could read and write fine. The next day we were having issues from various clients with access to the 2 Lustre FS I had set LNET routes on previously. We ended up removing all the lnet routes to the 2 new lnet routers on the Lustre filesystems and things started to working again. So we ended up removing the 2 new lnet routers from our LNET env.</p> <p>LNET routers are running lnet 2.12.4, Lustre FS are lustre 2.12.3 and a very old version </p> <p>We have not experienced this before and was wondering it there is a specific procedure we have to follow to add new lnet routers in our environment ? </p> <p>The messages we were seeing on the lustre FS were for example:<br/> Feb 19 09:03:17 boslfs02mds01 kernel: LNetError: 6413:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 9 previous similar messages<br/> Feb 19 09:14:32 boslfs02mds01 kernel: LNetError: 6413:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.242.46.216@o2ib1 added to recovery queue. Health = 900<br/> Feb 19 09:14:32 boslfs02mds01 kernel: LNetError: 6413:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 9 previous similar messages<br/> Feb 19 09:25:47 boslfs02mds01 kernel: LNetError: 6413:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.242.46.217@o2ib1 added to recovery queue. Health = 900</p> <p>We were getting messages like the above for all 4 of the lnet routers, both the existing and the 2 new ones that were added.</p> <p>Also the hardware configuration of the 2 new LNET router is different. They have a dual port ConnectX-4 card running in ethernet mode at 10G and the 2 ports are LACP bonded, with a CX5 card for 100 rate IB. The older LNET routers have a ConnectX-4 IB card with IB rate 100 and a traditional 10G ethernet card with 2 10G and are LACP bonded. Not sure if this matters, but I wanted to mention it.</p> All Centos 7.x. Hardware is either Dell or Lenovo. IB infrastructure is EDR IB with a MSB7800 switch. MLNX OFED is 4.7-1.0.0.1 for lnet routers LU-14454

LNET routers added - then access issues with Lustre storage

Bug Major Reopened Unresolved Serguei Smirnov Michael Ethier Fri, 19 Feb 2021 16:25:29 +0000 Wed, 5 May 2021 17:03:47 +0000 Lustre 2.12.3 Lustre 2.12.4 0 4 <p>Actually, those recoveryq messages I mentioned above may not be an issue in regards to the access problem I described. I can see those messages in /var/log/messages on one of the lustre FS - at much earlier times, like weeks ago - before we experienced the access issue.</p> <p>Cyril</p> <p>Could you please assist with this one?</p> <p>Thanks</p> <p>Peter</p> <p>Hello,<br/> Anything else you need ?<br/> Thanks,<br/> Mike</p> <p>Hello Michael,</p> <p>sorry for the late answer, I had to take unexpected leave.</p> <p>Could you provide the outputs of the following commands from the servers, the routers and several clients?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lnetctl peer show -v 4 lnetctl net show -v 4 lnetctl global show lnetctl route show</pre> </div></div> <p> </p> <p>Thank you.</p> <p>Cyril.</p> <p>Hi Cyril, no worries. So those 2 lnet routers with issues do not have their lnet active to I can't run the commands. If we add routes to the Lustre storage and to some clients that's when we run into issues with accessing the Lustre storage afterwards.</p> <p><span class="error">&#91;root@boslnet03 ~&#93;</span># lnetctl peer show -v 4<br/> ^C<br/> <span class="error">&#91;root@boslnet03 ~&#93;</span># lnetctl net show<br/> net:</p> <ul class="alternate" type="square"> <li>net type: lo<br/> local NI(s):</li> <li>nid: 0@lo<br/> status: up</li> <li>net type: tcp1<br/> local NI(s):</li> <li>nid: 10.242.62.213@tcp1<br/> status: down<br/> interfaces:<br/> 0: bond0</li> <li>net type: o2ib1<br/> local NI(s):</li> <li>nid: 10.242.46.227@o2ib1<br/> status: down<br/> interfaces:<br/> 0: ib0</li> </ul> <p>Thanks,<br/> Mike</p> <p>Hello Michael,</p> <p>I do not get your "those 2 lnet routers with issues do not have their lnet active to I can't run the commands.". You mean you cannot mess up your working configuration by enabling them?</p> <p>I will be difficult to diagnose with little information… To see what is going on I need to have all requested details with the exact commands I provided. If you cannot mess up with your production environment, could you provide the commands you used to configure everything, with the details of your network (nids of the servers, the routers, the clients), and all available information for servers and clients.</p> <p>Thank you.</p> <p>Cyril.</p> <p> </p> <p>Hi Cyril,<br/> Sorry for the delay. Yes when we added the 2 new lnet routers to some of our lustre storage via lnet routes and set a few client with routes to access the storage via the lnet routers we had problems accessing the storage from other clients. I will gather the details you have ask for and reply back.<br/> Thanks,<br/> Mike</p> <p>Hi Cyril,</p> <p>Your requested commands on a working lnet router:<br/> lnetctl peer show -v 4 pegs the cpu to 100% and never returns on a lnet router.</p> <p><span class="error">&#91;root@boslnet01 ~&#93;</span># lnetctl net show -v 4<br/> net:</p> <ul class="alternate" type="square"> <li>net type: lo<br/> local NI(s):</li> <li>nid: 0@lo<br/> status: up<br/> statistics:<br/> send_count: 0<br/> recv_count: 0<br/> drop_count: 0<br/> sent_stats:<br/> put: 0<br/> get: 0<br/> reply: 0<br/> ack: 0<br/> hello: 0<br/> received_stats:<br/> put: 0<br/> get: 0<br/> reply: 0<br/> ack: 0<br/> hello: 0<br/> dropped_stats:<br/> put: 0<br/> get: 0<br/> reply: 0<br/> ack: 0<br/> hello: 0<br/> health stats:<br/> health value: 0<br/> interrupts: 0<br/> dropped: 0<br/> aborted: 0<br/> no route: 0<br/> timeouts: 0<br/> error: 0<br/> tunables:<br/> peer_timeout: 0<br/> peer_credits: 0<br/> peer_buffer_credits: 0<br/> credits: 0<br/> dev cpt: 0<br/> tcp bonding: 0<br/> CPT: "<span class="error">&#91;0,1&#93;</span>"</li> <li>net type: tcp1<br/> local NI(s):</li> <li>nid: 10.242.62.227@tcp1<br/> status: up<br/> interfaces:<br/> 0: bond0<br/> statistics:<br/> send_count: 1263198586<br/> recv_count: 446196203<br/> drop_count: 827<br/> sent_stats:<br/> put: 1211118273<br/> get: 52080311<br/> reply: 2<br/> ack: 0<br/> hello: 0<br/> received_stats:<br/> put: 380078000<br/> get: 48000898<br/> reply: 3709225<br/> ack: 14408080<br/> hello: 0<br/> dropped_stats:<br/> put: 707<br/> get: 117<br/> reply: 3<br/> ack: 0<br/> hello: 0<br/> health stats:<br/> health value: 1000<br/> interrupts: 0<br/> dropped: 0<br/> aborted: 0<br/> no route: 0<br/> timeouts: 454<br/> error: 5397<br/> tunables:<br/> peer_timeout: 180<br/> peer_credits: 8<br/> peer_buffer_credits: 0<br/> credits: 256<br/> dev cpt: -1<br/> tcp bonding: 0<br/> CPT: "<span class="error">&#91;0,1&#93;</span>"</li> <li>net type: tcp2<br/> local NI(s):</li> <li>nid: 10.242.107.16@tcp2<br/> status: up<br/> interfaces:<br/> 0: p3p2<br/> statistics:<br/> send_count: 61007347<br/> recv_count: 60503869<br/> drop_count: 5<br/> sent_stats:<br/> put: 56550435<br/> get: 4456912<br/> reply: 0<br/> ack: 0<br/> hello: 0<br/> received_stats:<br/> put: 12820596<br/> get: 52672<br/> reply: 4359334<br/> ack: 43271267<br/> hello: 0<br/> dropped_stats:<br/> put: 4<br/> get: 0<br/> reply: 1<br/> ack: 0<br/> hello: 0<br/> health stats:<br/> health value: 1000<br/> interrupts: 0<br/> dropped: 0<br/> aborted: 0<br/> no route: 0<br/> timeouts: 0<br/> error: 0<br/> tunables:<br/> peer_timeout: 180<br/> peer_credits: 8<br/> peer_buffer_credits: 0<br/> credits: 256<br/> dev cpt: 0<br/> tcp bonding: 0<br/> CPT: "<span class="error">&#91;0,1&#93;</span>"</li> <li>net type: o2ib1<br/> local NI(s):</li> <li>nid: 10.242.46.216@o2ib1<br/> status: up<br/> interfaces:<br/> 0: ib0<br/> statistics:<br/> send_count: 459118144<br/> recv_count: 1276624005<br/> drop_count: 457262<br/> sent_stats:<br/> put: 392898596<br/> get: 471643<br/> reply: 8068558<br/> ack: 57679347<br/> hello: 0<br/> received_stats:<br/> put: 1267668708<br/> get: 8955285<br/> reply: 12<br/> ack: 0<br/> hello: 0<br/> dropped_stats:<br/> put: 457250<br/> get: 0<br/> reply: 12<br/> ack: 0<br/> hello: 0<br/> health stats:<br/> health value: 1000<br/> interrupts: 0<br/> dropped: 62<br/> aborted: 0<br/> no route: 0<br/> timeouts: 350<br/> error: 0<br/> tunables:<br/> peer_timeout: 180<br/> peer_credits: 8<br/> peer_buffer_credits: 0<br/> credits: 256<br/> peercredits_hiw: 4<br/> map_on_demand: 0<br/> concurrent_sends: 8<br/> fmr_pool_size: 512<br/> fmr_flush_trigger: 384<br/> fmr_cache: 1<br/> ntx: 512<br/> conns_per_peer: 1<br/> lnd tunables:<br/> dev cpt: 1<br/> tcp bonding: 0<br/> CPT: "<span class="error">&#91;0,1&#93;</span>"<br/> <span class="error">&#91;root@boslnet01 ~&#93;</span># lnetctl global show<br/> global:<br/> numa_range: 0<br/> max_intf: 200<br/> discovery: 0<br/> drop_asym_route: 0<br/> retry_count: 0<br/> transaction_timeout: 50<br/> health_sensitivity: 0<br/> recovery_interval: 1<br/> <span class="error">&#91;root@boslnet01 ~&#93;</span># lnetctl route show<br/> <span class="error">&#91;root@boslnet01 ~&#93;</span>#</li> </ul> <p>Your commands on a non-working lnet router:<br/> <span class="error">&#91;root@boslnet03 ~&#93;</span># lnetctl net show -v 4<br/> net:</p> <ul class="alternate" type="square"> <li>net type: lo<br/> local NI(s):</li> <li>nid: 0@lo<br/> status: up<br/> statistics:<br/> send_count: 0<br/> recv_count: 0<br/> drop_count: 0<br/> sent_stats:<br/> put: 0<br/> get: 0<br/> reply: 0<br/> ack: 0<br/> hello: 0<br/> received_stats:<br/> put: 0<br/> get: 0<br/> reply: 0<br/> ack: 0<br/> hello: 0<br/> dropped_stats:<br/> put: 0<br/> get: 0<br/> reply: 0<br/> ack: 0<br/> hello: 0<br/> health stats:<br/> health value: 0<br/> interrupts: 0<br/> dropped: 0<br/> aborted: 0<br/> no route: 0<br/> timeouts: 0<br/> error: 0<br/> tunables:<br/> peer_timeout: 0<br/> peer_credits: 0<br/> peer_buffer_credits: 0<br/> credits: 0<br/> dev cpt: 0<br/> tcp bonding: 0<br/> CPT: "<span class="error">&#91;0&#93;</span>"</li> <li>net type: tcp1<br/> local NI(s):</li> <li>nid: 10.242.62.213@tcp1<br/> status: down<br/> interfaces:<br/> 0: bond0<br/> statistics:<br/> send_count: 9937720<br/> recv_count: 13722<br/> drop_count: 276772<br/> sent_stats:<br/> put: 9913812<br/> get: 23902<br/> reply: 6<br/> ack: 0<br/> hello: 0<br/> received_stats:<br/> put: 7145<br/> get: 1445<br/> reply: 5121<br/> ack: 11<br/> hello: 0<br/> dropped_stats:<br/> put: 276766<br/> get: 0<br/> reply: 6<br/> ack: 0<br/> hello: 0<br/> health stats:<br/> health value: 1000<br/> interrupts: 0<br/> dropped: 0<br/> aborted: 0<br/> no route: 0<br/> timeouts: 738<br/> error: 3<br/> tunables:<br/> peer_timeout: 180<br/> peer_credits: 8<br/> peer_buffer_credits: 0<br/> credits: 256<br/> dev cpt: -1<br/> tcp bonding: 0<br/> CPT: "<span class="error">&#91;0&#93;</span>"</li> <li>net type: o2ib1<br/> local NI(s):</li> <li>nid: 10.242.46.227@o2ib1<br/> status: down<br/> interfaces:<br/> 0: ib0<br/> statistics:<br/> send_count: 217189<br/> recv_count: 10141187<br/> drop_count: 10<br/> sent_stats:<br/> put: 7145<br/> get: 204912<br/> reply: 5121<br/> ack: 11<br/> hello: 0<br/> received_stats:<br/> put: 9913812<br/> get: 227369<br/> reply: 6<br/> ack: 0<br/> hello: 0<br/> dropped_stats:<br/> put: 10<br/> get: 0<br/> reply: 0<br/> ack: 0<br/> hello: 0<br/> health stats:<br/> health value: 1000<br/> interrupts: 0<br/> dropped: 0<br/> aborted: 0<br/> no route: 0<br/> timeouts: 0<br/> error: 0<br/> tunables:<br/> peer_timeout: 180<br/> peer_credits: 8<br/> peer_buffer_credits: 0<br/> credits: 256<br/> peercredits_hiw: 4<br/> map_on_demand: 0<br/> concurrent_sends: 8<br/> fmr_pool_size: 512<br/> fmr_flush_trigger: 384<br/> fmr_cache: 1<br/> ntx: 512<br/> conns_per_peer: 1<br/> lnd tunables:<br/> dev cpt: 0<br/> tcp bonding: 0<br/> CPT: "<span class="error">&#91;0&#93;</span>"<br/> <span class="error">&#91;root@boslnet03 ~&#93;</span># lnetctl global show<br/> global:<br/> numa_range: 0<br/> max_intf: 200<br/> discovery: 0<br/> drop_asym_route: 0<br/> retry_count: 0<br/> transaction_timeout: 50<br/> health_sensitivity: 0<br/> recovery_interval: 1<br/> <span class="error">&#91;root@boslnet03 ~&#93;</span># lnetctl route show<br/> <span class="error">&#91;root@boslnet03 ~&#93;</span>#</li> </ul> <p>Your commands on a Lustre storage node (MDS):<br/> <span class="error">&#91;root@boslfs02mds01 ~&#93;</span># lnetctl net show -v 4<br/> net:</p> <ul class="alternate" type="square"> <li>net type: lo<br/> local NI(s):</li> <li>nid: 0@lo<br/> status: up<br/> statistics:<br/> send_count: 2888768<br/> recv_count: 2888764<br/> drop_count: 4<br/> sent_stats:<br/> put: 2888768<br/> get: 0<br/> reply: 0<br/> ack: 0<br/> hello: 0<br/> received_stats:<br/> put: 2190548<br/> get: 0<br/> reply: 0<br/> ack: 698216<br/> hello: 0<br/> dropped_stats:<br/> put: 4<br/> get: 0<br/> reply: 0<br/> ack: 0<br/> hello: 0<br/> health stats:<br/> health value: 0<br/> interrupts: 0<br/> dropped: 0<br/> aborted: 0<br/> no route: 0<br/> timeouts: 0<br/> error: 0<br/> tunables:<br/> peer_timeout: 0<br/> peer_credits: 0<br/> peer_buffer_credits: 0<br/> credits: 0<br/> dev cpt: 0<br/> tcp bonding: 0<br/> CPT: "<span class="error">&#91;0,1,2,3&#93;</span>"</li> <li>net type: o2ib1<br/> local NI(s):</li> <li>nid: 10.242.46.233@o2ib1<br/> status: up<br/> interfaces:<br/> 0: ib0<br/> statistics:<br/> send_count: 120142564<br/> recv_count: 120852294<br/> drop_count: 22158<br/> sent_stats:<br/> put: 120091641<br/> get: 50923<br/> reply: 0<br/> ack: 0<br/> hello: 0<br/> received_stats:<br/> put: 119667645<br/> get: 13605<br/> reply: 37318<br/> ack: 1133726<br/> hello: 0<br/> dropped_stats:<br/> put: 18448<br/> get: 0<br/> reply: 0<br/> ack: 3710<br/> hello: 0<br/> health stats:<br/> health value: 1000<br/> interrupts: 0<br/> dropped: 1804948<br/> aborted: 0<br/> no route: 0<br/> timeouts: 0<br/> error: 0<br/> tunables:<br/> peer_timeout: 180<br/> peer_credits: 8<br/> peer_buffer_credits: 0<br/> credits: 256<br/> peercredits_hiw: 4<br/> map_on_demand: 0<br/> concurrent_sends: 8<br/> fmr_pool_size: 512<br/> fmr_flush_trigger: 384<br/> fmr_cache: 1<br/> ntx: 512<br/> conns_per_peer: 1<br/> lnd tunables:<br/> dev cpt: 3<br/> tcp bonding: 0<br/> CPT: "<span class="error">&#91;0,1,2,3&#93;</span>"<br/> <span class="error">&#91;root@boslfs02mds01 ~&#93;</span># lnetctl global show<br/> global:<br/> numa_range: 0<br/> max_intf: 200<br/> discovery: 0<br/> drop_asym_route: 0<br/> retry_count: 0<br/> transaction_timeout: 50<br/> health_sensitivity: 100<br/> recovery_interval: 1<br/> <span class="error">&#91;root@boslfs02mds01 ~&#93;</span># lnetctl route show<br/> route:</li> <li>net: tcp1<br/> gateway: 10.242.46.216@o2ib1</li> <li>net: tcp1<br/> gateway: 10.242.46.217@o2ib1</li> <li>net: tcp2<br/> gateway: 10.242.46.217@o2ib1</li> <li>net: tcp2<br/> gateway: 10.242.46.216@o2ib1</li> </ul> <p>Details of existing working lnet router:<br/> <span class="error">&#91;root@boslnet01 ~&#93;</span># dkms status<br/> lustre-client, 2.12.4, 3.10.0-1062.18.1.el7.x86_64, x86_64: installed<br/> <span class="error">&#91;root@boslnet01 ~&#93;</span># more /etc/modprobe.d/lustre.conf <br/> options lnet networks="tcp1(bond0), tcp2(p3p2), o2ib1(ib0)"<br/> options lnet forwarding="enabled"<br/> options lnet lnet_peer_discovery_disabled=1<br/> options lnet lnet_health_sensitivity=0<br/> <span class="error">&#91;root@boslnet01 ~&#93;</span># lnetctl net show<br/> net:</p> <ul class="alternate" type="square"> <li>net type: lo<br/> local NI(s):</li> <li>nid: 0@lo<br/> status: up</li> <li>net type: tcp1<br/> local NI(s):</li> <li>nid: 10.242.62.227@tcp1<br/> status: up<br/> interfaces:<br/> 0: bond0</li> <li>net type: tcp2<br/> local NI(s):</li> <li>nid: 10.242.107.16@tcp2<br/> status: up<br/> interfaces:<br/> 0: p3p2</li> <li>net type: o2ib1<br/> local NI(s):</li> <li>nid: 10.242.46.216@o2ib1<br/> status: up<br/> interfaces:<br/> 0: ib0<br/> <span class="error">&#91;root@boslnet01 ~&#93;</span># more /etc/redhat-release <br/> CentOS Linux release 7.7.1908 (Core)<br/> <span class="error">&#91;root@boslnet01 ~&#93;</span># lspci |grep -i mell<br/> 81:00.0 Infiniband controller: Mellanox Technologies MT27700 Family <span class="error">&#91;ConnectX-4&#93;</span><br/> <span class="error">&#91;root@boslnet01 ~&#93;</span># ibstat<br/> CA 'mlx5_0'<br/> CA type: MT4115<br/> Number of ports: 1<br/> Firmware version: 12.17.2052<br/> Hardware version: 0<br/> Node GUID: 0x98039b0300bec2c8<br/> System image GUID: 0x98039b0300bec2c8<br/> Port 1:<br/> State: Active<br/> Physical state: LinkUp<br/> Rate: 100<br/> Base lid: 44<br/> LMC: 0<br/> SM lid: 1<br/> Capability mask: 0x2651e848<br/> Port GUID: 0x98039b0300bec2c8<br/> Link layer: InfiniBand<br/> <span class="error">&#91;root@boslnet01 ~&#93;</span># ifconfig<br/> bond0: flags=5187&lt;UP,BROADCAST,RUNNING,MASTER,MULTICAST&gt; mtu 1500<br/> inet 10.242.62.227 netmask 255.255.255.0 broadcast 10.242.62.255<br/> inet6 fe80::a236:9fff:fe8d:9774 prefixlen 64 scopeid 0x20&lt;link&gt;<br/> ether a0:36:9f:8d:97:74 txqueuelen 1000 (Ethernet)<br/> RX packets 5311208385 bytes 4185414327252 (3.8 TiB)<br/> RX errors 0 dropped 0 overruns 0 frame 0<br/> TX packets 11038342648 bytes 14227372363039 (12.9 TiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</li> </ul> <p>ib0: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt; mtu 2044<br/> inet 10.242.46.216 netmask 255.255.252.0 broadcast 10.242.47.255<br/> inet6 fe80::9a03:9b03:be:c2c8 prefixlen 64 scopeid 0x20&lt;link&gt;<br/> Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).<br/> infiniband 20:00:00:68:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 txqueuelen 256 (InfiniBand)<br/> RX packets 29806 bytes 4627169 (4.4 MiB)<br/> RX errors 0 dropped 0 overruns 0 frame 0<br/> TX packets 837 bytes 50408 (49.2 KiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</p> <p>lo: flags=73&lt;UP,LOOPBACK,RUNNING&gt; mtu 65536<br/> inet 127.0.0.1 netmask 255.0.0.0<br/> inet6 ::1 prefixlen 128 scopeid 0x10&lt;host&gt;<br/> loop txqueuelen 1000 (Local Loopback)<br/> RX packets 743598 bytes 71604808 (68.2 MiB)<br/> RX errors 0 dropped 0 overruns 0 frame 0<br/> TX packets 743598 bytes 71604808 (68.2 MiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</p> <p>p1p1: flags=6211&lt;UP,BROADCAST,RUNNING,SLAVE,MULTICAST&gt; mtu 1500<br/> ether a0:36:9f:8d:97:74 txqueuelen 1000 (Ethernet)<br/> RX packets 2983098310 bytes 2267051063818 (2.0 TiB)<br/> RX errors 0 dropped 0 overruns 0 frame 0<br/> TX packets 6235321147 bytes 8498451034677 (7.7 TiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</p> <p>p1p2: flags=6211&lt;UP,BROADCAST,RUNNING,SLAVE,MULTICAST&gt; mtu 1500<br/> ether a0:36:9f:8d:97:74 txqueuelen 1000 (Ethernet)<br/> RX packets 2328110076 bytes 1918363263500 (1.7 TiB)<br/> RX errors 0 dropped 0 overruns 0 frame 0<br/> TX packets 4803021510 bytes 5728921330120 (5.2 TiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</p> <p>p3p1: flags=4099&lt;UP,BROADCAST,MULTICAST&gt; mtu 1500<br/> ether 3c:fd:fe:16:01:88 txqueuelen 1000 (Ethernet)<br/> RX packets 0 bytes 0 (0.0 B)<br/> RX errors 0 dropped 0 overruns 0 frame 0<br/> TX packets 0 bytes 0 (0.0 B)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</p> <p>p3p2: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt; mtu 1500<br/> inet 10.242.107.16 netmask 255.255.255.192 broadcast 10.242.107.63<br/> inet6 fe80::3efd:feff:fe16:18a prefixlen 64 scopeid 0x20&lt;link&gt;<br/> ether 3c:fd:fe:16:01:8a txqueuelen 1000 (Ethernet)<br/> RX packets 3292939462 bytes 4737754838918 (4.3 TiB)<br/> RX errors 0 dropped 667 overruns 0 frame 0<br/> TX packets 31792664489 bytes 47939135623196 (43.6 TiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</p> <p>Details of one of our lustre storage servers:</p> <p><span class="error">&#91;root@boslfs02mds01 ~&#93;</span># rpm -qa |grep lustre<br/> kernel-3.10.0-1062.1.1.el7_lustre.x86_64<br/> kmod-lustre-2.12.3-1.el7.x86_64<br/> kmod-lustre-osd-ldiskfs-2.12.3-1.el7.x86_64<br/> kernel-devel-3.10.0-1062.1.1.el7_lustre.x86_64<br/> lustre-osd-zfs-mount-2.12.3-1.el7.x86_64<br/> lustre-2.12.3-1.el7.x86_64<br/> lustre-zfs-dkms-2.12.3-1.el7.noarch<br/> lustre-resource-agents-2.12.3-1.el7.x86_64<br/> lustre-ldiskfs-zfs-5.0.0-1.el7.x86_64<br/> lustre-osd-ldiskfs-mount-2.12.3-1.el7.x86_64<br/> kmod-spl-3.10.0-1062.1.1.el7_lustre.x86_64-0.7.13-1.el7.x86_64<br/> <span class="error">&#91;root@boslfs02mds01 ~&#93;</span># more /etc/redhat-release <br/> CentOS Linux release 7.7.1908 (Core)<br/> <span class="error">&#91;root@boslfs02mds01 ~&#93;</span># more /etc/modprobe.d/lustre.conf <br/> options lnet networks="o2ib1(ib0)" routes="tcp1 10.242.46.<span class="error">&#91;216-217&#93;</span>@o2ib1; tcp2 10.242.46.<span class="error">&#91;216-217&#93;</span>@o2ib1"<br/> options lnet lnet_transaction_timeout=50 lnet_retry_count=0 lnet_peer_discovery_disabled=1<br/> <span class="error">&#91;root@boslfs02mds01 ~&#93;</span># lnetctl route show<br/> route:</p> <ul class="alternate" type="square"> <li>net: tcp1<br/> gateway: 10.242.46.216@o2ib1</li> <li>net: tcp1<br/> gateway: 10.242.46.217@o2ib1</li> <li>net: tcp2<br/> gateway: 10.242.46.217@o2ib1</li> <li>net: tcp2<br/> gateway: 10.242.46.216@o2ib1<br/> <span class="error">&#91;root@boslfs02mds01 ~&#93;</span># lspci |grep -i mell<br/> d8:00.0 Infiniband controller: Mellanox Technologies MT27800 Family <span class="error">&#91;ConnectX-5&#93;</span><br/> <span class="error">&#91;root@boslfs02mds01 ~&#93;</span># lspci |grep -i eth<br/> 01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)<br/> 01:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)<br/> 19:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)<br/> 19:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)<br/> <span class="error">&#91;root@boslfs02mds01 ~&#93;</span># ibstat<br/> CA 'mlx5_0'<br/> CA type: MT4119<br/> Number of ports: 1<br/> Firmware version: 16.25.1020<br/> Hardware version: 0<br/> Node GUID: 0xb8599f030005a4ec<br/> System image GUID: 0xb8599f030005a4ec<br/> Port 1:<br/> State: Active<br/> Physical state: LinkUp<br/> Rate: 100<br/> Base lid: 20<br/> LMC: 0<br/> SM lid: 1<br/> Capability mask: 0x2651e848<br/> Port GUID: 0xb8599f030005a4ec<br/> Link layer: InfiniBand<br/> <span class="error">&#91;root@boslfs02mds01 ~&#93;</span># ifconfig<br/> bond0: flags=5187&lt;UP,BROADCAST,RUNNING,MASTER,MULTICAST&gt; mtu 1500<br/> inet 10.242.62.139 netmask 255.255.255.0 broadcast 10.242.62.255<br/> inet6 fe80::e643:4bff:fe21:6750 prefixlen 64 scopeid 0x20&lt;link&gt;<br/> ether e4:43:4b:21:67:50 txqueuelen 1000 (Ethernet)<br/> RX packets 244991196 bytes 40288519866 (37.5 GiB)<br/> RX errors 0 dropped 689357 overruns 0 frame 0<br/> TX packets 243506307 bytes 25816873448 (24.0 GiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</li> </ul> <p>em1: flags=6211&lt;UP,BROADCAST,RUNNING,SLAVE,MULTICAST&gt; mtu 1500<br/> ether e4:43:4b:21:67:50 txqueuelen 1000 (Ethernet)<br/> RX packets 124720954 bytes 20746393497 (19.3 GiB)<br/> RX errors 0 dropped 19702 overruns 0 frame 0<br/> TX packets 121422150 bytes 12632871392 (11.7 GiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</p> <p>em2: flags=6211&lt;UP,BROADCAST,RUNNING,SLAVE,MULTICAST&gt; mtu 1500<br/> ether e4:43:4b:21:67:50 txqueuelen 1000 (Ethernet)<br/> RX packets 120270244 bytes 19542126677 (18.2 GiB)<br/> RX errors 0 dropped 315129 overruns 0 frame 0<br/> TX packets 122084164 bytes 13184003240 (12.2 GiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</p> <p>em3: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt; mtu 1500<br/> inet 10.0.0.139 netmask 255.255.255.0 broadcast 10.0.0.255<br/> inet6 fe80::e643:4bff:fe21:6754 prefixlen 64 scopeid 0x20&lt;link&gt;<br/> ether e4:43:4b:21:67:54 txqueuelen 1000 (Ethernet)<br/> RX packets 228132 bytes 31718009 (30.2 MiB)<br/> RX errors 0 dropped 0 overruns 0 frame 0<br/> TX packets 328790 bytes 52526191 (50.0 MiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0<br/> device memory 0x92980000-929fffff </p> <p>em4: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt; mtu 1500<br/> inet 10.242.112.176 netmask 255.255.255.128 broadcast 10.242.112.255<br/> inet6 fe80::e643:4bff:fe21:6755 prefixlen 64 scopeid 0x20&lt;link&gt;<br/> ether e4:43:4b:21:67:55 txqueuelen 1000 (Ethernet)<br/> RX packets 26539 bytes 1600620 (1.5 MiB)<br/> RX errors 0 dropped 0 overruns 0 frame 0<br/> TX packets 210 bytes 20138 (19.6 KiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0<br/> device memory 0x92900000-9297ffff </p> <p>ib0: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt; mtu 2044<br/> inet 10.242.46.233 netmask 255.255.252.0 broadcast 10.242.47.255<br/> inet6 fe80::ba59:9f03:5:a4ec prefixlen 64 scopeid 0x20&lt;link&gt;<br/> Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).<br/> infiniband 20:00:07:EB:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 txqueuelen 256 (InfiniBand)<br/> RX packets 10537 bytes 1677139 (1.5 MiB)<br/> RX errors 0 dropped 0 overruns 0 frame 0<br/> TX packets 14 bytes 984 (984.0 B)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</p> <p>idrac: flags=67&lt;UP,BROADCAST,RUNNING&gt; mtu 1500<br/> inet6 fe80::4ed9:8fff:fe7c:c104 prefixlen 64 scopeid 0x20&lt;link&gt;<br/> ether 4c:d9:8f:7c:c1:04 txqueuelen 1000 (Ethernet)<br/> RX packets 690330 bytes 56738080 (54.1 MiB)<br/> RX errors 0 dropped 0 overruns 0 frame 0<br/> TX packets 808171 bytes 76845763 (73.2 MiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</p> <p>lo: flags=73&lt;UP,LOOPBACK,RUNNING&gt; mtu 65536<br/> inet 127.0.0.1 netmask 255.0.0.0<br/> inet6 ::1 prefixlen 128 scopeid 0x10&lt;host&gt;<br/> loop txqueuelen 1000 (Local Loopback)<br/> RX packets 20976 bytes 4281290 (4.0 MiB)<br/> RX errors 0 dropped 0 overruns 0 frame 0<br/> TX packets 20976 bytes 4281290 (4.0 MiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</p> <p>Deails of a lnet router that doesn't work:</p> <p><span class="error">&#91;root@boslnet03 ~&#93;</span># dkms status<br/> lustre-client, 2.12.4, 3.10.0-1062.18.1.el7.x86_64, x86_64: installed<br/> <span class="error">&#91;root@boslnet03 ~&#93;</span># more /etc/modprobe.d/lustre.conf <br/> options lnet networks="tcp1(bond0), o2ib1(ib0)"<br/> options lnet forwarding="enabled"<br/> options lnet lnet_peer_discovery_disabled=1<br/> options lnet lnet_health_sensitivity=0<br/> <span class="error">&#91;root@boslnet03 ~&#93;</span># lnetctl net show<br/> net:</p> <ul class="alternate" type="square"> <li>net type: lo<br/> local NI(s):</li> <li>nid: 0@lo<br/> status: up</li> <li>net type: tcp1<br/> local NI(s):</li> <li>nid: 10.242.62.213@tcp1<br/> status: down<br/> interfaces:<br/> 0: bond0</li> <li>net type: o2ib1<br/> local NI(s):</li> <li>nid: 10.242.46.227@o2ib1<br/> status: down<br/> interfaces:<br/> 0: ib0<br/> <span class="error">&#91;root@boslnet03 ~&#93;</span># more /etc/redhat-release <br/> CentOS Linux release 7.7.1908 (Core)<br/> <span class="error">&#91;root@boslnet03 ~&#93;</span># lspci |grep -i mell<br/> 08:00.0 Ethernet controller: Mellanox Technologies MT27710 Family <span class="error">&#91;ConnectX-4 Lx&#93;</span><br/> 08:00.1 Ethernet controller: Mellanox Technologies MT27710 Family <span class="error">&#91;ConnectX-4 Lx&#93;</span><br/> 5b:00.0 Infiniband controller: Mellanox Technologies MT27800 Family <span class="error">&#91;ConnectX-5&#93;</span><br/> <span class="error">&#91;root@boslfs02mds01 ~&#93;</span># lspci |grep -i eth<br/> 01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)<br/> 01:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)<br/> 19:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)<br/> 19:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)<br/> <span class="error">&#91;root@boslnet03 ~&#93;</span># ibstat<br/> CA 'mlx5_2'<br/> CA type: MT4119<br/> Number of ports: 1<br/> Firmware version: 16.29.1016<br/> Hardware version: 0<br/> Node GUID: 0x98039b0300b4146e<br/> System image GUID: 0x98039b0300b4146e<br/> Port 1:<br/> State: Active<br/> Physical state: LinkUp<br/> Rate: 100<br/> Base lid: 26<br/> LMC: 0<br/> SM lid: 1<br/> Capability mask: 0x2651e848<br/> Port GUID: 0x98039b0300b4146e<br/> Link layer: InfiniBand<br/> CA 'mlx5_bond_0'<br/> CA type: MT4117<br/> Number of ports: 1<br/> Firmware version: 14.29.1016<br/> Hardware version: 0<br/> Node GUID: 0x98039b0300cbcb34<br/> System image GUID: 0x98039b0300cbcb34<br/> Port 1:<br/> State: Active<br/> Physical state: LinkUp<br/> Rate: 10<br/> Base lid: 0<br/> LMC: 0<br/> SM lid: 0<br/> Capability mask: 0x00010000<br/> Port GUID: 0x9a039bfffecbcb34<br/> Link layer: Ethernet<br/> <span class="error">&#91;root@boslnet03 ~&#93;</span># ifconfig<br/> bond0: flags=5187&lt;UP,BROADCAST,RUNNING,MASTER,MULTICAST&gt; mtu 1500<br/> inet 10.242.62.213 netmask 255.255.255.0 broadcast 10.242.62.255<br/> inet6 fe80::9a03:9bff:fecb:cb34 prefixlen 64 scopeid 0x20&lt;link&gt;<br/> ether 98:03:9b:cb:cb:34 txqueuelen 1000 (Ethernet)<br/> RX packets 493962473 bytes 39340657521 (36.6 GiB)<br/> RX errors 0 dropped 0 overruns 0 frame 0<br/> TX packets 515219003 bytes 85671938800 (79.7 GiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</li> </ul> <p>enp0s20f0u1u6: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt; mtu 1500<br/> inet 169.254.95.120 netmask 255.255.255.0 broadcast 169.254.95.255<br/> inet6 fe80::3868:ddff:fe0c:29f7 prefixlen 64 scopeid 0x20&lt;link&gt;<br/> ether 3a:68:dd:0c:29:f7 txqueuelen 1000 (Ethernet)<br/> RX packets 2505736 bytes 240636075 (229.4 MiB)<br/> RX errors 1 dropped 0 overruns 0 frame 0<br/> TX packets 2501477 bytes 274965254 (262.2 MiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</p> <p>ib0: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt; mtu 2044<br/> inet 10.242.46.227 netmask 255.255.252.0 broadcast 10.242.47.255<br/> inet6 fe80::9a03:9b03:b4:146e prefixlen 64 scopeid 0x20&lt;link&gt;<br/> Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).<br/> infiniband 20:00:10:29:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 txqueuelen 256 (InfiniBand)<br/> RX packets 39137 bytes 6016144 (5.7 MiB)<br/> RX errors 0 dropped 0 overruns 0 frame 0<br/> TX packets 50 bytes 3144 (3.0 KiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</p> <p>lo: flags=73&lt;UP,LOOPBACK,RUNNING&gt; mtu 65536<br/> inet 127.0.0.1 netmask 255.0.0.0<br/> inet6 ::1 prefixlen 128 scopeid 0x10&lt;host&gt;<br/> loop txqueuelen 1000 (Local Loopback)<br/> RX packets 260 bytes 21314 (20.8 KiB)<br/> RX errors 0 dropped 0 overruns 0 frame 0<br/> TX packets 260 bytes 21314 (20.8 KiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</p> <p>p1p1: flags=6211&lt;UP,BROADCAST,RUNNING,SLAVE,MULTICAST&gt; mtu 1500<br/> ether 98:03:9b:cb:cb:34 txqueuelen 1000 (Ethernet)<br/> RX packets 276852635 bytes 24386554325 (22.7 GiB)<br/> RX errors 0 dropped 0 overruns 0 frame 0<br/> TX packets 295567994 bytes 70653344774 (65.8 GiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</p> <p>p1p2: flags=6211&lt;UP,BROADCAST,RUNNING,SLAVE,MULTICAST&gt; mtu 1500<br/> ether 98:03:9b:cb:cb:34 txqueuelen 1000 (Ethernet)<br/> RX packets 217109839 bytes 14954103262 (13.9 GiB)<br/> RX errors 0 dropped 0 overruns 0 frame 0<br/> TX packets 219651019 bytes 15018595778 (13.9 GiB)<br/> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</p> <p>Michael,</p> <p>Would it be possible to set up a live debugging session so that we can go through the procedure of adding a new router together? </p> <p>I haven't seen anything wrong in the logs you provided, but the errors mentioned in the description could be explained by mis-configuration.</p> <p>The procedure should be roughly as follows:</p> <p>1) Setup router node: configure lnet, start lnet, verify by dumping "lnetctl net show", and by "lnetctl pinging" to and from peers on both nets (tcp1, o2ib1) multiple times.</p> <p>2) Add the route on the server and on the client, list the new router as the gw to use to reach respective nets.</p> <p>3) Verify by "lnetctl pinging" multiple times across the new router (a tcp1 client to server and back).</p> <p>4) Verify by "lnetctl pinging" multiple times across the new router using a client on tcp2 (from the logs provided, the new router handles only tcp1, so let's verify that tcp2 is still good).</p> <p>5) If all looks good so far, try mounting</p> <p>Hi, i'm attaching 2 logs files per Serguei's request. Thanks, Mike. <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/38051/38051_log.txt" title="log.txt attached to LU-14454">log.txt^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/38052/38052_log1.txt" title="log1.txt attached to LU-14454">log1.txt^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> </p> <p>During the today's call we found out that the new router's IP may need to be added to the access control list for a group of clients. Regular ping from the client to the router was going through, but lnetctl ping was not. Because lnetctl ping was part of the procedure we used earlier, failed lnetctl ping we're seeing now may not explain the behaviour we were seeing before. We'll proceed once the ACL issue is out of the way.</p> <p>I verified that from the client we can lnet ping the 2 new lnet routers now. There were ACLs blocking access.<br/> <span class="error">&#91;root@holylogin01 ~&#93;</span># lnetctl ping 10.242.62.214@tcp1<br/> ping:</p> <ul class="alternate" type="square"> <li>primary nid: 10.242.62.214@tcp1<br/> Multi-Rail: False<br/> peer ni:</li> <li>nid: 10.242.62.214@tcp1</li> <li>nid: 10.242.46.228@o2ib1<br/> <span class="error">&#91;root@holylogin01 ~&#93;</span># lnetctl ping 10.242.62.213@tcp1<br/> ping:</li> <li>primary nid: 10.242.62.213@tcp1<br/> Multi-Rail: False<br/> peer ni:</li> <li>nid: 10.242.62.213@tcp1</li> <li>nid: 10.242.46.227@o2ib1</li> </ul> <p>It has been verified that network ACLs were causing the issue. We have successfully added a new LNET router and Lustre FS access seems to be fine now. This ticket can be closed. Thanks to Serguei for all his help, much appreciated.</p> <p>Great - thanks for the update</p> <p>Michael reported later yesterday via e-mail that the clients which didn't get the route to the new gateway setup had issues with accessing the FS. This should not have happened unless asymmetric routes are configured to be dropped on the clients.</p> <p>Michael recently reported via email:</p> <p><em>"I wanted to let you know we finally added those 2 lnet routers we were working previously, globally to our cluster in Holyoke/boston and they are now running in production.</em></p> <p><em>It appears the procedure requires that you add the lnet routes across all the clients that are mounting the storage on the other side of the routers and then add lnet routes</em></p> <p><em>to the storage side after that."</em></p> <p>Michael believes that the ticket can be closed.</p> Development Rank 1|i01n4f: Rank (Obsolete) 9223372036854775807 Severity