https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12411 Lustre <p>Hit with master.</p> <p>Steps to reproduce:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>sles15build01:/home/hornc/lustre-filesystem # insmod ./libcfs/libcfs/libcfs.ko sles15build01:/home/hornc/lustre-filesystem # insmod ./lnet/lnet/lnet.ko sles15build01:/home/hornc/lustre-filesystem # cd lnet/utils/ sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl lnet configure sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl export net: - net type: lo local NI(s): - nid: 0@lo status: up statistics: send_count: 0 recv_count: 0 drop_count: 0 sent_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 received_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 dropped_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 health stats: health value: 0 interrupts: 0 dropped: 0 aborted: 0 no route: 0 timeouts: 0 error: 0 tunables: peer_timeout: 0 peer_credits: 0 peer_buffer_credits: 0 credits: 0 dev cpt: 0 tcp bonding: 0 CPT: "[0,1,2,3]" global: numa_range: 0 max_intf: 200 discovery: 1 drop_asym_route: 0 sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl route add --net gni4 --gateway 10.12.0.[1-4]@o2ib40 sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl export net: - net type: lo local NI(s): - nid: 0@lo status: up statistics: send_count: 0 recv_count: 0 drop_count: 0 sent_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 received_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 dropped_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 health stats: health value: 0 interrupts: 0 dropped: 0 aborted: 0 no route: 0 timeouts: 0 error: 0 tunables: peer_timeout: 0 peer_credits: 0 peer_buffer_credits: 0 credits: 0 dev cpt: 0 tcp bonding: 0 CPT: "[0,1,2,3]" route: - net: gni4 gateway: 10.12.0.4@o2ib40 hop: -1 priority: 0 health_sensitivity: 1 state: down - net: gni4 gateway: 10.12.0.3@o2ib40 hop: -1 priority: 0 health_sensitivity: 1 state: down - net: gni4 gateway: 10.12.0.2@o2ib40 hop: -1 priority: 0 health_sensitivity: 1 state: down - net: gni4 gateway: 10.12.0.1@o2ib40 hop: -1 priority: 0 health_sensitivity: 1 state: down peer: - primary nid: 10.12.0.1@o2ib40 Multi-Rail: True peer ni: - nid: 10.12.0.1@o2ib40 state: up max_ni_tx_credits: 0 available_tx_credits: 0 min_tx_credits: 0 tx_q_num_of_buf: 0 available_rtr_credits: 0 min_rtr_credits: 0 refcount: 2 statistics: send_count: 0 recv_count: 0 drop_count: 0 sent_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 received_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 dropped_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 health stats: health value: 1000 dropped: 0 timeout: 0 error: 0 network timeout: 0 - primary nid: 10.12.0.2@o2ib40 Multi-Rail: True peer ni: - nid: 10.12.0.2@o2ib40 state: up max_ni_tx_credits: 0 available_tx_credits: 0 min_tx_credits: 0 tx_q_num_of_buf: 0 available_rtr_credits: 0 min_rtr_credits: 0 refcount: 2 statistics: send_count: 0 recv_count: 0 drop_count: 0 sent_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 received_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 dropped_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 health stats: health value: 1000 dropped: 0 timeout: 0 error: 0 network timeout: 0 - primary nid: 10.12.0.4@o2ib40 Multi-Rail: True peer ni: - nid: 10.12.0.4@o2ib40 state: up max_ni_tx_credits: 0 available_tx_credits: 0 min_tx_credits: 0 tx_q_num_of_buf: 0 available_rtr_credits: 0 min_rtr_credits: 0 refcount: 2 statistics: send_count: 0 recv_count: 0 drop_count: 0 sent_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 received_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 dropped_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 health stats: health value: 1000 dropped: 0 timeout: 0 error: 0 network timeout: 0 - primary nid: 10.12.0.3@o2ib40 Multi-Rail: True peer ni: - nid: 10.12.0.3@o2ib40 state: up max_ni_tx_credits: 0 available_tx_credits: 0 min_tx_credits: 0 tx_q_num_of_buf: 0 available_rtr_credits: 0 min_rtr_credits: 0 refcount: 2 statistics: send_count: 0 recv_count: 0 drop_count: 0 sent_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 received_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 dropped_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 health stats: health value: 1000 dropped: 0 timeout: 0 error: 0 network timeout: 0 global: numa_range: 0 max_intf: 200 discovery: 1 drop_asym_route: 0 sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl route del --gateway 10.12.0.[1-4]@o2ib4 --net gni4 ^^^^ Command hangs </pre> </div></div> LU-12411

Hang on lnetctl route del

Bug Major Resolved Fixed Amir Shehata Chris Horn Mon, 10 Jun 2019 04:43:55 +0000 Fri, 24 Apr 2020 19:26:26 +0000 Tue, 25 Jun 2019 04:19:28 +0000 Lustre 2.13.0 Lustre 2.12.3 Lustre 2.13.0 Lustre 2.12.4 0 3 <p>I hit this issue on a route add and was able to grab the dmesg from the node.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>sles15build01:/home/hornc/lustre-filesystem # insmod libcfs/libcfs/libcfs.ko sles15build01:/home/hornc/lustre-filesystem # insmod lnet/lnet/lnet.ko sles15build01:/home/hornc/lustre-filesystem # cd lnet/utils/ sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl lnet configure sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl route add --net o2ib40 --gw 485@gni4 route add: add a route --net: net name (e.g. tcp0) --gateway: gateway nid (e.g. 10.1.1.2@tcp) --hop: number to final destination (1 &lt; hops &lt; 255) --priority: priority of route (0 - highest prio --health_sensitivity: gateway health sensitivity (&gt;= 1) sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl route add --net o2ib40 --gateway 485@gni4 sles15build01:/home/hornc/lustre-filesystem/lnet/utils # lnetctl route show route: - net: o2ib40 gateway: 485@gni4 sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl route show route: - net: o2ib40 gateway: 485@gni4 sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl peer show peer: - primary nid: 485@gni4 Multi-Rail: True peer ni: - nid: 485@gni4 state: up sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl route add --net o2ib40 --gateway 486@gni4,487@gni4,488@gni,10.12.0.[1-4]@o2ib,192.168.0.[20-24]@tcp sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl route show route: - net: o2ib40 gateway: 10.12.0.1@o2ib - net: o2ib40 gateway: 488@gni - net: o2ib40 gateway: 487@gni4 - net: o2ib40 gateway: 192.168.0.23@tcp - net: o2ib40 gateway: 192.168.0.21@tcp - net: o2ib40 gateway: 192.168.0.20@tcp - net: o2ib40 gateway: 192.168.0.24@tcp - net: o2ib40 gateway: 10.12.0.2@o2ib - net: o2ib40 gateway: 486@gni4 - net: o2ib40 gateway: 192.168.0.22@tcp - net: o2ib40 gateway: 485@gni4 - net: o2ib40 gateway: 10.12.0.3@o2ib - net: o2ib40 gateway: 10.12.0.4@o2ib sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl peer show peer: - primary nid: 488@gni Multi-Rail: True peer ni: - nid: 488@gni state: up - primary nid: 10.12.0.1@o2ib Multi-Rail: True peer ni: - nid: 10.12.0.1@o2ib state: up - primary nid: 10.12.0.4@o2ib Multi-Rail: True peer ni: - nid: 10.12.0.4@o2ib state: up - primary nid: 192.168.0.21@tcp Multi-Rail: True peer ni: - nid: 192.168.0.21@tcp state: up - primary nid: 192.168.0.24@tcp Multi-Rail: True peer ni: - nid: 192.168.0.24@tcp state: up - primary nid: 486@gni4 Multi-Rail: True peer ni: - nid: 486@gni4 state: up - primary nid: 10.12.0.2@o2ib Multi-Rail: True peer ni: - nid: 10.12.0.2@o2ib state: up - primary nid: 192.168.0.22@tcp Multi-Rail: True peer ni: - nid: 192.168.0.22@tcp state: up - primary nid: 487@gni4 Multi-Rail: True peer ni: - nid: 487@gni4 state: up - primary nid: 192.168.0.20@tcp Multi-Rail: True peer ni: - nid: 192.168.0.20@tcp state: up - primary nid: 485@gni4 Multi-Rail: True peer ni: - nid: 485@gni4 state: up - primary nid: 10.12.0.3@o2ib Multi-Rail: True peer ni: - nid: 10.12.0.3@o2ib state: up - primary nid: 192.168.0.23@tcp Multi-Rail: True peer ni: - nid: 192.168.0.23@tcp state: up sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl route add --net o2ib40 --gateway [486-489]@gni4 add: - route: errno: -1 descr: "Cannot parse nid: [486-489]@gni4" sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl route add --net o2ib40 --gateway [485,486,93,94]@gni4 add: - route: errno: -1 descr: "Cannot parse nid: [485,486,93,94]@gni4" sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl route add --net o2ib --gateway 192.168.2.24@tcp,192.168.2.25@tcp ^^^ Command hangs </pre> </div></div> <p>FWIW, this is the module options on this node:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>sles15build01:~ # cat /etc/modprobe.d/lnet.conf options lnet forwarding=enabled options lnet networks="tcp(eth0)" sles15build01:~ # </pre> </div></div> <p>Attached a dump from the route add case mentioned above <a href="https://jira.whamcloud.com/secure/attachment/32770/LU-12411.dump.tar.bz2" class="external-link" rel="nofollow">https://jira.whamcloud.com/secure/attachment/32770/LU-12411.dump.tar.bz2</a></p> <p>Might be related to the fact that I wasn't configuring any local interfaces before adding the routes? I see another hang doing the following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>sles15build01:/home/hornc/lustre-filesystem # insmod libcfs/libcfs/libcfs.ko sles15build01:/home/hornc/lustre-filesystem # insmod lnet/lnet/lnet.ko^C sles15build01:/home/hornc/lustre-filesystem # vim /etc/modprobe.d/lnet.conf sles15build01:/home/hornc/lustre-filesystem # insmod lnet/lnet/lnet.ko sles15build01:/home/hornc/lustre-filesystem # cd lnet/utils/ sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl lnet configure sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl route add --net o2ib --gateway 192.168.2.24@tcp,192.168.2.25@tcp sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl route del --net o2ib --gateway 192.168.2.24@tcp,192.168.2.25@tcp del: - route: errno: -8 descr: "Success" sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl route show route: - net: o2ib gateway: 192.168.2.25@tcp - net: o2ib gateway: 192.168.2.24@tcp sles15build01:/home/hornc/lustre-filesystem/lnet/utils # ./lnetctl net add --net tcp --if eth0 ^^^ Command hangs </pre> </div></div> <p>This might also be a bug exposed when adding any route where the gateway is on an unreachable lnet. We probably shouldn't allow that to happen. I will push a patch to prevent this.</p> <p>Chris Horn (hornc@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/35198" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35198</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12411" title="Hang on lnetctl route del" class="issue-link" data-issue-key="LU-12411"><del>LU-12411</del></a> lnet: Do not allow gateways on remote nets<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 203acca356f3793c1f1c469af1b83790c814c9ad</p> <p>Steps to reproduce</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@lustre01 ~]# modprobe lnet [root@lustre01 ~]# lnetctl lnet configure [root@lustre01 ~]# lnetctl route add --net tcp1 --gateway 192.168.122.[106-107]@tcp [root@lustre01 ~]# lnetctl route del --net tcp1 --gateway 192.168.122.[106-107]@tcp </pre> </div></div> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35198/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35198/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12411" title="Hang on lnetctl route del" class="issue-link" data-issue-key="LU-12411"><del>LU-12411</del></a> lnet: Do not allow gateways on remote nets<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 43b35351e9ca258773e89c2d68047e939fb822fb</p> <p>Landed for 2.13</p> <p>Amir, Is there another bug here to chase, or is it sufficient to just prevent the behavior that leads to breakage?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=pjones" class="user-hover" rel="pjones">pjones</a> FYI, I discovered that this is a regression that was originally introduced in 2.10.0</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 376633ab5c487a2e9497e118ce351c4b1597bf33 Author: Amir Shehata &lt;amir.shehata@intel.com&gt; Date: Mon Jul 4 14:51:06 2016 -0700 LU-7734 lnet: Routing fixes part 1 </pre> </div></div> <p>Unfortunately, my fix that landed under this ticket contained a flaw. I pushed a patch for that under <a href="https://jira.whamcloud.com/browse/LU-12595" title="Attempt to add route using non-local gateway should return EHOSTUNREACH, not EINVAL" class="issue-link" data-issue-key="LU-12595"><del>LU-12595</del></a>. Assuming <a href="https://jira.whamcloud.com/browse/LU-12595" title="Attempt to add route using non-local gateway should return EHOSTUNREACH, not EINVAL" class="issue-link" data-issue-key="LU-12595"><del>LU-12595</del></a> lands, you might consider landing both of these patches to LTS.</p> <p>ok thanks. I've flagged the second one for the LTS so we'll pick it up when that lands to master.</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36870" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36870</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12411" title="Hang on lnetctl route del" class="issue-link" data-issue-key="LU-12411"><del>LU-12411</del></a> lnet: Do not allow gateways on remote nets<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: cb829cc7683e66ecb95db7a80e925d716b6560e9</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36870/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36870/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12411" title="Hang on lnetctl route del" class="issue-link" data-issue-key="LU-12411"><del>LU-12411</del></a> lnet: Do not allow gateways on remote nets<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: c6c9084c959ac972af557da100f251eccc79d2f7</p> Related LU-12595 Development Rank 1|i00hvr: Rank (Obsolete) 9223372036854775807 Severity