https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14555 Lustre <p>We have the following configuration:</p> <p>2.14_servers == o2ib100 == 2.12_routers == tcp129 == 2.12_routers == o2ib18 == 2.12_clients</p> <p>Discovery is disabled, and the routes are configured statically, on all the systems.</p> <p>This causes LNet to complain vociferously on the console from lnet_check_route_inconsistency()</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LNet: 29144:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 is detected to be multi-hop but hop count is set to -1</pre> </div></div> <p>If LNet is configured so that there is only one route to any given endpoint, even on a multi-hop network, there is no value to spending sysadmin time determining and setting the hop counts as far as I can tell.  And setting hops is optional according to the Lustre Operations Manual.</p> <p>Is hop count actually required in 2.14 due to <a href="https://jira.whamcloud.com/browse/LU-13029" title="LNet Routing: asym routing is not working for multi-hop routing" class="issue-link" data-issue-key="LU-13029"><del>LU-13029</del></a> and <a href="https://jira.whamcloud.com/browse/LU-13785" title="router ib interface was not configured on boot. gni clients mis-classified the router as multi-hop leading to evictions" class="issue-link" data-issue-key="LU-13785"><del>LU-13785</del></a>?</p> RHEL 8 <br/> multi-hop network <br/> hops not set LU-14555

lnet_check_route_inconsistency() complains when hops == -1

Bug Minor Resolved Fixed Gian-Carlo Defazio Olaf Faaland llnl Fri, 26 Mar 2021 01:02:40 +0000 Fri, 13 Jan 2023 14:57:21 +0000 Fri, 13 Jan 2023 14:57:21 +0000 Lustre 2.14.0 Lustre 2.16.0 0 4 <p>Olaf Faaland-LLNL (faaland1@llnl.gov) uploaded a new patch: <a href="https://review.whamcloud.com/43127" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43127</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14555" title="lnet_check_route_inconsistency() complains when hops == -1" class="issue-link" data-issue-key="LU-14555"><del>LU-14555</del></a> lnet: do not complain if hops == -1<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4ebbbb068cf9d2f53b6923ffe7744dc562bc94fe</p> <p>I don't actually know if hops == 0 is either valid or possible, so in the patch I checked for that and reported it as well as reporting if hops == 1.</p> <p>Peter,</p> <p>There appears to be more to hop count than I realized.  Please assign this to an engineer.</p> <p>thanks</p> <p>Serguei</p> <p>Could you please assist?</p> <p>Thanks</p> <p>Peter</p> <p>Olaf, the check for inconsistency should happen only when we ping the route for aliveness.</p> <p>Do you see it happening more often?</p> <p>We can probably reduce the severity of the debug message. But would like to make sure that it's not being printed more frequently than it should.</p> <p>Hi Amir,</p> <p>I'm not certain if this is only occurring when the route is pinged for aliveness.  I'll look.  But if setting the hop count is not required, then a console message is inappropriate.   And if setting the hop count is required, then shouldn't that be enforced at the time routes are created?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@garteri:~]# pdsh -N -w e1 'dmesg -T | grep -w hop | fgrep 1.54 | tail' | sed 's/is detected to be multi-hop.*$//' [Thu Mar 25 15:21:37 2021] LNet: 29145:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 [Thu Mar 25 15:23:07 2021] LNet: 29145:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 [Thu Mar 25 15:23:37 2021] LNet: 29143:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 [Thu Mar 25 15:24:08 2021] LNet: 29146:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 [Thu Mar 25 15:24:37 2021] LNet: 29144:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 [Thu Mar 25 15:25:38 2021] LNet: 29146:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 [Thu Mar 25 15:27:07 2021] LNet: 29146:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 [Thu Mar 25 15:29:37 2021] LNet: 29144:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 [Thu Mar 25 15:34:08 2021] LNet: 29146:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 </pre> </div></div> <p>thanks<br/>  </p> <p>Amir,<br/> The timing of the message does not seem to correlate well with the timing of the call to lnet_check_routers(). If I'm looking at the wrong code, let me know.</p> <p>The debug log "discover" messages from lnet_check_routers for one router, .1.54:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2021-03-26 18:12:19.043797 00000400:00000200:3.0::0:30243:0:(router.c:1231:lnet_check_routers()) discover 172.19.1.54@o2 2021-03-26 18:12:49.798562 00000400:00000200:3.0::0:30243:0:(router.c:1231:lnet_check_routers()) discover 172.19.1.54@o2 2021-03-26 18:13:19.494570 00000400:00000200:3.0::0:30243:0:(router.c:1231:lnet_check_routers()) discover 172.19.1.54@o22021-03-26 18:13:49.190567 00000400:00000200:3.0::0:30243:0:(router.c:1231:lnet_check_routers()) discover 172.19.1.54@o2 2021-03-26 18:14:19.910568 00000400:00000200:3.0::0:30243:0:(router.c:1231:lnet_check_routers()) discover 172.19.1.54@o22021-03-26 18:14:49.606563 00000400:00000200:3.0::0:30243:0:(router.c:1231:lnet_check_routers()) discover 172.19.1.54@o2 2021-03-26 18:15:19.302567 00000400:00000200:3.0::0:30243:0:(router.c:1231:lnet_check_routers()) discover 172.19.1.54@o22021-03-26 18:15:50.022572 00000400:00000200:3.0::0:30243:0:(router.c:1231:lnet_check_routers()) discover 172.19.1.54@o2 2021-03-26 18:16:20.742564 00000400:00000200:3.0::0:30243:0:(router.c:1231:lnet_check_routers()) discover 172.19.1.54@o22021-03-26 18:16:50.438577 00000400:00000200:3.0::0:30243:0:(router.c:1231:lnet_check_routers()) discover 172.19.1.54@o2 2021-03-26 18:17:20.134570 00000400:00000200:3.0::0:30243:0:(router.c:1231:lnet_check_routers()) discover 172.19.1.54@o2 2021-03-26 18:17:50.854558 00000400:00000200:3.0::0:30243:0:(router.c:1231:lnet_check_routers()) discover 172.19.1.54@o2 2021-03-26 18:18:20.550570 00000400:00000200:3.0::0:30243:0:(router.c:1231:lnet_check_routers()) discover 172.19.1.54@o2 2021-03-26 18:18:50.246560 00000400:00000200:3.0::0:30243:0:(router.c:1231:lnet_check_routers()) discover 172.19.1.54@o2 2021-03-26 18:19:20.966560 00000400:00000200:3.0::0:30243:0:(router.c:1231:lnet_check_routers()) discover 172.19.1.54@o2 2021-03-26 18:19:50.662570 00000400:00000200:3.0::0:30243:0:(router.c:1231:lnet_check_routers()) discover 172.19.1.54@o2 2021-03-26 18:20:20.358566 00000400:00000200:3.0::0:30243:0:(router.c:1231:lnet_check_routers()) discover 172.19.1.54@o2 </pre> </div></div> <p>and the console log multi-hop messages from that period:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[Fri Mar 26 18:12:18 2021] LNet: 30237:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 is detected to be multi-hop but hop count is set to -1 [Fri Mar 26 18:12:48 2021] LNet: 30240:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 is detected to be multi-hop but hop count is set to -1 [Fri Mar 26 18:13:18 2021] LNet: 30238:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 is detected to be multi-hop but hop count is set to -1 [Fri Mar 26 18:13:48 2021] LNet: 30238:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 is detected to be multi-hop but hop count is set to -1 [Fri Mar 26 18:14:19 2021] LNet: 30238:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 is detected to be multi-hop but hop count is set to -1 [Fri Mar 26 18:14:48 2021] LNet: 30237:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 is detected to be multi-hop but hop count is set to -1 [Fri Mar 26 18:15:18 2021] LNet: 30240:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 is detected to be multi-hop but hop count is set to -1 [Fri Mar 26 18:16:19 2021] LNet: 30240:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 is detected to be multi-hop but hop count is set to -1 [Fri Mar 26 18:17:50 2021] LNet: 30240:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 is detected to be multi-hop but hop count is set to -1 [Fri Mar 26 18:20:19 2021] LNet: 30237:0:(router.c:384:lnet_check_route_inconsistency()) route o2ib18-&gt;172.19.1.54@o2ib100 is detected to be multi-hop but hop count is set to -1</pre> </div></div> <p>Looks like it's getting the keep alive every 30 seconds and that's when we do the check route consistency. I think it'll be enough to reduce the message severity to just "net". It is not mandatory to set the hop count. However, the reason we have the check is to verify configuration consistency. However, if it's not standard to explicitly specify the hop count when configuring the route, then the check becomes less effective.</p> <p>"Gian-Carlo DeFazio &lt;defazio1@llnl.gov&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/46918" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46918</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14555" title="lnet_check_route_inconsistency() complains when hops == -1" class="issue-link" data-issue-key="LU-14555"><del>LU-14555</del></a> lnet: change route inconsistency warnings<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 572b52a488ac7a186be2f808668e163e9ac850b2</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/46918/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46918/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14555" title="lnet_check_route_inconsistency() complains when hops == -1" class="issue-link" data-issue-key="LU-14555"><del>LU-14555</del></a> lnet: asym route inconsistency warning<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6ab060e58e6b3f38b0c8d57b56fec887c6fe9fb6</p> <p>Landed for 2.16</p> <p>Just adding the requirement that <b>avoid_asym_router_failure</b> be true doesn't prevent the warning from beaing logged because we have <b>avoid_asym_router_failure=1.</b> </p> <p>However, due to changes in the code since ticket was made, I've made a change which further reduces the conditions in which the warning is logged: having hops undefined for a multi-hop route is no longer considered an inconsistent configuration.</p> <p>"Gian-Carlo DeFazio &lt;defazio1@llnl.gov&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49352" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49352</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14555" title="lnet_check_route_inconsistency() complains when hops == -1" class="issue-link" data-issue-key="LU-14555"><del>LU-14555</del></a> lnet: asym route inconsistency warning<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: af572e46f883f8736af8055a2030eb640792130c</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49352/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49352/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14555" title="lnet_check_route_inconsistency() complains when hops == -1" class="issue-link" data-issue-key="LU-14555"><del>LU-14555</del></a> lnet: asym route inconsistency warning<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6aed5df1771c299b527251b0e18ff9f6cb95dd75</p> <p>Landed for 2.16</p> Development Rank 1|i01qjz: Rank (Obsolete) 9223372036854775807 Severity