https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5364 Lustre <p>We have 2 IB fabrics connected with 2 lustre routers. One side of fabrics connected via obsidain longbows and the other fabrics is directed connected to routers via qdr switch.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Fabric1_o2ib233 &lt;---&gt;LONGBOW1&lt;----&lt;ROUTER1&gt;----&gt;QDR&lt;--Fabric2_o2ib Fabric1_o2ib233 &lt;---&gt;LONGBOW2&lt;----&lt;ROUTER2&gt;-----&gt;QDR&lt;--Fabric2_o2ib </pre> </div></div> <p>We get Router disconnects on the fabric2_o2ib side with errors like this on the routers</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LNet: 1310:0:(o2iblnd_cb.c:2360:kiblnd_passive_connect()) Conn race 10.151.27.74@o2ib LNet: 1308:0:(o2iblnd_cb.c:2360:kiblnd_passive_connect()) Conn race 10.151.27.86@o2ib LNet: 1312:0:(o2iblnd_cb.c:2360:kiblnd_passive_connect()) Conn race 10.151.25.242@o2ib LNet: 1312:0:(o2iblnd_cb.c:2360:kiblnd_passive_connect()) Conn race 10.151.25.156@o2ib LNet: 1314:0:(o2iblnd_cb.c:2360:kiblnd_passive_connect()) Conn race 10.151.27.80@o2ib </pre> </div></div> <p>ROUTER MODULE SETTINGS</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">options lnet networks=<span class="code-quote">"o2ib(ib1),o2ib233(ib0)"</span> forwarding=enabled options ko2iblnd require_privileged_port=0 options ko2iblnd use_privileged_port=0 options ko2iblnd timeout=150 options ko2iblnd retry_count=7 options ko2iblnd peer_timeout=0 options ptlrpc at_min=100 </pre> </div></div> <p>SERVERS SETTINGS</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">options ko2iblnd require_privileged_port=0 options ko2iblnd use_privileged_port=0 options lnet networks=o2ib(ib1),o2ib100(ib1) routes=<span class="code-quote">"o2ib233 10.151.27.[58,93]@o2ib"</span> dead_router_check_interval=60 live_router_check_interval=60 # Get rid of messages <span class="code-keyword">for</span> missing, special-purpose hardware (LU-1599) blacklist padlock-sha options ko2iblnd timeout=150 options ko2iblnd retry_count=7 options ko2iblnd peer_timeout=0 options ptlrpc at_min=100 </pre> </div></div> <p>CLIENTS</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">options ko2iblnd require_privileged_port=0 options ko2iblnd use_privileged_port=0 options lnet networks=o2ib233(ib1) routes=<span class="code-quote">"o2ib 10.153.27.[58,93]@o2ib233"</span> dead_router_check_interval=60 live_router_check_interval=60 </pre> </div></div> LU-5364

Lustre Router connection hangs one side of fabric

Bug Major Resolved Not a Bug Amir Shehata Mahmoud Hanafi Thu, 17 Jul 2014 20:52:13 +0000 Tue, 12 Aug 2014 13:54:16 +0000 Tue, 22 Jul 2014 18:52:26 +0000 Lustre 2.4.3 0 7 <p>Amir</p> <p>Could you please assist with this one?</p> <p>Thanks</p> <p>Peter</p> <p>currently investigating. Will update when I have more information.</p> <p>The race the error message is referring to, is when the router receives an ib connect but there already exists a peer with the same nid (of the destination) in connecting state. This happens if the router is already in the process of establishing a connection with that nid. In this case the incoming connection gets dropped. There are multiple scenarios where that occurs:<br/> 1. if the router is in the process of transmitting a message to the destination nid, and is currently connecting<br/> 2. if the router receives 2 consecutive connects from the same nid (although, I'm not sure if this is a possible case).<br/> 3. if the router is reconnecting to the peer when it gets another connection request.</p> <p>I'm still investigating the code more thoroughly to try and understand which scenario is more likely.</p> <p>Would it be possible to grab syslog messages from the router to see the errors in context.</p> <p>Also do you hit this issue rightaway or does the system work for a while before the problem is encountered?</p> <p>After examining the code some more, these "Conn race" should not result in hanging. The side that the router disconnected due to a race, only did so, because there is already another connection to that side in progress.</p> <p>I just want to clarify if the symptoms being experienced are temporary disconnects or permanent hangs with no recovery?</p> <p>Also, as indicated in my previous comments, if we could get the logs from both sides of the router, as well as logs from the router, when this problem occurs, that'll help in giving context to the problem.</p> <p>Further testing showed that this may have been due IB fabric. </p> <p>You may close this for now. I will reopen it when we have more data.</p> <p>ok thanks Mahmoud!</p> Development Rank 1|hzwrpj: Rank (Obsolete) 14961 Severity