https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14615 Lustre <p>On a few host when adding tcp nid using </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lnetctl lnet configure lnetctl net add --net tcp --<span class="code-keyword">if</span> ib1 </pre> </div></div> <p>We get this error</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [1137072.940179] LNet: 39800:0:(config.c:1641:lnet_inet_enumerate()) lnet: Ignoring <span class="code-keyword">interface</span> eth2: it's down [1137072.950118] LNet: 39800:0:(config.c:1641:lnet_inet_enumerate()) Skipped 2 previous similar messages [1137072.959931] LNet: Added LNI 10.151.27.21@tcp [8/256/0/180] [1137072.959988] LNetError: 39814:0:(lib-socket.c:315:lnet_sock_listen()) Can't create socket: port 988 already in use [1137072.970687] LNetError: 122-1: Can't start acceptor on port 988: port already in use [1137072.970724] LNetError: 39800:0:(api-ni.c:3123:lnet_add_net_common()) Failed to start up acceptor thread [1137073.977512] LNet: Removed LNI 10.151.27.21@tcp </pre> </div></div> <p>Nothing is using that port</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # lsof -i tcp@localhost:988 # lsof -i udp@localhost:988 </pre> </div></div> LU-14615

can't add tcp nid

Bug Major Resolved Fixed Amir Shehata Mahmoud Hanafi Wed, 14 Apr 2021 20:26:48 +0000 Fri, 16 Sep 2022 16:14:00 +0000 Fri, 16 Sep 2022 16:14:00 +0000 Lustre 2.12.5 Lustre 2.12.6 0 4 <p>Are you using ib1 for an o2iblnd network as well?</p> <p>What version is this Mahmoud?</p> <p>This is 2.12.5 and yes we are using o2ib on ib1 also. It worked on most of the node.  </p> <p>Can you check the service port for the o2iblnd? What is that set to? I'm thinking if it's 988 instead of 987, then you could run into this problem.</p> <p>We don't have o2ib listed in /etc/services.</p> <p>How do I check the port. This also occurs randomly when nodes are rebooted. </p> <p> </p> <p>Does this show any useful results</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> netstat -lnp</pre> </div></div> <p>the error in the description is printed if kernel_bind() returns EADDRINUSE.</p> <p>I looked at the kernel_bind() code and it seems that a port can be shared in the following circumstances</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">// from include/net/inet_hashtables.h </span> 45 *»·····1) Sockets bound to different interfaces may share a local port. 46 *»····· Failing that, <span class="code-keyword">goto</span> test 2. 47 *»·····2) If all sockets have sk-&gt;sk_reuse set, and none of them are in 48 *»····· TCP_LISTEN state, the port may be shared. 49 *»····· Failing that, <span class="code-keyword">goto</span> test 3. 50 *»·····3) If all sockets are bound to a specific inet_sk(sk)-&gt;rcv_saddr local 51 *»····· address, and none of them are the same, the port may be 52 *»····· shared. 53 *»····· Failing <span class="code-keyword">this</span>, the port cannot be shared. </pre> </div></div> <p>When we create the port we do set the SO_REUSEADDR and we bind to any address on the system</p> <p>Some debugging steps I would take</p> <ol> <li>Try a different listening port instead of 988</li> <li>Does it happen on all privileged ports? Is the behaviour different if we use a non-privileged port?</li> <li>What's the ib status of the interface at the time of the bind? Could it be possible that the IB HCA hasn't fully initialized yet? We've seen cases when the IB stack might not have been initialized by the time we bring up the LND. I know you're using the interface for ethernet, but it's worth looking at the status of the card.</li> <li>Dump the results of the netstat -lnp and ibstatus at the beginning when a node is rebooted.</li> </ol> <p>Are you able to consistently reproduce this problem?</p> <p> </p> <p>netstat show nothing. None else using that port.</p> <ol> <li>How do I try a different port.</li> <li>We don't privileged port configured. (why is it using a privilege port?)<br/> I don't see the same options for tcp <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> options ko2iblnd require_privileged_port=0 use_privileged_port=0 </pre> </div></div></li> </ol> <p>3 . We have seen this issue but that give a different error and we can bring up the interface later.</p> <p>When the node is in this state if we remove the tcp option from lustre.conf and try to load the module we the same error.</p> <p>you can set:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> options lnet accept_port=XXX </pre> </div></div> <p>By non-privileged I was thinking anything above 1024.</p> <p>The accept_port would need to be set consistently in order for the nodes to connect.</p> <p>Another thing to look out for is the actual network interface not coming up or maybe the IPoIB is not finished configuring (not loaded) before LNet tries to bind to the port? (Maybe something like: <a href="https://unix.stackexchange.com/questions/126009/cause-a-script-to-execute-after-networking-has-started" class="external-link" target="_blank" rel="nofollow noopener">https://unix.stackexchange.com/questions/126009/cause-a-script-to-execute-after-networking-has-started</a>)</p> <p>In Syslog you should see</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready</pre> </div></div> <p>Or something to that effect. Does that happen before or after LNet throws the error?</p> <p>Maybe if we can grab the entire syslog when this problem happens, we can look at the context for other clues.</p> <p>Please close this case</p> Development Rank 1|i01s93: Rank (Obsolete) 9223372036854775807 Severity