https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9914 Lustre <p>NOTE: I see that after this patch:<br/> <a href="https://review.whamcloud.com/#/c/28702/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28702/</a></p> <p>Without this patch the problem was being hidden, by an immediate failure.</p> <p>Steps:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Peer 2: net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.122.30@tcp status: up interfaces: 0: eth0 - nid: 192.168.122.31@tcp status: up interfaces: 0: eth1 - nid: 192.168.122.32@tcp status: up interfaces: 0: eth2 - nid: 192.168.122.33@tcp status: up interfaces: 0: eth3 - nid: 192.168.122.34@tcp status: up interfaces: 0: eth4 - nid: 192.168.122.35@tcp status: up interfaces: 0: eth5 - nid: 192.168.122.36@tcp status: up interfaces: 0: eth6 - nid: 192.168.122.37@tcp status: up interfaces: 0: eth7 - nid: 192.168.122.38@tcp status: up interfaces: 0: eth8 - nid: 192.168.122.39@tcp status: up interfaces: 0: eth9 - nid: 192.168.122.40@tcp status: up interfaces: 0: eth10 - nid: 192.168.122.41@tcp status: up interfaces: 0: eth11 - nid: 192.168.122.42@tcp status: up interfaces: 0: eth12 - nid: 192.168.122.43@tcp status: up interfaces: 0: eth13 - nid: 192.168.122.44@tcp status: up interfaces: 0: eth14 - nid: 192.168.122.45@tcp status: up interfaces: 0: eth15 - nid: 192.168.122.46@tcp status: up interfaces: 0: eth16 #peer 1 modprobe lnet lnetctl lnet configure lnetctl net add --net tcp --<span class="code-keyword">if</span> eth0,eth1 # max_interfaces <span class="code-keyword">default</span> to 200 lnetctl discover 192.168.122.30@tcp lnetctl set max_interfaces 16 # discover hangs (I kill it... so it might come back after a <span class="code-keyword">while</span>, but haven't waited) lnetctl discover 192.168.122.30@tcp </pre> </div></div> LU-9914

Dynamic Discovery - discovery hangs if max_interfaces is changed from 200->16

Bug Minor Open Unresolved Amir Shehata Amir Shehata patch Fri, 25 Aug 2017 01:10:11 +0000 Sat, 29 Jan 2022 10:48:06 +0000 0 4 <p>problem is here:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">1154 <span class="code-object">int</span> 1155 lnet_ping_info_validate(struct lnet_ping_info *pinfo) 1156 { 1157 »·······<span class="code-keyword">if</span> (!pinfo) 1158 »·······»·······<span class="code-keyword">return</span> -EINVAL; 1159 »·······<span class="code-keyword">if</span> (pinfo-&gt;pi_magic != LNET_PROTO_PING_MAGIC) 1160 »·······»·······<span class="code-keyword">return</span> -EPROTO; 1161 »·······<span class="code-keyword">if</span> (!(pinfo-&gt;pi_features &amp; LNET_PING_FEAT_NI_STATUS)) 1162 »·······»·······<span class="code-keyword">return</span> -EPROTO; 1163 »·······<span class="code-comment">/* Loopback is guaranteed to be present */</span> 1164 »·······<span class="code-keyword">if</span> (pinfo-&gt;pi_nnis &lt; 1 || pinfo-&gt;pi_nnis &gt; lnet_interfaces_max) 1165 »·······»·······<span class="code-keyword">return</span> -ERANGE; 1166 »·······<span class="code-keyword">if</span> (LNET_NETTYP(LNET_NIDNET(LNET_PING_INFO_LONI(pinfo))) != LOLND) 1167 »·······»·······<span class="code-keyword">return</span> -EPROTO; 1168 »·······<span class="code-keyword">return</span> 0; 1169 } 2103 »·······/* 2104 »······· * A reply with invalid or corrupted info. Set PING_FAILED to 2105 »······· * trigger a retry. 2106 »······· */ 2107 »·······rc = lnet_ping_info_validate(&amp;pbuf-&gt;pb_info); 2108 »·······<span class="code-keyword">if</span> (rc) { 2109 »·······»·······lp-&gt;lp_state |= LNET_PEER_PING_FAILED; 2110 »·······»·······lp-&gt;lp_ping_error = 0; 2111 »·······»·······CDEBUG(D_NET, <span class="code-quote">"Corrupted Ping Reply from %s: %d\n"</span>, 2112 »·······»······· libcfs_nid2str(lp-&gt;lp_primary_nid), rc); 2113 »·······»·······<span class="code-keyword">goto</span> out; 2114 »·······} </pre> </div></div> <p>Doesn't look like the state machine is handling the ping failure properly. Basically, the local lnet_interfaces_max is less than the number of interfaces on the far end 16 &lt; 18. So we should get an -ERANGE.</p> <p>Looks like we're stuck in a loop retrying the ping for discover and it keeps failing with the same error:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">(peer.c:2112:lnet_discovery_event_reply()) Corrupted Ping Reply from 192.168.122.30@tcp: -34 </pre> </div></div> <p>To be honest, <tt>lnet_interfaces_max</tt> exists only to avoid hard-coding a limit, and you ought to run with compatible values across the cluster. Meaning that <tt>lnet_interfaces_max</tt> on each node should be at least the number of interfaces of its peers.</p> <p>Still, what happens here isn't exactly graceful handling of the problematic configuration. My proposal would be to fail discovery of nodes that have more interfaces than <tt>lnet_interfaces_max</tt>, add some checks to prevent discovery from retrying, and emit an error message indicating that this problem has been encountered.</p> <p> </p> <p>Olaf Weber (olaf.weber@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/28714" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28714</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9914" title="Dynamic Discovery - discovery hangs if max_interfaces is changed from 200-&gt;16" class="issue-link" data-issue-key="LU-9914">LU-9914</a> lnet: gracefully handle peers with too many NIs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b973f67c227f5b988afb052171cf74cc7a097157</p> Development Rank 1|hzzizz: Rank (Obsolete) 9223372036854775807 Severity