https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5874 Lustre <p>1. setup the system and run sanity<br/> 2. add a new network interface on the client side<br/> 3. the traffic was interrupted and keep showing following messages:<br/> 4. after remove the new added interface, system goes back to normal</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 27B: call setstripe on open unlinked file/rename victim == 12:18:00 (1415218680) Lustre: DEBUG MARKER: == sanity test 27B: call setstripe on open unlinked file/rename victim == 12:18:00 (1415218680) LNet: Added LNI 192.168.4.74@o2ib [8/256/0/180] LNet: No route to 192.168.4.47@o2ib via from 10.2.4.74@tcp Lustre: 4806:0:(client.c:1934:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1415218690/real 1415218690] req@ffff880824129000 x1483963522156376/t0(0) o400-&gt;lustre-MDT0000-mdc-ffff880434a40800@192.168.4.47@o2ib:12/10 lens 224/224 e 0 to 1 dl 1415218753 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 4806:0:(client.c:1934:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Lustre: lustre-MDT0000-mdc-ffff880434a40800: Connection to lustre-MDT0000 (at 192.168.4.47@o2ib) was lost; in progress operations using this service will wait for recovery to complete LNet: Skipped 5 previous similar messages LustreError: 166-1: MGC192.168.4.47@o2ib: Connection to MGS (at 192.168.4.47@o2ib) was lost; in progress operations using this service will fail LNet: Removed LNI 192.168.4.74@o2ib Lustre: lustre-OST0000-osc-ffff880434a40800: Connection restored to lustre-OST0000 (at 192.168.4.47@o2ib) Lustre: Skipped 2 previous similar messages LL_IOC_LOV_SETSTRIPE: No such file or directory LL_IOC_LOV_SETSTRIPE: No such file or directory Resetting fail_loc on all nodes...done. PASS 27B (26s) </pre> </div></div> LU-5874

DLC: the ongoing traffic was interrupted after adding a new network interface

Bug Major Resolved Fixed Amir Shehata Sarah Liu Wed, 5 Nov 2014 20:23:29 +0000 Mon, 19 Jan 2015 18:12:02 +0000 Mon, 19 Jan 2015 18:12:02 +0000 Lustre 2.7.0 Lustre 2.7.0 0 7 <p>Amir,<br/> Can you please have a look at this one?<br/> Thank you!</p> <p>Can you please exact steps used to reproduce this issue? </p> <p>What would be ideal is the set of lnetctl commands used, and any show output to see the change.</p> <p>1. setup a lustre filesystem with 1 MDT and 1 OST, servers use o2ib; a router; 1 client uses tcp; mount the system and run sanity on the client side<br/> 2. adding the new network, I only use this command on the client side<br/> lnetctl net add --net o2ib --if ib0</p> <p>then you can see the above errors on the client side</p> <p>The issue here is that both the client and the servers are on ib0 with the addition of the network dynamically. this makes the configuration invalid due to the presence of the route that bridges the ib0 and tcp.</p> <p>There are a couple of options to fix this:<br/> 1. reject the addition of a network which makes the configuration inalid<br/> 2. remove the route</p> <p>Currently investigating the best solution.</p> <p>If ib0 is added on the client, I thought lnet would automatically switch to ib0 to talk with the servers in ib0. There shouldn't be those "No route to ......" errors. Maybe I missed something?</p> <p>It's not a good idea to remove the route, because if the ib0 interface is brought down later then the TCP client would not be able to talk to the servers any more (as the route was removed).</p> <p>If I'm reading the following code correctly, from lnet_send()</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* Is <span class="code-keyword">this</span> <span class="code-keyword">for</span> someone on a local network? */</span> local_ni = lnet_net2ni_locked(LNET_NIDNET(dst_nid), cpt); <span class="code-keyword">if</span> (local_ni != NULL) { <span class="code-keyword">if</span> (src_ni == NULL) { src_ni = local_ni; src_nid = src_ni-&gt;ni_nid; } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (src_ni == local_ni) { lnet_ni_decref_locked(local_ni, cpt); } <span class="code-keyword">else</span> { lnet_ni_decref_locked(local_ni, cpt); lnet_ni_decref_locked(src_ni, cpt); lnet_net_unlock(cpt); LCONSOLE_WARN(<span class="code-quote">"No route to %s via from %s\n"</span>, libcfs_nid2str(dst_nid), libcfs_nid2str(src_nid)); <span class="code-keyword">return</span> -EINVAL; } </pre> </div></div> <p>It seems to say if you're trying to send to a local ni, which is the case in this test case with the addition of ib0, then it is expecting that the local_ni and src_ni be the same. However, the src_nid is still the @tcp nid, which is as far as I could tell is stored in ptlrpc (not 100% sure yet). But if so, then what would trigger it to update the src_nid?</p> <p>I'm also leaning towards rejecting the network addition. Basically, what I'm doing is before adding an NI dynamically, I check if it's a remote net, and if so, I reject adding the NI.</p> <p>Agree, and I'd suggest:<br/> 1. The command line should print an error message, saying something like "the NI conflicts with route ***, remove the route before adding the NI". This message should be printed to stderr, rather than from the kernel to dmesg, because that's the user issued the command.<br/> 2. Document it in the Lustre manual.</p> <p>Some more details:<br/> mdc_setup() calls client_obd_setup() calls client_import_add_conn() calls import_set_conn calls() ptlrpc_uuid_to_connection() calls LNetDist()</p> <p>mdc_setup() is only triggered on startup. This essentially picks the src_nid and sticks with that through out. So the addition of a new "closer NI" doesn't retrigger updating the connection hash maintained in ptlrpc.</p> <p>Another option (but not as part of this bug) when a network is added connections should be evaluated and updated if there exists an NI which creates a more preferred path to the destination.</p> <p>This will allow LNet to take advantage of going directly over the shortest path when updates occur. </p> <p>Ideally, however, it seems that NIDs shouldn't be visible outside of LNet. I realize however, that this would be a major change.</p> <p>Amir Shehata (amir.shehata@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/12912" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12912</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5874" title="DLC: the ongoing traffic was interrupted after adding a new network interface" class="issue-link" data-issue-key="LU-5874"><del>LU-5874</del></a> lnet: reject invalid net configuration<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d4b6589173915723dfebf4d5ce64ce2591180c41</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12912/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12912/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5874" title="DLC: the ongoing traffic was interrupted after adding a new network interface" class="issue-link" data-issue-key="LU-5874"><del>LU-5874</del></a> lnet: reject invalid net configuration<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4aebc21514f49453aeeca83c6f6279d473d61617</p> <p>Landed for 2.7</p> Related LU-5875 LU-2456 Development Rank 1|hzx0bz: Rank (Obsolete) 16426 Severity