[LU-5874] DLC: the ongoing traffic was interrupted after adding a new network interface

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.7.0
• Fix Version/s: Lustre 2.7.0
• Type: Bug
• Priority: Major
• Reporter: Sarah Liu
• Assignee: Amir Shehata (Inactive)
• Resolution: Fixed
• Severity: 3

Description

1. setup the system and run sanity
2. add a new network interface on the client side
3. the traffic was interrupted and keep showing following messages:
4. after remove the new added interface, system goes back to normal

== sanity test 27B: call setstripe on open unlinked file/rename victim == 12:18:00 (1415218680)
Lustre: DEBUG MARKER: == sanity test 27B: call setstripe on open unlinked file/rename victim == 12:18:00 (1415218680)
LNet: Added LNI 192.168.4.74@o2ib [8/256/0/180]
LNet: No route to 192.168.4.47@o2ib via from 10.2.4.74@tcp
Lustre: 4806:0:(client.c:1934:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1415218690/real 1415218690]  req@ffff880824129000 x1483963522156376/t0(0) o400->lustre-MDT0000-mdc-ffff880434a40800@192.168.4.47@o2ib:12/10 lens 224/224 e 0 to 1 dl 1415218753 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 4806:0:(client.c:1934:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff880434a40800: Connection to lustre-MDT0000 (at 192.168.4.47@o2ib) was lost; in progress operations using this service will wait for recovery to complete
LNet: Skipped 5 previous similar messages
LustreError: 166-1: MGC192.168.4.47@o2ib: Connection to MGS (at 192.168.4.47@o2ib) was lost; in progress operations using this service will fail
LNet: Removed LNI 192.168.4.74@o2ib
Lustre: lustre-OST0000-osc-ffff880434a40800: Connection restored to lustre-OST0000 (at 192.168.4.47@o2ib)
Lustre: Skipped 2 previous similar messages
LL_IOC_LOV_SETSTRIPE: No such file or directory
LL_IOC_LOV_SETSTRIPE: No such file or directory
Resetting fail_loc on all nodes...done.
PASS 27B (26s)

Issue Links

• is related to: LU-5875 DLC: failed adding an existing networ...
• is related to: LU-2456 Dynamic LNet Config Main Development ...

Comments

[Jodi Levi (Inactive)]

Amir,
Can you please have a look at this one?
Thank you!

[Amir Shehata (Inactive)]

Can you please exact steps used to reproduce this issue?

What would be ideal is the set of lnetctl commands used, and any show output to see the change.

[Sarah Liu]

1. setup a lustre filesystem with 1 MDT and 1 OST, servers use o2ib; a router; 1 client uses tcp; mount the system and run sanity on the client side
2. adding the new network, I only use this command on the client side
lnetctl net add --net o2ib --if ib0

then you can see the above errors on the client side

[Amir Shehata (Inactive)]

The issue here is that both the client and the servers are on ib0 with the addition of the network dynamically. this makes the configuration invalid due to the presence of the route that bridges the ib0 and tcp.

There are a couple of options to fix this:
1. reject the addition of a network which makes the configuration inalid
2. remove the route

Currently investigating the best solution.

[Isaac Huang (Inactive)]

If ib0 is added on the client, I thought lnet would automatically switch to ib0 to talk with the servers in ib0. There shouldn't be those "No route to ......" errors. Maybe I missed something?

It's not a good idea to remove the route, because if the ib0 interface is brought down later then the TCP client would not be able to talk to the servers any more (as the route was removed).

[Amir Shehata (Inactive)]

If I'm reading the following code correctly, from lnet_send()

        /* Is this for someone on a local network? */
	local_ni = lnet_net2ni_locked(LNET_NIDNET(dst_nid), cpt);

        if (local_ni != NULL) {
                if (src_ni == NULL) {
                        src_ni = local_ni;
                        src_nid = src_ni->ni_nid;
                } else if (src_ni == local_ni) {
			lnet_ni_decref_locked(local_ni, cpt);
		} else {
			lnet_ni_decref_locked(local_ni, cpt);
			lnet_ni_decref_locked(src_ni, cpt);
			lnet_net_unlock(cpt);
			LCONSOLE_WARN("No route to %s via from %s\n",
				      libcfs_nid2str(dst_nid),
				      libcfs_nid2str(src_nid));
			return -EINVAL;
		}

It seems to say if you're trying to send to a local ni, which is the case in this test case with the addition of ib0, then it is expecting that the local_ni and src_ni be the same. However, the src_nid is still the @tcp nid, which is as far as I could tell is stored in ptlrpc (not 100% sure yet). But if so, then what would trigger it to update the src_nid?

I'm also leaning towards rejecting the network addition. Basically, what I'm doing is before adding an NI dynamically, I check if it's a remote net, and if so, I reject adding the NI.

[Isaac Huang (Inactive)]

Agree, and I'd suggest:
1. The command line should print an error message, saying something like "the NI conflicts with route ***, remove the route before adding the NI". This message should be printed to stderr, rather than from the kernel to dmesg, because that's the user issued the command.
2. Document it in the Lustre manual.

[Amir Shehata (Inactive)]

Some more details:
mdc_setup() calls client_obd_setup() calls client_import_add_conn() calls import_set_conn calls() ptlrpc_uuid_to_connection() calls LNetDist()

mdc_setup() is only triggered on startup. This essentially picks the src_nid and sticks with that through out. So the addition of a new "closer NI" doesn't retrigger updating the connection hash maintained in ptlrpc.

Another option (but not as part of this bug) when a network is added connections should be evaluated and updated if there exists an NI which creates a more preferred path to the destination.

This will allow LNet to take advantage of going directly over the shortest path when updates occur.

Ideally, however, it seems that NIDs shouldn't be visible outside of LNet. I realize however, that this would be a major change.

[Gerrit Updater]

Amir Shehata ([e-mail]) uploaded a new patch: http://review.whamcloud.com/12912
Subject: LU-5874 lnet: reject invalid net configuration
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: d4b6589173915723dfebf4d5ce64ce2591180c41

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/12912/
Subject: LU-5874 lnet: reject invalid net configuration
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4aebc21514f49453aeeca83c6f6279d473d61617

[Peter Jones]

Landed for 2.7