[LU-33] client can't recover on N-hop router configuraton

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 1.8.6
• Type: Bug
• Priority: Minor
• Reporter: Shuichi Ihara (Inactive)
• Assignee: Liang Zhen (Inactive)
• Resolution: Fixed
• Severity: 3

Description

I'm testing on Lustre N-hop routing (e.g. o2ib0 <> tcp <> o2ib1) below.

MDS/OSS  <-- IB (o2ib0) --> Router1 <-- TCP (tcp0) --> Router2 <-- IB (o2ib1) --> Client

- Network configuration -
There are two IB fabrics and 1GbE connects both fabrics with LNET routers.

MDS/OSS IP address: 192.168.100.120@o2ib0
options lnet networks=o2ib0 routes="tcp0 192.168.100.121@o2ib0; o2ib1 192.168.100.121@o2ib0"

Router1 IP address: 192.168.100.121@o2ib0, 192.168.10.121@tcp0
options lnet ip2nets="tcp0 192.168.20.*; o2ib0(ib0) 192.168.100.*" routes="o2ib1 192.168.20.122@tcp0" forwarding="enabled"

Router2 IP address: 192.168.200.122@o2ib1, 192.168.10.122@tcp0
options lnet ip2nets="tcp0 192.168.20.*; o2ib1(ib0) 192.168.200.*" routes="o2ib0 192.168.20.121@tcp0" forwarding="enabled"

Client IP address: 192.168.200.123@o2ib1
options lnet networks=o2ib1(ib0) routes="o2ib0 192.168.200.122@o2ib1"

It worked with above the configurations, but it seems that there is an issue if Router2 downs (e.g. 'lctl net down'), then restart it. The problem is that the client can't be recovery unless the client umount and remount the filesytem on it.

Comments

[Liang Zhen (Inactive)]

Ihara,

could you help me to get a few things after restarting the router in the test:

1. cat /proc/sys/lnet/peers on router2
2. cat /proc/sys/lnet/peers and /proc/sys/lnet/routes on client
3. lctl ping route2 on client to see if it can work, if it can't please get error messages on client
4. lctl ping client on route2 to see if it can work, if it can't please get error messages on router2

NB: also, could you tell me detail version of your lustre?

Thanks
Liang

[Shuichi Ihara (Inactive)]

Liang,

here are results you requested. I just remount the lustre on the clients. do you need the following results when the problem happens?

1. /proc/sys/lnet/peers on router2

1. cat/proc/sys/lnet/peers
   nid refs state max rtr min tx min queue
   192.168.200.123@o2ib1 1 up 8 8 5 8 5 0
   192.168.20.121@tcp 3 up 8 8 5 8 -1029 0

2./proc/sys/lnet/peers and /proc/sys/lnet/routes on client

1. cat /proc/sys/lnet/peers
   nid refs state max rtr min tx min queue
   192.168.200.122@o2ib1 3 up 8 8 8 8 5 0

1. cat /proc/sys/lnet/routes
   Routing disabled
   net hops state router
   o2ib 1 up 192.168.200.122@o2ib1

3. lctl ping route2 on client

1. lctl ping 192.168.200.122@o2ib1
   12345-0@lo
   12345-192.168.20.122@tcp
   12345-192.168.200.122@o2ib1

4. lctl ping client on the route2

1. lctl ping 192.168.200.123@o2ib1
   12345-0@lo
   12345-192.168.200.123@o2ib1

I'm using the lustre-1.8.4.ddn2 which is based on lustre-1.8.4 and backported patches mostly from 1.8.5.

[Liang Zhen (Inactive)]

Ihara, yes please collect these information when you got the problem, and if possible, please attach source tar-ball of lnet at here.

Thanks
Liang

[Shuichi Ihara (Inactive)]

Here are results when the problem happened.

1. /proc/sys/lnet/peers on router2
[root@r03 ~]# cat /proc/sys/lnet/peers
nid refs state max rtr min tx min queue
192.168.20.121@tcp 3 up 8 8 8 8 5 0

2. /proc/sys/lnet/peers and /proc/sys/lnet/routes on client
[root@r04 ~]# cat /proc/sys/lnet/peers
nid refs state max rtr min tx min queue
192.168.200.122@o2ib1 3 down 8 8 8 8 5 0

[root@r04 ~]# cat /proc/sys/lnet/routes
Routing disabled
net hops state router
o2ib 1 down 192.168.200.122@o2ib1

3. lctl ping route2 on client

[root@r04 ~]# lctl ping 192.168.200.122@o2ib1
failed to ping 192.168.200.122@o2ib1: Input/output error
[root@r04 ~]# lctl ping 192.168.200.122@o2ib1
12345-0@lo
12345-192.168.20.122@tcp
12345-192.168.200.122@o2ib1

4. lctl ping client on the route2

lctl ping route2 on client

[root@r03 ~]# lctl ping 192.168.200.123@o2ib1
12345-0@lo
12345-192.168.200.123@o2ib1

when I did "lctl ping route2" on client, got an Input/output error, but tried again once, the connection restore, then the client did connection to MGS and recovered the filesystem correctly.

And if "lctl ping client" on router2, it can also help connection restore once I did.
So, we can get the connection back once ping client from route2 or router2 from clients.

[Shuichi Ihara (Inactive)]

attached is part of lnet code in lustre-1.8.4.ddn2.

[Liang Zhen (Inactive)]

Ihara,

I think the reason is:

• the client can only connect to one router (route2), client will mark route2 as DOWN when you turn down it.
• it never come up again because:
  • any message from upper layer wouldn't be sent because the only router is down
  • LNet itself will not send anything to check the router because dead_router_check_interval is not enabled

So I think you can add this to your modprobe.conf (on clients and servers)
"dead_router_check_interval=5", so the worst case, you will see the router come up again in 10 seconds (two pings)

You may also want to add this, "live_router_check_interval=60", so client/server will check live router for each minute.

Liang

[Shuichi Ihara (Inactive)]

Liang,

Thanks. It can be fixed by your advises, but I needed to add two parameters (dead_router_check_interval=5 and live_router_check_interval=60) not only on servers/clients, but also both routers.
Because router2 needs to detect all router1 fails (router1 also has to detect router2 fails), otherwise the clients can't restore the connections due to clients are not checking router1 and servers also can't detect router2 fails.

I just wonder if we could see any notices on the routers when the connection back.

when all router1 fails, we can see the following error messages on router2.

Jan 5 22:23:14 r03 kernel: [127541.771483] Lustre: No route to 12345-192.168.100.120@o2ib via LNET_NID_ANY (all routers down)
Jan 5 22:23:14 r03 kernel: [127541.780334] Lustre: Skipped 3 previous similar messages

However, even router1 is back from fails and all connection restores on the clients, there are no any messages on router2.
on clients, the we can see the following messages to indicate the connection is restored.

Jan 5 22:22:34 r11 kernel: [ 3202.159898] Lustre: MGC192.168.100.120@o2ib: Connection restored to service MGS using nid 192.168.100.120@o2ib.
Jan 5 22:22:34 r11 kernel: [ 3202.159903] Lustre: Skipped 1 previous similar message
Jan 5 22:22:40 r11 kernel: [ 3208.029050] Lustre: 12298:0:(import.c:517:import_select_connection()) lustre-OST0000-osc-ffff810319b57400: tried all connections, increasing latency to 6s

Ihara

[Liang Zhen (Inactive)]

Ihara,
I think the most reliable information indicating status of routers are those files under proc, we can't rely on console output anyway.
How do you think if I change status of this to "resolved"?

Thanks
Liang

[Shuichi Ihara (Inactive)]

Liang, it's enough now. And, yes, we do get information from /proc as much as possible. please close this ticket.

Many thanks!

Ihara

[Liang Zhen (Inactive)]

mark it as resolved