https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-33 Lustre <p>I'm testing on Lustre N-hop routing (e.g. o2ib0 &lt;&gt; tcp &lt;&gt; o2ib1) below.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> MDS/OSS &lt;-- IB (o2ib0) --&gt; Router1 &lt;-- TCP (tcp0) --&gt; Router2 &lt;-- IB (o2ib1) --&gt; Client - Network configuration - There are two IB fabrics and 1GbE connects both fabrics with LNET routers. MDS/OSS IP address: 192.168.100.120@o2ib0 options lnet networks=o2ib0 routes="tcp0 192.168.100.121@o2ib0; o2ib1 192.168.100.121@o2ib0" Router1 IP address: 192.168.100.121@o2ib0, 192.168.10.121@tcp0 options lnet ip2nets="tcp0 192.168.20.*; o2ib0(ib0) 192.168.100.*" routes="o2ib1 192.168.20.122@tcp0" forwarding="enabled" Router2 IP address: 192.168.200.122@o2ib1, 192.168.10.122@tcp0 options lnet ip2nets="tcp0 192.168.20.*; o2ib1(ib0) 192.168.200.*" routes="o2ib0 192.168.20.121@tcp0" forwarding="enabled" Client IP address: 192.168.200.123@o2ib1 options lnet networks=o2ib1(ib0) routes="o2ib0 192.168.200.122@o2ib1" </pre> </div></div> <p>It worked with above the configurations, but it seems that there is an issue if Router2 downs (e.g. 'lctl net down'), then restart it. The problem is that the client can't be recovery unless the client umount and remount the filesytem on it.</p> LU-33

client can't recover on N-hop router configuraton

Bug Minor Resolved Fixed Liang Zhen Shuichi Ihara Tue, 28 Dec 2010 01:56:31 +0000 Tue, 28 Jun 2011 15:01:39 +0000 Fri, 28 Jan 2011 00:45:30 +0000 Lustre 1.8.6 0 1 <p>Ihara,</p> <p>could you help me to get a few things after restarting the router in the test:</p> <ol> <li>cat /proc/sys/lnet/peers on router2</li> <li>cat /proc/sys/lnet/peers and /proc/sys/lnet/routes on client</li> <li>lctl ping route2 on client to see if it can work, if it can't please get error messages on client</li> <li>lctl ping client on route2 to see if it can work, if it can't please get error messages on router2</li> </ol> <p>NB: also, could you tell me detail version of your lustre? </p> <p>Thanks<br/> Liang</p> <p>Liang,</p> <p>here are results you requested. I just remount the lustre on the clients. do you need the following results when the problem happens?</p> <p>1. /proc/sys/lnet/peers on router2</p> <ol> <li>cat/proc/sys/lnet/peers<br/> nid refs state max rtr min tx min queue<br/> 192.168.200.123@o2ib1 1 up 8 8 5 8 5 0<br/> 192.168.20.121@tcp 3 up 8 8 5 8 -1029 0</li> </ol> <p>2./proc/sys/lnet/peers and /proc/sys/lnet/routes on client</p> <ol> <li>cat /proc/sys/lnet/peers<br/> nid refs state max rtr min tx min queue<br/> 192.168.200.122@o2ib1 3 up 8 8 8 8 5 0</li> </ol> <ol> <li>cat /proc/sys/lnet/routes<br/> Routing disabled<br/> net hops state router<br/> o2ib 1 up 192.168.200.122@o2ib1</li> </ol> <p>3. lctl ping route2 on client</p> <ol> <li>lctl ping 192.168.200.122@o2ib1<br/> 12345-0@lo<br/> 12345-192.168.20.122@tcp<br/> 12345-192.168.200.122@o2ib1</li> </ol> <p>4. lctl ping client on the route2</p> <ol> <li>lctl ping 192.168.200.123@o2ib1<br/> 12345-0@lo<br/> 12345-192.168.200.123@o2ib1</li> </ol> <p>I'm using the lustre-1.8.4.ddn2 which is based on lustre-1.8.4 and backported patches mostly from 1.8.5.</p> <p>Ihara, yes please collect these information when you got the problem, and if possible, please attach source tar-ball of lnet at here.</p> <p>Thanks<br/> Liang</p> <p>Here are results when the problem happened.</p> <p>1. /proc/sys/lnet/peers on router2<br/> <span class="error">&#91;root@r03 ~&#93;</span># cat /proc/sys/lnet/peers <br/> nid refs state max rtr min tx min queue<br/> 192.168.20.121@tcp 3 up 8 8 8 8 5 0</p> <p>2. /proc/sys/lnet/peers and /proc/sys/lnet/routes on client<br/> <span class="error">&#91;root@r04 ~&#93;</span># cat /proc/sys/lnet/peers <br/> nid refs state max rtr min tx min queue<br/> 192.168.200.122@o2ib1 3 down 8 8 8 8 5 0</p> <p><span class="error">&#91;root@r04 ~&#93;</span># cat /proc/sys/lnet/routes<br/> Routing disabled<br/> net hops state router<br/> o2ib 1 down 192.168.200.122@o2ib1</p> <p>3. lctl ping route2 on client</p> <p><span class="error">&#91;root@r04 ~&#93;</span># lctl ping 192.168.200.122@o2ib1<br/> failed to ping 192.168.200.122@o2ib1: Input/output error<br/> <span class="error">&#91;root@r04 ~&#93;</span># lctl ping 192.168.200.122@o2ib1<br/> 12345-0@lo<br/> 12345-192.168.20.122@tcp<br/> 12345-192.168.200.122@o2ib1</p> <p>4. lctl ping client on the route2</p> <p>lctl ping route2 on client</p> <p><span class="error">&#91;root@r03 ~&#93;</span># lctl ping 192.168.200.123@o2ib1<br/> 12345-0@lo<br/> 12345-192.168.200.123@o2ib1</p> <p>when I did "lctl ping route2" on client, got an Input/output error, but tried again once, the connection restore, then the client did connection to MGS and recovered the filesystem correctly.</p> <p>And if "lctl ping client" on router2, it can also help connection restore once I did.<br/> So, we can get the connection back once ping client from route2 or router2 from clients.</p> <p>attached is part of lnet code in lustre-1.8.4.ddn2.</p> <p>Ihara,</p> <p>I think the reason is:</p> <ul> <li>the client can only connect to one router (route2), client will mark route2 as DOWN when you turn down it.</li> <li>it never come up again because: <ul> <li>any message from upper layer wouldn't be sent because the only router is down</li> <li>LNet itself will not send anything to check the router because dead_router_check_interval is not enabled</li> </ul> </li> </ul> <p>So I think you can add this to your modprobe.conf (on clients and servers)<br/> "dead_router_check_interval=5", so the worst case, you will see the router come up again in 10 seconds (two pings)</p> <p>You may also want to add this, "live_router_check_interval=60", so client/server will check live router for each minute.</p> <p>Liang</p> <p>Liang,</p> <p>Thanks. It can be fixed by your advises, but I needed to add two parameters (dead_router_check_interval=5 and live_router_check_interval=60) not only on servers/clients, but also both routers.<br/> Because router2 needs to detect all router1 fails (router1 also has to detect router2 fails), otherwise the clients can't restore the connections due to clients are not checking router1 and servers also can't detect router2 fails.</p> <p>I just wonder if we could see any notices on the routers when the connection back.</p> <p>when all router1 fails, we can see the following error messages on router2.</p> <p>Jan 5 22:23:14 r03 kernel: <span class="error">&#91;127541.771483&#93;</span> Lustre: No route to 12345-192.168.100.120@o2ib via LNET_NID_ANY (all routers down)<br/> Jan 5 22:23:14 r03 kernel: <span class="error">&#91;127541.780334&#93;</span> Lustre: Skipped 3 previous similar messages</p> <p>However, even router1 is back from fails and all connection restores on the clients, there are no any messages on router2.<br/> on clients, the we can see the following messages to indicate the connection is restored.</p> <p>Jan 5 22:22:34 r11 kernel: [ 3202.159898] Lustre: MGC192.168.100.120@o2ib: Connection restored to service MGS using nid 192.168.100.120@o2ib.<br/> Jan 5 22:22:34 r11 kernel: [ 3202.159903] Lustre: Skipped 1 previous similar message<br/> Jan 5 22:22:40 r11 kernel: [ 3208.029050] Lustre: 12298:0:(import.c:517:import_select_connection()) lustre-OST0000-osc-ffff810319b57400: tried all connections, increasing latency to 6s</p> <p>Ihara</p> <p>Ihara, <br/> I think the most reliable information indicating status of routers are those files under proc, we can't rely on console output anyway.<br/> How do you think if I change status of this to "resolved"?</p> <p>Thanks<br/> Liang</p> <p>Liang, it's enough now. And, yes, we do get information from /proc as much as possible. please close this ticket.</p> <p>Many thanks!</p> <p>Ihara</p> <p>mark it as resolved</p> Development Rank 1|hzw0lj: Rank (Obsolete) 10177 Severity