[LU-4136] MDT temporarily unhealthy when restarting

Tue, 31 Dec 2013 15:42:10 +0000

Hi,

When restarting an MDT, we consistently see that its status under /proc/fs/lustre/health_check is temporarily unhealthy.

Here are some logs:

00000004:02000400:1.0F:Tue Oct 22 15:23:52 CEST 2013:0:11263:0:(mdt_recovery.c:233:mdt_server_data_init()) fs1-MDT0000: used disk, loading
00000020:02000000:8.0F:Tue Oct 22 15:23:52 CEST 2013:0:11086:0:(obd_mount_server.c:1776:server_calc_timeout()) fs1-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450
00000100:00000400:4.0:Tue Oct 22 15:23:57 CEST 2013:0:5640:0:(client.c:1869:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1382448232/real 0]  req@ffff8810789a9000 x1449588289516076/t0(0) o38->fs1-MDT0000-lwp-MDT0000@10.3.0.11@o2ib:12/10 lens 400/544 e 0 to 1 dl 1382448237 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
00000100:00000400:4.0:Tue Oct 22 15:23:57 CEST 2013:0:5640:0:(client.c:1869:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1382448232/real 0]  req@ffff881071d6f400 x1449588289513784/t0(0) o8->fs1-OST0006-osc-MDT0000@10.4.0.6@o2ib1:28/4 lens 400/544 e 0 to 1 dl 1382448237 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
00000100:00000400:4.0:Tue Oct 22 15:23:57 CEST 2013:0:5640:0:(client.c:1869:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1382448232/real 0]  req@ffff8808653cb800 x1449588289513644/t0(0) o8->fs1-OST0005-osc-MDT0000@10.3.0.6@o2ib:28/4 lens 400/544 e 0 to 1 dl 1382448237 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
00000100:00000400:1.0:Tue Oct 22 15:23:59 CEST 2013:0:6207:0:(client.c:1869:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1382448232/real 0]  req@ffff880876dbc000 x1449588289516292/t0(0) o104->MGS@10.3.0.11@o2ib:15/16 lens 296/224 e 0 to 1 dl 1382448239 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
00010000:02020000:1.0:Tue Oct 22 15:23:59 CEST 2013:0:6207:0:(ldlm_lockd.c:641:ldlm_failed_ast()) 138-a: MGS: A client on nid 10.3.0.11@o2ib was evicted due to a lock blocking callback time out: rc -107
00000100:02000000:9.0F:Tue Oct 22 15:24:18 CEST 2013:0:5640:0:(import.c:1407:ptlrpc_import_recovery_state_machine()) fs1-OST0006-osc-MDT0000: Connection restored to fs1-OST0006 (at 10.4.0.3@o2ib1)
00010000:02000400:5.0F:Tue Oct 22 15:24:29 CEST 2013:0:11219:0:(ldlm_lib.c:1581:target_start_recovery_timer()) fs1-MDT0000: Will be in recovery for at least 2:30, or until 1 client reconnects
00010000:02000000:3.0F:Tue Oct 22 15:24:29 CEST 2013:0:11285:0:(ldlm_lib.c:1420:target_finish_recovery()) fs1-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
00000100:02000000:9.0:Tue Oct 22 15:24:55 CEST 2013:0:5640:0:(import.c:1407:ptlrpc_import_recovery_state_machine()) fs1-OST0005-osc-MDT0000: Connection restored to fs1-OST0005 (at 10.3.0.3@o2ib)

As we can see, as soon as MDT is started it has troubles connecting to several OSTs. Moreover a recovery is beginning, but it finishes soon. However, the MDT becomes healthy only when the connection to all OSTs is restored, ie at 15:24:55. Indeed, from 15:23:52 when it is started to 15:24:55 when the connection to last OST is restored, the MDT is reporting unhealthy status.

We can understand that an MDT that has not been able to connect to its OSTs is unhealthy, but we do not understand why it has troubles connecting to them, as there are no errors on the network.
It seems that with Lustre 2.4 the connection between MDT and OSTs is hard to establish, and takes some time before being restored (we have other examples where it took more than 2 minutes to do so).

The problem with this situation is that we monitor Lustre MDT and OST health status for HA purpose. If a target is seen as unhealthy, the node hosting this resource can be fenced.

Thanks,
Sebastien.

Whamcloud Community JIRA

[LU-4136] MDT temporarily unhealthy when restarting