18:03:37:Lustre: DEBUG MARKER: == recovery-small test 101: IR: Make sure IR works w/o normal recovery == 18:03:35 (1346807015) 18:03:48:Lustre: DEBUG MARKER: /usr/sbin/lctl list_param mgs.*.ir_timeout 18:03:48:Lustre: DEBUG MARKER: lctl set_param -n mgs.MGS.live.lustre=state=full 18:04:10:Lustre: 2035:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346807025/real 1346807025] req@ffff88007822ec00 x1412229275582789/t0(0) o400->lustre-OST0001-osc-MDT0000@10.10.4.89@tcp:28/4 lens 224/224 e 0 to 1 dl 1346807049 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 18:04:10:Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 10.10.4.89@tcp) was lost; in progress operations using this service will wait for recovery to complete 18:04:21:Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 10.10.4.89@tcp) was lost; in progress operations using this service will wait for recovery to complete 18:04:21:Lustre: Skipped 5 previous similar messages 18:04:52:Lustre: 2036:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346807040/real 1346807040] req@ffff8800763ebc00 x1412229275582812/t0(0) o400->lustre-OST0000-osc-MDT0000@10.10.4.93@tcp:28/4 lens 224/224 e 0 to 1 dl 1346807084 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 18:04:52:Lustre: 2036:0:(client.c:1917:ptlrpc_expire_one_request()) Skipped 53 previous similar messages 18:05:04:LustreError: 11-0: an error occurred while communicating with 10.10.4.93@tcp. The ost_connect operation failed with -19 18:05:35:LustreError: 11-0: an error occurred while communicating with 10.10.4.93@tcp. The ost_connect operation failed with -19 18:05:35:LustreError: Skipped 5 previous similar messages 18:05:57:Lustre: 2034:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1346807149/real 1346807149] req@ffff880076456c00 x1412229275582911/t0(0) o8->lustre-OST0006-osc-MDT0000@10.10.4.89@tcp:28/4 lens 400/544 e 0 to 1 dl 1346807174 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 18:05:57:Lustre: 2034:0:(client.c:1917:ptlrpc_expire_one_request()) Skipped 23 previous similar messages 18:06:08:LustreError: 11-0: an error occurred while communicating with 10.10.4.93@tcp. The ost_connect operation failed with -19 18:06:08:LustreError: Skipped 10 previous similar messages 18:06:29:Lustre: lustre-OST0000-osc-MDT0000: Connection restored to lustre-OST0000 (at 10.10.4.93@tcp) 18:06:29:Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0000_UUID now active, resetting orphans 18:07:10:Lustre: lustre-OST0001-osc-MDT0000: Connection restored to lustre-OST0001 (at 10.10.4.93@tcp) 18:07:10:Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0001_UUID now active, resetting orphans 18:07:10:LustreError: 11-0: an error occurred while communicating with 10.10.4.93@tcp. The ost_connect operation failed with -19 18:07:10:LustreError: Skipped 27 previous similar messages 18:08:12:Lustre: 2034:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1346807284/real 1346807284] req@ffff8800763ee400 x1412229275583093/t0(0) o8->lustre-OST0002-osc-MDT0000@10.10.4.89@tcp:28/4 lens 400/544 e 0 to 1 dl 1346807310 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 18:08:12:Lustre: 2034:0:(client.c:1917:ptlrpc_expire_one_request()) Skipped 56 previous similar messages 18:08:53:Lustre: lustre-OST0002-osc-MDT0000: Connection restored to lustre-OST0002 (at 10.10.4.93@tcp) 18:08:53:Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0002_UUID now active, resetting orphans 18:09:55:LustreError: 11-0: an error occurred while communicating with 10.10.4.93@tcp. The ost_connect operation failed with -19 18:09:55:LustreError: Skipped 47 previous similar messages 18:10:16:Lustre: lustre-OST0003-osc-MDT0000: Connection restored to lustre-OST0003 (at 10.10.4.93@tcp) 18:10:16:Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0003_UUID now active, resetting orphans 18:11:37:Lustre: lustre-OST0004-osc-MDT0000: Connection restored to lustre-OST0004 (at 10.10.4.93@tcp) 18:11:37:Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0004_UUID now active, resetting orphans 18:12:28:Lustre: 2034:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1346807544/real 1346807544] req@ffff8800782b2400 x1412229275583459/t0(0) o8->lustre-OST0005-osc-MDT0000@10.10.4.89@tcp:28/4 lens 400/544 e 0 to 1 dl 1346807570 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 18:12:28:Lustre: 2034:0:(client.c:1917:ptlrpc_expire_one_request()) Skipped 53 previous similar messages 18:13:30:Lustre: lustre-OST0005-osc-MDT0000: Connection restored to lustre-OST0005 (at 10.10.4.93@tcp) 18:13:30:Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0005_UUID now active, resetting orphans 18:14:12:LustreError: 11-0: an error occurred while communicating with 10.10.4.93@tcp. The ost_connect operation failed with -19 18:14:12:LustreError: Skipped 38 previous similar messages 18:14:44:Lustre: lustre-OST0006-osc-MDT0000: Connection restored to lustre-OST0006 (at 10.10.4.93@tcp) 19:03:41:********** Timeout by autotest system **********19:04:29: connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4298698804, last ping 4298703804, now 4298708804 19:04:29: connection1:0: detected conn error (1011) 19:04:40:Lustre: MGS: haven't heard from client 361ca9cc-129d-3660-c5b7-0edb83795263 (at 10.10.4.91@tcp) in 47 seconds. I think it's dead, and I am evicting it. exp ffff8800782a8800, cur 1346810674 expire 1346810644 last 1346810627 19:04:40:Lustre: MGS: haven't heard from client 69ed96bf-e5fc-bf3a-f577-46a0fdda4cab (at 10.10.4.90@tcp) in 34 seconds. I think it's dead, and I am evicting it. exp ffff880078341800, cur 1346810674 expire 1346810644 last 1346810640 19:05:12:Lustre: 2036:0:(client.c:1917:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1346810699/real 1346810699] req@ffff880037b90400 x1412229275588496/t0(0) o400->lustre-OST0001-osc-MDT0000@10.10.4.93@tcp:28/4 lens 224/224 e 0 to 1 dl 1346810706 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 19:05:12:Lustre: 2036:0:(client.c:1917:ptlrpc_expire_one_request()) Skipped 9 previous similar messages 19:05:12:Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 10.10.4.93@tcp) was lost; in progress operations using this service will wait for recovery to complete 19:05:23:Lustre: MGS: haven't heard from client 6cc3d066-2650-741b-fb5d-e09c81f5001a (at 10.10.4.86@tcp) in 48 seconds. I think it's dead, and I am evicting it. exp ffff8800782a8400, cur 1346810716 expire 1346810686 last 1346810668 19:05:34: 19:05:34:<ConMan> Console [fat-intel-3vm7] disconnected from <fat-intel-3:6006> at 09-04 19:05.