2014-10-24 14:20:55 zwicky-lcy-oss2 login: Lustre: Lustre: Build Version: 2.5.3-1chaos0morrone-1chaos0morrone--PRISTINE-2.6.32-431.29.2.1chaos.ch5.2.x86_64 2014-10-24 14:21:32 LustreError: 137-5: lcy-OST0001_UUID: not available for connect from 192.168.121.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. 2014-10-24 14:21:33 LustreError: 137-5: lcy-OST0000_UUID: not available for connect from 192.168.121.140@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. 2014-10-24 14:21:33 LustreError: Skipped 3 previous similar messages 2014-10-24 14:21:34 LustreError: 137-5: lcy-OST0001_UUID: not available for connect from 192.168.121.34@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. 2014-10-24 14:21:37 LustreError: 137-5: lcy-OST0001_UUID: not available for connect from 192.168.121.66@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. 2014-10-24 14:21:37 LustreError: Skipped 10 previous similar messages 2014-10-24 14:21:38 LustreError: 7421:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880fef274800 x1482881169883144/t0(0) o253->MGC10.1.1.169@o2ib9@10.1.1.169@o2ib9:26/25 lens 4768/4768 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 2014-10-24 14:21:38 LustreError: 7421:0:(obd_mount_server.c:1120:server_register_target()) lcy-OST0001: error registering with the MGS: rc = -5 (not fatal) 2014-10-24 14:21:41 LustreError: 137-5: lcy-OST0001_UUID: not available for connect from 192.168.121.134@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. 2014-10-24 14:21:41 LustreError: Skipped 6 previous similar messages 2014-10-24 14:21:44 LustreError: 7421:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880fef274800 x1482881169883148/t0(0) o101->MGC10.1.1.169@o2ib9@10.1.1.169@o2ib9:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 2014-10-24 14:21:50 LustreError: 7421:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880fef274800 x1482881169883152/t0(0) o101->MGC10.1.1.169@o2ib9@10.1.1.169@o2ib9:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 2014-10-24 14:21:50 LustreError: 137-5: lcy-OST0001_UUID: not available for connect from 192.168.121.133@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. 2014-10-24 14:21:50 LustreError: Skipped 23 previous similar messages 2014-10-24 14:21:56 LustreError: 7421:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880fef274000 x1482881169883156/t0(0) o101->MGC10.1.1.169@o2ib9@10.1.1.169@o2ib9:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 2014-10-24 14:21:56 LustreError: 13a-8: Failed to get MGS log params and no local copy. 2014-10-24 14:22:02 LustreError: 7421:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff880fef274000 x1482881169883160/t0(0) o101->MGC10.1.1.169@o2ib9@10.1.1.169@o2ib9:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 2014-10-24 14:22:06 LustreError: 137-5: lcy-OST0000_UUID: not available for connect from 192.168.121.132@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. 2014-10-24 14:22:06 LustreError: Skipped 37 previous similar messages 2014-10-24 14:22:08 LustreError: 13a-8: Failed to get MGS log params and no local copy. 2014-10-24 14:22:19 LustreError: 7421:0:(client.c:1083:ptlrpc_import_delay_req()) @@@ send limit expired req@ffff88081f303000 x1482881169883172/t0(0) o253->MGC10.1.1.169@o2ib9@10.1.1.169@o2ib9:26/25 lens 4768/4768 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1 2014-10-24 14:22:19 LustreError: 7421:0:(client.c:1083:ptlrpc_import_delay_req()) Skipped 1 previous similar message 2014-10-24 14:22:19 Lustre: lcy-OST0001: Will be in recovery for at least 5:00, or until 135 clients reconnect. 2014-10-24 14:26:12 LustreError: 11-0: lcy-MDT0000-lwp-OST0001: Communicating with 10.1.1.169@o2ib9, operation mds_connect failed with -11. 2014-10-24 14:26:15 LustreError: 137-5: lcy-OST0000_UUID: not available for connect from 10.1.1.169@o2ib9 (no target). If you are running an HA pair check that the target is mounted on the other server. 2014-10-24 14:26:15 LustreError: Skipped 50 previous similar messages 2014-10-24 14:26:38 INFO: task tgt_recov:7578 blocked for more than 120 seconds. 2014-10-24 14:26:38 Tainted: P --------------- 2.6.32-431.29.2.1chaos.ch5.2.x86_64 #1 2014-10-24 14:26:38 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 2014-10-24 14:26:38 tgt_recov D 0000000000000009 0 7578 2 0x00000000 2014-10-24 14:26:38 ffff8810292b7db0 0000000000000046 0000000000000000 ffff8810292b7d74 2014-10-24 14:26:38 ffff881000000000 ffff88083fe83000 ffff88085c4b6840 0000000000000400 2014-10-24 14:26:38 ffff881032fd65f8 ffff8810292b7fd8 000000000000fbc8 ffff881032fd65f8 2014-10-24 14:26:38 Call Trace: 2014-10-24 14:26:38 [] ? check_for_clients+0x0/0x70 [ptlrpc] 2014-10-24 14:26:38 [] target_recovery_overseer+0x95/0x250 [ptlrpc] 2014-10-24 14:26:38 [] ? exp_connect_healthy+0x0/0x20 [ptlrpc] 2014-10-24 14:26:38 [] ? autoremove_wake_function+0x0/0x40 2014-10-24 14:26:38 [] ? target_recovery_thread+0x0/0x19d0 [ptlrpc] 2014-10-24 14:26:38 [] target_recovery_thread+0x540/0x19d0 [ptlrpc] 2014-10-24 14:26:38 [] ? default_wake_function+0x12/0x20 2014-10-24 14:26:38 [] ? target_recovery_thread+0x0/0x19d0 [ptlrpc] 2014-10-24 14:26:38 [] kthread+0x96/0xa0 2014-10-24 14:26:38 [] child_rip+0xa/0x20 2014-10-24 14:26:38 [] ? kthread+0x0/0xa0 2014-10-24 14:26:38 [] ? child_rip+0x0/0x20 2014-10-24 14:26:42 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778574327, ql: 11, comp: 124, conn: 135, next: 287778574336, last_committed: 287778573956) 2014-10-24 14:26:42 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778574337, ql: 11, comp: 124, conn: 135, next: 287778574338, last_committed: 287778573956) 2014-10-24 14:26:42 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778574339, ql: 11, comp: 124, conn: 135, next: 287778574359, last_committed: 287778573956) 2014-10-24 14:26:42 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778574360, ql: 11, comp: 124, conn: 135, next: 287778574371, last_committed: 287778573956) 2014-10-24 14:26:42 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778574372, ql: 11, comp: 124, conn: 135, next: 287778574383, last_committed: 287778573956) 2014-10-24 14:26:42 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778574384, ql: 11, comp: 124, conn: 135, next: 287778574390, last_committed: 287778573956) 2014-10-24 14:26:42 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778574392, ql: 11, comp: 124, conn: 135, next: 287778574395, last_committed: 287778573956) 2014-10-24 14:26:42 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778574396, ql: 11, comp: 124, conn: 135, next: 287778574406, last_committed: 287778573956) 2014-10-24 14:26:42 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778574407, ql: 11, comp: 124, conn: 135, next: 287778574422, last_committed: 287778573956) 2014-10-24 14:26:42 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778574423, ql: 11, comp: 124, conn: 135, next: 287778574424, last_committed: 287778573956) 2014-10-24 14:26:42 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778574425, ql: 11, comp: 124, conn: 135, next: 287778574428, last_committed: 287778573956) 2014-10-24 14:26:42 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778574429, ql: 11, comp: 124, conn: 135, next: 287778574436, last_committed: 287778573956) 2014-10-24 14:26:42 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778574437, ql: 11, comp: 124, conn: 135, next: 287778574443, last_committed: 287778573956) 2014-10-24 14:26:42 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778574444, ql: 11, comp: 124, conn: 135, next: 287778574445, last_committed: 287778573956) 2014-10-24 14:26:42 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778574446, ql: 11, comp: 124, conn: 135, next: 287778574451, last_committed: 287778573956) 2014-10-24 14:26:42 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778574452, ql: 11, comp: 124, conn: 135, next: 287778574453, last_committed: 287778573956) 2014-10-24 14:26:45 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778575465, ql: 5, comp: 130, conn: 135, next: 287778575528, last_committed: 287778574168) 2014-10-24 14:26:45 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778575529, ql: 5, comp: 130, conn: 135, next: 287778575569, last_committed: 287778574168) 2014-10-24 14:26:46 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778575718, ql: 5, comp: 130, conn: 135, next: 287778575732, last_committed: 287778574635) 2014-10-24 14:26:46 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778575733, ql: 5, comp: 130, conn: 135, next: 287778575818, last_committed: 287778574635) 2014-10-24 14:26:46 LustreError: 7578:0:(ldlm_lib.c:1738:check_for_next_transno()) lcy-OST0001: waking for gap in transno, VBR is OFF (skip: 287778575981, ql: 5, comp: 130, conn: 135, next: 287778575982, last_committed: 287778574635) 2014-10-24 14:27:21 Lustre: lcy-OST0001: Recovery over after 5:02, of 135 clients 135 recovered and 0 were evicted. 2014-10-24 14:27:21 Lustre: lcy-OST0001: deleting orphan objects from 0x0:92927327 to 0x0:92930881