2012-11-12 16:25:54 Lustre: Lustre: Build Version: 2.3.54-4chaos-3surya1-3surya1--PRISTINE-2.6.32-220.23.1.2chaos.ch5.x86_64 2012-11-12 16:25:55 LustreError: 6447:0:(mgc_request.c:248:do_config_log_add()) failed processing sptlrpc log: -2 2012-11-12 16:25:55 LustreError: 11-0: lstest-MDT0000-osp-OST0193: Communicating with 172.20.5.2@o2ib500, operation mds_connect failed with -11 2012-11-12 16:25:55 Lustre: lstest-OST0193: Will be in recovery for at least 5:00, or until 275 clients reconnect. 2012-11-12 16:25:56 LustreError: 6528:0:(ldlm_lockd.c:824:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: filter-ffff8807fef4c000 lock: ffff880ff005dcc0/0xbdf5847332a7d090 lrc: 3/0,0 mode: PW/PW res: 10773567/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 nid: 172.20.4.149@o2ib500 remote: 0x9c0890bf799c59d0 expref: 4 pid: 6531 timeout 0 2012-11-12 16:25:56 LustreError: 137-5: UUID 'lstest-OST0194_UUID' is not available for connect (no target) 2012-11-12 16:26:02 LustreError: 137-5: UUID 'lstest-OST0194_UUID' is not available for connect (no target) 2012-11-12 16:26:06 LustreError: 137-5: UUID 'lstest-OST0194_UUID' is not available for connect (no target) 2012-11-12 16:26:20 LustreError: 137-5: UUID 'lstest-OST0194_UUID' is not available for connect (no target) 2012-11-12 16:26:20 LustreError: 11-0: lstest-MDT0000-osp-OST0193: Communicating with 172.20.5.2@o2ib500, operation mds_connect failed with -11 2012-11-12 16:26:31 LustreError: 137-5: UUID 'lstest-OST0194_UUID' is not available for connect (no target) 2012-11-12 16:26:45 LustreError: 11-0: lstest-MDT0000-osp-OST0193: Communicating with 172.20.5.2@o2ib500, operation mds_connect failed with -11 2012-11-12 16:26:51 LustreError: 137-5: UUID 'lstest-OST0194_UUID' is not available for connect (no target) 2012-11-12 16:27:01 LustreError: 137-5: UUID 'lstest-OST0194_UUID' is not available for connect (no target) 2012-11-12 16:27:01 LustreError: Skipped 2 previous similar messages 2012-11-12 16:27:10 LustreError: 11-0: lstest-MDT0000-osp-OST0193: Communicating with 172.20.5.2@o2ib500, operation mds_connect failed with -11 2012-11-12 16:27:35 LustreError: 11-0: lstest-MDT0000-osp-OST0193: Communicating with 172.20.5.2@o2ib500, operation mds_connect failed with -11 2012-11-12 16:27:51 Lustre: lstest-OST0193: Client 7877635e-a0c6-353f-51e9-47e6f0ef5fb2 (at 172.20.17.2@o2ib500) reconnecting, waiting for 275 clients in recovery for 3:04 2012-11-12 16:27:51 Lustre: lstest-OST0193: Client 7877635e-a0c6-353f-51e9-47e6f0ef5fb2 (at 172.20.17.2@o2ib500) refused reconnection, still busy with 1 active RPCs 2012-11-12 16:27:54 Lustre: lstest-OST0193: Client df4b9103-f4bf-8082-3f31-a1512a4dda76 (at 172.20.17.7@o2ib500) reconnecting, waiting for 275 clients in recovery for 3:01 2012-11-12 16:27:54 Lustre: lstest-OST0193: Client df4b9103-f4bf-8082-3f31-a1512a4dda76 (at 172.20.17.7@o2ib500) refused reconnection, still busy with 1 active RPCs 2012-11-12 16:27:57 Lustre: lstest-OST0193: Client 4028a636-dc0d-66a7-557b-f4d960ae30a7 (at 172.20.17.9@o2ib500) reconnecting, waiting for 275 clients in recovery for 2:58 2012-11-12 16:27:57 Lustre: lstest-OST0193: Client 4028a636-dc0d-66a7-557b-f4d960ae30a7 (at 172.20.17.9@o2ib500) refused reconnection, still busy with 1 active RPCs 2012-11-12 16:27:58 Lustre: lstest-OST0193: Client 6b5359f5-fbbd-23ea-2c3e-9f96a635e074 (at 172.20.17.12@o2ib500) reconnecting, waiting for 275 clients in recovery for 2:57 2012-11-12 16:27:58 Lustre: lstest-OST0193: Client 6b5359f5-fbbd-23ea-2c3e-9f96a635e074 (at 172.20.17.12@o2ib500) refused reconnection, still busy with 1 active RPCs 2012-11-12 16:28:00 LustreError: 11-0: lstest-MDT0000-osp-OST0193: Communicating with 172.20.5.2@o2ib500, operation mds_connect failed with -11 2012-11-12 16:28:01 Lustre: lstest-OST0193: Client 2cc7281f-3e5e-2b27-a226-dd1ad869ab9c (at 172.20.17.3@o2ib500) reconnecting, waiting for 275 clients in recovery for 2:54 2012-11-12 16:28:01 Lustre: Skipped 2 previous similar messages 2012-11-12 16:28:01 Lustre: lstest-OST0193: Client 2cc7281f-3e5e-2b27-a226-dd1ad869ab9c (at 172.20.17.3@o2ib500) refused reconnection, still busy with 1 active RPCs 2012-11-12 16:28:01 Lustre: Skipped 2 previous similar messages 2012-11-12 16:28:25 LustreError: 11-0: lstest-MDT0000-osp-OST0193: Communicating with 172.20.5.2@o2ib500, operation mds_connect failed with -11 2012-11-12 16:28:26 Lustre: lstest-OST0193: Client 4a526e6a-626c-9ac9-9ce4-4119a8694707 (at 172.20.17.10@o2ib500) reconnecting, waiting for 275 clients in recovery for 2:29 2012-11-12 16:28:26 Lustre: lstest-OST0193: Client 4a526e6a-626c-9ac9-9ce4-4119a8694707 (at 172.20.17.10@o2ib500) refused reconnection, still busy with 1 active RPCs 2012-11-12 16:28:36 Lustre: lstest-OST0193: Client 4028a636-dc0d-66a7-557b-f4d960ae30a7 (at 172.20.17.9@o2ib500) reconnecting, waiting for 275 clients in recovery for 2:19 2012-11-12 16:28:36 Lustre: Skipped 3 previous similar messages 2012-11-12 16:28:36 Lustre: lstest-OST0193: Client 4028a636-dc0d-66a7-557b-f4d960ae30a7 (at 172.20.17.9@o2ib500) refused reconnection, still busy with 1 active RPCs 2012-11-12 16:28:36 Lustre: Skipped 3 previous similar messages 2012-11-12 16:28:50 LustreError: 11-0: lstest-MDT0000-osp-OST0193: Communicating with 172.20.5.2@o2ib500, operation mds_connect failed with -11 2012-11-12 16:28:52 Lustre: lstest-OST0193: Client dbd2bec3-364d-3eeb-690a-ef833e86cb64 (at 172.20.17.6@o2ib500) reconnecting, waiting for 275 clients in recovery for 2:03 2012-11-12 16:28:52 Lustre: Skipped 5 previous similar messages 2012-11-12 16:28:52 Lustre: lstest-OST0193: Client dbd2bec3-364d-3eeb-690a-ef833e86cb64 (at 172.20.17.6@o2ib500) refused reconnection, still busy with 1 active RPCs 2012-11-12 16:28:52 Lustre: Skipped 5 previous similar messages 2012-11-12 16:29:26 Lustre: lstest-OST0193: Client 4028a636-dc0d-66a7-557b-f4d960ae30a7 (at 172.20.17.9@o2ib500) reconnecting, waiting for 275 clients in recovery for 1:29 2012-11-12 16:29:26 Lustre: Skipped 17 previous similar messages 2012-11-12 16:29:26 Lustre: lstest-OST0193: Client 4028a636-dc0d-66a7-557b-f4d960ae30a7 (at 172.20.17.9@o2ib500) refused reconnection, still busy with 1 active RPCs 2012-11-12 16:29:26 Lustre: Skipped 17 previous similar messages 2012-11-12 16:29:40 LustreError: 11-0: lstest-MDT0000-osp-OST0193: Communicating with 172.20.5.2@o2ib500, operation mds_connect failed with -11 2012-11-12 16:29:40 LustreError: Skipped 1 previous similar message 2012-11-12 16:30:21 INFO: task tgt_recov:6560 blocked for more than 120 seconds. 2012-11-12 16:30:22 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 2012-11-12 16:30:22 tgt_recov D 0000000000000001 0 6560 2 0x00000000 2012-11-12 16:30:22 ffff880ff0007e10 0000000000000046 0000000000000000 ffff880ff0007dd4 2012-11-12 16:30:22 ffff880f00000000 ffff88083fe81000 ffff880044635fc0 0000000000000400 2012-11-12 16:30:22 ffff880ff1251ab8 ffff880ff0007fd8 000000000000f4e8 ffff880ff1251ab8 2012-11-12 16:30:22 Call Trace: 2012-11-12 16:30:22 [] ? check_for_clients+0x0/0x90 [ptlrpc] 2012-11-12 16:30:22 [] target_recovery_overseer+0x95/0x250 [ptlrpc] 2012-11-12 16:30:22 [] ? exp_connect_healthy+0x0/0x20 [ptlrpc] 2012-11-12 16:30:22 [] ? autoremove_wake_function+0x0/0x40 2012-11-12 16:30:22 [] target_recovery_thread+0x58e/0x19d0 [ptlrpc] 2012-11-12 16:30:22 [] ? __mmdrop+0x44/0x60 2012-11-12 16:30:22 [] ? target_recovery_thread+0x0/0x19d0 [ptlrpc] 2012-11-12 16:30:22 [] child_rip+0xa/0x20 2012-11-12 16:30:22 [] ? target_recovery_thread+0x0/0x19d0 [ptlrpc] 2012-11-12 16:30:22 [] ? target_recovery_thread+0x0/0x19d0 [ptlrpc] 2012-11-12 16:30:22 [] ? child_rip+0x0/0x20 2012-11-12 16:30:30 Lustre: lstest-OST0193: Client 6ea39525-a16e-3a7b-16a4-51292b88d905 (at 172.20.17.1@o2ib500) reconnecting, waiting for 275 clients in recovery for 0:25 2012-11-12 16:30:30 Lustre: Skipped 41 previous similar messages 2012-11-12 16:30:30 Lustre: lstest-OST0193: Client 6ea39525-a16e-3a7b-16a4-51292b88d905 (at 172.20.17.1@o2ib500) refused reconnection, still busy with 1 active RPCs 2012-11-12 16:30:30 Lustre: Skipped 41 previous similar messages 2012-11-12 16:30:55 LustreError: 11-0: lstest-MDT0000-osp-OST0193: Communicating with 172.20.5.2@o2ib500, operation mds_connect failed with -11 2012-11-12 16:30:55 LustreError: Skipped 2 previous similar messages 2012-11-12 16:30:56 Lustre: lstest-OST0193: Denying connection for new client f8c9bfa8-10ef-87a9-bf7d-930f5355d3bf (at 172.20.4.149@o2ib500), waiting for all 275 known clients (221 recovered, 46 in progress, and 8 unseen) to recover in 0:59 2012-11-12 16:30:58 Lustre: lstest-OST0193: Denying connection for new client 27d6608b-c4e0-5160-1688-57a3961c105c (at 172.20.17.27@o2ib500), waiting for all 275 known clients (221 recovered, 46 in progress, and 8 unseen) to recover in 0:57 2012-11-12 16:30:58 Lustre: lstest-OST0193: Denying connection for new client b84b98f2-5b39-39c3-3ac8-1342661763d4 (at 172.20.17.93@o2ib500), waiting for all 275 known clients (221 recovered, 46 in progress, and 8 unseen) to recover in 0:57 2012-11-12 16:30:58 Lustre: Skipped 1 previous similar message 2012-11-12 16:30:59 Lustre: lstest-OST0193: Denying connection for new client 6bbb52b9-73f7-9e7c-a7c2-f8e3b4c3b170 (at 172.20.17.23@o2ib500), waiting for all 275 known clients (221 recovered, 46 in progress, and 8 unseen) to recover in 0:56 2012-11-12 16:30:59 Lustre: Skipped 3 previous similar messages 2012-11-12 16:31:08 Lustre: lstest-OST0193: Denying connection for new client fcb94e44-c7c4-8c2c-039c-c2f0eb93cb17 (at 172.20.17.40@o2ib500), waiting for all 275 known clients (221 recovered, 46 in progress, and 8 unseen) to recover in 0:47 2012-11-12 16:31:08 Lustre: Skipped 1 previous similar message 2012-11-12 16:31:12 Lustre: lstest-OST0193: Denying connection for new client cd84432a-85dc-480a-1fe6-64fc40f6c468 (at 172.20.17.99@o2ib500), waiting for all 275 known clients (221 recovered, 46 in progress, and 8 unseen) to recover in 0:43 2012-11-12 16:31:12 Lustre: Skipped 4 previous similar messages 2012-11-12 16:31:20 Lustre: lstest-OST0193: Denying connection for new client b628c39d-43c1-7c0c-7137-b104b0b74b22 (at 172.20.17.26@o2ib500), waiting for all 275 known clients (221 recovered, 46 in progress, and 8 unseen) to recover in 0:35 2012-11-12 16:31:20 Lustre: Skipped 74 previous similar messages 2012-11-12 16:31:46 Lustre: lstest-OST0193: Denying connection for new client f8c9bfa8-10ef-87a9-bf7d-930f5355d3bf (at 172.20.4.149@o2ib500), waiting for all 275 known clients (221 recovered, 46 in progress, and 8 unseen) to recover in 0:09 2012-11-12 16:31:46 Lustre: Skipped 24 previous similar messages 2012-11-12 16:31:56 LustreError: 6520:0:(ofd_obd.c:521:ofd_set_info_async()) lstest-OST0193: Unsupported key revimp_update 2012-11-12 16:31:57 Lustre: 6524:0:(ofd_obd.c:1069:ofd_orphans_destroy()) lstest-OST0193: deleting orphan objects from 10773597 to 10774078 2012-11-12 16:31:57 LustreError: 6525:0:(ldlm_resource.c:1106:ldlm_resource_get()) lvbo_init failed for resource 8846486: rc -2 2012-11-12 16:31:57 Lustre: lstest-OST0193: Recovery over after 6:02, of 275 clients 266 recovered and 9 were evicted. 2012-11-12 16:31:57 LustreError: 6604:0:(ldlm_resource.c:1106:ldlm_resource_get()) lvbo_init failed for resource 8846317: rc -2 2012-11-12 16:31:57 LustreError: 6604:0:(ldlm_resource.c:1106:ldlm_resource_get()) Skipped 2 previous similar messages 2012-11-12 16:31:58 LustreError: 6607:0:(ldlm_resource.c:1106:ldlm_resource_get()) lvbo_init failed for resource 9549289: rc -2 2012-11-12 16:31:58 LustreError: 6607:0:(ldlm_resource.c:1106:ldlm_resource_get()) Skipped 943 previous similar messages 2012-11-12 16:31:58 Lustre: lstest-OST0193: Client 927b9a9e-21ec-7bdc-930d-cbe291044bea (at 172.20.17.8@o2ib500) reconnecting 2012-11-12 16:31:58 LustreError: 6602:0:(ofd_obd.c:521:ofd_set_info_async()) lstest-OST0193: Unsupported key revimp_update 2012-11-12 16:31:58 Lustre: lstest-OST0193: Client 234f43fb-aeae-3f64-e93c-f28ec6bd62dc (at 172.20.17.13@o2ib500) reconnecting 2012-11-12 16:32:00 Lustre: lstest-OST0193: Client 4a526e6a-626c-9ac9-9ce4-4119a8694707 (at 172.20.17.10@o2ib500) reconnecting 2012-11-12 16:32:00 LustreError: 6603:0:(ofd_obd.c:521:ofd_set_info_async()) lstest-OST0193: Unsupported key revimp_update 2012-11-12 16:32:00 LustreError: 6603:0:(ofd_obd.c:521:ofd_set_info_async()) Skipped 1 previous similar message 2012-11-12 16:32:01 Lustre: lstest-OST0193: Client dbd2bec3-364d-3eeb-690a-ef833e86cb64 (at 172.20.17.6@o2ib500) reconnecting 2012-11-12 16:32:01 Lustre: Skipped 1 previous similar message 2012-11-12 16:32:01 LustreError: 6524:0:(ofd_obd.c:521:ofd_set_info_async()) lstest-OST0193: Unsupported key revimp_update 2012-11-12 16:32:01 LustreError: 6524:0:(ofd_obd.c:521:ofd_set_info_async()) Skipped 1 previous similar message 2012-11-12 16:32:10 Lustre: lstest-OST0193: Client 6ea39525-a16e-3a7b-16a4-51292b88d905 (at 172.20.17.1@o2ib500) reconnecting 2012-11-12 16:32:10 Lustre: Skipped 3 previous similar messages 2012-11-12 16:32:10 LustreError: 6524:0:(ofd_obd.c:521:ofd_set_info_async()) lstest-OST0193: Unsupported key revimp_update 2012-11-12 16:32:10 LustreError: 6524:0:(ofd_obd.c:521:ofd_set_info_async()) Skipped 3 previous similar messages 2012-11-12 16:32:15 Lustre: lstest-OST0193: Client 7877635e-a0c6-353f-51e9-47e6f0ef5fb2 (at 172.20.17.2@o2ib500) reconnecting 2012-11-12 16:32:15 LustreError: 6603:0:(ofd_obd.c:521:ofd_set_info_async()) lstest-OST0193: Unsupported key revimp_update 2012-11-12 16:33:13 Lustre: lstest-OST0193: haven't heard from client 5537e8c9-c216-654c-853b-a53147e9c41f (at 172.20.4.15@o2ib500) in 438 seconds. I think it's dead, and I am evicting it. exp ffff880829d35000, cur 1352766793 expire 1352766643 last 1352766355 2012-11-12 16:33:13 Lustre: lstest-OST0193: haven't heard from client 33d0746e-3e77-82b4-e9bd-4ff0a8d9b7f4 (at 172.20.3.135@o2ib500) in 432 seconds. I think it's dead, and I am evicting it. exp ffff88081e805c00, cur 1352766793 expire 1352766643 last 1352766361 2012-11-12 16:33:50 Lustre: lstest-MDT0000-osp-OST0193: Connection to lstest-MDT0000 (at 172.20.5.2@o2ib500) was lost; in progress operations using this service will wait for recovery to complete 2012-11-12 16:33:50 LustreError: 166-1: MGC172.20.5.2@o2ib500: Connection to MGS (at 172.20.5.2@o2ib500) was lost; in progress operations using this service will fail 2012-11-12 16:34:28 LustreError: 0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer expired after 151s: evicting client at 172.20.4.23@o2ib500 ns: filter-ffff8807fef4c000 lock: ffff8807fa258b80/0xbdf5847332a7db2c lrc: 3/0,0 mode: PW/PW res: 10773567/0 rrc: 21 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 nid: 172.20.4.23@o2ib500 remote: 0x6b97208d7425ab3d expref: 4 pid: 6520 timeout 4295275712 2012-11-12 16:36:40 Lustre: lstest-OST0193: haven't heard from client lstest-MDT0000-mdtlov_UUID (at 172.20.5.2@o2ib500) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807feecd800, cur 1352767000 expire 1352766850 last 1352766773 2012-11-12 16:36:40 Lustre: Skipped 2 previous similar messages 2012-11-12 16:36:59 LustreError: 0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer expired after 302s: evicting client at 172.20.4.142@o2ib500 ns: filter-ffff8807fef4c000 lock: ffff88102585ab40/0xbdf5847332a7db33 lrc: 3/0,0 mode: PW/PW res: 10773567/0 rrc: 20 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 nid: 172.20.4.142@o2ib500 remote: 0xdbc19fc5fc548c7b expref: 8 pid: 6526 timeout 4295426043 2012-11-12 16:38:16 LustreError: 6613:0:(ldlm_resource.c:1106:ldlm_resource_get()) lvbo_init failed for resource 10774195: rc -2 2012-11-12 16:38:16 LustreError: 6613:0:(ldlm_resource.c:1106:ldlm_resource_get()) Skipped 1583 previous similar messages 2012-11-12 16:38:25 LustreError: 167-0: lstest-MDT0000-osp-OST0193: This client was evicted by lstest-MDT0000; in progress operations using this service will fail. 2012-11-12 16:38:25 Lustre: Evicted from MGS (at 172.20.5.2@o2ib500) after server handle changed from 0xee6f11761b813c09 to 0x4dcd884db4b6cee9 2012-11-12 16:38:25 Lustre: lstest-MDT0000-osp-OST0193: Connection restored to lstest-MDT0000 (at 172.20.5.2@o2ib500) 2012-11-12 16:38:25 Lustre: MGC172.20.5.2@o2ib500: Connection restored to MGS (at 172.20.5.2@o2ib500) 2012-11-12 16:39:30 LustreError: 0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer expired after 453s: evicting client at 172.20.4.47@o2ib500 ns: filter-ffff8807fef4c000 lock: ffff881023fd9b80/0xbdf5847332a7db3a lrc: 3/0,0 mode: PW/PW res: 10773567/0 rrc: 19 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 nid: 172.20.4.47@o2ib500 remote: 0xd6f599828daab42f expref: 4 pid: 6599 timeout 4295577043 2012-11-12 16:42:01 LustreError: 0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer expired after 604s: evicting client at 172.20.4.141@o2ib500 ns: filter-ffff8807fef4c000 lock: ffff881023fd9980/0xbdf5847332a7db41 lrc: 3/0,0 mode: PW/PW res: 10773567/0 rrc: 18 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 nid: 172.20.4.141@o2ib500 remote: 0x6104baf44c81bdef expref: 5 pid: 6599 timeout 4295728043 2012-11-12 16:43:21 Lustre: 6525:0:(ofd_obd.c:1069:ofd_orphans_destroy()) lstest-OST0193: deleting orphan objects from 10774198 to 10774239 2012-11-12 16:44:32 LustreError: 0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer expired after 755s: evicting client at 172.20.4.135@o2ib500 ns: filter-ffff8807fef4c000 lock: ffff8808113d7bc0/0xbdf5847332a7db48 lrc: 3/0,0 mode: PW/PW res: 10773567/0 rrc: 17 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 nid: 172.20.4.135@o2ib500 remote: 0xbdd7dafd67525827 expref: 5 pid: 6525 timeout 4295879043 2012-11-12 16:46:13 LustreError: 0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer expired after 856s: evicting client at 172.20.4.133@o2ib500 ns: filter-ffff8807fef4c000 lock: ffff8807fa258980/0xbdf5847332a7db4f lrc: 3/0,0 mode: PW/PW res: 10773567/0 rrc: 16 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 nid: 172.20.4.133@o2ib500 remote: 0xb46962d8c1fb7c0f expref: 7 pid: 6520 timeout 4295980043 2012-11-12 16:47:09 scsi host10: ib_srp: new target: id_ext 22140080e51f8e9c ioc_guid 0080e51f8e9c0003 pkey ffff service_id 20140080e51f8e9c dgid fe80:0000:0000:0000:0080:e51f:8e9c:0002 2012-11-12 16:47:09 scsi host10: ib_srp: REJ received 2012-11-12 16:47:09 scsi host10: REJ reason 0x3 2012-11-12 16:47:09 scsi host10: ib_srp: Connection failed 2012-11-12 16:47:09 scsi host11: ib_srp: new target: id_ext 22150080e51f8e9c ioc_guid 0080e51f7ba40003 pkey ffff service_id 20150080e51f8e9c dgid fe80:0000:0000:0000:0080:e51f:7ba4:0002 2012-11-12 16:47:09 scsi host11: ib_srp: REJ received 2012-11-12 16:47:09 scsi host11: REJ reason 0x3 2012-11-12 16:47:09 scsi host11: ib_srp: Connection failed 2012-11-12 16:47:35 scsi host12: ib_srp: new target: id_ext 22140080e51f8e9c ioc_guid 0080e51f8e9c0003 pkey ffff service_id 20140080e51f8e9c dgid fe80:0000:0000:0000:0080:e51f:8e9c:0002 2012-11-12 16:47:35 scsi host12: ib_srp: REJ received 2012-11-12 16:47:35 scsi host12: REJ reason 0x3 2012-11-12 16:47:35 scsi host12: ib_srp: Connection failed 2012-11-12 16:47:35 scsi host13: ib_srp: new target: id_ext 22150080e51f8e9c ioc_guid 0080e51f7ba40003 pkey ffff service_id 20150080e51f8e9c dgid fe80:0000:0000:0000:0080:e51f:7ba4:0002 2012-11-12 16:47:35 scsi host13: ib_srp: REJ received 2012-11-12 16:47:35 scsi host13: REJ reason 0x3 2012-11-12 16:47:35 scsi host13: ib_srp: Connection failed 2012-11-12 16:47:54 LustreError: 0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer expired after 957s: evicting client at 172.20.4.137@o2ib500 ns: filter-ffff8807fef4c000 lock: ffff88102585a940/0xbdf5847332a7db56 lrc: 3/0,0 mode: PW/PW res: 10773567/0 rrc: 15 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 nid: 172.20.4.137@o2ib500 remote: 0xec945a90fa18a8ad expref: 5 pid: 6526 timeout 4296081043 2012-11-12 16:49:35 LustreError: 0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer expired after 1058s: evicting client at 172.20.4.24@o2ib500 ns: filter-ffff8807fef4c000 lock: ffff88102585a740/0xbdf5847332a7db5d lrc: 3/0,0 mode: PW/PW res: 10773567/0 rrc: 14 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 nid: 172.20.4.24@o2ib500 remote: 0xed9fad365a9428e8 expref: 4 pid: 6526 timeout 4296182043 2012-11-12 16:51:16 LustreError: 0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer expired after 1159s: evicting client at 172.20.4.139@o2ib500 ns: filter-ffff8807fef4c000 lock: ffff8810287f9b00/0xbdf5847332a7db6b lrc: 3/0,0 mode: PW/PW res: 10773567/0 rrc: 13 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 nid: 172.20.4.139@o2ib500 remote: 0x939fbbb378025004 expref: 9 pid: 6603 timeout 4296283043 2012-11-12 16:52:57 LustreError: 0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer expired after 1260s: evicting client at 172.20.4.22@o2ib500 ns: filter-ffff8807fef4c000 lock: ffff8807fef5d880/0xbdf5847332a7db80 lrc: 3/0,0 mode: PW/PW res: 10773567/0 rrc: 12 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 nid: 172.20.4.22@o2ib500 remote: 0xb980afe2ab20ec00 expref: 4 pid: 6525 timeout 4296384043 2012-11-12 16:56:19 LustreError: 0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer expired after 1462s: evicting client at 172.20.4.134@o2ib500 ns: filter-ffff8807fef4c000 lock: ffff88100bc99bc0/0xbdf5847332a7db95 lrc: 3/0,0 mode: PW/PW res: 10773567/0 rrc: 10 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 nid: 172.20.4.134@o2ib500 remote: 0xd9ff05298fd6f7cc expref: 6 pid: 6602 timeout 4296586000 2012-11-12 16:56:19 LustreError: 0:0:(ldlm_lockd.c:374:waiting_locks_callback()) Skipped 1 previous similar message Console [grove403] log at 2012-11-12 17:00:00 PST. 2012-11-12 17:01:22 LustreError: 0:0:(ldlm_lockd.c:374:waiting_locks_callback()) ### lock callback timer expired after 1765s: evicting client at 172.20.4.39@o2ib500 ns: filter-ffff8807fef4c000 lock: ffff88100c0c2c80/0xbdf5847332a7dbbf lrc: 3/0,0 mode: PW/PW res: 10773567/0 rrc: 7 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 nid: 172.20.4.39@o2ib500 remote: 0x936651ef6fc00619 expref: 5 pid: 6608 timeout 4296889000 2012-11-12 17:01:22 LustreError: 0:0:(ldlm_lockd.c:374:waiting_locks_callback()) Skipped 2 previous similar messages