[-- MARK -- Thu Mar 28 12:00:00 2013]
Lustre: 26893:0:(ldlm_lib.c:952:target_handle_connect()) nbp2-OST002f: connection from 2db2d024-4a3a-a3cd-26c7-fdb17a5f6c84@10.151.59.201@o2ib t0 exp (null) cur 1364497320 last 0
Lustre: 26893:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 48 previous similar messages
Lustre: nbp2-OST0027: haven't heard from client 8be6ae01-a3b1-7b71-474e-3d1573f976b0 (at 10.151.28.215@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b93190800, cur 1364497379 expire 1364497229 last 1364497152
Lustre: Skipped 14 previous similar messages
Lustre: 1745:0:(o2iblnd_cb.c:2330:kiblnd_passive_connect()) Conn stale 10.151.28.215@o2ib [old ver: 12, new ver: 12]
Lustre: 1745:0:(o2iblnd_cb.c:2330:kiblnd_passive_connect()) Conn stale 10.151.55.8@o2ib [old ver: 12, new ver: 12]
Lustre: nbp2-OST0017: haven't heard from client 9aee0651-5355-0bb7-f897-0a7e8f9f92e7 (at 10.151.55.8@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803e3a1ac00, cur 1364497671 expire 1364497521 last 1364497444
Lustre: Skipped 14 previous similar messages
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 1855s: evicting client at 10.151.56.239@o2ib  ns: filter-nbp2-OST002f_UUID lock: ffff88031fec5d80/0x7052b372f4a12286 lrc: 3/0,0 mode: PW/PW res: 186988778/0 rrc: 415 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x8b5bc4a406f50c08 expref: 17 pid: 27235 timeout 5503894000
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) Skipped 3 previous similar messages
Lustre: nbp2-OST000f: haven't heard from client 9d4c2967-aaad-d39e-3d61-cd33e74264e9 (at 10.151.13.31@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b8a004400, cur 1364497848 expire 1364497698 last 1364497621
Lustre: Skipped 89 previous similar messages
Lustre: 27263:0:(ldlm_lib.c:952:target_handle_connect()) nbp2-OST0007: connection from 4cf66610-1a71-db8b-b1f4-f082786b7200@10.151.13.31@o2ib t0 exp (null) cur 1364497953 last 0
Lustre: 27263:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 118 previous similar messages
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 2459s: evicting client at 10.151.57.248@o2ib  ns: filter-nbp2-OST002f_UUID lock: ffff8807e4b13900/0x7052b372f4a122a9 lrc: 3/0,0 mode: PW/PW res: 186988778/0 rrc: 359 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x3e60fe3ace3dff68 expref: 17 pid: 26519 timeout 5504498000
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) Skipped 3 previous similar messages
Lustre: 1731:0:(o2iblnd_cb.c:2330:kiblnd_passive_connect()) Conn stale 10.151.29.182@o2ib [old ver: 12, new ver: 12]
Lustre: 1731:0:(o2iblnd_cb.c:2330:kiblnd_passive_connect()) Skipped 5 previous similar messages
Lustre: nbp2-OST001f: haven't heard from client 2a7e5276-dc6e-63a8-f184-74ee198f8436 (at 10.151.29.182@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bd843c400, cur 1364498434 expire 1364498284 last 1364498207
Lustre: Skipped 449 previous similar messages
Lustre: 1745:0:(o2iblnd_cb.c:2330:kiblnd_passive_connect()) Conn stale 10.151.28.79@o2ib [old ver: 12, new ver: 12]
Lustre: 1745:0:(o2iblnd_cb.c:2330:kiblnd_passive_connect()) Skipped 1 previous similar message
LustreError: 26191:0:(ldlm_lockd.c:2074:ldlm_cancel_handler()) ldlm_cancel from 10.151.41.228@o2ib arrived at 1364498539 with bad export cookie 8093728786310805375
Lustre: 27263:0:(ldlm_lib.c:952:target_handle_connect()) nbp2-OST002f: connection from a09ef092-8a47-5e73-a3a3-17f1af761709@10.151.56.247@o2ib t0 exp (null) cur 1364498678 last 0
Lustre: 27263:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 457 previous similar messages
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 3063s: evicting client at 10.151.56.249@o2ib  ns: filter-nbp2-OST002f_UUID lock: ffff880259bbf6c0/0x7052b372f4a122e8 lrc: 3/0,0 mode: PW/PW res: 186988778/0 rrc: 303 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x5ac8cc31440ee531 expref: 17 pid: 27235 timeout 5505102000
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) Skipped 4 previous similar messages
Lustre: 26986:0:(ldlm_lib.c:952:target_handle_connect()) nbp2-OST002f: connection from a944c2b2-502e-0d65-9c71-ed220ea3bd1e@10.151.56.250@o2ib t0 exp (null) cur 1364499428 last 0
Lustre: 26986:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 4 previous similar messages
LustreError: 81914:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 3667s: evicting client at 10.151.59.210@o2ib  ns: filter-nbp2-OST002f_UUID lock: ffff880374d51900/0x7052b372f4a12320 lrc: 3/0,0 mode: PW/PW res: 186988778/0 rrc: 135 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x8c078f95266ba3fc expref: 17 pid: 27035 timeout 5505706000
LustreError: 81914:0:(ldlm_lockd.c:358:waiting_locks_callback()) Skipped 3 previous similar messages
CONSOLE MARK 201303281230
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 4271s: evicting client at 10.151.59.207@o2ib  ns: filter-nbp2-OST002f_UUID lock: ffff8802c9f2d240/0x7052b372f4a1236d lrc: 3/0,0 mode: PW/PW res: 186988778/0 rrc: 79 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0xdae2f9b7b672d4ce expref: 17 pid: 27235 timeout 5506310000
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) Skipped 3 previous similar messages
Lustre: 26587:0:(ldlm_lib.c:952:target_handle_connect()) nbp2-OST002f: connection from 9201a1d9-1298-4d74-1eae-b30e5fcefde0@10.151.59.207@o2ib t0 exp (null) cur 1364500188 last 0
Lustre: 26587:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 4 previous similar messages
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 4875s: evicting client at 10.151.58.253@o2ib  ns: filter-nbp2-OST002f_UUID lock: ffff8802ed08d480/0x7052b372f4a123e4 lrc: 3/0,0 mode: PW/PW res: 186988778/0 rrc: 23 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x553ef0f376ce76dc expref: 16 pid: 27202 timeout 5506914000
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) Skipped 3 previous similar messages
LustreError: 26195:0:(client.c:1045:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff8804109dc400 x1429513605905330/t0(0) o105->nbp2-OST002f@10.151.58.253@o2ib:15/16 lens 344/192 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1
LustreError: 26195:0:(client.c:1045:ptlrpc_import_delay_req()) Skipped 10 previous similar messages
LustreError: 26195:0:(ldlm_lockd.c:656:ldlm_handle_ast_error()) ### client (nid 10.151.58.253@o2ib) returned 0 from completion AST ns: filter-nbp2-OST002f_UUID lock: ffff880176dcbd80/0x7052b372f4a12596 lrc: 4/0,0 mode: PW/PW res: 186988778/0 rrc: 22 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x553ef0f376ce76e3 expref: 14 pid: 27235 timeout 5507065135
LustreError: 26195:0:(ldlm_lockd.c:656:ldlm_handle_ast_error()) Skipped 10 previous similar messages
Lustre: 26494:0:(ldlm_lib.c:952:target_handle_connect()) nbp2-OST002f: connection from 311c4f75-5e65-1774-f74e-e4fba8f88bd6@10.151.58.253@o2ib t0 exp (null) cur 1364500793 last 0
Lustre: 26494:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 16 previous similar messages
[-- MARK -- Thu Mar 28 13:00:00 2013]
