[-- MARK -- Thu Mar 28 12:00:00 2013]
Lustre: nbp2-OST0010: haven't heard from client 8be6ae01-a3b1-7b71-474e-3d1573f976b0 (at 10.151.28.215@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88043bbea800, cur 1364497379 expire 1364497229 last 1364497152
Lustre: Skipped 14 previous similar messages
Lustre: 1806:0:(o2iblnd_cb.c:2330:kiblnd_passive_connect()) Conn stale 10.151.28.215@o2ib [old ver: 12, new ver: 12]
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 1359s: evicting client at 10.151.59.203@o2ib  ns: filter-nbp2-OST0048_UUID lock: ffff88025057fd80/0x8bb24e51c42fe3f5 lrc: 3/0,0 mode: PW/PW res: 198039956/0 rrc: 490 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0xa2731f3d5452f5e9 expref: 17 pid: 25807 timeout 5503646000
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) Skipped 6 previous similar messages
Lustre: 1806:0:(o2iblnd_cb.c:2330:kiblnd_passive_connect()) Conn stale 10.151.55.8@o2ib [old ver: 12, new ver: 12]
Lustre: nbp2-OST0018: haven't heard from client 9aee0651-5355-0bb7-f897-0a7e8f9f92e7 (at 10.151.55.8@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803def9f800, cur 1364497671 expire 1364497521 last 1364497444
Lustre: Skipped 14 previous similar messages
Lustre: 26798:0:(ldlm_lib.c:952:target_handle_connect()) nbp2-OST0048: connection from 2db2d024-4a3a-a3cd-26c7-fdb17a5f6c84@10.151.59.201@o2ib t0 exp (null) cur 1364497795 last 0
Lustre: 26798:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 110 previous similar messages
Lustre: nbp2-OST0018: haven't heard from client 9d4c2967-aaad-d39e-3d61-cd33e74264e9 (at 10.151.13.31@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804487be400, cur 1364497848 expire 1364497698 last 1364497621
Lustre: Skipped 89 previous similar messages
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 1963s: evicting client at 10.151.58.250@o2ib  ns: filter-nbp2-OST0048_UUID lock: ffff880467188480/0x8bb24e51c42fe42d lrc: 3/0,0 mode: PW/PW res: 198039956/0 rrc: 434 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0xe4fa90fc1a73881d expref: 17 pid: 25873 timeout 5504250000
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) Skipped 8 previous similar messages
LustreError: 25574:0:(client.c:1045:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff8803e78e7c00 x1429513600625845/t0(0) o105->nbp2-OST0068@10.151.28.202@o2ib:15/16 lens 344/192 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1
LustreError: 25574:0:(ldlm_lockd.c:656:ldlm_handle_ast_error()) ### client (nid 10.151.28.202@o2ib) returned 0 from completion AST ns: filter-nbp2-OST0068_UUID lock: ffff880002a36000/0x8bb24e51c43426f2 lrc: 4/0,0 mode: PW/PW res: 206073256/0 rrc: 890 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x15c4abe1009acf23 expref: 13 pid: 25799 timeout 5504421000
LustreError: 25574:0:(client.c:1045:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff8803e78e7c00 x1429513600625848/t0(0) o105->nbp2-OST0068@10.151.28.202@o2ib:15/16 lens 344/192 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1
LustreError: 25574:0:(ldlm_lockd.c:656:ldlm_handle_ast_error()) ### client (nid 10.151.28.202@o2ib) returned 0 from completion AST ns: filter-nbp2-OST0068_UUID lock: ffff880002a36d80/0x8bb24e51c43426f9 lrc: 4/0,0 mode: PW/PW res: 206073256/0 rrc: 889 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x15c4abe1009acf2a expref: 12 pid: 25799 timeout 5504421165
Lustre: 26779:0:(ldlm_lib.c:952:target_handle_connect()) nbp2-OST0068: connection from a0e5b76f-0b93-ac5e-61a9-4ac6a8989789@10.151.28.204@o2ib t0 exp (null) cur 1364498423 last 0
Lustre: 26779:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 439 previous similar messages
Lustre: 1806:0:(o2iblnd_cb.c:2330:kiblnd_passive_connect()) Conn stale 10.151.29.182@o2ib [old ver: 12, new ver: 12]
Lustre: 1806:0:(o2iblnd_cb.c:2330:kiblnd_passive_connect()) Skipped 5 previous similar messages
Lustre: nbp2-OST0010: haven't heard from client 2a7e5276-dc6e-63a8-f184-74ee198f8436 (at 10.151.29.182@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88040d001800, cur 1364498434 expire 1364498284 last 1364498207
Lustre: Skipped 449 previous similar messages
Lustre: 1806:0:(o2iblnd_cb.c:2330:kiblnd_passive_connect()) Conn stale 10.151.28.79@o2ib [old ver: 12, new ver: 12]
Lustre: 1806:0:(o2iblnd_cb.c:2330:kiblnd_passive_connect()) Skipped 1 previous similar message
LustreError: 25574:0:(client.c:1045:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff8803ed86c400 x1429513600640511/t0(0) o105->nbp2-OST0040@10.151.28.202@o2ib:15/16 lens 344/192 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1
LustreError: 25574:0:(client.c:1045:ptlrpc_import_delay_req()) Skipped 4 previous similar messages
LustreError: 25574:0:(ldlm_lockd.c:656:ldlm_handle_ast_error()) ### client (nid 10.151.28.202@o2ib) returned 0 from completion AST ns: filter-nbp2-OST0040_UUID lock: ffff88023cf06900/0x8bb24e51c4355390 lrc: 4/0,0 mode: PW/PW res: 191183676/0 rrc: 881 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x15c4abe1009ad0ff expref: 7 pid: 26779 timeout 5504820000
LustreError: 25574:0:(ldlm_lockd.c:656:ldlm_handle_ast_error()) Skipped 4 previous similar messages
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 2567s: evicting client at 10.151.56.243@o2ib  ns: filter-nbp2-OST0048_UUID lock: ffff88053e3fb480/0x8bb24e51c42fe450 lrc: 3/0,0 mode: PW/PW res: 198039956/0 rrc: 378 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x7a0d5c9e297dcf03 expref: 17 pid: 25820 timeout 5504854000
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) Skipped 14 previous similar messages
Lustre: 26647:0:(ldlm_lib.c:952:target_handle_connect()) nbp2-OST0040: connection from 4192c6c2-fad0-a0e4-84bf-013806530ecd@10.151.28.210@o2ib t0 exp (null) cur 1364499051 last 0
Lustre: 26647:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 55 previous similar messages
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 3180s: evicting client at 10.151.56.248@o2ib  ns: filter-nbp2-OST0048_UUID lock: ffff8807993add80/0x8bb24e51c42fe481 lrc: 3/0,0 mode: PW/PW res: 198039956/0 rrc: 266 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0xe3deebd6e154da8a expref: 17 pid: 26825 timeout 5505467000
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) Skipped 14 previous similar messages
Lustre: 26670:0:(ldlm_lib.c:952:target_handle_connect()) nbp2-OST0000: connection from f3bc9513-6904-6380-4541-9fddf8f111f5@10.151.59.210@o2ib t0 exp (null) cur 1364499728 last 0
Lustre: 26670:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 16 previous similar messages
CONSOLE MARK 201303281230
LustreError: 25574:0:(client.c:1045:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff8805319cb800 x1429513600673585/t0(0) o105->nbp2-OST0068@10.151.41.222@o2ib:15/16 lens 344/192 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1
LustreError: 25574:0:(client.c:1045:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
LustreError: 25574:0:(ldlm_lockd.c:656:ldlm_handle_ast_error()) ### client (nid 10.151.41.222@o2ib) returned 0 from completion AST ns: filter-nbp2-OST0068_UUID lock: ffff880372967d80/0x8bb24e51c4342c7f lrc: 4/0,0 mode: PW/PW res: 206073256/0 rrc: 11 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x5d255c3ebc51a1b3 expref: 13 pid: 25866 timeout 5506135000
LustreError: 25574:0:(ldlm_lockd.c:656:ldlm_handle_ast_error()) Skipped 2 previous similar messages
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 3805s: evicting client at 10.151.56.249@o2ib  ns: filter-nbp2-OST0048_UUID lock: ffff8808c96896c0/0x8bb24e51c42fe4b9 lrc: 3/0,0 mode: PW/PW res: 198039956/0 rrc: 210 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x5ac8cc31440ee6c0 expref: 17 pid: 26808 timeout 5506092000
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) Skipped 14 previous similar messages
Lustre: 27001:0:(ldlm_lib.c:952:target_handle_connect()) nbp2-OST0000: connection from 2fad52f5-322a-2b1f-6750-8518887858bf@10.151.59.211@o2ib t0 exp (null) cur 1364500329 last 0
Lustre: 27001:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 9 previous similar messages
LustreError: 88864:0:(ldlm_lockd.c:2074:ldlm_cancel_handler()) ldlm_cancel from 10.151.56.239@o2ib arrived at 1364500344 with bad export cookie 10066194230171859630
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 4409s: evicting client at 10.151.56.247@o2ib  ns: filter-nbp2-OST0048_UUID lock: ffff880b19d50240/0x8bb24e51c42fe4ff lrc: 3/0,0 mode: PW/PW res: 198039956/0 rrc: 140 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0xc5a82ff7570e4d4a expref: 17 pid: 26808 timeout 5506696000
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) Skipped 8 previous similar messages
[-- MARK -- Thu Mar 28 13:00:00 2013]
