[-- MARK -- Thu Mar 28 12:00:00 2013]
Lustre: nbp2-OST0054: haven't heard from client 8be6ae01-a3b1-7b71-474e-3d1573f976b0 (at 10.151.28.215@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a956c1c00, cur 1364497379 expire 1364497229 last 1364497152
Lustre: Skipped 14 previous similar messages
Lustre: 1807:0:(o2iblnd_cb.c:2330:kiblnd_passive_connect()) Conn stale 10.151.28.215@o2ib [old ver: 12, new ver: 12]
Lustre: 1789:0:(o2iblnd_cb.c:2330:kiblnd_passive_connect()) Conn stale 10.151.55.8@o2ib [old ver: 12, new ver: 12]
Lustre: nbp2-OST0034: haven't heard from client 9aee0651-5355-0bb7-f897-0a7e8f9f92e7 (at 10.151.55.8@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a33229000, cur 1364497671 expire 1364497521 last 1364497444
Lustre: Skipped 14 previous similar messages
Lustre: nbp2-OST001c: haven't heard from client 9d4c2967-aaad-d39e-3d61-cd33e74264e9 (at 10.151.13.31@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a942db800, cur 1364497848 expire 1364497698 last 1364497621
Lustre: Skipped 89 previous similar messages
Lustre: 26708:0:(ldlm_lib.c:952:target_handle_connect()) nbp2-OST0004: connection from 4cf66610-1a71-db8b-b1f4-f082786b7200@10.151.13.31@o2ib t0 exp (null) cur 1364497953 last 0
Lustre: 26708:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 113 previous similar messages
Lustre: 1807:0:(o2iblnd_cb.c:2330:kiblnd_passive_connect()) Conn stale 10.151.29.182@o2ib [old ver: 12, new ver: 12]
Lustre: 1807:0:(o2iblnd_cb.c:2330:kiblnd_passive_connect()) Skipped 5 previous similar messages
Lustre: nbp2-OST001c: haven't heard from client 2a7e5276-dc6e-63a8-f184-74ee198f8436 (at 10.151.29.182@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a7ccfa400, cur 1364498434 expire 1364498284 last 1364498207
Lustre: Skipped 449 previous similar messages
Lustre: 1807:0:(o2iblnd_cb.c:2330:kiblnd_passive_connect()) Conn stale 10.151.28.79@o2ib [old ver: 12, new ver: 12]
Lustre: 1807:0:(o2iblnd_cb.c:2330:kiblnd_passive_connect()) Skipped 1 previous similar message
CONSOLE MARK 201303281230
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 151s: evicting client at 10.151.56.228@o2ib  ns: filter-nbp2-OST005c_UUID lock: ffff880a0b398000/0xde8090a69a819439 lrc: 3/0,0 mode: PW/PW res: 198596049/0 rrc: 64 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x7ddfede63ffc0ba1 expref: 4 pid: 27016 timeout 5506167914
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) Skipped 1 previous similar message
Lustre: 27087:0:(ldlm_lib.c:952:target_handle_connect()) nbp2-OST005c: connection from 149128fb-4c2e-bfac-f636-b4cdc78df50d@10.151.56.228@o2ib t0 exp (null) cur 1364500056 last 0
Lustre: 27087:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 454 previous similar messages
LustreError: 23:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 302s: evicting client at 10.151.55.249@o2ib  ns: filter-nbp2-OST005c_UUID lock: ffff88092abe8240/0xde8090a69a819440 lrc: 3/0,0 mode: PW/PW res: 198596049/0 rrc: 64 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x61c570729ccf9b10 expref: 4 pid: 27049 timeout 5506318133
Lustre: 27061:0:(ldlm_lib.c:952:target_handle_connect()) nbp2-OST005c: connection from 57d98725-86da-1171-14bc-5653ccd00bb0@10.151.55.249@o2ib t0 exp (null) cur 1364500194 last 0
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 151s: evicting client at 10.151.54.236@o2ib  ns: filter-nbp2-OST005c_UUID lock: ffff880863f78240/0xde8090a69a8549ed lrc: 3/0,0 mode: PW/PW res: 198596049/0 rrc: 256 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80000020 remote: 0x18112a55a3a31b14 expref: 8 pid: 26228 timeout 5506619392
LustreError: 26038:0:(client.c:1045:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff8805afc96c00 x1429513587698051/t0(0) o105->nbp2-OST005c@10.151.54.236@o2ib:15/16 lens 344/192 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1
LustreError: 26038:0:(ldlm_lockd.c:656:ldlm_handle_ast_error()) ### client (nid 10.151.54.236@o2ib) returned 0 from completion AST ns: filter-nbp2-OST005c_UUID lock: ffff88067efbc6c0/0xde8090a69a8549fb lrc: 4/0,0 mode: PW/PW res: 198596049/0 rrc: 257 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x18112a55a3a31b1b expref: 8 pid: 27083 timeout 5506770109
LustreError: 26038:0:(client.c:1045:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff8805afc96c00 x1429513587698053/t0(0) o105->nbp2-OST005c@10.151.54.236@o2ib:15/16 lens 344/192 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1
Lustre: 26875:0:(ldlm_lib.c:952:target_handle_connect()) nbp2-OST005c: connection from 59ccce36-cc90-e025-8d7d-80c6500a873c@10.151.54.236@o2ib t0 exp (null) cur 1364500498 last 0
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 453s: evicting client at 10.151.54.238@o2ib  ns: filter-nbp2-OST005c_UUID lock: ffff8809d9f5f6c0/0xde8090a69a854a33 lrc: 3/0,0 mode: PW/PW res: 198596049/0 rrc: 248 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x46a2531ac439d390 expref: 8 pid: 27045 timeout 5506921000
LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) Skipped 1 previous similar message
Lustre: 26676:0:(ldlm_lib.c:952:target_handle_connect()) nbp2-OST005c: connection from 8cbfd74b-c488-2363-497c-caa516fccefe@10.151.54.238@o2ib t0 exp (null) cur 1364500799 last 0
Lustre: 26676:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 14 previous similar messages
[-- MARK -- Thu Mar 28 13:00:00 2013]
