Jul 16 18:06:56 cs04r-sc-oss01-08 kernel: Lustre: 20733:0:(ldlm_lib.c:874:target_handle_connect()) play01-OST0000: refuse reconnection from 69d87395-1fce-0c9e-2505-0acc7fd7164e@172.23.138.5@tcp to 0xffff81021b506a00; still busy with 7 active RPCs
Jul 16 18:06:56 cs04r-sc-oss01-08 kernel: Lustre: 20733:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 2 previous similar messages
Jul 16 18:07:02 cs04r-sc-oss01-08 kernel: LustreError: 5796:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff81032e732000
Jul 16 18:07:02 cs04r-sc-oss01-08 kernel: LustreError: 5796:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff8100ab9f2000
Jul 16 18:07:02 cs04r-sc-oss01-08 kernel: Lustre: 21472:0:(ost_handler.c:887:ost_brw_read()) play01-OST0005: ignoring bulk IO comm error with 69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID id 12345-172.23.138.5@tcp - client will retry
Jul 16 18:07:02 cs04r-sc-oss01-08 kernel: Lustre: 21472:0:(ost_handler.c:887:ost_brw_read()) Skipped 20 previous similar messages
Jul 16 18:07:02 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff810182e22000
Jul 16 18:07:02 cs04r-sc-oss01-08 kernel: LustreError: 5796:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff8100ab9f8000
Jul 16 18:07:02 cs04r-sc-oss01-08 kernel: LustreError: 5794:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff8102968f4000
Jul 16 18:07:02 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff81028215a000
Jul 16 18:07:02 cs04r-sc-oss01-08 kernel: Lustre: 21188:0:(ost_handler.c:887:ost_brw_read()) play01-OST0004: ignoring bulk IO comm error with 69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID id 12345-172.23.138.5@tcp - client will retry
Jul 16 18:07:02 cs04r-sc-oss01-08 kernel: Lustre: 21188:0:(ost_handler.c:887:ost_brw_read()) Skipped 3 previous similar messages
Jul 16 18:07:02 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff810399006000
Jul 16 18:07:02 cs04r-sc-oss01-08 kernel: LustreError: 5796:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff8101618d4000
Jul 16 18:07:02 cs04r-sc-oss01-08 kernel: LustreError: 5796:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff810141600000
Jul 16 18:07:02 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff8102bf25e000
Jul 16 18:08:27 cs04r-sc-oss01-08 kernel: Lustre: 20659:0:(ldlm_lib.c:574:target_handle_reconnect()) play01-OST0002: 69d87395-1fce-0c9e-2505-0acc7fd7164e reconnecting
Jul 16 18:08:27 cs04r-sc-oss01-08 kernel: Lustre: 20659:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 9 previous similar messages
Jul 16 18:08:30 cs04r-sc-oss01-08 init: Id "co" respawning too fast: disabled for 5 minutes
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow i_mutex 141s due to heavy IO load
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow quota init 141s due to heavy IO load
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: Lustre: Skipped 3 previous similar messages
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow journal start 97s due to heavy IO load
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: Lustre: Skipped 3 previous similar messages
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow journal start 137s due to heavy IO load
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: Lustre: Skipped 15 previous similar messages
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow brw_start 137s due to heavy IO load
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: Lustre: Skipped 15 previous similar messages
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow direct_io 141s due to heavy IO load
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow journal start 141s due to heavy IO load
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow commitrw commit 141s due to heavy IO load
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: Lustre: Skipped 23 previous similar messages
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: LustreError: 0:0:(ldlm_lockd.c:313:waiting_locks_callback()) ### lock callback timer expired after 137s: evicting client at 172.23.142.199@tcp  ns: filter-play01-OST0003_UUID lock: ffff8101755c7c00/0x8036ad3b0f6b4cd9 lrc: 3/0,0 mode: PW/PW res: 22843717/0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x20 remote: 0xd577a930b0978dc0 expref: 203 pid: 20985 timeout 4310694078
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: LustreError: 0:0:(ldlm_lockd.c:313:waiting_locks_callback()) Skipped 3 previous similar messages
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: LustreError: 21385:0:(ost_handler.c:1078:ost_brw_write()) @@@ ptlrpc_bulk_get failed: rc -107  req@ffff810302a67400 x1407112325899528/t0 o4->7189d78c-6463-a455-0a8c-a61a7453c968@NET_0x20000ac178ec7_UUID:0/0 lens 448/416 e 0 to 0 dl 1342458726 ref 1 fl Interpret:/0/0 rc 0/0
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: Lustre: 21216:0:(ost_handler.c:1224:ost_brw_write()) play01-OST0003: ignoring bulk IO comm error with 7189d78c-6463-a455-0a8c-a61a7453c968@NET_0x20000ac178ec7_UUID id 12345-172.23.142.199@tcp - client will retry
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: LustreError: 21296:0:(ost_handler.c:1060:ost_brw_write()) @@@ Eviction on bulk GET  req@ffff81042f19a450 x1407112325899544/t0 o4->7189d78c-6463-a455-0a8c-a61a7453c968@NET_0x20000ac178ec7_UUID:0/0 lens 448/416 e 0 to 0 dl 1342458726 ref 1 fl Interpret:/0/0 rc 0/0
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: LustreError: 21922:0:(ldlm_lockd.c:1883:ldlm_cancel_handler()) ldlm_cancel from 172.23.142.199@tcp arrived at 1342458511 with bad export cookie 9238762154770351400
Jul 16 18:08:31 cs04r-sc-oss01-08 kernel: LustreError: 21385:0:(ost_handler.c:1078:ost_brw_write()) Skipped 3 previous similar messages
Jul 16 18:08:32 cs04r-sc-oss01-08 kernel: LustreError: 21217:0:(ost_handler.c:1078:ost_brw_write()) @@@ ptlrpc_bulk_get failed: rc -107  req@ffff8102df515000 x1407112325899542/t0 o4->7189d78c-6463-a455-0a8c-a61a7453c968@NET_0x20000ac178ec7_UUID:0/0 lens 448/416 e 0 to 0 dl 1342458726 ref 1 fl Interpret:/0/0 rc 0/0
Jul 16 18:08:32 cs04r-sc-oss01-08 kernel: LustreError: 21217:0:(ost_handler.c:1078:ost_brw_write()) Skipped 3 previous similar messages
Jul 16 18:08:32 cs04r-sc-oss01-08 kernel: Lustre: 21217:0:(ost_handler.c:1224:ost_brw_write()) play01-OST0003: ignoring bulk IO comm error with 7189d78c-6463-a455-0a8c-a61a7453c968@NET_0x20000ac178ec7_UUID id 12345-172.23.142.199@tcp - client will retry
Jul 16 18:08:32 cs04r-sc-oss01-08 kernel: Lustre: 21217:0:(ost_handler.c:1224:ost_brw_write()) Skipped 6 previous similar messages
Jul 16 18:08:32 cs04r-sc-oss01-08 kernel: LustreError: 20753:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff810415674800 x1407653101671180/t0 o104->@NET_0x20000ac178ec7_UUID:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jul 16 18:08:32 cs04r-sc-oss01-08 kernel: LustreError: 20753:0:(ldlm_lockd.c:612:ldlm_handle_ast_error()) ### client (nid 172.23.142.199@tcp) returned 0 from blocking AST ns: filter-play01-OST0003_UUID lock: ffff810392b0bc00/0x8036ad3b0f6b6218 lrc: 4/0,0 mode: PW/PW res: 22843749/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x20 remote: 0xd577a930b097b83e expref: 54 pid: 20630 timeout 4310909079
Jul 16 18:08:33 cs04r-sc-oss01-08 kernel: LustreError: 20625:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff81022c45a000 x1407653101671185/t0 o104->@NET_0x20000ac178ec7_UUID:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jul 16 18:08:33 cs04r-sc-oss01-08 kernel: LustreError: 20625:0:(ldlm_lockd.c:612:ldlm_handle_ast_error()) ### client (nid 172.23.142.199@tcp) returned 0 from blocking AST ns: filter-play01-OST0003_UUID lock: ffff81009a01c000/0x8036ad3b0f6b5028 lrc: 4/0,0 mode: PW/PW res: 22843731/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x20 remote: 0xd577a930b0979c14 expref: 53 pid: 21036 timeout 4310909890
Jul 16 18:08:35 cs04r-sc-oss01-08 kernel: failure to allocate a tage (104)
Jul 16 18:08:35 cs04r-sc-oss01-08 kernel: failure to allocate a tage (169)
Jul 16 18:08:35 cs04r-sc-oss01-08 kernel: LustreError: 21076:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff8102d4919400 x1407653101671211/t0 o104->@NET_0x20000ac178ec7_UUID:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jul 16 18:08:35 cs04r-sc-oss01-08 kernel: LustreError: 21076:0:(ldlm_lockd.c:612:ldlm_handle_ast_error()) ### client (nid 172.23.142.199@tcp) returned 0 from blocking AST ns: filter-play01-OST0003_UUID lock: ffff810080a9d600/0x8036ad3b0f6b5ab6 lrc: 4/0,0 mode: PW/PW res: 22843737/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x20 remote: 0xd577a930b097a84d expref: 52 pid: 21005 timeout 4310911800
Jul 16 18:08:35 cs04r-sc-oss01-08 kernel: LustreError: 21076:0:(ldlm_lockd.c:612:ldlm_handle_ast_error()) Skipped 1 previous similar message
Jul 16 18:08:35 cs04r-sc-oss01-08 kernel: failure to allocate a tage (92)
Jul 16 18:08:35 cs04r-sc-oss01-08 kernel: failure to allocate a tage (180)
Jul 16 18:08:35 cs04r-sc-oss01-08 kernel: failure to allocate a tage (95)
Jul 16 18:08:36 cs04r-sc-oss01-08 kernel: failure to allocate a tage (180)
Jul 16 18:08:36 cs04r-sc-oss01-08 kernel: failure to allocate a tage (99)
Jul 16 18:08:36 cs04r-sc-oss01-08 kernel: LustreError: 0:0:(ldlm_lockd.c:313:waiting_locks_callback()) ### lock callback timer expired after 133s: evicting client at 172.23.142.199@tcp  ns: filter-play01-OST0003_UUID lock: ffff810038491000/0x8036ad3b0f6b4e7d lrc: 3/0,0 mode: PW/PW res: 22843726/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x20 remote: 0xd577a930b0979545 expref: 48 pid: 21102 timeout 4310702590
Jul 16 18:08:36 cs04r-sc-oss01-08 kernel: failure to allocate a tage (191)
Jul 16 18:08:36 cs04r-sc-oss01-08 kernel: LustreError: 0:0:(ldlm_lockd.c:313:waiting_locks_callback()) Skipped 2 previous similar messages
Jul 16 18:08:36 cs04r-sc-oss01-08 kernel: failure to allocate a tage (192)
Jul 16 18:08:37 cs04r-sc-oss01-08 kernel: failure to allocate a tage (197)
Jul 16 18:08:37 cs04r-sc-oss01-08 kernel: failure to allocate a tage (204)
Jul 16 18:08:37 cs04r-sc-oss01-08 kernel: failure to allocate a tage (205)
Jul 16 18:08:38 cs04r-sc-oss01-08 kernel: LustreError: 20819:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff81029c30b000 x1407653101671231/t0 o104->@NET_0x20000ac178ec7_UUID:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jul 16 18:08:38 cs04r-sc-oss01-08 kernel: LustreError: 20819:0:(client.c:841:ptlrpc_import_delay_req()) Skipped 3 previous similar messages
Jul 16 18:08:38 cs04r-sc-oss01-08 kernel: LustreError: 20819:0:(ldlm_lockd.c:612:ldlm_handle_ast_error()) ### client (nid 172.23.142.199@tcp) returned 0 from blocking AST ns: filter-play01-OST0003_UUID lock: ffff8103ef17da00/0x8036ad3b0f6b5e9f lrc: 4/0,0 mode: PW/PW res: 22843741/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x20 remote: 0xd577a930b097ad6a expref: 48 pid: 20861 timeout 4310914257
Jul 16 18:08:38 cs04r-sc-oss01-08 kernel: LustreError: 20819:0:(ldlm_lockd.c:612:ldlm_handle_ast_error()) Skipped 2 previous similar messages
Jul 16 18:08:38 cs04r-sc-oss01-08 kernel: failure to allocate a tage (213)
Jul 16 18:08:38 cs04r-sc-oss01-08 kernel: failure to allocate a tage (146)
Jul 16 18:08:38 cs04r-sc-oss01-08 kernel: failure to allocate a tage (222)
Jul 16 18:08:38 cs04r-sc-oss01-08 kernel: Lustre: 21995:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1407653101671146 sent from play01-OST0003 to NID 172.23.138.5@tcp 7s ago has timed out (7s prior to deadline).
Jul 16 18:08:38 cs04r-sc-oss01-08 kernel:   req@ffff8103f7fb2400 x1407653101671146/t0 o105->@NET_0x20000ac178a05_UUID:15/16 lens 344/384 e 0 to 1 dl 1342458518 ref 2 fl Rpc:N/0/0 rc 0/0
Jul 16 18:08:38 cs04r-sc-oss01-08 kernel: Lustre: 21924:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1407653101671142 sent from play01-OST0003 to NID 172.23.138.5@tcp 7s ago has timed out (7s prior to deadline).
Jul 16 18:08:38 cs04r-sc-oss01-08 kernel:   req@ffff8103e8c40000 x1407653101671142/t0 o105->@NET_0x20000ac178a05_UUID:15/16 lens 344/384 e 0 to 1 dl 1342458518 ref 2 fl Rpc:N/0/0 rc 0/0
Jul 16 18:08:38 cs04r-sc-oss01-08 kernel: LustreError: 138-a: play01-OST0003: A client on nid 172.23.138.5@tcp was evicted due to a lock completion callback to 172.23.138.5@tcp timed out: rc -107
Jul 16 18:08:38 cs04r-sc-oss01-08 kernel: Lustre: 21924:0:(service.c:1434:ptlrpc_server_handle_request()) @@@ Request x1407112325899535 took longer than estimated (6+1s); client may timeout.  req@ffff8103e9fb1000 x1407112325899535/t0 o103->7189d78c-6463-a455-0a8c-a61a7453c968@NET_0x20000ac178ec7_UUID:0/0 lens 296/192 e 0 to 0 dl 1342458517 ref 1 fl Complete:/0/0 rc 0/0
Jul 16 18:08:38 cs04r-sc-oss01-08 kernel: Lustre: 21924:0:(service.c:1434:ptlrpc_server_handle_request()) Skipped 77 previous similar messages
Jul 16 18:08:38 cs04r-sc-oss01-08 kernel: LustreError: 138-a: play01-OST0003: A client on nid 172.23.138.5@tcp was evicted due to a lock completion callback to 172.23.138.5@tcp timed out: rc -107
Jul 16 18:08:38 cs04r-sc-oss01-08 kernel: LustreError: Skipped 3 previous similar messages
Jul 16 18:08:38 cs04r-sc-oss01-08 kernel: Lustre: 21995:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jul 16 18:08:39 cs04r-sc-oss01-08 kernel: Lustre: 21995:0:(service.c:1434:ptlrpc_server_handle_request()) @@@ Request x1407112325899536 took longer than estimated (6+1s); client may timeout.  req@ffff81036b3ad000 x1407112325899536/t0 o103->7189d78c-6463-a455-0a8c-a61a7453c968@NET_0x20000ac178ec7_UUID:0/0 lens 296/192 e 0 to 0 dl 1342458517 ref 1 fl Complete:/0/0 rc 0/0
Jul 16 18:08:39 cs04r-sc-oss01-08 kernel: Lustre: 21995:0:(service.c:1434:ptlrpc_server_handle_request()) Skipped 4 previous similar messages
Jul 16 18:08:39 cs04r-sc-oss01-08 kernel: failure to allocate a tage (208)
Jul 16 18:08:40 cs04r-sc-oss01-08 kernel: failure to allocate a tage (250)
Jul 16 18:08:40 cs04r-sc-oss01-08 kernel: failure to allocate a tage (253)
Jul 16 18:08:40 cs04r-sc-oss01-08 kernel: LustreError: 20934:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff810022428c00 x1407653101671262/t0 o104->@NET_0x20000ac178ec7_UUID:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jul 16 18:08:40 cs04r-sc-oss01-08 kernel: LustreError: 20934:0:(ldlm_lockd.c:612:ldlm_handle_ast_error()) ### client (nid 172.23.142.199@tcp) returned 0 from blocking AST ns: filter-play01-OST0003_UUID lock: ffff8102b7460000/0x8036ad3b0f6b5dd4 lrc: 4/0,0 mode: PW/PW res: 22843739/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x20 remote: 0xd577a930b097ab48 expref: 26 pid: 20995 timeout 4310918154
Jul 16 18:08:41 cs04r-sc-oss01-08 kernel: LustreError: 20934:0:(ldlm_lockd.c:612:ldlm_handle_ast_error()) Skipped 1 previous similar message
Jul 16 18:08:41 cs04r-sc-oss01-08 kernel: failure to allocate a tage (286)
Jul 16 18:08:41 cs04r-sc-oss01-08 kernel: failure to allocate a tage (514)
Jul 16 18:08:41 cs04r-sc-oss01-08 kernel: failure to allocate a tage (295)
Jul 16 18:08:41 cs04r-sc-oss01-08 kernel: failure to allocate a tage (527)
Jul 16 18:08:42 cs04r-sc-oss01-08 kernel: failure to allocate a tage (310)
Jul 16 18:08:42 cs04r-sc-oss01-08 kernel: failure to allocate a tage (536)
Jul 16 18:08:42 cs04r-sc-oss01-08 kernel: failure to allocate a tage (314)
Jul 16 18:08:42 cs04r-sc-oss01-08 kernel: failure to allocate a tage (537)
Jul 16 18:08:42 cs04r-sc-oss01-08 kernel: failure to allocate a tage (557)
Jul 16 18:08:43 cs04r-sc-oss01-08 kernel: failure to allocate a tage (349)
Jul 16 18:08:43 cs04r-sc-oss01-08 kernel: failure to allocate a tage (592)
Jul 16 18:08:43 cs04r-sc-oss01-08 kernel: failure to allocate a tage (385)
Jul 16 18:08:44 cs04r-sc-oss01-08 kernel: failure to allocate a tage (625)
Jul 16 18:08:44 cs04r-sc-oss01-08 kernel: failure to allocate a tage (632)
Jul 16 18:08:45 cs04r-sc-oss01-08 kernel: LustreError: 21919:0:(filter.c:2583:filter_grant_sanity_check()) filter_statfs: tot_dirty 64122880 != fo_tot_dirty 72511488
Jul 16 18:08:45 cs04r-sc-oss01-08 kernel: LustreError: 0:0:(ldlm_lockd.c:313:waiting_locks_callback()) ### lock callback timer expired after 133s: evicting client at 172.23.142.199@tcp  ns: filter-play01-OST0003_UUID lock: ffff81011b1c2200/0x8036ad3b0f6b6257 lrc: 3/0,0 mode: PW/PW res: 22843751/0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x20 remote: 0xd577a930b097ba2f expref: 24 pid: 21054 timeout 4310711252
Jul 16 18:08:45 cs04r-sc-oss01-08 kernel: Lustre: 20591:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1407653101671248 sent from play01-OST0003 to NID 172.23.138.5@tcp 7s ago has timed out (7s prior to deadline).
Jul 16 18:08:45 cs04r-sc-oss01-08 kernel:   req@ffff810163eea000 x1407653101671248/t0 o105->@NET_0x20000ac178a05_UUID:15/16 lens 344/384 e 0 to 1 dl 1342458525 ref 2 fl Rpc:N/0/0 rc 0/0
Jul 16 18:08:45 cs04r-sc-oss01-08 kernel: Lustre: 20591:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jul 16 18:08:45 cs04r-sc-oss01-08 kernel: LustreError: 138-a: play01-OST0003: A client on nid 172.23.138.5@tcp was evicted due to a lock completion callback to 172.23.138.5@tcp timed out: rc -107
Jul 16 18:08:45 cs04r-sc-oss01-08 kernel: LustreError: Skipped 5 previous similar messages
Jul 16 18:08:46 cs04r-sc-oss01-08 kernel: failure to allocate a tage (321)
Jul 16 18:08:46 cs04r-sc-oss01-08 kernel: failure to allocate a tage (648)
Jul 16 18:08:46 cs04r-sc-oss01-08 kernel: failure to allocate a tage (417)
Jul 16 18:08:48 cs04r-sc-oss01-08 kernel: failure to allocate a tage (698)
Jul 16 18:08:48 cs04r-sc-oss01-08 kernel: failure to allocate a tage (699)
Jul 16 18:08:48 cs04r-sc-oss01-08 kernel: failure to allocate a tage (699)
Jul 16 18:08:49 cs04r-sc-oss01-08 kernel: failure to allocate a tage (461)
Jul 16 18:08:49 cs04r-sc-oss01-08 kernel: failure to allocate a tage (463)
Jul 16 18:08:49 cs04r-sc-oss01-08 kernel: failure to allocate a tage (740)
Jul 16 18:08:50 cs04r-sc-oss01-08 kernel: failure to allocate a tage (744)
Jul 16 18:08:50 cs04r-sc-oss01-08 kernel: LustreError: 21882:0:(filter.c:2583:filter_grant_sanity_check()) filter_statfs: tot_dirty 65204224 != fo_tot_dirty 73592832
Jul 16 18:08:50 cs04r-sc-oss01-08 kernel: LustreError: 20594:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff810081090800 x1407653101671287/t0 o104->@NET_0x20000ac178ec7_UUID:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jul 16 18:08:50 cs04r-sc-oss01-08 kernel: LustreError: 20594:0:(client.c:841:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Jul 16 18:08:50 cs04r-sc-oss01-08 kernel: LustreError: 20594:0:(ldlm_lockd.c:612:ldlm_handle_ast_error()) ### client (nid 172.23.142.199@tcp) returned 0 from blocking AST ns: filter-play01-OST0003_UUID lock: ffff81039d809600/0x8036ad3b0f6b68c4 lrc: 4/0,0 mode: PW/PW res: 22843759/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x20 remote: 0xd577a930b097c2e8 expref: 21 pid: 21038 timeout 4310927998
Jul 16 18:08:51 cs04r-sc-oss01-08 kernel: failure to allocate a tage (492)
Jul 16 18:08:51 cs04r-sc-oss01-08 kernel: failure to allocate a tage (777)
Jul 16 18:08:51 cs04r-sc-oss01-08 kernel: Lustre: 20954:0:(ldlm_lib.c:874:target_handle_connect()) play01-OST0001: refuse reconnection from 69d87395-1fce-0c9e-2505-0acc7fd7164e@172.23.138.5@tcp to 0xffff810235b63800; still busy with 2 active RPCs
Jul 16 18:08:51 cs04r-sc-oss01-08 kernel: Lustre: 20954:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 5 previous similar messages
Jul 16 18:08:51 cs04r-sc-oss01-08 kernel: failure to allocate a tage (504)
Jul 16 18:08:51 cs04r-sc-oss01-08 kernel: failure to allocate a tage (785)
Jul 16 18:08:52 cs04r-sc-oss01-08 kernel: failure to allocate a tage (505)
Jul 16 18:08:52 cs04r-sc-oss01-08 kernel: failure to allocate a tage (796)
Jul 16 18:08:52 cs04r-sc-oss01-08 kernel: LustreError: 21566:0:(ost_handler.c:829:ost_brw_read()) @@@ Reconnect on bulk PUT  req@ffff810142bdc400 x1405109999587549/t0 o3->69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID:0/0 lens 448/400 e 0 to 0 dl 1342458700 ref 1 fl Interpret:/0/0 rc 0/0
Jul 16 18:08:52 cs04r-sc-oss01-08 kernel: LustreError: 21566:0:(ost_handler.c:829:ost_brw_read()) Skipped 14 previous similar messages
Jul 16 18:08:52 cs04r-sc-oss01-08 kernel: failure to allocate a tage (511)
Jul 16 18:08:52 cs04r-sc-oss01-08 kernel: failure to allocate a tage (515)
Jul 16 18:08:52 cs04r-sc-oss01-08 kernel: failure to allocate a tage (515)
Jul 16 18:08:52 cs04r-sc-oss01-08 kernel: Lustre: 20591:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1407653101671279 sent from play01-OST0003 to NID 172.23.138.5@tcp 7s ago has timed out (7s prior to deadline).
Jul 16 18:08:52 cs04r-sc-oss01-08 kernel:   req@ffff81019d327000 x1407653101671279/t0 o105->@NET_0x20000ac178a05_UUID:15/16 lens 344/384 e 0 to 1 dl 1342458532 ref 2 fl Rpc:N/0/0 rc 0/0
Jul 16 18:08:52 cs04r-sc-oss01-08 kernel: LustreError: 138-a: play01-OST0003: A client on nid 172.23.138.5@tcp was evicted due to a lock completion callback to 172.23.138.5@tcp timed out: rc -107
Jul 16 18:08:53 cs04r-sc-oss01-08 kernel: failure to allocate a tage (522)
Jul 16 18:08:53 cs04r-sc-oss01-08 kernel: failure to allocate a tage (526)
Jul 16 18:08:53 cs04r-sc-oss01-08 kernel: failure to allocate a tage (527)
Jul 16 18:08:53 cs04r-sc-oss01-08 kernel: failure to allocate a tage (531)
Jul 16 18:08:54 cs04r-sc-oss01-08 kernel: failure to allocate a tage (531)
Jul 16 18:08:54 cs04r-sc-oss01-08 kernel: failure to allocate a tage (630)
Jul 16 18:08:54 cs04r-sc-oss01-08 kernel: failure to allocate a tage (539)
Jul 16 18:08:54 cs04r-sc-oss01-08 kernel: failure to allocate a tage (457)
Jul 16 18:08:54 cs04r-sc-oss01-08 kernel: failure to allocate a tage (457)
Jul 16 18:08:55 cs04r-sc-oss01-08 kernel: failure to allocate a tage (651)
Jul 16 18:08:55 cs04r-sc-oss01-08 kernel: failure to allocate a tage (568)
Jul 16 18:08:55 cs04r-sc-oss01-08 kernel: failure to allocate a tage (666)
Jul 16 18:08:56 cs04r-sc-oss01-08 kernel: failure to allocate a tage (676)
Jul 16 18:08:56 cs04r-sc-oss01-08 kernel: failure to allocate a tage (600)
Jul 16 18:08:57 cs04r-sc-oss01-08 kernel: failure to allocate a tage (603)
Jul 16 18:08:57 cs04r-sc-oss01-08 kernel: failure to allocate a tage (696)
Jul 16 18:08:57 cs04r-sc-oss01-08 kernel: failure to allocate a tage (608)
Jul 16 18:08:57 cs04r-sc-oss01-08 kernel: LustreError: 0:0:(ldlm_lockd.c:313:waiting_locks_callback()) ### lock callback timer expired after 133s: evicting client at 172.23.142.199@tcp  ns: filter-play01-OST0003_UUID lock: ffff81003b1edc00/0x8036ad3b0f6b60dd lrc: 3/0,0 mode: PW/PW res: 22843746/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x20 remote: 0xd577a930b097b359 expref: 12 pid: 21024 timeout 4310723946
Jul 16 18:08:57 cs04r-sc-oss01-08 kernel: failure to allocate a tage (612)
Jul 16 18:08:57 cs04r-sc-oss01-08 kernel: failure to allocate a tage (705)
Jul 16 18:08:58 cs04r-sc-oss01-08 kernel: failure to allocate a tage (706)
Jul 16 18:08:58 cs04r-sc-oss01-08 kernel: failure to allocate a tage (616)
Jul 16 18:08:58 cs04r-sc-oss01-08 kernel: failure to allocate a tage (512)
Jul 16 18:08:59 cs04r-sc-oss01-08 kernel: failure to allocate a tage (713)
Jul 16 18:08:59 cs04r-sc-oss01-08 kernel: failure to allocate a tage (627)
Jul 16 18:08:59 cs04r-sc-oss01-08 kernel: failure to allocate a tage (629)
Jul 16 18:08:59 cs04r-sc-oss01-08 kernel: failure to allocate a tage (631)
Jul 16 18:09:00 cs04r-sc-oss01-08 kernel: Lustre: 20591:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1407653101671296 sent from play01-OST0003 to NID 172.23.138.5@tcp 7s ago has timed out (7s prior to deadline).
Jul 16 18:09:00 cs04r-sc-oss01-08 kernel:   req@ffff8102ec528c00 x1407653101671296/t0 o105->@NET_0x20000ac178a05_UUID:15/16 lens 344/384 e 0 to 1 dl 1342458540 ref 2 fl Rpc:N/0/0 rc 0/0
Jul 16 18:09:00 cs04r-sc-oss01-08 kernel: LustreError: 138-a: play01-OST0003: A client on nid 172.23.138.5@tcp was evicted due to a lock completion callback to 172.23.138.5@tcp timed out: rc -107
Jul 16 18:09:00 cs04r-sc-oss01-08 kernel: failure to allocate a tage (726)
Jul 16 18:09:00 cs04r-sc-oss01-08 kernel: failure to allocate a tage (730)
Jul 16 18:09:00 cs04r-sc-oss01-08 kernel: failure to allocate a tage (735)
Jul 16 18:09:01 cs04r-sc-oss01-08 kernel: failure to allocate a tage (659)
Jul 16 18:09:01 cs04r-sc-oss01-08 kernel: failure to allocate a tage (659)
Jul 16 18:09:01 cs04r-sc-oss01-08 kernel: LustreError: 20620:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff810216029800 x1407653101671325/t0 o104->@NET_0x20000ac178ec7_UUID:15/16 lens 296/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Jul 16 18:09:01 cs04r-sc-oss01-08 kernel: LustreError: 20620:0:(client.c:841:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Jul 16 18:09:01 cs04r-sc-oss01-08 kernel: LustreError: 20620:0:(ldlm_lockd.c:612:ldlm_handle_ast_error()) ### client (nid 172.23.142.199@tcp) returned 0 from blocking AST ns: filter-play01-OST0003_UUID lock: ffff8102d142a600/0x8036ad3b0f6b6950 lrc: 4/0,0 mode: PW/PW res: 22843762/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x20 remote: 0xd577a930b097c573 expref: 9 pid: 20724 timeout 4310952445
Jul 16 18:09:01 cs04r-sc-oss01-08 kernel: LustreError: 20620:0:(ldlm_lockd.c:612:ldlm_handle_ast_error()) Skipped 1 previous similar message
Jul 16 18:09:01 cs04r-sc-oss01-08 kernel: failure to allocate a tage (546)
Jul 16 18:09:02 cs04r-sc-oss01-08 kernel: failure to allocate a tage (750)
Jul 16 18:09:02 cs04r-sc-oss01-08 kernel: failure to allocate a tage (752)
Jul 16 18:09:03 cs04r-sc-oss01-08 kernel: failure to allocate a tage (668)
Jul 16 18:09:03 cs04r-sc-oss01-08 kernel: failure to allocate a tage (675)
Jul 16 18:09:04 cs04r-sc-oss01-08 kernel: failure to allocate a tage (773)
Jul 16 18:09:04 cs04r-sc-oss01-08 kernel: failure to allocate a tage (775)
Jul 16 18:09:04 cs04r-sc-oss01-08 kernel: failure to allocate a tage (778)
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff81040df70000
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff810237d9e000
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff8102386f0000
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: Lustre: 21220:0:(ost_handler.c:887:ost_brw_read()) play01-OST0001: ignoring bulk IO comm error with 69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID id 12345-172.23.138.5@tcp - client will retry
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: Lustre: 21220:0:(ost_handler.c:887:ost_brw_read()) Skipped 42 previous similar messages
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 5796:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff81030c860000
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 5796:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff8102bfde6000
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 5796:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff81033c560000
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff81029930c000
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff81008f028000
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff810433b6c000
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff81004b1ca000
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff810294180000
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff8103f9ad6000
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: Lustre: 20591:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1407653101671322 sent from play01-OST0003 to NID 172.23.138.5@tcp 5s ago has failed due to network error (7s prior to deadline).
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel:   req@ffff810336a0e800 x1407653101671322/t0 o105->@NET_0x20000ac178a05_UUID:15/16 lens 344/384 e 0 to 1 dl 1342458547 ref 1 fl Rpc:N/0/0 rc 0/0
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff810247fd6000
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 5796:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff81031ddb6000
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 5796:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff810010068000
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 5796:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff8103b25e0000
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 5796:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff81038a3cc000
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 20591:0:(filter.c:2583:filter_grant_sanity_check()) filter_disconnect: tot_dirty 65802240 != fo_tot_dirty 74190848
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: failure to allocate a tage (628)
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: 20591:0:(ldlm_lockd.c:234:expired_lock_main()) dump the log upon eviction
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342458545.20591
Jul 16 18:09:05 cs04r-sc-oss01-08 kernel: failure to allocate a tage (0)
Jul 16 18:09:06 cs04r-sc-oss01-08 last message repeated 5 times
Jul 16 18:09:06 cs04r-sc-oss01-08 kernel: failure to allocate a tage (10)
Jul 16 18:09:06 cs04r-sc-oss01-08 kernel: failure to allocate a tage (12)
Jul 16 18:09:07 cs04r-sc-oss01-08 kernel: failure to allocate a tage (27)
Jul 16 18:09:07 cs04r-sc-oss01-08 kernel: failure to allocate a tage (16)
Jul 16 18:09:08 cs04r-sc-oss01-08 kernel: failure to allocate a tage (24)
Jul 16 18:09:08 cs04r-sc-oss01-08 kernel: failure to allocate a tage (52)
Jul 16 18:09:09 cs04r-sc-oss01-08 kernel: failure to allocate a tage (63)
Jul 16 18:09:09 cs04r-sc-oss01-08 kernel: failure to allocate a tage (41)
Jul 16 18:09:09 cs04r-sc-oss01-08 kernel: failure to allocate a tage (71)
Jul 16 18:09:10 cs04r-sc-oss01-08 kernel: failure to allocate a tage (77)
Jul 16 18:09:10 cs04r-sc-oss01-08 kernel: failure to allocate a tage (63)
Jul 16 18:09:10 cs04r-sc-oss01-08 kernel: failure to allocate a tage (82)
Jul 16 18:09:10 cs04r-sc-oss01-08 kernel: failure to allocate a tage (93)
Jul 16 18:09:11 cs04r-sc-oss01-08 kernel: failure to allocate a tage (98)
Jul 16 18:09:12 cs04r-sc-oss01-08 kernel: failure to allocate a tage (115)
Jul 16 18:09:12 cs04r-sc-oss01-08 kernel: failure to allocate a tage (128)
Jul 16 18:09:12 cs04r-sc-oss01-08 kernel: failure to allocate a tage (141)
Jul 16 18:09:13 cs04r-sc-oss01-08 kernel: failure to allocate a tage (148)
Jul 16 18:09:13 cs04r-sc-oss01-08 kernel: failure to allocate a tage (150)
Jul 16 18:09:13 cs04r-sc-oss01-08 kernel: failure to allocate a tage (141)
Jul 16 18:09:13 cs04r-sc-oss01-08 kernel: failure to allocate a tage (164)
Jul 16 18:09:14 cs04r-sc-oss01-08 kernel: failure to allocate a tage (168)
Jul 16 18:09:14 cs04r-sc-oss01-08 kernel: failure to allocate a tage (170)
Jul 16 18:09:14 cs04r-sc-oss01-08 kernel: failure to allocate a tage (171)
Jul 16 18:09:15 cs04r-sc-oss01-08 kernel: failure to allocate a tage (176)
Jul 16 18:09:15 cs04r-sc-oss01-08 kernel: failure to allocate a tage (178)
Jul 16 18:09:15 cs04r-sc-oss01-08 kernel: failure to allocate a tage (144)
Jul 16 18:09:16 cs04r-sc-oss01-08 kernel: failure to allocate a tage (186)
Jul 16 18:09:17 cs04r-sc-oss01-08 kernel: failure to allocate a tage (178)
Jul 16 18:09:17 cs04r-sc-oss01-08 kernel: failure to allocate a tage (188)
Jul 16 18:09:17 cs04r-sc-oss01-08 kernel: failure to allocate a tage (183)
Jul 16 18:09:18 cs04r-sc-oss01-08 kernel: failure to allocate a tage (199)
Jul 16 18:09:19 cs04r-sc-oss01-08 kernel: failure to allocate a tage (160)
Jul 16 18:09:19 cs04r-sc-oss01-08 kernel: failure to allocate a tage (192)
Jul 16 18:09:20 cs04r-sc-oss01-08 kernel: failure to allocate a tage (194)
Jul 16 18:09:22 cs04r-sc-oss01-08 kernel: failure to allocate a tage (218)
Jul 16 18:09:22 cs04r-sc-oss01-08 kernel: failure to allocate a tage (202)
Jul 16 18:09:22 cs04r-sc-oss01-08 kernel: failure to allocate a tage (204)
Jul 16 18:09:23 cs04r-sc-oss01-08 kernel: failure to allocate a tage (223)
Jul 16 18:09:23 cs04r-sc-oss01-08 kernel: failure to allocate a tage (225)
Jul 16 18:09:23 cs04r-sc-oss01-08 kernel: failure to allocate a tage (206)
Jul 16 18:09:23 cs04r-sc-oss01-08 kernel: Lustre: 20718:0:(ldlm_lib.c:874:target_handle_connect()) play01-OST0004: refuse reconnection from 69d87395-1fce-0c9e-2505-0acc7fd7164e@172.23.138.5@tcp to 0xffff810118a87400; still busy with 4 active RPCs
Jul 16 18:09:23 cs04r-sc-oss01-08 kernel: Lustre: 20718:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 1 previous similar message
Jul 16 18:09:23 cs04r-sc-oss01-08 kernel: failure to allocate a tage (213)
Jul 16 18:09:23 cs04r-sc-oss01-08 kernel: LustreError: 21372:0:(ost_handler.c:844:ost_brw_read()) @@@ bulk PUT failed: rc -107  req@ffff8101a0930000 x1405109999589145/t0 o3->69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID:0/0 lens 448/400 e 0 to 0 dl 1342458778 ref 1 fl Interpret:/0/0 rc 0/0
Jul 16 18:09:23 cs04r-sc-oss01-08 kernel: Lustre: 21372:0:(ost_handler.c:887:ost_brw_read()) play01-OST0004: ignoring bulk IO comm error with 69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID id 12345-172.23.138.5@tcp - client will retry
Jul 16 18:09:23 cs04r-sc-oss01-08 kernel: Lustre: 21372:0:(ost_handler.c:887:ost_brw_read()) Skipped 14 previous similar messages
Jul 16 18:09:24 cs04r-sc-oss01-08 kernel: failure to allocate a tage (220)
Jul 16 18:09:24 cs04r-sc-oss01-08 kernel: LustreError: 21286:0:(ost_handler.c:829:ost_brw_read()) @@@ Reconnect on bulk PUT  req@ffff810016bdb400 x1405109999587556/t0 o3->69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID:0/0 lens 448/400 e 0 to 0 dl 1342458700 ref 1 fl Interpret:/0/0 rc 0/0
Jul 16 18:09:24 cs04r-sc-oss01-08 kernel: LustreError: 21286:0:(ost_handler.c:829:ost_brw_read()) Skipped 2 previous similar messages
Jul 16 18:09:24 cs04r-sc-oss01-08 kernel: LustreError: 21251:0:(ost_handler.c:844:ost_brw_read()) @@@ bulk PUT failed: rc -107  req@ffff81043fd44850 x1405109999589146/t0 o3->69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID:0/0 lens 448/400 e 0 to 0 dl 1342458778 ref 1 fl Interpret:/0/0 rc 0/0
Jul 16 18:09:25 cs04r-sc-oss01-08 kernel: failure to allocate a tage (247)
Jul 16 18:09:25 cs04r-sc-oss01-08 kernel: failure to allocate a tage (236)
Jul 16 18:09:26 cs04r-sc-oss01-08 kernel: failure to allocate a tage (243)
Jul 16 18:09:26 cs04r-sc-oss01-08 kernel: failure to allocate a tage (244)
Jul 16 18:09:26 cs04r-sc-oss01-08 kernel: failure to allocate a tage (262)
Jul 16 18:09:27 cs04r-sc-oss01-08 kernel: failure to allocate a tage (263)
Jul 16 18:09:27 cs04r-sc-oss01-08 kernel: failure to allocate a tage (256)
Jul 16 18:09:28 cs04r-sc-oss01-08 kernel: failure to allocate a tage (283)
Jul 16 18:09:28 cs04r-sc-oss01-08 kernel: failure to allocate a tage (277)
Jul 16 18:09:29 cs04r-sc-oss01-08 kernel: failure to allocate a tage (278)
Jul 16 18:09:29 cs04r-sc-oss01-08 kernel: failure to allocate a tage (278)
Jul 16 18:09:29 cs04r-sc-oss01-08 kernel: failure to allocate a tage (280)
Jul 16 18:09:29 cs04r-sc-oss01-08 kernel: failure to allocate a tage (310)
Jul 16 18:09:30 cs04r-sc-oss01-08 kernel: failure to allocate a tage (283)
Jul 16 18:09:30 cs04r-sc-oss01-08 kernel: failure to allocate a tage (318)
Jul 16 18:09:30 cs04r-sc-oss01-08 kernel: failure to allocate a tage (328)
Jul 16 18:09:30 cs04r-sc-oss01-08 kernel: failure to allocate a tage (329)
Jul 16 18:09:30 cs04r-sc-oss01-08 kernel: failure to allocate a tage (299)
Jul 16 18:09:31 cs04r-sc-oss01-08 kernel: failure to allocate a tage (300)
Jul 16 18:09:31 cs04r-sc-oss01-08 kernel: failure to allocate a tage (341)
Jul 16 18:09:31 cs04r-sc-oss01-08 kernel: failure to allocate a tage (303)
Jul 16 18:09:31 cs04r-sc-oss01-08 kernel: failure to allocate a tage (303)
Jul 16 18:09:32 cs04r-sc-oss01-08 kernel: failure to allocate a tage (367)
Jul 16 18:09:33 cs04r-sc-oss01-08 kernel: failure to allocate a tage (370)
Jul 16 18:09:33 cs04r-sc-oss01-08 kernel: failure to allocate a tage (373)
Jul 16 18:09:34 cs04r-sc-oss01-08 kernel: failure to allocate a tage (376)
Jul 16 18:09:34 cs04r-sc-oss01-08 kernel: failure to allocate a tage (338)
Jul 16 18:09:34 cs04r-sc-oss01-08 kernel: failure to allocate a tage (388)
Jul 16 18:09:35 cs04r-sc-oss01-08 kernel: failure to allocate a tage (346)
Jul 16 18:09:36 cs04r-sc-oss01-08 kernel: failure to allocate a tage (399)
Jul 16 18:09:36 cs04r-sc-oss01-08 kernel: failure to allocate a tage (300)
Jul 16 18:09:36 cs04r-sc-oss01-08 last message repeated 2 times
Jul 16 18:09:36 cs04r-sc-oss01-08 kernel: failure to allocate a tage (303)
Jul 16 18:09:37 cs04r-sc-oss01-08 kernel: failure to allocate a tage (407)
Jul 16 18:09:37 cs04r-sc-oss01-08 kernel: failure to allocate a tage (359)
Jul 16 18:09:37 cs04r-sc-oss01-08 kernel: failure to allocate a tage (408)
Jul 16 18:09:37 cs04r-sc-oss01-08 kernel: failure to allocate a tage (408)
Jul 16 18:09:37 cs04r-sc-oss01-08 kernel: failure to allocate a tage (359)
Jul 16 18:09:37 cs04r-sc-oss01-08 kernel: failure to allocate a tage (359)
Jul 16 18:09:38 cs04r-sc-oss01-08 kernel: failure to allocate a tage (409)
Jul 16 18:09:38 cs04r-sc-oss01-08 kernel: failure to allocate a tage (412)
Jul 16 18:09:38 cs04r-sc-oss01-08 kernel: failure to allocate a tage (363)
Jul 16 18:09:38 cs04r-sc-oss01-08 kernel: failure to allocate a tage (415)
Jul 16 18:09:39 cs04r-sc-oss01-08 kernel: failure to allocate a tage (425)
Jul 16 18:09:40 cs04r-sc-oss01-08 kernel: failure to allocate a tage (428)
Jul 16 18:09:40 cs04r-sc-oss01-08 kernel: failure to allocate a tage (374)
Jul 16 18:09:40 cs04r-sc-oss01-08 kernel: failure to allocate a tage (446)
Jul 16 18:09:41 cs04r-sc-oss01-08 kernel: failure to allocate a tage (384)
Jul 16 18:09:41 cs04r-sc-oss01-08 kernel: failure to allocate a tage (449)
Jul 16 18:09:41 cs04r-sc-oss01-08 kernel: failure to allocate a tage (390)
Jul 16 18:09:42 cs04r-sc-oss01-08 kernel: failure to allocate a tage (403)
Jul 16 18:09:42 cs04r-sc-oss01-08 kernel: failure to allocate a tage (477)
Jul 16 18:09:42 cs04r-sc-oss01-08 kernel: failure to allocate a tage (410)
Jul 16 18:09:43 cs04r-sc-oss01-08 kernel: failure to allocate a tage (414)
Jul 16 18:09:43 cs04r-sc-oss01-08 kernel: failure to allocate a tage (488)
Jul 16 18:09:43 cs04r-sc-oss01-08 kernel: failure to allocate a tage (493)
Jul 16 18:09:43 cs04r-sc-oss01-08 kernel: failure to allocate a tage (493)
Jul 16 18:09:43 cs04r-sc-oss01-08 kernel: failure to allocate a tage (438)
Jul 16 18:09:44 cs04r-sc-oss01-08 kernel: failure to allocate a tage (506)
Jul 16 18:09:44 cs04r-sc-oss01-08 kernel: failure to allocate a tage (510)
Jul 16 18:09:44 cs04r-sc-oss01-08 kernel: failure to allocate a tage (448)
Jul 16 18:09:44 cs04r-sc-oss01-08 kernel: failure to allocate a tage (454)
Jul 16 18:09:44 cs04r-sc-oss01-08 kernel: failure to allocate a tage (518)
Jul 16 18:09:45 cs04r-sc-oss01-08 kernel: failure to allocate a tage (461)
Jul 16 18:09:46 cs04r-sc-oss01-08 kernel: failure to allocate a tage (468)
Jul 16 18:09:46 cs04r-sc-oss01-08 kernel: failure to allocate a tage (538)
Jul 16 18:09:46 cs04r-sc-oss01-08 kernel: failure to allocate a tage (472)
Jul 16 18:09:46 cs04r-sc-oss01-08 kernel: failure to allocate a tage (540)
Jul 16 18:09:46 cs04r-sc-oss01-08 kernel: failure to allocate a tage (541)
Jul 16 18:09:46 cs04r-sc-oss01-08 kernel: failure to allocate a tage (547)
Jul 16 18:09:47 cs04r-sc-oss01-08 kernel: failure to allocate a tage (489)
Jul 16 18:09:47 cs04r-sc-oss01-08 kernel: failure to allocate a tage (492)
Jul 16 18:09:47 cs04r-sc-oss01-08 kernel: failure to allocate a tage (553)
Jul 16 18:09:48 cs04r-sc-oss01-08 kernel: failure to allocate a tage (559)
Jul 16 18:09:48 cs04r-sc-oss01-08 kernel: failure to allocate a tage (576)
Jul 16 18:09:49 cs04r-sc-oss01-08 kernel: failure to allocate a tage (583)
Jul 16 18:09:50 cs04r-sc-oss01-08 kernel: failure to allocate a tage (532)
Jul 16 18:09:51 cs04r-sc-oss01-08 kernel: failure to allocate a tage (532)
Jul 16 18:09:51 cs04r-sc-oss01-08 kernel: failure to allocate a tage (589)
Jul 16 18:09:51 cs04r-sc-oss01-08 kernel: failure to allocate a tage (590)
Jul 16 18:09:51 cs04r-sc-oss01-08 kernel: failure to allocate a tage (535)
Jul 16 18:09:51 cs04r-sc-oss01-08 kernel: failure to allocate a tage (594)
Jul 16 18:09:51 cs04r-sc-oss01-08 kernel: failure to allocate a tage (594)
Jul 16 18:09:52 cs04r-sc-oss01-08 kernel: failure to allocate a tage (538)
Jul 16 18:09:52 cs04r-sc-oss01-08 kernel: failure to allocate a tage (539)
Jul 16 18:09:53 cs04r-sc-oss01-08 kernel: failure to allocate a tage (545)
Jul 16 18:09:54 cs04r-sc-oss01-08 kernel: failure to allocate a tage (546)
Jul 16 18:09:54 cs04r-sc-oss01-08 kernel: failure to allocate a tage (546)
Jul 16 18:09:55 cs04r-sc-oss01-08 kernel: failure to allocate a tage (547)
Jul 16 18:09:56 cs04r-sc-oss01-08 kernel: failure to allocate a tage (547)
Jul 16 18:09:56 cs04r-sc-oss01-08 kernel: failure to allocate a tage (615)
Jul 16 18:09:57 cs04r-sc-oss01-08 kernel: failure to allocate a tage (616)
Jul 16 18:09:57 cs04r-sc-oss01-08 kernel: failure to allocate a tage (547)
Jul 16 18:09:58 cs04r-sc-oss01-08 kernel: failure to allocate a tage (547)
Jul 16 18:09:59 cs04r-sc-oss01-08 kernel: failure to allocate a tage (549)
Jul 16 18:10:00 cs04r-sc-oss01-08 kernel: failure to allocate a tage (622)
Jul 16 18:10:00 cs04r-sc-oss01-08 kernel: failure to allocate a tage (549)
Jul 16 18:10:00 cs04r-sc-oss01-08 kernel: failure to allocate a tage (623)
Jul 16 18:10:00 cs04r-sc-oss01-08 kernel: failure to allocate a tage (549)
Jul 16 18:10:00 cs04r-sc-oss01-08 kernel: failure to allocate a tage (623)
Jul 16 18:10:00 cs04r-sc-oss01-08 kernel: failure to allocate a tage (549)
Jul 16 18:10:01 cs04r-sc-oss01-08 kernel: failure to allocate a tage (625)
Jul 16 18:10:01 cs04r-sc-oss01-08 kernel: failure to allocate a tage (551)
Jul 16 18:10:01 cs04r-sc-oss01-08 kernel: failure to allocate a tage (625)
Jul 16 18:10:03 cs04r-sc-oss01-08 last message repeated 4 times
Jul 16 18:10:04 cs04r-sc-oss01-08 kernel: failure to allocate a tage (630)
Jul 16 18:10:05 cs04r-sc-oss01-08 kernel: failure to allocate a tage (631)
Jul 16 18:10:05 cs04r-sc-oss01-08 kernel: failure to allocate a tage (494)
Jul 16 18:10:05 cs04r-sc-oss01-08 kernel: failure to allocate a tage (635)
Jul 16 18:10:06 cs04r-sc-oss01-08 kernel: failure to allocate a tage (637)
Jul 16 18:10:06 cs04r-sc-oss01-08 kernel: failure to allocate a tage (498)
Jul 16 18:10:07 cs04r-sc-oss01-08 kernel: failure to allocate a tage (498)
Jul 16 18:10:07 cs04r-sc-oss01-08 kernel: failure to allocate a tage (640)
Jul 16 18:10:07 cs04r-sc-oss01-08 kernel: failure to allocate a tage (499)
Jul 16 18:10:08 cs04r-sc-oss01-08 kernel: failure to allocate a tage (640)
Jul 16 18:10:09 cs04r-sc-oss01-08 kernel: failure to allocate a tage (576)
Jul 16 18:10:10 cs04r-sc-oss01-08 kernel: failure to allocate a tage (661)
Jul 16 18:10:13 cs04r-sc-oss01-08 kernel: failure to allocate a tage (590)
Jul 16 18:10:13 cs04r-sc-oss01-08 kernel: failure to allocate a tage (593)
Jul 16 18:10:13 cs04r-sc-oss01-08 kernel: failure to allocate a tage (666)
Jul 16 18:10:13 cs04r-sc-oss01-08 kernel: failure to allocate a tage (594)
Jul 16 18:10:14 cs04r-sc-oss01-08 kernel: failure to allocate a tage (595)
Jul 16 18:10:14 cs04r-sc-oss01-08 kernel: failure to allocate a tage (675)
Jul 16 18:10:15 cs04r-sc-oss01-08 kernel: failure to allocate a tage (616)
Jul 16 18:10:15 cs04r-sc-oss01-08 kernel: failure to allocate a tage (628)
Jul 16 18:10:15 cs04r-sc-oss01-08 kernel: failure to allocate a tage (630)
Jul 16 18:10:15 cs04r-sc-oss01-08 kernel: failure to allocate a tage (695)
Jul 16 18:10:15 cs04r-sc-oss01-08 kernel: failure to allocate a tage (634)
Jul 16 18:10:16 cs04r-sc-oss01-08 kernel: failure to allocate a tage (635)
Jul 16 18:10:16 cs04r-sc-oss01-08 kernel: failure to allocate a tage (701)
Jul 16 18:10:16 cs04r-sc-oss01-08 kernel: failure to allocate a tage (568)
Jul 16 18:10:16 cs04r-sc-oss01-08 kernel: failure to allocate a tage (705)
Jul 16 18:10:16 cs04r-sc-oss01-08 kernel: failure to allocate a tage (571)
Jul 16 18:10:17 cs04r-sc-oss01-08 kernel: failure to allocate a tage (707)
Jul 16 18:10:18 cs04r-sc-oss01-08 kernel: failure to allocate a tage (717)
Jul 16 18:10:19 cs04r-sc-oss01-08 kernel: failure to allocate a tage (718)
Jul 16 18:10:19 cs04r-sc-oss01-08 kernel: Lustre: 20917:0:(ldlm_lib.c:574:target_handle_reconnect()) play01-OST0004: 69d87395-1fce-0c9e-2505-0acc7fd7164e reconnecting
Jul 16 18:10:19 cs04r-sc-oss01-08 kernel: Lustre: 20917:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 9 previous similar messages
Jul 16 18:10:19 cs04r-sc-oss01-08 kernel: failure to allocate a tage (725)
Jul 16 18:10:19 cs04r-sc-oss01-08 kernel: failure to allocate a tage (687)
Jul 16 18:10:20 cs04r-sc-oss01-08 kernel: failure to allocate a tage (603)
Jul 16 18:10:20 cs04r-sc-oss01-08 kernel: failure to allocate a tage (605)
Jul 16 18:10:21 cs04r-sc-oss01-08 kernel: failure to allocate a tage (626)
Jul 16 18:10:21 cs04r-sc-oss01-08 kernel: failure to allocate a tage (628)
Jul 16 18:10:22 cs04r-sc-oss01-08 kernel: failure to allocate a tage (765)
Jul 16 18:10:22 cs04r-sc-oss01-08 kernel: failure to allocate a tage (774)
Jul 16 18:10:22 cs04r-sc-oss01-08 kernel: failure to allocate a tage (776)
Jul 16 18:10:22 cs04r-sc-oss01-08 kernel: failure to allocate a tage (789)
Jul 16 18:10:23 cs04r-sc-oss01-08 kernel: failure to allocate a tage (791)
Jul 16 18:10:23 cs04r-sc-oss01-08 kernel: failure to allocate a tage (792)
Jul 16 18:10:24 cs04r-sc-oss01-08 kernel: failure to allocate a tage (842)
Jul 16 18:10:25 cs04r-sc-oss01-08 kernel: failure to allocate a tage (850)
Jul 16 18:10:25 cs04r-sc-oss01-08 kernel: failure to allocate a tage (710)
Jul 16 18:10:26 cs04r-sc-oss01-08 kernel: failure to allocate a tage (723)
Jul 16 18:10:27 cs04r-sc-oss01-08 kernel: failure to allocate a tage (729)
Jul 16 18:10:28 cs04r-sc-oss01-08 kernel: failure to allocate a tage (739)
Jul 16 18:10:29 cs04r-sc-oss01-08 kernel: failure to allocate a tage (752)
Jul 16 18:10:30 cs04r-sc-oss01-08 kernel: failure to allocate a tage (766)
Jul 16 18:10:32 cs04r-sc-oss01-08 kernel: failure to allocate a tage (786)
Jul 16 18:10:33 cs04r-sc-oss01-08 kernel: failure to allocate a tage (794)
Jul 16 18:10:33 cs04r-sc-oss01-08 kernel: failure to allocate a tage (801)
Jul 16 18:10:34 cs04r-sc-oss01-08 kernel: failure to allocate a tage (809)
Jul 16 18:10:34 cs04r-sc-oss01-08 kernel: failure to allocate a tage (809)
Jul 16 18:10:34 cs04r-sc-oss01-08 kernel: failure to allocate a tage (813)
Jul 16 18:10:35 cs04r-sc-oss01-08 kernel: failure to allocate a tage (824)
Jul 16 18:10:35 cs04r-sc-oss01-08 kernel: failure to allocate a tage (824)
Jul 16 18:10:35 cs04r-sc-oss01-08 kernel: failure to allocate a tage (827)
Jul 16 18:10:37 cs04r-sc-oss01-08 kernel: failure to allocate a tage (853)
Jul 16 18:10:38 cs04r-sc-oss01-08 kernel: failure to allocate a tage (853)
Jul 16 18:10:38 cs04r-sc-oss01-08 kernel: failure to allocate a tage (857)
Jul 16 18:10:40 cs04r-sc-oss01-08 kernel: failure to allocate a tage (880)
Jul 16 18:10:40 cs04r-sc-oss01-08 kernel: failure to allocate a tage (886)
Jul 16 18:10:41 cs04r-sc-oss01-08 kernel: failure to allocate a tage (900)
Jul 16 18:10:42 cs04r-sc-oss01-08 kernel: failure to allocate a tage (906)
Jul 16 18:10:42 cs04r-sc-oss01-08 kernel: failure to allocate a tage (906)
Jul 16 18:10:42 cs04r-sc-oss01-08 kernel: failure to allocate a tage (909)
Jul 16 18:10:42 cs04r-sc-oss01-08 kernel: failure to allocate a tage (909)
Jul 16 18:10:43 cs04r-sc-oss01-08 kernel: failure to allocate a tage (912)
Jul 16 18:10:45 cs04r-sc-oss01-08 kernel: failure to allocate a tage (915)
Jul 16 18:10:46 cs04r-sc-oss01-08 kernel: failure to allocate a tage (916)
Jul 16 18:10:46 cs04r-sc-oss01-08 last message repeated 2 times
Jul 16 18:10:46 cs04r-sc-oss01-08 kernel: failure to allocate a tage (919)
Jul 16 18:10:47 cs04r-sc-oss01-08 kernel: failure to allocate a tage (921)
Jul 16 18:10:47 cs04r-sc-oss01-08 kernel: failure to allocate a tage (926)
Jul 16 18:10:48 cs04r-sc-oss01-08 last message repeated 2 times
Jul 16 18:10:49 cs04r-sc-oss01-08 kernel: failure to allocate a tage (928)
Jul 16 18:10:50 cs04r-sc-oss01-08 last message repeated 2 times
Jul 16 18:10:50 cs04r-sc-oss01-08 kernel: failure to allocate a tage (929)
Jul 16 18:10:50 cs04r-sc-oss01-08 kernel: failure to allocate a tage (930)
Jul 16 18:10:51 cs04r-sc-oss01-08 last message repeated 2 times
Jul 16 18:10:52 cs04r-sc-oss01-08 kernel: failure to allocate a tage (934)
Jul 16 18:10:53 cs04r-sc-oss01-08 last message repeated 2 times
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0004: slow i_mutex 82s due to heavy IO load
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0004: slow direct_io 82s due to heavy IO load
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0004: slow journal start 61s due to heavy IO load
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: Skipped 27 previous similar messages
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0004: slow journal start 82s due to heavy IO load
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: Skipped 16 previous similar messages
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0004: slow brw_start 82s due to heavy IO load
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: Skipped 17 previous similar messages
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0004: slow parent lock 78s due to heavy IO load
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: Skipped 2 previous similar messages
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0004: slow journal start 82s due to heavy IO load
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0004: slow commitrw commit 82s due to heavy IO load
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0004: slow parent lock 78s due to heavy IO load
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0004: slow preprw_read setup 78s due to heavy IO load
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: Skipped 1 previous similar message
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0004: slow preprw_read setup 78s due to heavy IO load
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: Skipped 1 previous similar message
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: Skipped 10 previous similar messages
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0004: slow parent lock 75s due to heavy IO load
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: Skipped 11 previous similar messages
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0004: slow preprw_read setup 75s due to heavy IO load
Jul 16 18:11:38 cs04r-sc-oss01-08 kernel: Lustre: Skipped 9 previous similar messages
Jul 16 18:11:39 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0004: slow parent lock 48s due to heavy IO load
Jul 16 18:11:39 cs04r-sc-oss01-08 kernel: Lustre: Skipped 7 previous similar messages
Jul 16 18:11:39 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0004: slow parent lock 48s due to heavy IO load
Jul 16 18:11:39 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0004: slow preprw_read setup 48s due to heavy IO load
Jul 16 18:11:39 cs04r-sc-oss01-08 kernel: Lustre: Skipped 6 previous similar messages
Jul 16 18:11:39 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0004: slow preprw_read setup 46s due to heavy IO load
Jul 16 18:11:39 cs04r-sc-oss01-08 kernel: Lustre: Skipped 1 previous similar message
Jul 16 18:11:40 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0004: slow parent lock 34s due to heavy IO load
Jul 16 18:11:40 cs04r-sc-oss01-08 kernel: Lustre: Skipped 8 previous similar messages
Jul 16 18:12:18 cs04r-sc-oss01-08 kernel: Lustre: 21991:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1407653101673471 sent from play01-OST0004 to NID 172.23.138.5@tcp 7s ago has timed out (7s prior to deadline).
Jul 16 18:12:18 cs04r-sc-oss01-08 kernel:   req@ffff8101b298b400 x1407653101673471/t0 o105->@NET_0x20000ac178a05_UUID:15/16 lens 344/384 e 0 to 1 dl 1342458738 ref 2 fl Rpc:N/0/0 rc 0/0
Jul 16 18:12:18 cs04r-sc-oss01-08 kernel: Lustre: 21991:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jul 16 18:12:18 cs04r-sc-oss01-08 kernel: LustreError: 138-a: play01-OST0004: A client on nid 172.23.138.5@tcp was evicted due to a lock completion callback to 172.23.138.5@tcp timed out: rc -107
Jul 16 18:12:18 cs04r-sc-oss01-08 kernel: LustreError: 20591:0:(ldlm_lockd.c:234:expired_lock_main()) dump the log upon eviction
Jul 16 18:12:18 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342458738.20591
Jul 16 18:12:18 cs04r-sc-oss01-08 kernel: LustreError: 21290:0:(ost_handler.c:825:ost_brw_read()) @@@ Eviction on bulk PUT  req@ffff8101d8d56800 x1405109999625842/t0 o3->69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID:0/0 lens 448/400 e 0 to 0 dl 1342458945 ref 1 fl Interpret:/0/0 rc 0/0
Jul 16 18:12:18 cs04r-sc-oss01-08 kernel: LustreError: 21402:0:(ost_handler.c:825:ost_brw_read()) @@@ Eviction on bulk PUT  req@ffff81043ab52450 x1405109999625847/t0 o3->69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID:0/0 lens 448/400 e 0 to 0 dl 1342458945 ref 1 fl Interpret:/0/0 rc 0/0
Jul 16 18:12:18 cs04r-sc-oss01-08 kernel: LustreError: 21402:0:(ost_handler.c:825:ost_brw_read()) Skipped 2 previous similar messages
Jul 16 18:12:18 cs04r-sc-oss01-08 kernel: LustreError: 21290:0:(ost_handler.c:825:ost_brw_read()) Skipped 5 previous similar messages
Jul 16 18:12:18 cs04r-sc-oss01-08 kernel: failure to allocate a tage (1)
Jul 16 18:12:18 cs04r-sc-oss01-08 kernel: failure to allocate a tage (3)
Jul 16 18:12:19 cs04r-sc-oss01-08 kernel: failure to allocate a tage (6)
Jul 16 18:12:19 cs04r-sc-oss01-08 kernel: failure to allocate a tage (7)
Jul 16 18:12:20 cs04r-sc-oss01-08 kernel: failure to allocate a tage (11)
Jul 16 18:12:20 cs04r-sc-oss01-08 kernel: failure to allocate a tage (79)
Jul 16 18:12:20 cs04r-sc-oss01-08 kernel: failure to allocate a tage (14)
Jul 16 18:12:21 cs04r-sc-oss01-08 kernel: failure to allocate a tage (91)
Jul 16 18:12:21 cs04r-sc-oss01-08 kernel: failure to allocate a tage (109)
Jul 16 18:12:21 cs04r-sc-oss01-08 kernel: failure to allocate a tage (18)
Jul 16 18:12:21 cs04r-sc-oss01-08 kernel: failure to allocate a tage (115)
Jul 16 18:12:21 cs04r-sc-oss01-08 kernel: failure to allocate a tage (18)
Jul 16 18:12:22 cs04r-sc-oss01-08 kernel: failure to allocate a tage (121)
Jul 16 18:12:22 cs04r-sc-oss01-08 kernel: failure to allocate a tage (20)
Jul 16 18:12:23 cs04r-sc-oss01-08 last message repeated 3 times
Jul 16 18:12:23 cs04r-sc-oss01-08 kernel: failure to allocate a tage (150)
Jul 16 18:12:23 cs04r-sc-oss01-08 kernel: failure to allocate a tage (160)
Jul 16 18:12:23 cs04r-sc-oss01-08 kernel: failure to allocate a tage (169)
Jul 16 18:12:24 cs04r-sc-oss01-08 kernel: failure to allocate a tage (27)
Jul 16 18:12:24 cs04r-sc-oss01-08 kernel: failure to allocate a tage (176)
Jul 16 18:12:25 cs04r-sc-oss01-08 kernel: failure to allocate a tage (231)
Jul 16 18:12:25 cs04r-sc-oss01-08 kernel: failure to allocate a tage (42)
Jul 16 18:12:25 cs04r-sc-oss01-08 kernel: failure to allocate a tage (43)
Jul 16 18:12:25 cs04r-sc-oss01-08 kernel: failure to allocate a tage (257)
Jul 16 18:12:26 cs04r-sc-oss01-08 kernel: failure to allocate a tage (45)
Jul 16 18:12:26 cs04r-sc-oss01-08 kernel: failure to allocate a tage (48)
Jul 16 18:12:27 cs04r-sc-oss01-08 last message repeated 3 times
Jul 16 18:12:27 cs04r-sc-oss01-08 kernel: failure to allocate a tage (52)
Jul 16 18:12:28 cs04r-sc-oss01-08 kernel: failure to allocate a tage (55)
Jul 16 18:12:28 cs04r-sc-oss01-08 kernel: failure to allocate a tage (56)
Jul 16 18:12:28 cs04r-sc-oss01-08 kernel: failure to allocate a tage (59)
Jul 16 18:12:28 cs04r-sc-oss01-08 kernel: failure to allocate a tage (60)
Jul 16 18:12:29 cs04r-sc-oss01-08 kernel: failure to allocate a tage (62)
Jul 16 18:12:29 cs04r-sc-oss01-08 kernel: failure to allocate a tage (370)
Jul 16 18:12:29 cs04r-sc-oss01-08 kernel: failure to allocate a tage (65)
Jul 16 18:12:29 cs04r-sc-oss01-08 kernel: failure to allocate a tage (376)
Jul 16 18:12:29 cs04r-sc-oss01-08 kernel: failure to allocate a tage (69)
Jul 16 18:12:29 cs04r-sc-oss01-08 kernel: failure to allocate a tage (69)
Jul 16 18:12:29 cs04r-sc-oss01-08 kernel: failure to allocate a tage (396)
Jul 16 18:12:29 cs04r-sc-oss01-08 kernel: failure to allocate a tage (403)
Jul 16 18:12:29 cs04r-sc-oss01-08 kernel: failure to allocate a tage (409)
Jul 16 18:12:30 cs04r-sc-oss01-08 kernel: failure to allocate a tage (73)
Jul 16 18:12:30 cs04r-sc-oss01-08 kernel: failure to allocate a tage (419)
Jul 16 18:12:31 cs04r-sc-oss01-08 kernel: failure to allocate a tage (423)
Jul 16 18:12:31 cs04r-sc-oss01-08 kernel: failure to allocate a tage (426)
Jul 16 18:12:31 cs04r-sc-oss01-08 kernel: failure to allocate a tage (75)
Jul 16 18:12:32 cs04r-sc-oss01-08 kernel: failure to allocate a tage (84)
Jul 16 18:12:32 cs04r-sc-oss01-08 kernel: failure to allocate a tage (81)
Jul 16 18:12:33 cs04r-sc-oss01-08 kernel: failure to allocate a tage (94)
Jul 16 18:12:33 cs04r-sc-oss01-08 kernel: failure to allocate a tage (91)
Jul 16 18:12:33 cs04r-sc-oss01-08 kernel: failure to allocate a tage (92)
Jul 16 18:12:34 cs04r-sc-oss01-08 kernel: failure to allocate a tage (163)
Jul 16 18:12:34 cs04r-sc-oss01-08 kernel: failure to allocate a tage (104)
Jul 16 18:12:34 cs04r-sc-oss01-08 kernel: failure to allocate a tage (165)
Jul 16 18:12:34 cs04r-sc-oss01-08 kernel: failure to allocate a tage (105)
Jul 16 18:12:35 cs04r-sc-oss01-08 kernel: failure to allocate a tage (107)
Jul 16 18:12:35 cs04r-sc-oss01-08 kernel: failure to allocate a tage (109)
Jul 16 18:12:35 cs04r-sc-oss01-08 kernel: failure to allocate a tage (174)
Jul 16 18:12:35 cs04r-sc-oss01-08 kernel: failure to allocate a tage (175)
Jul 16 18:12:35 cs04r-sc-oss01-08 kernel: failure to allocate a tage (585)
Jul 16 18:12:36 cs04r-sc-oss01-08 kernel: failure to allocate a tage (179)
Jul 16 18:12:36 cs04r-sc-oss01-08 kernel: failure to allocate a tage (181)
Jul 16 18:12:36 cs04r-sc-oss01-08 kernel: failure to allocate a tage (608)
Jul 16 18:12:36 cs04r-sc-oss01-08 kernel: failure to allocate a tage (619)
Jul 16 18:12:36 cs04r-sc-oss01-08 kernel: failure to allocate a tage (623)
Jul 16 18:12:37 cs04r-sc-oss01-08 kernel: failure to allocate a tage (189)
Jul 16 18:12:37 cs04r-sc-oss01-08 kernel: failure to allocate a tage (191)
Jul 16 18:12:37 cs04r-sc-oss01-08 kernel: failure to allocate a tage (126)
Jul 16 18:12:38 cs04r-sc-oss01-08 kernel: LustreError: 20769:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107)  req@ffff8101fceb4c00 x1405109999626007/t0 o400-><?>@<?>:0/0 lens 192/0 e 0 to 0 dl 1342458823 ref 1 fl Interpret:H/0/0 rc -107/0
Jul 16 18:12:38 cs04r-sc-oss01-08 kernel: LustreError: 20769:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 15 previous similar messages
Jul 16 18:12:38 cs04r-sc-oss01-08 kernel: failure to allocate a tage (127)
Jul 16 18:12:38 cs04r-sc-oss01-08 kernel: failure to allocate a tage (131)
Jul 16 18:12:39 cs04r-sc-oss01-08 kernel: failure to allocate a tage (202)
Jul 16 18:12:39 cs04r-sc-oss01-08 kernel: Lustre: 21047:0:(ldlm_lib.c:574:target_handle_reconnect()) play01-OST0001: 69d87395-1fce-0c9e-2505-0acc7fd7164e reconnecting
Jul 16 18:12:39 cs04r-sc-oss01-08 kernel: Lustre: 20804:0:(ldlm_lib.c:874:target_handle_connect()) play01-OST0003: refuse reconnection from 69d87395-1fce-0c9e-2505-0acc7fd7164e@172.23.138.5@tcp to 0xffff810236181e00; still busy with 7 active RPCs
Jul 16 18:12:39 cs04r-sc-oss01-08 kernel: Lustre: 20804:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 1 previous similar message
Jul 16 18:12:39 cs04r-sc-oss01-08 kernel: failure to allocate a tage (131)
Jul 16 18:12:39 cs04r-sc-oss01-08 kernel: Lustre: 21047:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages
Jul 16 18:12:39 cs04r-sc-oss01-08 kernel: LustreError: 21349:0:(ost_handler.c:829:ost_brw_read()) @@@ Reconnect on bulk PUT  req@ffff81038eac1000 x1405109999625851/t0 o3->69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID:0/0 lens 448/400 e 0 to 0 dl 1342458945 ref 1 fl Interpret:/0/0 rc 0/0
Jul 16 18:12:40 cs04r-sc-oss01-08 kernel: failure to allocate a tage (754)
Jul 16 18:12:40 cs04r-sc-oss01-08 kernel: failure to allocate a tage (158)
Jul 16 18:12:40 cs04r-sc-oss01-08 kernel: failure to allocate a tage (763)
Jul 16 18:12:41 cs04r-sc-oss01-08 kernel: failure to allocate a tage (162)
Jul 16 18:12:41 cs04r-sc-oss01-08 kernel: failure to allocate a tage (167)
Jul 16 18:12:41 cs04r-sc-oss01-08 kernel: failure to allocate a tage (159)
Jul 16 18:12:41 cs04r-sc-oss01-08 kernel: failure to allocate a tage (810)
Jul 16 18:12:41 cs04r-sc-oss01-08 kernel: failure to allocate a tage (173)
Jul 16 18:12:42 cs04r-sc-oss01-08 kernel: failure to allocate a tage (836)
Jul 16 18:12:42 cs04r-sc-oss01-08 kernel: failure to allocate a tage (178)
Jul 16 18:12:43 cs04r-sc-oss01-08 kernel: failure to allocate a tage (181)
Jul 16 18:12:43 cs04r-sc-oss01-08 kernel: failure to allocate a tage (182)
Jul 16 18:12:43 cs04r-sc-oss01-08 kernel: failure to allocate a tage (183)
Jul 16 18:12:43 cs04r-sc-oss01-08 kernel: failure to allocate a tage (871)
Jul 16 18:12:44 cs04r-sc-oss01-08 kernel: failure to allocate a tage (187)
Jul 16 18:12:44 cs04r-sc-oss01-08 kernel: failure to allocate a tage (187)
Jul 16 18:12:44 cs04r-sc-oss01-08 kernel: failure to allocate a tage (188)
Jul 16 18:12:45 cs04r-sc-oss01-08 kernel: failure to allocate a tage (158)
Jul 16 18:12:46 cs04r-sc-oss01-08 kernel: failure to allocate a tage (191)
Jul 16 18:12:46 cs04r-sc-oss01-08 kernel: failure to allocate a tage (159)
Jul 16 18:12:47 cs04r-sc-oss01-08 kernel: failure to allocate a tage (199)
Jul 16 18:12:47 cs04r-sc-oss01-08 kernel: failure to allocate a tage (199)
Jul 16 18:12:47 cs04r-sc-oss01-08 kernel: failure to allocate a tage (167)
Jul 16 18:12:48 cs04r-sc-oss01-08 kernel: failure to allocate a tage (167)
Jul 16 18:12:48 cs04r-sc-oss01-08 kernel: failure to allocate a tage (174)
Jul 16 18:12:49 cs04r-sc-oss01-08 last message repeated 2 times
Jul 16 18:12:49 cs04r-sc-oss01-08 kernel: failure to allocate a tage (215)
Jul 16 18:12:49 cs04r-sc-oss01-08 kernel: failure to allocate a tage (216)
Jul 16 18:12:49 cs04r-sc-oss01-08 kernel: failure to allocate a tage (216)
Jul 16 18:12:50 cs04r-sc-oss01-08 kernel: failure to allocate a tage (217)
Jul 16 18:12:50 cs04r-sc-oss01-08 kernel: failure to allocate a tage (294)
Jul 16 18:12:50 cs04r-sc-oss01-08 kernel: failure to allocate a tage (223)
Jul 16 18:12:51 cs04r-sc-oss01-08 kernel: failure to allocate a tage (228)
Jul 16 18:12:52 cs04r-sc-oss01-08 kernel: failure to allocate a tage (229)
Jul 16 18:12:52 cs04r-sc-oss01-08 kernel: failure to allocate a tage (359)
Jul 16 18:12:52 cs04r-sc-oss01-08 kernel: failure to allocate a tage (362)
Jul 16 18:12:52 cs04r-sc-oss01-08 kernel: failure to allocate a tage (187)
Jul 16 18:12:52 cs04r-sc-oss01-08 kernel: failure to allocate a tage (365)
Jul 16 18:12:53 cs04r-sc-oss01-08 kernel: failure to allocate a tage (190)
Jul 16 18:12:53 cs04r-sc-oss01-08 kernel: failure to allocate a tage (378)
Jul 16 18:12:53 cs04r-sc-oss01-08 kernel: failure to allocate a tage (192)
Jul 16 18:12:54 cs04r-sc-oss01-08 kernel: failure to allocate a tage (193)
Jul 16 18:12:54 cs04r-sc-oss01-08 kernel: failure to allocate a tage (193)
Jul 16 18:12:54 cs04r-sc-oss01-08 kernel: failure to allocate a tage (197)
Jul 16 18:12:54 cs04r-sc-oss01-08 kernel: failure to allocate a tage (392)
Jul 16 18:12:55 cs04r-sc-oss01-08 kernel: failure to allocate a tage (392)
Jul 16 18:12:55 cs04r-sc-oss01-08 kernel: failure to allocate a tage (197)
Jul 16 18:12:55 cs04r-sc-oss01-08 last message repeated 3 times
Jul 16 18:12:55 cs04r-sc-oss01-08 kernel: failure to allocate a tage (395)
Jul 16 18:12:55 cs04r-sc-oss01-08 kernel: failure to allocate a tage (397)
Jul 16 18:12:56 cs04r-sc-oss01-08 kernel: failure to allocate a tage (275)
Jul 16 18:12:56 cs04r-sc-oss01-08 kernel: failure to allocate a tage (287)
Jul 16 18:12:56 cs04r-sc-oss01-08 kernel: failure to allocate a tage (223)
Jul 16 18:12:57 cs04r-sc-oss01-08 kernel: failure to allocate a tage (224)
Jul 16 18:12:57 cs04r-sc-oss01-08 kernel: failure to allocate a tage (301)
Jul 16 18:12:57 cs04r-sc-oss01-08 kernel: failure to allocate a tage (303)
Jul 16 18:12:57 cs04r-sc-oss01-08 kernel: failure to allocate a tage (304)
Jul 16 18:12:58 cs04r-sc-oss01-08 kernel: failure to allocate a tage (219)
Jul 16 18:12:58 cs04r-sc-oss01-08 kernel: failure to allocate a tage (311)
Jul 16 18:12:59 cs04r-sc-oss01-08 kernel: failure to allocate a tage (233)
Jul 16 18:12:59 cs04r-sc-oss01-08 kernel: failure to allocate a tage (321)
Jul 16 18:12:59 cs04r-sc-oss01-08 kernel: failure to allocate a tage (321)
Jul 16 18:12:59 cs04r-sc-oss01-08 kernel: failure to allocate a tage (234)
Jul 16 18:13:00 cs04r-sc-oss01-08 kernel: failure to allocate a tage (241)
Jul 16 18:13:00 cs04r-sc-oss01-08 kernel: failure to allocate a tage (328)
Jul 16 18:13:01 cs04r-sc-oss01-08 kernel: failure to allocate a tage (336)
Jul 16 18:13:03 cs04r-sc-oss01-08 kernel: failure to allocate a tage (252)
Jul 16 18:13:03 cs04r-sc-oss01-08 kernel: failure to allocate a tage (252)
Jul 16 18:13:03 cs04r-sc-oss01-08 kernel: failure to allocate a tage (340)
Jul 16 18:13:03 cs04r-sc-oss01-08 last message repeated 4 times
Jul 16 18:13:04 cs04r-sc-oss01-08 kernel: failure to allocate a tage (254)
Jul 16 18:13:04 cs04r-sc-oss01-08 kernel: failure to allocate a tage (341)
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: failure to allocate a tage (255)
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff810358a9c000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff81025c0fa000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff8103af012000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff8102166f8000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5794:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff81034942a000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5794:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff810044180000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: Lustre: 21514:0:(ost_handler.c:887:ost_brw_read()) play01-OST0003: ignoring bulk IO comm error with 69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID id 12345-172.23.138.5@tcp - client will retry
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: Lustre: 21514:0:(ost_handler.c:887:ost_brw_read()) Skipped 3 previous similar messages
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5794:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff810157f06000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff81000b7a8000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff8102df3f2000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff8103ab6fc000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff810400e68000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5794:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff810349428000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff81011cc92000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff8101eae86000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff8100b8a58000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff81025e548000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff8102b1ee0000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff81030dac0000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff810369642000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff8101c8938000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff8102b978a000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff8103f3180000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5794:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff810322c3c000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff8102dc992000
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff8102208c1580
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff81042b5a4580
Jul 16 18:13:05 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff810359a2acc0
Jul 16 18:13:06 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff8101cfcbc000
Jul 16 18:13:06 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff8101d5774000
Jul 16 18:13:06 cs04r-sc-oss01-08 kernel: LustreError: 5794:0:(events.c:381:server_bulk_callback()) event type 4, status -113, desc ffff81035c9f0dc0
Jul 16 18:13:06 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff810366226000
Jul 16 18:13:06 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff8103ab8e0000
Jul 16 18:13:06 cs04r-sc-oss01-08 kernel: LustreError: 5804:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff8102dc990000
Jul 16 18:13:06 cs04r-sc-oss01-08 kernel: LustreError: 5794:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff81021a1b6000
Jul 16 18:13:06 cs04r-sc-oss01-08 kernel: LustreError: 5794:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff8102d5dda000
Jul 16 18:13:06 cs04r-sc-oss01-08 kernel: LustreError: 5794:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff810322c10000
Jul 16 18:13:06 cs04r-sc-oss01-08 kernel: failure to allocate a tage (259)
Jul 16 18:13:06 cs04r-sc-oss01-08 kernel: failure to allocate a tage (351)
Jul 16 18:13:06 cs04r-sc-oss01-08 kernel: failure to allocate a tage (497)
Jul 16 18:13:06 cs04r-sc-oss01-08 kernel: failure to allocate a tage (262)
Jul 16 18:13:07 cs04r-sc-oss01-08 kernel: failure to allocate a tage (265)
Jul 16 18:13:07 cs04r-sc-oss01-08 kernel: failure to allocate a tage (504)
Jul 16 18:13:07 cs04r-sc-oss01-08 kernel: failure to allocate a tage (266)
Jul 16 18:13:07 cs04r-sc-oss01-08 kernel: failure to allocate a tage (506)
Jul 16 18:13:07 cs04r-sc-oss01-08 kernel: failure to allocate a tage (513)
Jul 16 18:13:08 cs04r-sc-oss01-08 kernel: failure to allocate a tage (276)
Jul 16 18:13:08 cs04r-sc-oss01-08 kernel: failure to allocate a tage (525)
Jul 16 18:13:08 cs04r-sc-oss01-08 kernel: failure to allocate a tage (763)
Jul 16 18:13:08 cs04r-sc-oss01-08 kernel: failure to allocate a tage (281)
Jul 16 18:13:09 cs04r-sc-oss01-08 kernel: failure to allocate a tage (780)
Jul 16 18:13:09 cs04r-sc-oss01-08 kernel: failure to allocate a tage (780)
Jul 16 18:13:10 cs04r-sc-oss01-08 kernel: failure to allocate a tage (291)
Jul 16 18:13:10 cs04r-sc-oss01-08 kernel: failure to allocate a tage (790)
Jul 16 18:13:10 cs04r-sc-oss01-08 kernel: failure to allocate a tage (291)
Jul 16 18:13:10 cs04r-sc-oss01-08 kernel: failure to allocate a tage (800)
Jul 16 18:13:10 cs04r-sc-oss01-08 kernel: failure to allocate a tage (801)
Jul 16 18:13:10 cs04r-sc-oss01-08 kernel: failure to allocate a tage (802)
Jul 16 18:13:11 cs04r-sc-oss01-08 kernel: failure to allocate a tage (805)
Jul 16 18:13:11 cs04r-sc-oss01-08 kernel: failure to allocate a tage (297)
Jul 16 18:13:11 cs04r-sc-oss01-08 kernel: failure to allocate a tage (810)
Jul 16 18:13:12 cs04r-sc-oss01-08 kernel: failure to allocate a tage (299)
Jul 16 18:13:12 cs04r-sc-oss01-08 kernel: failure to allocate a tage (815)
Jul 16 18:13:12 cs04r-sc-oss01-08 kernel: failure to allocate a tage (816)
Jul 16 18:13:12 cs04r-sc-oss01-08 last message repeated 2 times
Jul 16 18:13:13 cs04r-sc-oss01-08 kernel: failure to allocate a tage (823)
Jul 16 18:13:13 cs04r-sc-oss01-08 kernel: failure to allocate a tage (823)
Jul 16 18:13:13 cs04r-sc-oss01-08 kernel: failure to allocate a tage (826)
Jul 16 18:13:14 cs04r-sc-oss01-08 kernel: failure to allocate a tage (833)
Jul 16 18:13:14 cs04r-sc-oss01-08 kernel: failure to allocate a tage (834)
Jul 16 18:13:15 cs04r-sc-oss01-08 kernel: failure to allocate a tage (841)
Jul 16 18:13:16 cs04r-sc-oss01-08 kernel: failure to allocate a tage (841)
Jul 16 18:13:18 cs04r-sc-oss01-08 kernel: failure to allocate a tage (852)
Jul 16 18:13:18 cs04r-sc-oss01-08 kernel: failure to allocate a tage (854)
Jul 16 18:13:20 cs04r-sc-oss01-08 kernel: failure to allocate a tage (864)
Jul 16 18:13:20 cs04r-sc-oss01-08 kernel: failure to allocate a tage (867)
Jul 16 18:13:20 cs04r-sc-oss01-08 kernel: failure to allocate a tage (316)
Jul 16 18:13:21 cs04r-sc-oss01-08 kernel: failure to allocate a tage (879)
Jul 16 18:13:21 cs04r-sc-oss01-08 kernel: failure to allocate a tage (318)
Jul 16 18:13:21 cs04r-sc-oss01-08 kernel: failure to allocate a tage (886)
Jul 16 18:13:21 cs04r-sc-oss01-08 kernel: failure to allocate a tage (886)
Jul 16 18:13:22 cs04r-sc-oss01-08 kernel: failure to allocate a tage (321)
Jul 16 18:13:22 cs04r-sc-oss01-08 kernel: failure to allocate a tage (321)
Jul 16 18:13:22 cs04r-sc-oss01-08 kernel: failure to allocate a tage (894)
Jul 16 18:13:23 cs04r-sc-oss01-08 kernel: failure to allocate a tage (321)
Jul 16 18:13:23 cs04r-sc-oss01-08 kernel: failure to allocate a tage (896)
Jul 16 18:13:23 cs04r-sc-oss01-08 kernel: failure to allocate a tage (322)
Jul 16 18:13:23 cs04r-sc-oss01-08 kernel: failure to allocate a tage (899)
Jul 16 18:13:23 cs04r-sc-oss01-08 kernel: failure to allocate a tage (899)
Jul 16 18:13:24 cs04r-sc-oss01-08 kernel: failure to allocate a tage (902)
Jul 16 18:13:24 cs04r-sc-oss01-08 kernel: failure to allocate a tage (323)
Jul 16 18:13:24 cs04r-sc-oss01-08 kernel: failure to allocate a tage (903)
Jul 16 18:13:24 cs04r-sc-oss01-08 kernel: failure to allocate a tage (904)
Jul 16 18:13:24 cs04r-sc-oss01-08 kernel: failure to allocate a tage (324)
Jul 16 18:13:24 cs04r-sc-oss01-08 kernel: failure to allocate a tage (906)
Jul 16 18:13:25 cs04r-sc-oss01-08 kernel: failure to allocate a tage (329)
Jul 16 18:13:25 cs04r-sc-oss01-08 kernel: failure to allocate a tage (329)
Jul 16 18:13:26 cs04r-sc-oss01-08 kernel: failure to allocate a tage (340)
Jul 16 18:13:26 cs04r-sc-oss01-08 kernel: failure to allocate a tage (927)
Jul 16 18:13:26 cs04r-sc-oss01-08 kernel: failure to allocate a tage (341)
Jul 16 18:13:27 cs04r-sc-oss01-08 kernel: failure to allocate a tage (928)
Jul 16 18:13:27 cs04r-sc-oss01-08 kernel: failure to allocate a tage (342)
Jul 16 18:13:27 cs04r-sc-oss01-08 kernel: failure to allocate a tage (345)
Jul 16 18:13:27 cs04r-sc-oss01-08 kernel: failure to allocate a tage (345)
Jul 16 18:13:27 cs04r-sc-oss01-08 kernel: failure to allocate a tage (935)
Jul 16 18:13:28 cs04r-sc-oss01-08 kernel: failure to allocate a tage (345)
Jul 16 18:13:28 cs04r-sc-oss01-08 kernel: failure to allocate a tage (345)
Jul 16 18:13:29 cs04r-sc-oss01-08 kernel: failure to allocate a tage (946)
Jul 16 18:13:29 cs04r-sc-oss01-08 kernel: failure to allocate a tage (946)
Jul 16 18:13:29 cs04r-sc-oss01-08 kernel: failure to allocate a tage (353)
Jul 16 18:13:30 cs04r-sc-oss01-08 kernel: failure to allocate a tage (956)
Jul 16 18:13:30 cs04r-sc-oss01-08 kernel: failure to allocate a tage (355)
Jul 16 18:13:30 cs04r-sc-oss01-08 kernel: failure to allocate a tage (959)
Jul 16 18:13:30 cs04r-sc-oss01-08 kernel: failure to allocate a tage (357)
Jul 16 18:13:30 cs04r-sc-oss01-08 kernel: failure to allocate a tage (358)
Jul 16 18:13:30 cs04r-sc-oss01-08 kernel: failure to allocate a tage (961)
Jul 16 18:13:30 cs04r-sc-oss01-08 kernel: failure to allocate a tage (358)
Jul 16 18:13:31 cs04r-sc-oss01-08 kernel: failure to allocate a tage (968)
Jul 16 18:13:31 cs04r-sc-oss01-08 kernel: failure to allocate a tage (361)
Jul 16 18:13:31 cs04r-sc-oss01-08 kernel: failure to allocate a tage (361)
Jul 16 18:13:32 cs04r-sc-oss01-08 kernel: failure to allocate a tage (363)
Jul 16 18:13:33 cs04r-sc-oss01-08 kernel: failure to allocate a tage (363)
Jul 16 18:13:33 cs04r-sc-oss01-08 kernel: failure to allocate a tage (983)
Jul 16 18:13:34 cs04r-sc-oss01-08 kernel: failure to allocate a tage (988)
Jul 16 18:13:35 cs04r-sc-oss01-08 kernel: failure to allocate a tage (374)
Jul 16 18:13:35 cs04r-sc-oss01-08 kernel: failure to allocate a tage (1008)
Jul 16 18:13:36 cs04r-sc-oss01-08 kernel: failure to allocate a tage (1014)
Jul 16 18:13:36 cs04r-sc-oss01-08 kernel: failure to allocate a tage (397)
Jul 16 18:13:36 cs04r-sc-oss01-08 kernel: failure to allocate a tage (398)
Jul 16 18:13:36 cs04r-sc-oss01-08 kernel: failure to allocate a tage (398)
Jul 16 18:13:38 cs04r-sc-oss01-08 kernel: failure to allocate a tage (411)
Jul 16 18:13:38 cs04r-sc-oss01-08 kernel: failure to allocate a tage (411)
Jul 16 18:13:38 cs04r-sc-oss01-08 kernel: failure to allocate a tage (412)
Jul 16 18:13:38 cs04r-sc-oss01-08 kernel: failure to allocate a tage (418)
Jul 16 18:13:38 cs04r-sc-oss01-08 kernel: failure to allocate a tage (421)
Jul 16 18:13:39 cs04r-sc-oss01-08 kernel: failure to allocate a tage (429)
Jul 16 18:13:40 cs04r-sc-oss01-08 kernel: failure to allocate a tage (480)
Jul 16 18:13:40 cs04r-sc-oss01-08 kernel: failure to allocate a tage (480)
Jul 16 18:13:41 cs04r-sc-oss01-08 kernel: failure to allocate a tage (490)
Jul 16 18:13:41 cs04r-sc-oss01-08 kernel: failure to allocate a tage (490)
Jul 16 18:13:42 cs04r-sc-oss01-08 kernel: failure to allocate a tage (492)
Jul 16 18:13:42 cs04r-sc-oss01-08 kernel: failure to allocate a tage (496)
Jul 16 18:13:43 cs04r-sc-oss01-08 kernel: failure to allocate a tage (504)
Jul 16 18:13:43 cs04r-sc-oss01-08 kernel: failure to allocate a tage (510)
Jul 16 18:13:44 cs04r-sc-oss01-08 kernel: failure to allocate a tage (512)
Jul 16 18:13:48 cs04r-sc-oss01-08 kernel: failure to allocate a tage (567)
Jul 16 18:13:49 cs04r-sc-oss01-08 kernel: failure to allocate a tage (588)
Jul 16 18:13:49 cs04r-sc-oss01-08 kernel: failure to allocate a tage (597)
Jul 16 18:13:50 cs04r-sc-oss01-08 kernel: failure to allocate a tage (599)
Jul 16 18:13:50 cs04r-sc-oss01-08 kernel: failure to allocate a tage (617)
Jul 16 18:13:52 cs04r-sc-oss01-08 kernel: failure to allocate a tage (640)
Jul 16 18:13:52 cs04r-sc-oss01-08 kernel: failure to allocate a tage (646)
Jul 16 18:13:52 cs04r-sc-oss01-08 kernel: failure to allocate a tage (648)
Jul 16 18:13:52 cs04r-sc-oss01-08 kernel: failure to allocate a tage (659)
Jul 16 18:13:53 cs04r-sc-oss01-08 kernel: failure to allocate a tage (664)
Jul 16 18:13:53 cs04r-sc-oss01-08 kernel: failure to allocate a tage (673)
Jul 16 18:13:54 cs04r-sc-oss01-08 kernel: failure to allocate a tage (674)
Jul 16 18:13:54 cs04r-sc-oss01-08 kernel: failure to allocate a tage (679)
Jul 16 18:13:54 cs04r-sc-oss01-08 kernel: failure to allocate a tage (681)
Jul 16 18:13:54 cs04r-sc-oss01-08 kernel: failure to allocate a tage (687)
Jul 16 18:13:55 cs04r-sc-oss01-08 kernel: failure to allocate a tage (694)
Jul 16 18:13:55 cs04r-sc-oss01-08 kernel: failure to allocate a tage (706)
Jul 16 18:13:56 cs04r-sc-oss01-08 kernel: failure to allocate a tage (720)
Jul 16 18:13:56 cs04r-sc-oss01-08 kernel: failure to allocate a tage (720)
Jul 16 18:14:01 cs04r-sc-oss01-08 kernel: failure to allocate a tage (824)
Jul 16 18:14:02 cs04r-sc-oss01-08 kernel: failure to allocate a tage (833)
Jul 16 18:14:02 cs04r-sc-oss01-08 kernel: failure to allocate a tage (837)
Jul 16 18:14:04 cs04r-sc-oss01-08 kernel: failure to allocate a tage (852)
Jul 16 18:14:05 cs04r-sc-oss01-08 kernel: failure to allocate a tage (856)
Jul 16 18:14:05 cs04r-sc-oss01-08 kernel: failure to allocate a tage (862)
Jul 16 18:14:05 cs04r-sc-oss01-08 kernel: failure to allocate a tage (865)
Jul 16 18:14:07 cs04r-sc-oss01-08 kernel: failure to allocate a tage (893)
Jul 16 18:14:10 cs04r-sc-oss01-08 kernel: failure to allocate a tage (935)
Jul 16 18:14:10 cs04r-sc-oss01-08 kernel: failure to allocate a tage (940)
Jul 16 18:14:11 cs04r-sc-oss01-08 kernel: failure to allocate a tage (953)
Jul 16 18:14:11 cs04r-sc-oss01-08 kernel: failure to allocate a tage (957)
Jul 16 18:14:11 cs04r-sc-oss01-08 kernel: failure to allocate a tage (957)
Jul 16 18:14:13 cs04r-sc-oss01-08 kernel: failure to allocate a tage (985)
Jul 16 18:14:13 cs04r-sc-oss01-08 kernel: failure to allocate a tage (987)
Jul 16 18:14:13 cs04r-sc-oss01-08 kernel: failure to allocate a tage (992)
Jul 16 18:14:13 cs04r-sc-oss01-08 kernel: failure to allocate a tage (992)
Jul 16 18:14:15 cs04r-sc-oss01-08 kernel: failure to allocate a tage (1016)
Jul 16 18:14:15 cs04r-sc-oss01-08 kernel: failure to allocate a tage (1016)
Jul 16 18:14:15 cs04r-sc-oss01-08 kernel: failure to allocate a tage (1023)
Jul 16 18:15:06 cs04r-sc-oss01-08 init: Id "co" respawning too fast: disabled for 5 minutes
