224/368 e 0 to 1 dl 1414977077 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3330:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Lustre: meerkat-OST001e-osc: Connection to meerkat-OST001e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: meerkat-OST000e-osc: Connection to meerkat-OST000e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414977070/real 1414977072] req@ffff8805c83e4800 x1483597729769416/t0(0) o13->meerkat-OST0016-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1414977082 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Lustre: meerkat-OST0016-osc: Connection to meerkat-OST0016 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414977077/real 1414977079] req@ffff8806390a0800 x1483597729769712/t0(0) o8->meerkat-OST001e-osc@172.25.32.243@tcp:28/4 lens 400/544 e 0 to 1 dl 1414977085 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Lustre: meerkat-OST0036-osc: Connection to meerkat-OST0036 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414977083/real 1414977085] req@ffff880629ce2c00 x1483597729769952/t0(0) o8->meerkat-OST0024-osc@172.25.32.115@tcp:28/4 lens 400/544 e 0 to 1 dl 1414977091 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: Skipped 3 previous similar messages Lustre: meerkat-OST000e-osc: Connection restored to meerkat-OST000e (at 172.25.32.243@tcp) Lustre: Skipped 4 previous similar messages Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414977446/real 1414977453] req@ffff880638243400 x1483597729788564/t0(0) o13->meerkat-OST0004-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1414977459 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Lustre: meerkat-OST0004-osc: Connection to meerkat-OST0004 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 4 previous similar messages Lustre: MGS: Client 15b77b28-dbab-961b-571a-51927759ca6b (at 10.7.103.244@o2ib) reconnecting Lustre: Skipped 282 previous similar messages Lustre: MGS: Client 2d5a94d1-9feb-ce7e-4ca1-f5830ab6a290 (at 10.7.103.128@o2ib) reconnecting Lustre: Skipped 8 previous similar messages Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414979137/real 0] req@ffff88038b100000 x1483597731394596/t0(0) o6->meerkat-OST0036-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1414979153 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Lustre: meerkat-OST0036-osc: Connection to meerkat-OST0036 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: 3329:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414979143/real 0] req@ffff880315cfa400 x1483597731395564/t0(0) o13->meerkat-OST002c-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1414979159 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3329:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 106 previous similar messages Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414979147/real 0] req@ffff8800a363d400 x1483597731395760/t0(0) o13->meerkat-OST0006-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1414979163 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414979151/real 0] req@ffff88063904c400 x1483597731395988/t0(0) o6->meerkat-OST0024-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1414979171 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 23 previous similar messages Lustre: MGS: Client 44cb7b26-64f7-319f-ff91-1b0597f186f2 (at 10.7.103.213@o2ib) refused reconnection, still busy with 1 active RPCs LustreError: 3624:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88063773b850 x1474882440173736/t0(0) o256->44cb7b26-64f7-319f-ff91-1b0597f186f2@10.7.103.213@o2ib:0/0 lens 304/240 e 1 to 0 dl 1414979200 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: meerkat-MDT0000: Client 4f5b487e-e92c-04ac-fc2b-f09bc08ab044 (at 10.7.103.143@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 1 previous similar message Lustre: 3336:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414979864/real 1414979870] req@ffff8803130f8000 x1483597732138372/t0(0) o6->meerkat-OST002e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1414979871 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3336:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Lustre: meerkat-OST002e-osc: Connection to meerkat-OST002e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: meerkat-OST0024-osc: Connection restored to meerkat-OST0024 (at 172.25.32.115@tcp) Lustre: Skipped 15 previous similar messages LustreError: 11-0: meerkat-OST0004-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 7 previous similar messages Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414979879/real 0] req@ffff880629d9bc00 x1483597732156720/t0(0) o6->meerkat-OST0034-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1414979887 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: meerkat-OST003c-osc: Connection to meerkat-OST003c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: meerkat-OST003e-osc: Connection to meerkat-OST003e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414979883/real 0] req@ffff880314799400 x1483597732167976/t0(0) o13->meerkat-OST0024-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1414979891 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 80 previous similar messages Lustre: MGS: Client 72e453eb-eaf4-315d-0c8b-c95a1a9be588 (at 10.7.103.79@o2ib) reconnecting Lustre: Skipped 431 previous similar messages LustreError: 11-0: meerkat-OST000e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST002c-osc: Connection restored to meerkat-OST002c (at 172.25.32.115@tcp) Lustre: Skipped 7 previous similar messages Lustre: meerkat-OST000e-osc: Connection restored to meerkat-OST000e (at 172.25.32.243@tcp) Lustre: Skipped 2 previous similar messages LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff8802a9bd6240/0x661ae1127def002 lrc: 3/0,0 mode: PR/PR res: [0x200006069:0x1074b:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0b8a8e2fd expref: 1659 pid: 3569 timeout: 4406328334 lvb_type: 0 LustreError: 4738:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880312306400 x1474469698243440/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 1 to 0 dl 1414980460 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: meerkat-MDT0000: Client 4811bd80-e796-5dc3-9ae4-933c2a79e351 (at 10.7.103.80@o2ib) reconnecting Lustre: Skipped 17 previous similar messages Lustre: MGS: Client 5f1cdb0b-64bc-906e-274a-3981f53905eb (at 10.7.101.193@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: Skipped 1 previous similar message LustreError: 3624:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880638ea6850 x1483341007841536/t0(0) o256->5f1cdb0b-64bc-906e-274a-3981f53905eb@10.7.101.193@o2ib:0/0 lens 304/240 e 0 to 0 dl 1414980484 ref 1 fl Interpret:/0/0 rc 0/0 LustreError: 3628:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880638ea5850 x1474574891686232/t0(0) o256->3bad5d85-52d1-acb5-a1f8-105e3c7782f2@10.7.101.240@o2ib:0/0 lens 304/240 e 1 to 0 dl 1414980510 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) reconnecting Lustre: Skipped 60 previous similar messages Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: Skipped 1 previous similar message LustreError: 6021:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880315288000 x1482141738105416/t0(0) o37->8c5b20e1-4e92-4313-0ab5-a06d9154e563@198.202.118.30@tcp:0/0 lens 448/440 e 1 to 0 dl 1414982626 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414984294/real 0] req@ffff8803cab47c00 x1483597734745992/t0(0) o6->meerkat-OST000c-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1414984301 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 45 previous similar messages Lustre: meerkat-OST0026-osc: Connection to meerkat-OST0026 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 5 previous similar messages LustreError: 11-0: meerkat-OST000c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 7 previous similar messages Lustre: meerkat-OST0026-osc: Connection restored to meerkat-OST0026 (at 172.25.32.243@tcp) Lustre: Skipped 12 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414985193/real 0] req@ffff8802f3934000 x1483597735542452/t0(0) o6->meerkat-OST0034-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1414985200 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 10 previous similar messages Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414985193/real 0] req@ffff8802f2780800 x1483597735542444/t0(0) o6->meerkat-OST0034-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1414985200 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Lustre: meerkat-OST003c-osc: Connection to meerkat-OST003c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 8 previous similar messages LustreError: 3611:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST003e-osc: can't precreate: rc = -11 LustreError: 3611:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST003e-osc: cannot precreate objects: rc = -11 Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 10 previous similar messages LustreError: 11-0: meerkat-OST000e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 10 previous similar messages Lustre: meerkat-OST003c-osc: Connection restored to meerkat-OST003c (at 172.25.32.115@tcp) Lustre: Skipped 9 previous similar messages Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414985804/real 0] req@ffff8806382bbc00 x1483597735910460/t0(0) o6->meerkat-OST0036-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1414985815 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 67 previous similar messages Lustre: meerkat-OST0036-osc: Connection to meerkat-OST0036 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages Lustre: 3555:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414985804/real 0] req@ffff8800b83a6400 x1483597735912892/t0(0) o5->meerkat-OST0026-osc@172.25.32.243@tcp:28/4 lens 432/432 e 0 to 1 dl 1414985815 ref 3 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3555:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Lustre: meerkat-OST0026-osc: Connection to meerkat-OST0026 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages LustreError: 3555:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0026-osc: can't precreate: rc = -11 LustreError: 3555:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0026-osc: cannot precreate objects: rc = -11 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414985805/real 0] req@ffff8800ba489800 x1483597735913244/t0(0) o6->meerkat-OST002e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1414985816 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 75 previous similar messages Lustre: MGS: Client 2d5a94d1-9feb-ce7e-4ca1-f5830ab6a290 (at 10.7.103.128@o2ib) reconnecting Lustre: Skipped 1 previous similar message Lustre: meerkat-OST0016-osc: Connection to meerkat-OST0016 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 9 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414985806/real 0] req@ffff880312d33000 x1483597735913408/t0(0) o13->meerkat-OST002c-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1414985817 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 40 previous similar messages LustreError: 3588:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0034-osc: can't precreate: rc = -11 LustreError: 3588:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0034-osc: cannot precreate objects: rc = -11 Lustre: 3533:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414985809/real 0] req@ffff8803c9f2e800 x1483597735913964/t0(0) o5->meerkat-OST001c-osc@172.25.32.115@tcp:28/4 lens 432/432 e 0 to 1 dl 1414985820 ref 3 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3533:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages LustreError: 3533:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST001c-osc: can't precreate: rc = -11 LustreError: 3533:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST001c-osc: cannot precreate objects: rc = -11 LustreError: 3501:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000e-osc: can't precreate: rc = -11 LustreError: 3501:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000e-osc: cannot precreate objects: rc = -11 LustreError: 11-0: meerkat-OST0004-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 5 previous similar messages Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: Skipped 5 previous similar messages LustreError: 11-0: meerkat-OST002e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 6 previous similar messages Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 1 previous similar message Lustre: meerkat-MDT0000: Client 15e1bcf6-a618-80ca-a17a-70ccf8d686ab (at 10.7.101.234@o2ib) reconnecting Lustre: Skipped 43 previous similar messages Lustre: meerkat-MDT0000: Client 7d83acb1-250a-ceb0-89e4-e727e12079f2 (at 10.7.104.22@o2ib) reconnecting Lustre: Skipped 1 previous similar message Lustre: MGS: Client 080277ed-80fc-2273-db3f-3599bfd8b31f (at 10.7.103.80@o2ib) reconnecting Lustre: Skipped 32 previous similar messages Lustre: MGS: Client 1f588014-fe0e-e6d4-470d-2fe984ada26b (at 10.7.103.78@o2ib) reconnecting Lustre: Skipped 38 previous similar messages Lustre: meerkat-MDT0000: Client f8a37406-5da9-98a2-8c68-a139bd1b8714 (at 10.7.103.238@o2ib) reconnecting Lustre: Skipped 61 previous similar messages Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414987636/real 0] req@ffff880315645c00 x1483597736724396/t0(0) o6->meerkat-OST001c-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1414987658 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Lustre: meerkat-OST001c-osc: Connection to meerkat-OST001c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414987636/real 0] req@ffff8803d339cc00 x1483597736726540/t0(0) o13->meerkat-OST001c-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1414987658 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414987636/real 0] req@ffff8806251f6800 x1483597736724764/t0(0) o6->meerkat-OST000c-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1414987659 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 71 previous similar messages LustreError: 3568:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST002c-osc: can't precreate: rc = -11 LustreError: 3568:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3568:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST002c-osc: cannot precreate objects: rc = -11 LustreError: 3568:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414987638/real 0] req@ffff880315bb0800 x1483597736726940/t0(0) o6->meerkat-OST0006-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1414987661 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 35 previous similar messages LustreError: 3588:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0034-osc: can't precreate: rc = -11 LustreError: 3588:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0034-osc: cannot precreate objects: rc = -11 LustreError: 3575:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST002e-osc: can't precreate: rc = -11 LustreError: 3575:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST002e-osc: cannot precreate objects: rc = -11 Lustre: meerkat-OST0026-osc: Connection restored to meerkat-OST0026 (at 172.25.32.243@tcp) Lustre: Skipped 13 previous similar messages LustreError: 11-0: meerkat-OST002e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 6 previous similar messages Lustre: meerkat-OST001c-osc: Connection restored to meerkat-OST001c (at 172.25.32.115@tcp) Lustre: Skipped 5 previous similar messages Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414987858/real 1414987868] req@ffff8802f2780800 x1483597736800076/t0(0) o6->meerkat-OST0026-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1414987878 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Lustre: meerkat-OST0026-osc: Connection to meerkat-OST0026 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 14 previous similar messages LustreError: 11-0: meerkat-OST0004-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 9 previous similar messages Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 9 previous similar messages Lustre: meerkat-OST0026-osc: Connection restored to meerkat-OST0026 (at 172.25.32.243@tcp) Lustre: Skipped 1 previous similar message Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414989064/real 1414989067] req@ffff88056cf66000 x1483597739345172/t0(0) o13->meerkat-OST001a-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1414989074 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Lustre: meerkat-OST001a-osc: Connection to meerkat-OST001a (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: meerkat-OST0032-osc: Connection to meerkat-OST0032 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: MGS: Client 391f98f8-f601-2680-6640-6c7e17b6740c (at 10.7.104.38@o2ib) reconnecting Lustre: Skipped 33 previous similar messages Lustre: meerkat-OST001a-osc: Connection restored to meerkat-OST001a (at 172.25.32.248@tcp) Lustre: Skipped 1 previous similar message Lustre: MGS: Client 0382582d-97ec-f61a-f5e2-db3a61005deb (at 10.7.100.170@o2ib) reconnecting Lustre: Skipped 31 previous similar messages Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414989217/real 1414989224] req@ffff880636dba400 x1483597739373084/t0(0) o13->meerkat-OST0016-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1414989226 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Lustre: meerkat-OST0016-osc: Connection to meerkat-OST0016 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: meerkat-OST002c-osc: Connection to meerkat-OST002c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 8 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414989217/real 1414989224] req@ffff88062536d800 x1483597739373112/t0(0) o13->meerkat-OST003e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1414989229 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Lustre: meerkat-MDT0000: Client 20cdf7bd-ea3b-4906-5664-2cf3acd299ae (at 10.7.103.154@o2ib) reconnecting Lustre: Skipped 3 previous similar messages Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414989226/real 1414989231] req@ffff88062fb06000 x1483597739374696/t0(0) o8->meerkat-OST000c-osc@172.25.32.115@tcp:28/4 lens 400/544 e 0 to 1 dl 1414989234 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: meerkat-OST002c-osc: Connection restored to meerkat-OST002c (at 172.25.32.115@tcp) Lustre: Skipped 3 previous similar messages Lustre: meerkat-OST0026-osc: Connection restored to meerkat-OST0026 (at 172.25.32.243@tcp) Lustre: Skipped 1 previous similar message Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414989321/real 1414989324] req@ffff88062e6ef400 x1483597739392288/t0(0) o13->meerkat-OST0014-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1414989337 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Lustre: meerkat-OST0014-osc: Connection to meerkat-OST0014 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: meerkat-OST003c-osc: Connection to meerkat-OST003c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 9 previous similar messages LustreError: 3533:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST001c-osc: can't precreate: rc = -11 LustreError: 3533:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3533:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST001c-osc: cannot precreate objects: rc = -11 LustreError: 3533:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message Lustre: meerkat-OST0024-osc: Connection restored to meerkat-OST0024 (at 172.25.32.115@tcp) Lustre: Skipped 8 previous similar messages Lustre: meerkat-MDT0000: Client 0bf744b3-2fe8-8236-cd27-d04e85730316 (at 10.7.104.45@o2ib) reconnecting Lustre: Skipped 98 previous similar messages Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414989645/real 1414989650] req@ffff8804e7c81800 x1483597739444184/t0(0) o13->meerkat-OST0034-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1414989659 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages LustreError: 3588:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0034-osc: can't precreate: rc = -11 LustreError: 3588:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0034-osc: cannot precreate objects: rc = -11 LustreError: 3611:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST003e-osc: can't precreate: rc = -11 LustreError: 3611:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST003e-osc: cannot precreate objects: rc = -11 Lustre: meerkat-OST0016-osc: Connection to meerkat-OST0016 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 10 previous similar messages LustreError: 3550:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0024-osc: can't precreate: rc = -11 LustreError: 3550:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0024-osc: cannot precreate objects: rc = -11 Lustre: meerkat-MDT0000: Client 28c445ea-ef83-90b0-a2a1-dfac7ab60bee (at 10.7.104.44@o2ib) reconnecting Lustre: Skipped 64 previous similar messages Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414989676/real 1414989676] req@ffff880624aff400 x1483597739448480/t0(0) o8->meerkat-OST0016-osc@172.25.32.243@tcp:28/4 lens 400/544 e 0 to 1 dl 1414989693 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 28 previous similar messages Lustre: meerkat-OST001e-osc: Connection restored to meerkat-OST001e (at 172.25.32.243@tcp) Lustre: Skipped 12 previous similar messages Lustre: meerkat-OST002c-osc: Connection restored to meerkat-OST002c (at 172.25.32.115@tcp) Lustre: Skipped 6 previous similar messages Lustre: MGS: Client 93fc4711-d07f-2642-a190-f194f51dd182 (at 10.7.103.249@o2ib) reconnecting Lustre: Skipped 452 previous similar messages Lustre: 3331:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414990290/real 1414990293] req@ffff8805914e2400 x1483597739550988/t0(0) o13->meerkat-OST002e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1414990301 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3331:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Lustre: meerkat-OST002e-osc: Connection to meerkat-OST002e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages LustreError: 3588:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0034-osc: can't precreate: rc = -11 LustreError: 3588:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0034-osc: cannot precreate objects: rc = -11 LustreError: 3485:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0006-osc: can't precreate: rc = -11 LustreError: 3485:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0006-osc: cannot precreate objects: rc = -11 LustreError: 3611:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST003e-osc: can't precreate: rc = -11 LustreError: 3611:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST003e-osc: cannot precreate objects: rc = -11 LustreError: 3501:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000e-osc: can't precreate: rc = -11 LustreError: 3501:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000e-osc: cannot precreate objects: rc = -11 Lustre: meerkat-OST000e-osc: Connection restored to meerkat-OST000e (at 172.25.32.243@tcp) Lustre: Skipped 8 previous similar messages Lustre: meerkat-OST0034-osc: Connection restored to meerkat-OST0034 (at 172.25.32.115@tcp) Lustre: Skipped 8 previous similar messages Lustre: meerkat-OST003c-osc: Connection to meerkat-OST003c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 13 previous similar messages LustreError: 3575:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST002e-osc: can't precreate: rc = -11 LustreError: 3575:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST002e-osc: cannot precreate objects: rc = -11 LustreError: 3593:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0036-osc: can't precreate: rc = -11 LustreError: 3593:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 3 previous similar messages LustreError: 3593:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0036-osc: cannot precreate objects: rc = -11 LustreError: 3593:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 3 previous similar messages Lustre: meerkat-OST001c-osc: Connection restored to meerkat-OST001c (at 172.25.32.115@tcp) Lustre: Skipped 4 previous similar messages Lustre: meerkat-MDT0000: Client 19e16514-654d-6b70-935c-6ef2eee1bee4 (at 198.202.119.89@tcp) refused reconnection, still busy with 2 active RPCs Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414990555/real 1414990558] req@ffff880336521800 x1483597739755568/t0(0) o13->meerkat-OST0032-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1414990567 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 39 previous similar messages Lustre: meerkat-OST0032-osc: Connection to meerkat-OST0032 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 9 previous similar messages Lustre: meerkat-OST0032-osc: Connection restored to meerkat-OST0032 (at 172.25.32.248@tcp) Lustre: Skipped 9 previous similar messages Lustre: meerkat-MDT0000: Client 6cff5345-b4e4-0159-f619-69af8e9fcf37 (at 10.7.101.160@o2ib) reconnecting Lustre: Skipped 312 previous similar messages Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414990909/real 1414990916] req@ffff880313879c00 x1483597739809796/t0(0) o13->meerkat-OST0036-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1414990923 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Lustre: meerkat-OST0036-osc: Connection to meerkat-OST0036 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: meerkat-OST0036-osc: Connection restored to meerkat-OST0036 (at 172.25.32.243@tcp) Lustre: Skipped 4 previous similar messages LustreError: 3485:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0006-osc: can't precreate: rc = -11 LustreError: 3485:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0006-osc: cannot precreate objects: rc = -11 LNet: Service thread pid 3626 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 3626, comm: ll_mgs_0012 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] cfs_waitq_timedwait+0x11/0x20 [libcfs] [] target_bulk_io+0x3b8/0x910 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_prep_bulk_exp+0x6f/0x180 [ptlrpc] [] mgs_get_ir_logs+0x93d/0x130a [mgs] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [] ? lustre_swab_mgs_config_body+0x0/0x30 [ptlrpc] [] mgs_handle+0x4ea/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1414991248.3626 Lustre: meerkat-OST0026-osc: Connection restored to meerkat-OST0026 (at 172.25.32.243@tcp) Lustre: Skipped 19 previous similar messages LNet: Service thread pid 12424 was inactive for 206.79s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 12424, comm: ll_mgs_0017 Call Trace: [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] cfs_waitq_timedwait+0x11/0x20 [libcfs] [] target_bulk_io+0x3b8/0x910 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_prep_bulk_exp+0x6f/0x180 [ptlrpc] [] mgs_get_ir_logs+0x93d/0x130a [mgs] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [] ? lustre_swab_mgs_config_body+0x0/0x30 [ptlrpc] [] mgs_handle+0x4ea/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 Pid: 3413, comm: ll_mgs_0002 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] cfs_waitq_timedwait+0x11/0x20 [libcfs] [] target_bulk_io+0x3b8/0x910 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_prep_bulk_exp+0x6f/0x180 [ptlrpc] [] mgs_get_ir_logs+0x93d/0x130a [mgs] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [] ? lustre_swab_mgs_config_body+0x0/0x30 [ptlrpc] [] mgs_handle+0x4ea/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LNet: Service thread pid 3412 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: LNet: Skipped 1 previous similar message Pid: 3412, comm: ll_mgs_0001 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] cfs_waitq_timedwait+0x11/0x20 [libcfs] [] target_bulk_io+0x3b8/0x910 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_prep_bulk_exp+0x6f/0x180 [ptlrpc] [] mgs_get_ir_logs+0x93d/0x130a [mgs] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [] ? lustre_swab_mgs_config_body+0x0/0x30 [ptlrpc] [] mgs_handle+0x4ea/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1414991314.3412 LNet: Service thread pid 3622 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 3622, comm: ll_mgs_0008 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] cfs_waitq_timedwait+0x11/0x20 [libcfs] [] target_bulk_io+0x3b8/0x910 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_prep_bulk_exp+0x6f/0x180 [ptlrpc] [] mgs_get_ir_logs+0x93d/0x130a [mgs] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [] ? lustre_swab_mgs_config_body+0x0/0x30 [ptlrpc] [] mgs_handle+0x4ea/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1414991315.3622 LNet: Service thread pid 3411 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. LustreError: dumping log to /tmp/lustre-log.1414991319.3411 Lustre: meerkat-MDT0000: Client ebaac7aa-5413-d6c4-9489-f7f0f90e1f86 (at 10.7.103.131@o2ib) reconnecting Lustre: Skipped 483 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414991702/real 0] req@ffff8803362c8800 x1483597740641648/t0(0) o6->meerkat-OST003e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1414991718 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 54 previous similar messages Lustre: meerkat-OST003e-osc: Connection to meerkat-OST003e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 21 previous similar messages LustreError: 11-0: meerkat-OST003e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: 3618:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-207), not sending early reply req@ffff880638cd7850 x1480334774049384/t0(0) o256->8ce13baa-9997-7254-75e7-e417d924785f@10.7.103.239@o2ib:0/0 lens 304/240 e 5 to 0 dl 1414991860 ref 2 fl Interpret:/0/0 rc 0/0 LustreError: 3626:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 812+0s req@ffff880638cd7850 x1480334774049384/t0(0) o256->8ce13baa-9997-7254-75e7-e417d924785f@10.7.103.239@o2ib:0/0 lens 304/240 e 5 to 0 dl 1414991860 ref 1 fl Interpret:/0/0 rc 0/0 LNet: Service thread pid 3626 completed after 812.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: 12287:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-383), not sending early reply req@ffff880639225050 x1474671332411344/t0(0) o256->22031999-5b0f-f275-5c67-f64d47d021f6@10.7.103.102@o2ib:0/0 lens 304/240 e 4 to 0 dl 1414992042 ref 2 fl Interpret:/0/0 rc 0/0 LustreError: 3413:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 988+0s req@ffff880639225050 x1474671332411344/t0(0) o256->22031999-5b0f-f275-5c67-f64d47d021f6@10.7.103.102@o2ib:0/0 lens 304/240 e 4 to 0 dl 1414992042 ref 1 fl Interpret:/0/0 rc 0/0 LNet: Service thread pid 3413 completed after 988.03s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: 3413:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-383), not sending early reply req@ffff88063af36050 x1476461425712988/t0(0) o256->a572a62a-14e0-abc3-88a1-3b6334d3c5bc@10.7.103.81@o2ib:0/0 lens 304/240 e 4 to 0 dl 1414992053 ref 2 fl Interpret:/0/0 rc 0/0 LustreError: 12424:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 988+0s req@ffff88063af36050 x1476461425712988/t0(0) o256->a572a62a-14e0-abc3-88a1-3b6334d3c5bc@10.7.103.81@o2ib:0/0 lens 304/240 e 4 to 0 dl 1414992053 ref 1 fl Interpret:/0/0 rc 0/0 LNet: Service thread pid 12424 completed after 988.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: 12424:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-367), not sending early reply req@ffff880637f21850 x1474466191540832/t0(0) o256->7b773074-55fc-138b-3ab5-daf011387e37@10.7.103.150@o2ib:0/0 lens 304/240 e 3 to 0 dl 1414992086 ref 2 fl Interpret:/0/0 rc 0/0 Lustre: 12424:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-367), not sending early reply req@ffff880638cd6850 x1474559777858496/t0(0) o256->2bf0062f-3663-81f0-a48f-29fa118f627d@10.7.103.134@o2ib:0/0 lens 304/240 e 3 to 0 dl 1414992091 ref 2 fl Interpret:/0/0 rc 0/0 Lustre: 12424:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message LustreError: 3412:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 972+0s req@ffff880637f21850 x1474466191540832/t0(0) o256->7b773074-55fc-138b-3ab5-daf011387e37@10.7.103.150@o2ib:0/0 lens 304/240 e 3 to 0 dl 1414992086 ref 1 fl Interpret:/0/0 rc 0/0 LNet: Service thread pid 3412 completed after 972.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). LustreError: 3411:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 972+0s req@ffff880638cd6850 x1474559777858496/t0(0) o256->2bf0062f-3663-81f0-a48f-29fa118f627d@10.7.103.134@o2ib:0/0 lens 304/240 e 3 to 0 dl 1414992091 ref 1 fl Interpret:/0/0 rc 0/0 LustreError: 3411:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 1 previous similar message LNet: Service thread pid 3411 completed after 972.02s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). LNet: Skipped 1 previous similar message Lustre: meerkat-OST0034-osc: Connection restored to meerkat-OST0034 (at 172.25.32.115@tcp) Lustre: Skipped 14 previous similar messages LustreError: 11-0: meerkat-OST002e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 10 previous similar messages LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff8800bd35ad80/0x661ae1137c2f697 lrc: 3/0,0 mode: PR/PR res: [0x2000060a6:0x163de:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0b954b73c expref: 86 pid: 3420 timeout: 4418510477 lvb_type: 0 LustreError: 3420:0:(client.c:1048:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802f5534000 x1483597741138332/t0(0) o104->meerkat-MDT0000@192.168.230.53@tcp:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 LustreError: 3420:0:(ldlm_lockd.c:709:ldlm_handle_ast_error()) ### client (nid 192.168.230.53@tcp) returned 0 from blocking AST ns: mdt-meerkat-MDT0000_UUID lock: ffff8802b364e900/0x661ae1137c2e6fa lrc: 1/0,0 mode: --/PR res: [0x20000644a:0x42d5:0x0].0 bits 0x1b rrc: 2 type: IBT flags: 0xa01000000020 nid: 192.168.230.53@tcp remote: 0x7185dbd0b954b583 expref: 7 pid: 3420 timeout: 4418611000 lvb_type: 0 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414992924/real 0] req@ffff8800374de000 x1483597741382940/t0(0) o13->meerkat-OST0006-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1414992934 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 107 previous similar messages Lustre: meerkat-OST0006-osc: Connection to meerkat-OST0006 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 14 previous similar messages Lustre: meerkat-MDT0000: Client 61711fd7-2b48-ac39-4e56-995110471cab (at 10.7.104.18@o2ib) reconnecting Lustre: Skipped 28 previous similar messages Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 1 previous similar message Lustre: MGS: Client bad4e58a-9525-8c27-df0a-c827873d29d0 (at 10.7.101.195@o2ib) reconnecting Lustre: Skipped 404 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414994080/real 1414994083] req@ffff880625386400 x1483597742173132/t0(0) o6->meerkat-OST001c-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1414994087 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 126 previous similar messages Lustre: meerkat-OST001c-osc: Connection to meerkat-OST001c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 3515:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0014-osc: can't precreate: rc = -11 LustreError: 3515:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0014-osc: cannot precreate objects: rc = -11 Lustre: MGS: Client 412a921c-391a-fe16-2456-f6e77ec67b31 (at 10.7.100.122@o2ib) reconnecting Lustre: Skipped 10 previous similar messages LustreError: 11-0: meerkat-OST002e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST0036-osc: Connection restored to meerkat-OST0036 (at 172.25.32.243@tcp) Lustre: Skipped 15 previous similar messages Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414994097/real 1414994103] req@ffff8805be5ad400 x1483597742177964/t0(0) o6->meerkat-OST001e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1414994105 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 19 previous similar messages Lustre: meerkat-OST001e-osc: Connection to meerkat-OST001e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 10 previous similar messages Lustre: MGS: Client 2aecc5eb-df69-a310-9603-9da2ea3eb52f (at 10.7.103.212@o2ib) reconnecting Lustre: Skipped 10 previous similar messages LustreError: 11-0: meerkat-OST0034-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 5 previous similar messages Lustre: meerkat-MDT0000: Client b0d0ff72-8939-7bfe-da27-3264386fe86d (at 10.7.102.206@o2ib) reconnecting Lustre: Skipped 44 previous similar messages Lustre: MGS: Client 0df4cf50-8571-13d9-cbd6-21f603cf6545 (at 10.7.103.77@o2ib) reconnecting Lustre: Skipped 27 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414994736/real 0] req@ffff880336521800 x1483597742631760/t0(0) o6->meerkat-OST0006-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1414994750 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 38 previous similar messages Lustre: meerkat-OST0006-osc: Connection to meerkat-OST0006 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 9 previous similar messages Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414994736/real 0] req@ffff88063909c400 x1483597742631676/t0(0) o6->meerkat-OST0036-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1414994752 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 68 previous similar messages LustreError: 3501:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000e-osc: can't precreate: rc = -11 LustreError: 3501:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 5 previous similar messages LustreError: 3501:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000e-osc: cannot precreate objects: rc = -11 LustreError: 3501:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 5 previous similar messages Lustre: meerkat-OST002c-osc: Connection to meerkat-OST002c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 13 previous similar messages Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414994740/real 0] req@ffff8804ea1eec00 x1483597742635604/t0(0) o6->meerkat-OST0016-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1414994756 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 47 previous similar messages Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: Skipped 20 previous similar messages LustreError: 11-0: meerkat-OST0036-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414994759/real 1414994761] req@ffff8802e8dcd400 x1483597742642444/t0(0) o6->meerkat-OST0026-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1414994775 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Lustre: meerkat-OST0026-osc: Connection to meerkat-OST0026 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414995691/real 1414995696] req@ffff8805a4de2400 x1483597742955700/t0(0) o13->meerkat-OST001c-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1414995698 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Lustre: meerkat-OST001c-osc: Connection to meerkat-OST001c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414995694/real 1414995696] req@ffff880625c54800 x1483597742956188/t0(0) o2->meerkat-OST001c-osc@172.25.32.115@tcp:28/4 lens 560/432 e 0 to 1 dl 1414995701 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Lustre: meerkat-OST003c-osc: Connection to meerkat-OST003c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages LustreError: 3606:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST003c-osc: can't precreate: rc = -11 LustreError: 3606:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST003c-osc: cannot precreate objects: rc = -11 LustreError: 3515:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0014-osc: can't precreate: rc = -11 LustreError: 3515:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0014-osc: cannot precreate objects: rc = -11 Lustre: MGS: Client 9f8529f1-db32-f4b2-7e87-2121151f4139 (at 10.7.103.191@o2ib) reconnecting Lustre: Skipped 275 previous similar messages Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414995698/real 1414995699] req@ffff880625b13800 x1483597742957412/t0(0) o13->meerkat-OST0034-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1414995706 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 3588:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0034-osc: can't precreate: rc = -11 LustreError: 3588:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0034-osc: cannot precreate objects: rc = -11 LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 5217:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1414995711 with bad export cookie 459840025943960551 Lustre: meerkat-OST001e-osc: Connection to meerkat-OST001e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 5 previous similar messages LustreError: 3575:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST002e-osc: can't precreate: rc = -11 LustreError: 3575:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST002e-osc: cannot precreate objects: rc = -11 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414995703/real 1414995706] req@ffff8803392d0800 x1483597742958956/t0(0) o13->meerkat-OST000e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1414995716 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Lustre: MGS: Client 55a280fe-4b11-dd79-255b-6d0a54d5ac56 (at 10.7.103.215@o2ib) reconnecting Lustre: Skipped 42 previous similar messages Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414995741/real 1414995750] req@ffff880629c32c00 x1483597742962012/t0(0) o8->meerkat-OST0026-osc@172.25.32.243@tcp:28/4 lens 400/544 e 0 to 1 dl 1414995752 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Lustre: MGS: Client 5c3ca310-c7e1-97b3-5bb8-cf2d9a5ce212 (at 10.7.104.16@o2ib) reconnecting Lustre: Skipped 75 previous similar messages Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 20 previous similar messages Lustre: MGS: Client 2aecc5eb-df69-a310-9603-9da2ea3eb52f (at 10.7.103.212@o2ib) refused reconnection, still busy with 1 active RPCs LustreError: 3627:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88063a228050 x1474577475327948/t0(0) o256->2aecc5eb-df69-a310-9603-9da2ea3eb52f@10.7.103.212@o2ib:0/0 lens 304/240 e 0 to 0 dl 1414995812 ref 1 fl Interpret:/0/0 rc 0/0 LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff8802f57a5900/0x661ae113b631b95 lrc: 3/0,0 mode: PR/PR res: [0x200006094:0x1fb37:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0b9711cdf expref: 39 pid: 6376 timeout: 4421864744 lvb_type: 0 LustreError: 12099:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880337f2a800 x1474469723931864/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 0 to 0 dl 1414995942 ref 1 fl Interpret:/0/0 rc 0/0 LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff88033a3656c0/0x661ae113c34dd60 lrc: 3/0,0 mode: PR/PR res: [0x200006094:0x1f02e:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0b976ed15 expref: 43 pid: 13052 timeout: 4422180115 lvb_type: 0 LustreError: 4729:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff8803174b0800 x1474469724540764/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 0 to 0 dl 1414996290 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: meerkat-MDT0000: Client f8ee92d4-3bc1-7fca-234a-43af75dc6463 (at 10.7.101.153@o2ib) reconnecting Lustre: Skipped 436 previous similar messages Lustre: MGS: Client 9d59b3ed-f151-8d66-f659-a2a74e8a369f (at 10.7.100.211@o2ib) reconnecting Lustre: MGS: Client 193ea91f-77fc-1c52-d0e6-a0e6c43f482b (at 10.7.100.209@o2ib) reconnecting Lustre: Skipped 2 previous similar messages Lustre: Skipped 2 previous similar messages Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414997572/real 0] req@ffff88009bcbb800 x1483597744796956/t0(0) o6->meerkat-OST0006-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1414997579 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Lustre: meerkat-OST0006-osc: Connection to meerkat-OST0006 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: MGS: Client 90a4b178-38c3-539b-9e40-1729d8abcc72 (at 10.7.103.139@o2ib) reconnecting Lustre: Skipped 6 previous similar messages LustreError: 3533:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST001c-osc: can't precreate: rc = -11 LustreError: 3533:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST001c-osc: cannot precreate objects: rc = -11 Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414997576/real 0] req@ffff88039fd16000 x1483597744798104/t0(0) o13->meerkat-OST0024-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1414997583 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 131 previous similar messages Lustre: 6214:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414997593/real 1414997593] req@ffff88006ff08000 x1483597744801436/t0(0) o104->meerkat-MDT0000@10.7.103.252@o2ib:15/16 lens 296/224 e 0 to 1 dl 1414997604 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 6214:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Lustre: MGS: Client 10b2a9a1-b12e-4a32-a028-1f4777827d35 (at 198.202.118.232@tcp) reconnecting Lustre: Skipped 46 previous similar messages Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 15 previous similar messages LustreError: 11-0: meerkat-OST001c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 8 previous similar messages Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1414997923/real 0] req@ffff8802f4473000 x1483597745519976/t0(0) o6->meerkat-OST0024-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1414997940 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Lustre: meerkat-OST0024-osc: Connection to meerkat-OST0024 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: meerkat-MDT0000: Client edf8ac03-be63-dd50-44f0-104ac7e6df03 (at 10.7.103.89@o2ib) reconnecting Lustre: Skipped 376 previous similar messages Lustre: meerkat-OST000e-osc: Connection to meerkat-OST000e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 9 previous similar messages Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 15 previous similar messages LustreError: 11-0: meerkat-OST0014-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST0024-osc: Connection to meerkat-OST0024 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 5 previous similar messages LustreError: 11-0: meerkat-OST003e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 3 previous similar messages LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff8802b0c2eb40/0x661ae113e865eeb lrc: 3/0,0 mode: PR/PR res: [0x200006094:0x1f569:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0b9a17605 expref: 40 pid: 3709 timeout: 4424027563 lvb_type: 0 LustreError: 13043:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff88033808ec00 x1474469728847112/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 0 to 0 dl 1414998136 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: meerkat-MDT0000: Client 9eb442d7-85ec-bee1-dd6a-0d773d10e840 (at 10.7.100.181@o2ib) reconnecting Lustre: Skipped 3 previous similar messages Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414998139/real 1414998141] req@ffff880315a28c00 x1483597745685468/t0(0) o13->meerkat-OST0014-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1414998159 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 103 previous similar messages Lustre: meerkat-OST0014-osc: Connection to meerkat-OST0014 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages LustreError: 3555:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0026-osc: can't precreate: rc = -11 LustreError: 3555:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0026-osc: cannot precreate objects: rc = -11 Lustre: meerkat-OST0014-osc: Connection restored to meerkat-OST0014 (at 172.25.32.115@tcp) Lustre: Skipped 20 previous similar messages Lustre: meerkat-OST0022-osc: Connection to meerkat-OST0022 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: meerkat-MDT0000: Client f6611583-13e7-a203-946b-5912e20d2c00 (at 192.168.230.60@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client f6611583-13e7-a203-946b-5912e20d2c00 (at 192.168.230.60@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client f6611583-13e7-a203-946b-5912e20d2c00 (at 192.168.230.60@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client f6611583-13e7-a203-946b-5912e20d2c00 (at 192.168.230.60@tcp) refused reconnection, still busy with 1 active RPCs LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 121s: evicting client at 192.168.230.60@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff8802af683d80/0x661ae113ea13453 lrc: 3/0,0 mode: PR/PR res: [0x2000073ab:0x363c:0x0].0 bits 0x1b rrc: 4 type: IBT flags: 0x200000000020 nid: 192.168.230.60@tcp remote: 0xb705c6aff02d46f7 expref: 10879 pid: 3727 timeout: 4424251498 lvb_type: 0 Lustre: meerkat-MDT0000: Client 8efa4edd-6ddb-bf05-ecbd-1a6fdbb81c2b (at 10.7.102.94@o2ib) reconnecting Lustre: Skipped 146 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414998535/real 1414998541] req@ffff880638031400 x1483597745890464/t0(0) o13->meerkat-OST0024-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1414998548 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Lustre: meerkat-OST0024-osc: Connection to meerkat-OST0024 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: meerkat-OST0024-osc: Connection restored to meerkat-OST0024 (at 172.25.32.115@tcp) Lustre: Skipped 9 previous similar messages Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414998700/real 1414998709] req@ffff880624e6b000 x1483597746004720/t0(0) o13->meerkat-OST003c-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1414998720 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Lustre: meerkat-OST003c-osc: Connection to meerkat-OST003c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete LustreError: 3606:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST003c-osc: can't precreate: rc = -11 LustreError: 3606:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST003c-osc: cannot precreate objects: rc = -11 LustreError: 3519:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0016-osc: can't precreate: rc = -11 LustreError: 3519:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0016-osc: cannot precreate objects: rc = -11 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414999056/real 1414999060] req@ffff880419012000 x1483597746455028/t0(0) o13->meerkat-OST0014-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1414999078 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Lustre: meerkat-OST0014-osc: Connection to meerkat-OST0014 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages LustreError: 3515:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0014-osc: can't precreate: rc = -11 LustreError: 3515:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0014-osc: cannot precreate objects: rc = -11 Lustre: MGS: Client 354a4c27-89e2-ef99-78d4-0cfac0ff4d44 (at 10.7.103.75@o2ib) reconnecting Lustre: Skipped 25 previous similar messages LustreError: 3606:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST003c-osc: can't precreate: rc = -11 LustreError: 3606:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3606:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST003c-osc: cannot precreate objects: rc = -11 LustreError: 3606:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages LustreError: 3533:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST001c-osc: can't precreate: rc = -11 LustreError: 3533:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3533:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST001c-osc: cannot precreate objects: rc = -11 LustreError: 3533:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages LustreError: 3485:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0006-osc: can't precreate: rc = -11 LustreError: 3485:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3485:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0006-osc: cannot precreate objects: rc = -11 LustreError: 3485:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message Lustre: meerkat-OST0036-osc: Connection restored to meerkat-OST0036 (at 172.25.32.243@tcp) Lustre: Skipped 19 previous similar messages LustreError: 3593:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0036-osc: can't precreate: rc = -11 LustreError: 3593:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0036-osc: cannot precreate objects: rc = -11 Lustre: meerkat-OST0004-osc: Connection to meerkat-OST0004 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 17 previous similar messages LustreError: 11-0: meerkat-OST003a-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: Skipped 3 previous similar messages Lustre: MGS: Client 03283624-9881-7b2e-fb7d-d18bdf93c403 (at 10.7.103.178@o2ib) reconnecting Lustre: Skipped 118 previous similar messages Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1414999675/real 1414999679] req@ffff8803365cb000 x1483597747203228/t0(0) o6->meerkat-OST0006-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1414999700 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 53 previous similar messages Lustre: meerkat-MDT0000: Client 06bc4379-10ca-76ad-cd98-1d1013f1b911 (at 10.7.103.252@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-OST003a-osc: Connection restored to meerkat-OST003a (at 172.25.32.248@tcp) Lustre: Skipped 18 previous similar messages LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.61@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff88008c42e480/0x661ae11414055f0 lrc: 3/0,0 mode: PR/PR res: [0x2000073bb:0x185a4:0x0].0 bits 0x2 rrc: 5 type: IBT flags: 0x20 nid: 192.168.230.61@tcp remote: 0x8b9f2a2454a4a9fd expref: 3868 pid: 6402 timeout: 4426105522 lvb_type: 0 LustreError: 3630:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.61@tcp arrived at 1415000146 with bad export cookie 459840023463752373 LustreError: 13033:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880317f90000 x1479434060193844/t0(0) o37->0f47a0da-6735-24a0-7531-2fa98d66460c@192.168.230.61@tcp:0/0 lens 448/440 e 1 to 0 dl 1415000263 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: MGS: Client 0d770415-5152-eacb-b4dd-cb9ea56b09b5 (at 10.7.103.236@o2ib) reconnecting Lustre: Skipped 34 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415000288/real 1415000294] req@ffff88062b4bd800 x1483597747319132/t0(0) o13->meerkat-OST0024-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415000307 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Lustre: meerkat-OST0024-osc: Connection to meerkat-OST0024 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 16 previous similar messages LustreError: 3537:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST001e-osc: can't precreate: rc = -11 LustreError: 3537:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 6 previous similar messages LustreError: 3537:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST001e-osc: cannot precreate objects: rc = -11 LustreError: 3537:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 5 previous similar messages LustreError: 11-0: meerkat-OST0014-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message LustreError: 3606:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST003c-osc: can't precreate: rc = -11 LustreError: 3606:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST003c-osc: cannot precreate objects: rc = -11 LustreError: 3485:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0006-osc: can't precreate: rc = -11 LustreError: 3485:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3485:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0006-osc: cannot precreate objects: rc = -11 LustreError: 3485:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages Lustre: MGS: Client 27328d65-fd9a-d7d6-2c62-a94cb9ce1f44 (at 10.7.103.189@o2ib) reconnecting Lustre: Skipped 437 previous similar messages Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 12 previous similar messages LustreError: 11-0: meerkat-OST0014-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. Lustre: 6387:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415001046/real 1415001047] req@ffff880338247800 x1483597747421144/t0(0) o104->meerkat-MDT0000@198.202.118.117@tcp:15/16 lens 296/224 e 0 to 1 dl 1415001053 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 6387:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 32 previous similar messages Lustre: meerkat-OST003e-osc: Connection to meerkat-OST003e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 23 previous similar messages LustreError: 3533:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST001c-osc: can't precreate: rc = -11 LustreError: 3533:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3533:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST001c-osc: cannot precreate objects: rc = -11 LustreError: 3533:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3397:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415001278 with bad export cookie 459840026055688930 Lustre: meerkat-MDT0000: haven't heard from client 43259bb0-87f4-dcf5-00f7-9bbbc0d8c720 (at 10.7.102.121@o2ib) in 224 seconds. I think it's dead, and I am evicting it. exp ffff88063036a400, cur 1415001425 expire 1415001275 last 1415001201 Lustre: Skipped 3 previous similar messages LNet: Service thread pid 13233 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 13233, comm: ll_mgs_0027 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] cfs_waitq_timedwait+0x11/0x20 [libcfs] [] target_bulk_io+0x3b8/0x910 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_prep_bulk_exp+0x6f/0x180 [ptlrpc] [] mgs_get_ir_logs+0x93d/0x130a [mgs] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [] ? lustre_swab_mgs_config_body+0x0/0x30 [ptlrpc] [] mgs_handle+0x4ea/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415001482.13233 LNet: Service thread pid 12423 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 12423, comm: ll_mgs_0016 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] cfs_waitq_timedwait+0x11/0x20 [libcfs] [] target_bulk_io+0x3b8/0x910 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_prep_bulk_exp+0x6f/0x180 [ptlrpc] [] mgs_get_ir_logs+0x93d/0x130a [mgs] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [] ? lustre_swab_mgs_config_body+0x0/0x30 [ptlrpc] [] mgs_handle+0x4ea/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415001487.12423 Lustre: 3622:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-415), not sending early reply req@ffff88063a0a9850 x1482703973616892/t0(0) o256->1a0a147a-1994-05c3-bc1d-0d3e0ce80c77@10.7.103.252@o2ib:0/0 lens 304/240 e 4 to 0 dl 1415002302 ref 2 fl Interpret:/0/0 rc 0/0 LustreError: 13233:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 1020+0s req@ffff88063a0a9850 x1482703973616892/t0(0) o256->1a0a147a-1994-05c3-bc1d-0d3e0ce80c77@10.7.103.252@o2ib:0/0 lens 304/240 e 4 to 0 dl 1415002302 ref 1 fl Interpret:/0/0 rc 0/0 LNet: Service thread pid 13233 completed after 1020.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: 3618:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-415), not sending early reply req@ffff8806393a3850 x1462438242396144/t0(0) o256->597b5265-696c-da9b-283a-aab88384a18c@10.7.103.225@o2ib:0/0 lens 304/240 e 4 to 0 dl 1415002307 ref 2 fl Interpret:/0/0 rc 0/0 LustreError: 12423:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 1020+0s req@ffff8806393a3850 x1462438242396144/t0(0) o256->597b5265-696c-da9b-283a-aab88384a18c@10.7.103.225@o2ib:0/0 lens 304/240 e 4 to 0 dl 1415002307 ref 1 fl Interpret:/0/0 rc 0/0 LNet: Service thread pid 12423 completed after 1020.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: meerkat-MDT0000: Client ca5f52b2-308e-1ca0-dfd2-f55788202d8f (at 10.7.103.236@o2ib) reconnecting Lustre: Skipped 456 previous similar messages Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415002348/real 0] req@ffff880315617800 x1483597748159112/t0(0) o6->meerkat-OST0034-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415002360 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 177 previous similar messages Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST000e-osc: Connection restored to meerkat-OST000e (at 172.25.32.243@tcp) Lustre: Skipped 28 previous similar messages LustreError: 11-0: meerkat-OST002e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: 11-0: meerkat-OST0026-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 2 previous similar messages Lustre: meerkat-MDT0000: Client a3606482-21c2-2b66-b787-49ee29182f1a (at 10.7.101.15@o2ib) reconnecting Lustre: Skipped 40 previous similar messages Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415003395/real 0] req@ffff8806253d4800 x1483597748816100/t0(0) o6->meerkat-OST002e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415003405 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 169 previous similar messages Lustre: meerkat-OST002e-osc: Connection to meerkat-OST002e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 27 previous similar messages Lustre: MGS: Client d8e277d8-2f7a-da0c-7e59-51b861aaaed1 (at 10.7.103.184@o2ib) reconnecting Lustre: Skipped 5 previous similar messages LustreError: 11-0: meerkat-OST0014-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 5 previous similar messages Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: Skipped 27 previous similar messages Lustre: meerkat-OST0014-osc: Connection restored to meerkat-OST0014 (at 172.25.32.115@tcp) Lustre: Skipped 10 previous similar messages LustreError: 13039:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff8800a2cbab40/0x661ae114456e851 lrc: 3/0,0 mode: PR/PR res: [0x2000060ae:0x108c0:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0b9bca21b expref: 77 pid: 13051 timeout: 4429622900 lvb_type: 0 LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff8800abc6a6c0/0x661ae11447e79ac lrc: 3/0,0 mode: PR/PR res: [0x2000060ae:0x10920:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0b9c155c0 expref: 40 pid: 3420 timeout: 4429843514 lvb_type: 0 LustreError: 13028:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff8803380e4050 x1474469733539480/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 0 to 0 dl 1415003951 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: 3329:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415004051/real 1415004054] req@ffff8803376ba400 x1483597749244748/t0(0) o13->meerkat-OST0034-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415004061 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3329:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 77 previous similar messages Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: 3497:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415004053/real 1415004056] req@ffff8802b7783c00 x1483597749245024/t0(0) o5->meerkat-OST000c-osc@172.25.32.115@tcp:28/4 lens 432/432 e 0 to 1 dl 1415004063 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3497:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages LustreError: 3497:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000c-osc: can't precreate: rc = -11 LustreError: 3497:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 3 previous similar messages LustreError: 3497:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000c-osc: cannot precreate objects: rc = -11 LustreError: 3497:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 3 previous similar messages Lustre: MGS: Client 0cd71c52-b83f-be4a-eafe-b3e96712bb73 (at 10.7.103.186@o2ib) reconnecting Lustre: Skipped 5 previous similar messages Lustre: MGS: Client 320e1102-9e44-b162-b591-d2f795f9f12d (at 10.7.103.248@o2ib) reconnecting Lustre: Skipped 6 previous similar messages Lustre: 3480:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415004057/real 1415004058] req@ffff880639312c00 x1483597749245880/t0(0) o5->meerkat-OST0004-osc@172.25.32.115@tcp:28/4 lens 432/432 e 0 to 1 dl 1415004067 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: MGS: Client 40da1b6c-4701-a77d-e27d-e43b2fcdd151 (at 198.202.118.151@tcp) reconnecting Lustre: Skipped 8 previous similar messages Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415004061/real 1415004062] req@ffff880545287000 x1483597749246672/t0(0) o13->meerkat-OST000e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415004072 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages LustreError: 3555:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0026-osc: can't precreate: rc = -11 LustreError: 3555:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3555:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0026-osc: cannot precreate objects: rc = -11 LustreError: 3555:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415004072/real 1415004073] req@ffff880639049c00 x1483597749248928/t0(0) o8->meerkat-OST000e-osc@172.25.32.243@tcp:28/4 lens 400/544 e 0 to 1 dl 1415004082 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Lustre: meerkat-MDT0000: Client 8f727abb-a921-bac4-031e-55fb3aa14af2 (at 10.7.103.175@o2ib) reconnecting Lustre: Skipped 5 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 4270:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415004089 with bad export cookie 459840026156530286 Lustre: meerkat-MDT0000: Client 677d7d3a-37ea-920c-c096-20a623186fa9 (at 10.7.103.114@o2ib) reconnecting Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 4 previous similar messages Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415004120/real 1415004126] req@ffff8800a87ed000 x1483597749261384/t0(0) o13->meerkat-OST0034-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415004135 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Lustre: meerkat-MDT0000: Client e64a6a7f-020a-29d6-8da8-1faa63a4e6d6 (at 10.7.103.108@o2ib) reconnecting Lustre: Skipped 232 previous similar messages LustreError: 3501:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000e-osc: can't precreate: rc = -11 LustreError: 3501:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000e-osc: cannot precreate objects: rc = -11 LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 4270:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415004154 with bad export cookie 459840026158472569 Lustre: meerkat-OST0036-osc: Connection restored to meerkat-OST0036 (at 172.25.32.243@tcp) Lustre: Skipped 15 previous similar messages Lustre: meerkat-MDT0000: Client 5e6cdb3e-b53d-8452-b1f5-c464b4dcf978 (at 10.7.103.207@o2ib) reconnecting Lustre: Skipped 91 previous similar messages Lustre: 3330:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415004510/real 0] req@ffff880625f43000 x1483597749422252/t0(0) o6->meerkat-OST000c-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415004535 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3330:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 153 previous similar messages LustreError: 11-0: meerkat-OST000c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 4 previous similar messages Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST0024-osc: Connection restored to meerkat-OST0024 (at 172.25.32.115@tcp) Lustre: Skipped 5 previous similar messages Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 8 previous similar messages Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415005390/real 1415005393] req@ffff88041aa3bc00 x1483597749854084/t0(0) o13->meerkat-OST0014-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415005397 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 57 previous similar messages Lustre: meerkat-OST0014-osc: Connection to meerkat-OST0014 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 47 previous similar messages Lustre: meerkat-MDT0000: Client 4f5b487e-e92c-04ac-fc2b-f09bc08ab044 (at 10.7.103.143@o2ib) reconnecting Lustre: Skipped 4 previous similar messages LustreError: 3537:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST001e-osc: can't precreate: rc = -11 LustreError: 3537:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 7 previous similar messages LustreError: 3537:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST001e-osc: cannot precreate objects: rc = -11 LustreError: 3537:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 7 previous similar messages Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415005401/real 0] req@ffff88000f6f9400 x1483597749859816/t0(0) o8->meerkat-OST0004-osc@172.25.32.115@tcp:28/4 lens 400/544 e 0 to 1 dl 1415005407 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 138 previous similar messages Lustre: meerkat-MDT0000: Client 5fad899e-38c1-f247-901f-973e6428a901 (at 10.7.104.52@o2ib) reconnecting Lustre: Skipped 157 previous similar messages Lustre: meerkat-OST0016-osc: Connection restored to meerkat-OST0016 (at 172.25.32.243@tcp) Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415005450/real 0] req@ffff8803cada9800 x1483597750017340/t0(0) o6->meerkat-OST002c-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415005462 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages LustreError: 3501:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000e-osc: can't precreate: rc = -11 LustreError: 3501:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3501:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000e-osc: cannot precreate objects: rc = -11 LustreError: 3501:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message Lustre: MGS: Client cd9f0a7e-f9c0-42c0-a77b-a71fafb889d3 (at 10.7.102.204@o2ib) reconnecting Lustre: Skipped 74 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 4270:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415005468 with bad export cookie 459840026158852690 Lustre: meerkat-OST002c-osc: Connection restored to meerkat-OST002c (at 172.25.32.115@tcp) Lustre: Skipped 15 previous similar messages LustreError: 11-0: meerkat-OST003e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 8 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 Lustre: meerkat-OST0016-osc: Connection to meerkat-OST0016 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 28 previous similar messages LustreError: 4270:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415005486 with bad export cookie 459840026181307269 LustreError: 11-0: meerkat-OST0006-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 5 previous similar messages Lustre: meerkat-OST003e-osc: Connection restored to meerkat-OST003e (at 172.25.32.243@tcp) Lustre: Skipped 6 previous similar messages Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415005491/real 1415005494] req@ffff88000e173c00 x1483597750054056/t0(0) o13->meerkat-OST000c-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415005503 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 8 previous similar messages LustreError: 11-0: meerkat-OST0036-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415006199/real 0] req@ffff88009bed9c00 x1483597750744020/t0(0) o6->meerkat-OST003c-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415006206 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Lustre: meerkat-OST003c-osc: Connection to meerkat-OST003c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 8 previous similar messages Lustre: meerkat-MDT0000: Client 1082d204-79bb-d35f-e03e-3109ff7c5018 (at 10.7.103.244@o2ib) reconnecting Lustre: Skipped 56 previous similar messages Lustre: meerkat-OST0036-osc: Connection restored to meerkat-OST0036 (at 172.25.32.243@tcp) Lustre: Skipped 5 previous similar messages LustreError: 11-0: meerkat-OST002e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 3 previous similar messages Lustre: meerkat-OST001c-osc: Connection restored to meerkat-OST001c (at 172.25.32.115@tcp) Lustre: Skipped 11 previous similar messages Lustre: MGS: Client 3d2133a1-fa1f-239b-cf03-fcb5b93deddd (at 10.7.103.190@o2ib) reconnecting Lustre: Skipped 53 previous similar messages Lustre: MGS: Client 3501d8a8-b0e7-01a0-05d3-a57748b1a27f (at 10.7.103.235@o2ib) reconnecting Lustre: Skipped 2 previous similar messages Lustre: MGS: Client d3d9a620-fdb5-e520-9ae5-8942d8f1977d (at 10.7.101.183@o2ib) reconnecting Lustre: Skipped 40 previous similar messages LustreError: 4:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff88001157bb40/0x661ae1148c0924c lrc: 3/0,0 mode: PR/PR res: [0x2000060a1:0x17d84:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0b9e17b9e expref: 42 pid: 13054 timeout: 4433155479 lvb_type: 0 LustreError: 13034:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880336634000 x1474469738273016/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 0 to 0 dl 1415007314 ref 1 fl Interpret:/0/0 rc 0/0 LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff8802debf3d80/0x661ae1148d338b5 lrc: 3/0,0 mode: PR/PR res: [0x2000060a1:0x17d4d:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0b9e333cc expref: 40 pid: 3692 timeout: 4433264501 lvb_type: 0 Lustre: MGS: Client 227bab05-f1fd-f848-7cbc-be212fe31af9 (at 10.7.103.220@o2ib) reconnecting Lustre: Skipped 2 previous similar messages Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415007415/real 1415007420] req@ffff8803ce7a9c00 x1483597753036000/t0(0) o13->meerkat-OST002e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415007431 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 123 previous similar messages Lustre: meerkat-OST002e-osc: Connection to meerkat-OST002e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415007417/real 0] req@ffff8803c9a0bc00 x1483597753038364/t0(0) o6->meerkat-OST0034-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415007433 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415007424/real 0] req@ffff88041a9ea400 x1483597753039368/t0(0) o13->meerkat-OST000c-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415007440 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 108 previous similar messages Lustre: 3331:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415007429/real 0] req@ffff880637f74c00 x1483597753039996/t0(0) o13->meerkat-OST0006-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415007445 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3331:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Lustre: meerkat-MDT0000: Client 06bc4379-10ca-76ad-cd98-1d1013f1b911 (at 10.7.103.252@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: 6375:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415007445/real 1415007449] req@ffff88031435e400 x1483597753043232/t0(0) o104->meerkat-MDT0000@192.168.230.53@tcp:15/16 lens 296/224 e 0 to 1 dl 1415007458 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 6375:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 19 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3433:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8800a77fcc00 x1482703973899972/t0(0) o37->06bc4379-10ca-76ad-cd98-1d1013f1b911@10.7.103.252@o2ib:0/0 lens 448/440 e 1 to 0 dl 1415007488 ref 1 fl Interpret:/0/0 rc 0/0 LustreError: 4270:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415007465 with bad export cookie 459840026229036251 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415007458/real 0] req@ffff8802ece92800 x1483597753044752/t0(0) o8->meerkat-OST003e-osc@172.25.32.243@tcp:28/4 lens 400/544 e 0 to 1 dl 1415007478 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 LustreError: 11-0: meerkat-OST0004-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -114. LustreError: Skipped 3 previous similar messages Lustre: meerkat-OST0024-osc: Connection restored to meerkat-OST0024 (at 172.25.32.115@tcp) Lustre: Skipped 3 previous similar messages Lustre: meerkat-MDT0000: Client 06bc4379-10ca-76ad-cd98-1d1013f1b911 (at 10.7.103.252@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 1 previous similar message Lustre: meerkat-MDT0000: Client b0de5975-d7e7-33f7-66b4-296c80ca90e9 (at 10.7.104.21@o2ib) reconnecting Lustre: Skipped 562 previous similar messages LustreError: 3306:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88009aa0a000 Lustre: MGS: Client ec37672d-effa-6f29-06d5-0f52e5e4ef0e (at 10.7.101.34@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: MGS: Client 0d2d22b3-b24d-8f70-4232-92dc4e356009 (at 10.7.103.122@o2ib) refused reconnection, still busy with 1 active RPCs LustreError: 3625:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8800a6f32850 x1482713733628232/t0(0) o256->0d2d22b3-b24d-8f70-4232-92dc4e356009@10.7.103.122@o2ib:0/0 lens 304/240 e 1 to 0 dl 1415007736 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: MGS: haven't heard from client f21b4723-138b-9992-d974-2c7438e5e0da (at 10.7.102.77@o2ib) in 166 seconds. I think it's dead, and I am evicting it. exp ffff880315146800, cur 1415007710 expire 1415007560 last 1415007544 Lustre: Skipped 1 previous similar message Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415007742/real 1415007743] req@ffff880312555c00 x1483597753095456/t0(0) o13->meerkat-OST0022-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415007750 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Lustre: meerkat-OST0022-osc: Connection to meerkat-OST0022 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST0002-osc: Connection restored to meerkat-OST0002 (at 172.25.32.248@tcp) Lustre: Skipped 13 previous similar messages LNet: Service thread pid 3628 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 3628, comm: ll_mgs_0014 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] cfs_waitq_timedwait+0x11/0x20 [libcfs] [] target_bulk_io+0x3b8/0x910 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_prep_bulk_exp+0x6f/0x180 [ptlrpc] [] mgs_get_ir_logs+0x93d/0x130a [mgs] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [] ? lustre_swab_mgs_config_body+0x0/0x30 [ptlrpc] [] mgs_handle+0x4ea/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415007813.3628 LNet: Service thread pid 13033 was inactive for 422.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 13033, comm: mdt_rdpg00_018 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] cfs_waitq_timedwait+0x11/0x20 [libcfs] [] target_bulk_io+0x3b8/0x910 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? __ptlrpc_prep_bulk_page+0x68/0x170 [ptlrpc] [] ? mdd_dir_page_build+0x0/0x210 [mdd] [] mdt_sendpage+0x10d/0x240 [mdt] [] mdt_readpage+0x497/0x960 [mdt] [] mdt_handle_common+0x647/0x16d0 [mdt] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] mds_readpage_handle+0x15/0x20 [mdt] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415007938.13033 Lustre: meerkat-MDT0000: Client 293421fe-27dc-172e-4cdc-0d6474ea7f42 (at 10.7.104.31@o2ib) reconnecting Lustre: Skipped 321 previous similar messages Lustre: 12423:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-143), not sending early reply req@ffff880012e32050 x1474633250922956/t0(0) o256->129d2a15-cea6-5690-7239-ee557bbfe5a2@10.7.104.31@o2ib:0/0 lens 304/240 e 3 to 0 dl 1415008361 ref 2 fl Interpret:/0/0 rc 0/0 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415008342/real 1415008350] req@ffff880625197400 x1483597753189432/t0(0) o13->meerkat-OST000e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415008361 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Lustre: meerkat-OST000e-osc: Connection to meerkat-OST000e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 7 previous similar messages LustreError: 3628:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 748+0s req@ffff880012e32050 x1474633250922956/t0(0) o256->129d2a15-cea6-5690-7239-ee557bbfe5a2@10.7.104.31@o2ib:0/0 lens 304/240 e 3 to 0 dl 1415008361 ref 1 fl Interpret:/0/0 rc 0/0 LNet: Service thread pid 3628 completed after 748.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: meerkat-OST000e-osc: Connection restored to meerkat-OST000e (at 172.25.32.243@tcp) Lustre: Skipped 7 previous similar messages LustreError: 3611:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST003e-osc: can't precreate: rc = -11 LustreError: 3611:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 3 previous similar messages LustreError: 3611:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST003e-osc: cannot precreate objects: rc = -11 LustreError: 3611:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 3 previous similar messages Lustre: meerkat-OST0026-osc: Connection restored to meerkat-OST0026 (at 172.25.32.243@tcp) Lustre: Skipped 1 previous similar message Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415008507/real 1415008510] req@ffff880300c38000 x1483597753376900/t0(0) o13->meerkat-OST0002-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415008517 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Lustre: meerkat-OST0002-osc: Connection to meerkat-OST0002 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 5 previous similar messages Lustre: meerkat-OST0002-osc: Connection restored to meerkat-OST0002 (at 172.25.32.248@tcp) Lustre: Skipped 3 previous similar messages Lustre: meerkat-OST001e-osc: Connection restored to meerkat-OST001e (at 172.25.32.243@tcp) Lustre: Skipped 3 previous similar messages Lustre: meerkat-MDT0000: Client 426a4ee4-22a1-0043-48af-90853c7ea3f9 (at 10.7.101.20@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: 13032:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-527), not sending early reply req@ffff88009b7a6000 x1474469740722108/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 2 to 0 dl 1415008648 ref 2 fl Interpret:/0/0 rc 0/0 LustreError: 13033:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 1132+0s req@ffff88009b7a6000 x1474469740722108/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 2 to 0 dl 1415008648 ref 1 fl Interpret:/0/0 rc 0/0 LNet: Service thread pid 13033 completed after 1132.03s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). LustreError: 3588:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0034-osc: can't precreate: rc = -11 LustreError: 3588:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0034-osc: cannot precreate objects: rc = -11 LustreError: 3497:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000c-osc: can't precreate: rc = -11 LustreError: 3497:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3497:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000c-osc: cannot precreate objects: rc = -11 LustreError: 3497:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message Lustre: meerkat-OST0014-osc: Connection restored to meerkat-OST0014 (at 172.25.32.115@tcp) LustreError: 11-0: meerkat-OST000c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: MGS: Client ccc0ee10-f0eb-841b-3b8a-ada4760c38c0 (at 10.7.103.247@o2ib) reconnecting Lustre: Skipped 203 previous similar messages Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415009570/real 0] req@ffff8802e3c1ac00 x1483597754384396/t0(0) o6->meerkat-OST0014-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415009578 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 173 previous similar messages Lustre: meerkat-OST0014-osc: Connection to meerkat-OST0014 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 20 previous similar messages LustreError: 3497:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000c-osc: can't precreate: rc = -11 LustreError: 3497:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000c-osc: cannot precreate objects: rc = -11 LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3396:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415009585 with bad export cookie 459840026235109255 LustreError: 11-0: meerkat-OST0024-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message LustreError: 11-0: meerkat-OST0026-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 2 previous similar messages Lustre: meerkat-OST003e-osc: Connection restored to meerkat-OST003e (at 172.25.32.243@tcp) Lustre: Skipped 15 previous similar messages Lustre: MGS: Client 9f8529f1-db32-f4b2-7e87-2121151f4139 (at 10.7.103.191@o2ib) reconnecting Lustre: Skipped 141 previous similar messages Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415010889/real 0] req@ffff88031512f000 x1483597755080372/t0(0) o6->meerkat-OST0006-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415010901 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 150 previous similar messages Lustre: meerkat-OST0006-osc: Connection to meerkat-OST0006 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 15 previous similar messages LustreError: 11-0: meerkat-OST0016-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415010914/real 0] req@ffff8803151a7800 x1483597755109592/t0(0) o6->meerkat-OST001c-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415010921 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 119 previous similar messages Lustre: meerkat-OST001c-osc: Connection to meerkat-OST001c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415010923/real 0] req@ffff880628376000 x1483597755112104/t0(0) o8->meerkat-OST003e-osc@172.25.32.243@tcp:28/4 lens 400/544 e 0 to 1 dl 1415010931 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 102 previous similar messages LustreError: 3611:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST003e-osc: can't precreate: rc = -11 LustreError: 3611:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST003e-osc: cannot precreate objects: rc = -11 Lustre: meerkat-MDT0000: Client 4b4293da-929d-4ecd-6e11-849ae79785d1 (at 10.7.102.127@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-OST0026-osc: Connection restored to meerkat-OST0026 (at 172.25.32.243@tcp) Lustre: Skipped 11 previous similar messages LustreError: 11-0: meerkat-OST002e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 3 previous similar messages Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415010939/real 1415010948] req@ffff8803156adc00 x1483597755116124/t0(0) o6->meerkat-OST0034-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415010960 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 11 previous similar messages Lustre: meerkat-OST000e-osc: Connection restored to meerkat-OST000e (at 172.25.32.243@tcp) Lustre: Skipped 1 previous similar message Lustre: MGS: Client fcde8227-5d35-2279-5d49-b8dfef7051dc (at 10.7.101.190@o2ib) reconnecting Lustre: Skipped 278 previous similar messages Lustre: meerkat-OST0036-osc: Connection restored to meerkat-OST0036 (at 172.25.32.243@tcp) Lustre: Skipped 2 previous similar messages Lustre: meerkat-MDT0000: Client 4b4293da-929d-4ecd-6e11-849ae79785d1 (at 10.7.102.127@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415011005/real 0] req@ffff880511b20000 x1483597755138648/t0(0) o6->meerkat-OST000e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415011021 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Lustre: meerkat-OST000e-osc: Connection to meerkat-OST000e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: meerkat-MDT0000: Client 4b4293da-929d-4ecd-6e11-849ae79785d1 (at 10.7.102.127@o2ib) refused reconnection, still busy with 1 active RPCs LustreError: 3485:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0006-osc: can't precreate: rc = -11 LustreError: 3485:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0006-osc: cannot precreate objects: rc = -11 LustreError: 11-0: meerkat-OST0014-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 126s: evicting client at 10.7.102.127@o2ib ns: mdt-meerkat-MDT0000_UUID lock: ffff8803984e4900/0x661ae114c761496 lrc: 3/0,0 mode: PR/PR res: [0x200007b8e:0x2c:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x200000000020 nid: 10.7.102.127@o2ib remote: 0x3b4b28a6b555f63b expref: 35 pid: 6622 timeout: 4437001355 lvb_type: 0 Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 11 previous similar messages Lustre: meerkat-OST0006-osc: Connection to meerkat-OST0006 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 3485:0:(osp_precreate.c:737:osp_precreate_cleanup_orphans()) meerkat-OST0006-osc: cannot cleanup orphans: rc = -11 LustreError: 3480:0:(osp_precreate.c:737:osp_precreate_cleanup_orphans()) meerkat-OST0004-osc: cannot cleanup orphans: rc = -11 LustreError: 3501:0:(osp_precreate.c:737:osp_precreate_cleanup_orphans()) meerkat-OST000e-osc: cannot cleanup orphans: rc = -11 LustreError: 3515:0:(osp_precreate.c:737:osp_precreate_cleanup_orphans()) meerkat-OST0014-osc: cannot cleanup orphans: rc = -11 LustreError: 3515:0:(osp_precreate.c:737:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message LustreError: 3533:0:(osp_precreate.c:737:osp_precreate_cleanup_orphans()) meerkat-OST001c-osc: cannot cleanup orphans: rc = -11 LustreError: 3533:0:(osp_precreate.c:737:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message Lustre: 3550:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415011075/real 0] req@ffff880636d61c00 x1483597755151976/t0(0) o5->meerkat-OST0024-osc@172.25.32.115@tcp:28/4 lens 432/432 e 0 to 1 dl 1415011097 ref 3 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3550:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 204 previous similar messages LustreError: 3555:0:(osp_precreate.c:737:osp_precreate_cleanup_orphans()) meerkat-OST0026-osc: cannot cleanup orphans: rc = -11 LustreError: 3555:0:(osp_precreate.c:737:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages Lustre: meerkat-OST0036-osc: Connection restored to meerkat-OST0036 (at 172.25.32.243@tcp) Lustre: Skipped 11 previous similar messages LustreError: 11-0: meerkat-OST0026-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 12 previous similar messages Lustre: MGS: Client dd6641cf-b732-b7cc-978f-5708861aa03c (at 10.7.102.72@o2ib) reconnecting Lustre: Skipped 329 previous similar messages Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415011385/real 0] req@ffff8805d5de9c00 x1483597755359852/t0(0) o6->meerkat-OST0026-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415011408 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 41 previous similar messages Lustre: meerkat-OST0026-osc: Connection to meerkat-OST0026 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 10 previous similar messages LustreError: 3550:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0024-osc: can't precreate: rc = -11 LustreError: 3550:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0024-osc: cannot precreate objects: rc = -11 LustreError: 3485:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0006-osc: can't precreate: rc = -11 LustreError: 3485:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0006-osc: cannot precreate objects: rc = -11 Lustre: meerkat-MDT0000: Client 06bc4379-10ca-76ad-cd98-1d1013f1b911 (at 10.7.103.252@o2ib) reconnecting Lustre: Skipped 99 previous similar messages LustreError: 3575:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST002e-osc: can't precreate: rc = -11 LustreError: 3575:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST002e-osc: cannot precreate objects: rc = -11 LustreError: 3588:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0034-osc: can't precreate: rc = -11 LustreError: 3588:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3588:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0034-osc: cannot precreate objects: rc = -11 LustreError: 3588:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages LustreError: 11-0: meerkat-OST002e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 14 previous similar messages LustreError: 12099:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 103s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff8802f15346c0/0x661ae114cd3c1ee lrc: 3/0,0 mode: PR/PR res: [0x20000612e:0x53c9:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0ba218a91 expref: 72 pid: 13056 timeout: 4437542783 lvb_type: 0 LustreError: 6376:0:(client.c:1048:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880339ed3400 x1483597755436116/t0(0) o104->meerkat-MDT0000@192.168.230.53@tcp:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 LustreError: 6376:0:(ldlm_lockd.c:709:ldlm_handle_ast_error()) ### client (nid 192.168.230.53@tcp) returned 0 from blocking AST ns: mdt-meerkat-MDT0000_UUID lock: ffff8802e8df3480/0x661ae114cd3acc4 lrc: 1/0,0 mode: --/PR res: [0x20000645b:0x1b7ed:0x0].0 bits 0x1b rrc: 2 type: IBT flags: 0xa01000000020 nid: 192.168.230.53@tcp remote: 0x7185dbd0ba218623 expref: 4 pid: 13056 timeout: 4437643000 lvb_type: 0 Lustre: MGS: Client b4b276fc-a337-e01d-cc66-76f9cb9e69aa (at 10.7.103.103@o2ib) reconnecting Lustre: Skipped 11 previous similar messages Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415012204/real 1415012213] req@ffff8806249b9000 x1483597756192304/t0(0) o13->meerkat-OST003e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415012215 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 161 previous similar messages Lustre: meerkat-OST003e-osc: Connection to meerkat-OST003e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 3533:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST001c-osc: can't precreate: rc = -11 LustreError: 3533:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3533:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST001c-osc: cannot precreate objects: rc = -11 LustreError: 3533:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message LustreError: 11-0: meerkat-OST003e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 6 previous similar messages Lustre: meerkat-OST002c-osc: Connection restored to meerkat-OST002c (at 172.25.32.115@tcp) Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST0006-osc: Connection to meerkat-OST0006 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 5 previous similar messages LustreError: 3480:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0004-osc: can't precreate: rc = -11 LustreError: 3480:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0004-osc: cannot precreate objects: rc = -11 LustreError: 3588:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0034-osc: can't precreate: rc = -11 LustreError: 3588:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3588:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0034-osc: cannot precreate objects: rc = -11 LustreError: 3588:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message LustreError: 3485:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0006-osc: can't precreate: rc = -11 LustreError: 3485:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3485:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0006-osc: cannot precreate objects: rc = -11 LustreError: 3485:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages LustreError: 11-0: meerkat-OST000e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: meerkat-OST001e-osc: Connection restored to meerkat-OST001e (at 172.25.32.243@tcp) Lustre: Skipped 3 previous similar messages Lustre: meerkat-OST001c-osc: Connection to meerkat-OST001c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 13 previous similar messages LustreError: 3555:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0026-osc: can't precreate: rc = -11 LustreError: 3555:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3555:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0026-osc: cannot precreate objects: rc = -11 LustreError: 3555:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message LNet: Service thread pid 12099 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 12099, comm: mdt_rdpg00_007 Call Trace: [] ? shrink_inactive_list+0x428/0x830 [] ? mem_cgroup_lru_del_list+0x2b/0xb0 [] ? shrink_active_list+0x297/0x370 [] shrink_mem_cgroup_zone+0x3ae/0x610 [] ? mem_cgroup_iter+0xfd/0x280 [] shrink_zone+0x63/0xb0 [] zone_reclaim+0x349/0x400 [] get_page_from_freelist+0x69c/0x830 [] __alloc_pages_nodemask+0x113/0x8d0 [] ? perf_event_task_sched_out+0x33/0x80 [] ? dequeue_entity+0x113/0x2e0 [] kmem_getpages+0x62/0x170 [] cache_grow+0x2cf/0x320 [] cache_alloc_refill+0x202/0x240 [] kmem_cache_alloc+0x15f/0x190 [] ldiskfs_alloc_inode+0x20/0x150 [ldiskfs] [] alloc_inode+0x27/0xa0 [] iget_locked+0x78/0x170 [] ldiskfs_iget+0x37/0x800 [ldiskfs] [] ? fld_server_lookup+0x72/0x3d0 [fld] [] ? generic_detach_inode+0x18e/0x1f0 [] osd_iget+0x2e/0x2c0 [osd_ldiskfs] [] osd_ea_fid_get+0x176/0x2c0 [osd_ldiskfs] [] ? osd_remote_fid+0x9a/0x280 [osd_ldiskfs] [] osd_it_ea_rec+0xb45/0x1470 [osd_ldiskfs] [] ? call_filldir+0xb5/0x150 [ldiskfs] [] ? ldiskfs_readdir+0x5a9/0x730 [ldiskfs] [] ? osd_ldiskfs_filldir+0x0/0x480 [osd_ldiskfs] [] lod_it_rec+0x21/0x90 [lod] [] mdd_dir_page_build+0xfc/0x210 [mdd] [] dt_index_walk+0x162/0x3d0 [obdclass] [] ? mdd_dir_page_build+0x0/0x210 [mdd] [] mdd_readpage+0x38b/0x5a0 [mdd] [] mdt_readpage+0x47f/0x960 [mdt] [] mdt_handle_common+0x647/0x16d0 [mdt] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] mds_readpage_handle+0x15/0x20 [mdt] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415012531.12099 LNet: Service thread pid 12099 completed after 208.70s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: meerkat-MDT0000: Client dac36cb6-a9d7-f3f1-6efe-b858cd50465b (at 10.7.103.199@o2ib) reconnecting Lustre: Skipped 351 previous similar messages Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415014660/real 1415014664] req@ffff880563b7f800 x1483597759008628/t0(0) o13->meerkat-OST0004-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415014672 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 147 previous similar messages Lustre: meerkat-OST0004-osc: Connection to meerkat-OST0004 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages LustreError: 11-0: meerkat-OST003e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: Skipped 22 previous similar messages Lustre: meerkat-OST000c-osc: Connection to meerkat-OST000c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 3550:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0024-osc: can't precreate: rc = -11 LustreError: 3550:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3550:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0024-osc: cannot precreate objects: rc = -11 LustreError: 3550:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages LustreError: 3497:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000c-osc: can't precreate: rc = -11 LustreError: 3497:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000c-osc: cannot precreate objects: rc = -11 LustreError: 3485:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0006-osc: can't precreate: rc = -11 LustreError: 3485:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0006-osc: cannot precreate objects: rc = -11 LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3396:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415014724 with bad export cookie 459840026295915686 Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 4 previous similar messages Lustre: meerkat-MDT0000: Client 426a4ee4-22a1-0043-48af-90853c7ea3f9 (at 10.7.101.20@o2ib) refused reconnection, still busy with 1 active RPCs LustreError: 11-0: meerkat-OST000c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 10 previous similar messages Lustre: meerkat-MDT0000: Client 937943c3-76d0-47e3-c90f-964ba686b8b7 (at 10.7.104.17@o2ib) reconnecting Lustre: Skipped 363 previous similar messages Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415014727/real 1415014736] req@ffff8800136e6000 x1483597759032880/t0(0) o6->meerkat-OST001c-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415014748 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 163 previous similar messages Lustre: meerkat-OST001c-osc: Connection to meerkat-OST001c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: meerkat-OST000e-osc: Connection restored to meerkat-OST000e (at 172.25.32.243@tcp) Lustre: Skipped 1 previous similar message Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 14 previous similar messages Lustre: meerkat-OST0012-osc: Connection to meerkat-OST0012 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 11-0: meerkat-OST0002-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: Skipped 2 previous similar messages Lustre: meerkat-OST0002-osc: Connection restored to meerkat-OST0002 (at 172.25.32.248@tcp) Lustre: Skipped 7 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 198.202.118.83@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3396:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 198.202.118.83@tcp arrived at 1415014866 with bad export cookie 459840023463751085 LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3396:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415014873 with bad export cookie 459840026369529513 LustreError: 3515:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0014-osc: can't precreate: rc = -11 LustreError: 3515:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0014-osc: cannot precreate objects: rc = -11 Lustre: 3485:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415014866/real 0] req@ffff880546117400 x1483597759117700/t0(0) o5->meerkat-OST0006-osc@172.25.32.243@tcp:28/4 lens 432/432 e 0 to 1 dl 1415014898 ref 3 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3485:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 208 previous similar messages Lustre: meerkat-MDT0000: Client 08f6e3ea-5ac6-8cca-cdf6-fd62e34dd220 (at 10.7.102.124@o2ib) reconnecting Lustre: Skipped 293 previous similar messages LustreError: 11-0: meerkat-OST0036-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: Skipped 1 previous similar message LustreError: 4733:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff8800112a7800 x1482141740964988/t0(0) o37->8c5b20e1-4e92-4313-0ab5-a06d9154e563@198.202.118.30@tcp:0/0 lens 448/440 e 2 to 0 dl 1415015007 ref 1 fl Interpret:/0/0 rc 0/0 LustreError: 13033:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff8800b5a15900/0x661ae1151c1305d lrc: 3/0,0 mode: PR/PR res: [0x20000611e:0x26ad:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0ba65a0f3 expref: 44 pid: 13056 timeout: 4441114164 lvb_type: 0 LustreError: 13033:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff8800b363dc00 x1474469754920644/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 1 to 0 dl 1415015268 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: 3419:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415015214/real 1415015215] req@ffff8802f358ec00 x1483597760224496/t0(0) o104->meerkat-MDT0000@192.168.230.53@tcp:15/16 lens 296/224 e 0 to 1 dl 1415015222 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3397:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415015227 with bad export cookie 459840026378360125 Lustre: meerkat-OST000e-osc: Connection to meerkat-OST000e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 23 previous similar messages LustreError: 11-0: meerkat-OST002c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 10 previous similar messages Lustre: meerkat-OST0034-osc: Connection restored to meerkat-OST0034 (at 172.25.32.115@tcp) Lustre: Skipped 16 previous similar messages LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff88007a466b40/0x661ae1151fb5d7e lrc: 3/0,0 mode: PR/PR res: [0x20000611e:0x2cae:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0ba6ce41b expref: 41 pid: 3569 timeout: 4441409710 lvb_type: 0 LustreError: 13033:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff8802e1c50c00 x1474469755691024/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 0 to 0 dl 1415015525 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: 3329:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415015848/real 1415015851] req@ffff8803157ce400 x1483597760552896/t0(0) o13->meerkat-OST0024-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415015855 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3329:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 69 previous similar messages Lustre: meerkat-OST0024-osc: Connection to meerkat-OST0024 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 10 previous similar messages Lustre: meerkat-MDT0000: Client 79d7c292-9b21-1781-e097-07f18e61b5f2 (at 10.7.103.203@o2ib) reconnecting Lustre: Skipped 5 previous similar messages LustreError: 3519:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0016-osc: can't precreate: rc = -11 LustreError: 3519:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3519:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0016-osc: cannot precreate objects: rc = -11 LustreError: 3519:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message LustreError: 3606:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST003c-osc: can't precreate: rc = -11 LustreError: 3606:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST003c-osc: cannot precreate objects: rc = -11 Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: Skipped 10 previous similar messages Lustre: MGS: Client 08a8235c-c4e7-3082-6847-ef7907a1f6e7 (at 10.7.103.193@o2ib) reconnecting Lustre: Skipped 323 previous similar messages LustreError: 11-0: meerkat-OST001e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 4 previous similar messages Lustre: meerkat-MDT0000: Client da6c8f9e-8e18-d92d-83b0-dc2b12c81782 (at 10.7.101.27@o2ib) reconnecting Lustre: Skipped 399 previous similar messages Lustre: meerkat-MDT0000: Client 51058832-a631-50b7-dd44-4d95263441e7 (at 10.7.103.204@o2ib) reconnecting Lustre: Skipped 5 previous similar messages Lustre: meerkat-OST0016-osc: Connection to meerkat-OST0016 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 31 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3519:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0016-osc: can't precreate: rc = -11 LustreError: 3519:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0016-osc: cannot precreate objects: rc = -11 LustreError: 3501:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000e-osc: can't precreate: rc = -11 LustreError: 3501:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000e-osc: cannot precreate objects: rc = -11 LustreError: 3396:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415016391 with bad export cookie 459840026382605310 Lustre: meerkat-MDT0000: Client 250bc74e-e43e-04be-e11f-983b4d0d9e2c (at 10.7.103.223@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415016432/real 1415016451] req@ffff88059ae2e000 x1483597760705096/t0(0) o8->meerkat-OST003e-osc@172.25.32.243@tcp:28/4 lens 400/544 e 0 to 1 dl 1415016455 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 328 previous similar messages Lustre: meerkat-OST001e-osc: Connection restored to meerkat-OST001e (at 172.25.32.243@tcp) Lustre: Skipped 31 previous similar messages Lustre: meerkat-MDT0000: Client 5c2fe3cc-b9b0-0e65-f799-e9d0bca910c0 (at 198.202.119.69@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 6249e52a-f753-9378-7db9-6a4f5d439ce4 (at 198.202.119.70@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-OST0016-osc: slow creates, last=[0x100160000:0xac09e1:0x0], next=[0x100160000:0xac09e1:0x0], reserved=0, syn_changes=172, syn_rpc_in_progress=5, status=-11 LNet: Service thread pid 13229 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 13229, comm: ll_mgs_0023 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] cfs_waitq_timedwait+0x11/0x20 [libcfs] [] target_bulk_io+0x3b8/0x910 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_prep_bulk_exp+0x6f/0x180 [ptlrpc] [] mgs_get_ir_logs+0x93d/0x130a [mgs] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [] ? lustre_swab_mgs_config_body+0x0/0x30 [ptlrpc] [] mgs_handle+0x4ea/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415016616.13229 LNet: Service thread pid 13957 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 13957, comm: ll_mgs_0031 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] cfs_waitq_timedwait+0x11/0x20 [libcfs] [] target_bulk_io+0x3b8/0x910 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_prep_bulk_exp+0x6f/0x180 [ptlrpc] [] mgs_get_ir_logs+0x93d/0x130a [mgs] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [] ? lustre_swab_mgs_config_body+0x0/0x30 [ptlrpc] [] mgs_handle+0x4ea/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415016625.13957 LNet: Service thread pid 3626 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 3626, comm: ll_mgs_0012 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] cfs_waitq_timedwait+0x11/0x20 [libcfs] [] target_bulk_io+0x3b8/0x910 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_prep_bulk_exp+0x6f/0x180 [ptlrpc] [] mgs_get_ir_logs+0x93d/0x130a [mgs] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [] ? lustre_swab_mgs_config_body+0x0/0x30 [ptlrpc] [] mgs_handle+0x4ea/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415016633.3626 LNet: Service thread pid 3624 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 3624, comm: ll_mgs_0010 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] cfs_waitq_timedwait+0x11/0x20 [libcfs] [] target_bulk_io+0x3b8/0x910 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_prep_bulk_exp+0x6f/0x180 [ptlrpc] [] mgs_get_ir_logs+0x93d/0x130a [mgs] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [] ? lustre_swab_mgs_config_body+0x0/0x30 [ptlrpc] [] mgs_handle+0x4ea/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415016635.3624 Pid: 13233, comm: ll_mgs_0027 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] cfs_waitq_timedwait+0x11/0x20 [libcfs] [] target_bulk_io+0x3b8/0x910 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_prep_bulk_exp+0x6f/0x180 [ptlrpc] [] mgs_get_ir_logs+0x93d/0x130a [mgs] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [] ? lustre_swab_mgs_config_body+0x0/0x30 [ptlrpc] [] mgs_handle+0x4ea/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415016636.13233 LNet: Service thread pid 13231 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. LustreError: dumping log to /tmp/lustre-log.1415016637.13231 LNet: Service thread pid 13225 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. LustreError: dumping log to /tmp/lustre-log.1415016656.13225 Lustre: 3411:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-383), not sending early reply req@ffff880315a4a050 x1480335026493200/t0(0) o256->93fc4711-d07f-2642-a190-f194f51dd182@10.7.103.249@o2ib:0/0 lens 304/240 e 4 to 0 dl 1415017404 ref 2 fl Interpret:/0/0 rc 0/0 LustreError: 13229:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 988+0s req@ffff880315a4a050 x1480335026493200/t0(0) o256->93fc4711-d07f-2642-a190-f194f51dd182@10.7.103.249@o2ib:0/0 lens 304/240 e 4 to 0 dl 1415017404 ref 1 fl Interpret:/0/0 rc 0/0 LNet: Service thread pid 13229 completed after 988.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: 13229:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-383), not sending early reply req@ffff880638ee6850 x1474604961314608/t0(0) o256->c28b6818-6e56-8ec7-278a-f096d0702697@10.7.103.124@o2ib:0/0 lens 304/240 e 4 to 0 dl 1415017413 ref 2 fl Interpret:/0/0 rc 0/0 LustreError: 13957:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 988+0s req@ffff880638ee6850 x1474604961314608/t0(0) o256->c28b6818-6e56-8ec7-278a-f096d0702697@10.7.103.124@o2ib:0/0 lens 304/240 e 4 to 0 dl 1415017413 ref 1 fl Interpret:/0/0 rc 0/0 LNet: Service thread pid 13957 completed after 988.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: 13229:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-383), not sending early reply req@ffff8803151e9050 x1480335019159216/t0(0) o256->320e1102-9e44-b162-b591-d2f795f9f12d@10.7.103.248@o2ib:0/0 lens 304/240 e 4 to 0 dl 1415017421 ref 2 fl Interpret:/0/0 rc 0/0 Lustre: 13957:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-383), not sending early reply req@ffff88063af36850 x1474527071607596/t0(0) o256->6c923fb5-95df-59c8-b335-b2a058cf277a@10.7.104.30@o2ib:0/0 lens 304/240 e 4 to 0 dl 1415017423 ref 2 fl Interpret:/0/0 rc 0/0 LustreError: 3626:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 988+0s req@ffff8803151e9050 x1480335019159216/t0(0) o256->320e1102-9e44-b162-b591-d2f795f9f12d@10.7.103.248@o2ib:0/0 lens 304/240 e 4 to 0 dl 1415017421 ref 1 fl Interpret:/0/0 rc 0/0 LNet: Service thread pid 3626 completed after 987.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). LustreError: 3624:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 988+0s req@ffff88063af36850 x1474527071607596/t0(0) o256->6c923fb5-95df-59c8-b335-b2a058cf277a@10.7.104.30@o2ib:0/0 lens 304/240 e 4 to 0 dl 1415017423 ref 1 fl Interpret:/0/0 rc 0/0 LNet: Service thread pid 3624 completed after 988.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: 3624:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-367), not sending early reply req@ffff880315fd9850 x1462852211317776/t0(0) o256->96f059bf-5deb-1f96-4784-e4c0097225a8@10.7.104.18@o2ib:0/0 lens 304/240 e 3 to 0 dl 1415017428 ref 2 fl Interpret:/0/0 rc 0/0 Lustre: 3624:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages LustreError: 13231:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 988+0s req@ffff880638efa050 x1474651431842564/t0(0) o256->5052cd8c-32e0-e559-56a2-a07ea44f40d5@10.7.103.217@o2ib:0/0 lens 304/240 e 4 to 0 dl 1415017425 ref 1 fl Interpret:/0/0 rc 0/0 LustreError: 13231:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 1 previous similar message LNet: Service thread pid 13231 completed after 988.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). LNet: Skipped 1 previous similar message LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff88006f65e240/0x661ae11544b2e60 lrc: 3/0,0 mode: PR/PR res: [0x20000611e:0x2979:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0ba7cb984 expref: 7 pid: 13059 timeout: 4443681395 lvb_type: 0 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415017740/real 0] req@ffff8803fd60b400 x1483597761692556/t0(0) o6->meerkat-OST002c-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415017747 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Lustre: meerkat-OST002c-osc: Connection to meerkat-OST002c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: meerkat-MDT0000: Client 5e35aead-29b9-2bc2-275b-7fb0ed868b9d (at 198.202.119.81@tcp) reconnecting Lustre: Skipped 560 previous similar messages LustreError: 11-0: meerkat-OST003e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 12 previous similar messages Lustre: meerkat-OST001e-osc: Connection restored to meerkat-OST001e (at 172.25.32.243@tcp) Lustre: Skipped 15 previous similar messages Lustre: meerkat-MDT0000: Client c250eae3-37d7-9a19-7761-3b66e26fa954 (at 10.7.103.79@o2ib) reconnecting Lustre: Skipped 13 previous similar messages Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415017816/real 0] req@ffff88059b390000 x1483597761772848/t0(0) o6->meerkat-OST0036-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415017830 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 169 previous similar messages Lustre: meerkat-OST0036-osc: Connection to meerkat-OST0036 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 30 previous similar messages Lustre: meerkat-MDT0000: Client 45c46921-501b-d676-d268-076babef0f2b (at 10.7.103.189@o2ib) reconnecting Lustre: Skipped 103 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 10.7.101.162@o2ib was evicted due to a lock blocking callback time out: rc -107 LustreError: 3630:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 10.7.101.162@o2ib arrived at 1415017894 with bad export cookie 459840023463752471 Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 30 previous similar messages Lustre: MGS: Client a84824d0-0886-cafd-c1c9-1fea8700b395 (at 10.7.103.101@o2ib) reconnecting Lustre: Skipped 182 previous similar messages Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415018052/real 1415018061] req@ffff88062fcac000 x1483597761895208/t0(0) o6->meerkat-OST0036-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415018076 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 176 previous similar messages Lustre: meerkat-OST0036-osc: Connection to meerkat-OST0036 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 3593:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0036-osc: can't precreate: rc = -11 LustreError: 3593:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3593:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0036-osc: cannot precreate objects: rc = -11 LustreError: 3593:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message LustreError: 3568:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST002c-osc: can't precreate: rc = -11 LustreError: 3568:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST002c-osc: cannot precreate objects: rc = -11 LustreError: 3555:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0026-osc: can't precreate: rc = -11 LustreError: 3555:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3555:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0026-osc: cannot precreate objects: rc = -11 LustreError: 3555:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages LustreError: 3611:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST003e-osc: can't precreate: rc = -11 LustreError: 3611:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 3 previous similar messages LustreError: 3611:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST003e-osc: cannot precreate objects: rc = -11 LustreError: 3611:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 3 previous similar messages Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 15 previous similar messages Lustre: MGS: haven't heard from client 1b36b0b5-dda1-1aa6-43f2-9c82e50838ab (at 10.7.102.113@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88063a320c00, cur 1415018267 expire 1415018117 last 1415018040 Lustre: MGS: haven't heard from client d0a9336d-bf81-6ee7-f5e9-73127194c1ee (at 10.7.102.127@o2ib) in 221 seconds. I think it's dead, and I am evicting it. exp ffff880624cd9c00, cur 1415018267 expire 1415018117 last 1415018046 Lustre: Skipped 7 previous similar messages LNet: Service thread pid 3411 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: LNet: Skipped 1 previous similar message Pid: 3411, comm: ll_mgs_0000 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] cfs_waitq_timedwait+0x11/0x20 [libcfs] [] target_bulk_io+0x3b8/0x910 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_prep_bulk_exp+0x6f/0x180 [ptlrpc] [] mgs_get_ir_logs+0x93d/0x130a [mgs] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [] ? lustre_swab_mgs_config_body+0x0/0x30 [ptlrpc] [] mgs_handle+0x4ea/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415018342.3411 Lustre: meerkat-MDT0000: Client 8551fe3d-b651-bc4f-be81-2e1248b5550d (at 10.7.103.173@o2ib) reconnecting Lustre: Skipped 432 previous similar messages Lustre: 12424:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-255), not sending early reply req@ffff88063af36050 x1480332632722872/t0(0) o256->0d770415-5152-eacb-b4dd-cb9ea56b09b5@10.7.103.236@o2ib:0/0 lens 304/240 e 3 to 0 dl 1415019002 ref 2 fl Interpret:/0/0 rc 0/0 LustreError: 3411:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 860+0s req@ffff88063af36050 x1480332632722872/t0(0) o256->0d770415-5152-eacb-b4dd-cb9ea56b09b5@10.7.103.236@o2ib:0/0 lens 304/240 e 3 to 0 dl 1415019002 ref 1 fl Interpret:/0/0 rc 0/0 LustreError: 3411:0:(ldlm_lib.c:2702:target_bulk_io()) Skipped 1 previous similar message LNet: Service thread pid 3411 completed after 860.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). LNet: Skipped 1 previous similar message Lustre: meerkat-MDT0000: Client 753ac3ca-83ae-ca8e-90bf-67aa3d1aedf8 (at 10.7.103.128@o2ib) reconnecting Lustre: Skipped 5 previous similar messages Lustre: 3331:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415019541/real 0] req@ffff88063972a800 x1483597766237424/t0(0) o6->meerkat-OST0004-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415019552 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3331:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 177 previous similar messages Lustre: meerkat-OST0004-osc: Connection to meerkat-OST0004 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 3485:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0006-osc: can't precreate: rc = -11 LustreError: 3485:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0006-osc: cannot precreate objects: rc = -11 LustreError: 3555:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0026-osc: can't precreate: rc = -11 LustreError: 3555:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0026-osc: cannot precreate objects: rc = -11 LustreError: 11-0: meerkat-OST0004-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 8 previous similar messages Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 15 previous similar messages Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415019634/real 0] req@ffff880095041400 x1483597766441744/t0(0) o13->meerkat-OST0034-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415019650 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 158 previous similar messages Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 11-0: meerkat-OST0034-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 15 previous similar messages Lustre: meerkat-OST001c-osc: Connection restored to meerkat-OST001c (at 172.25.32.115@tcp) Lustre: Skipped 15 previous similar messages LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff8802aee25d80/0x661ae1158102ff8 lrc: 3/0,0 mode: PR/PR res: [0x200006144:0x833e:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0bae6434d expref: 6 pid: 3419 timeout: 4446403173 lvb_type: 0 Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415020497/real 1415020500] req@ffff880625351400 x1483597766932552/t0(0) o13->meerkat-OST000a-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415020505 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 56 previous similar messages Lustre: meerkat-OST000a-osc: Connection to meerkat-OST000a (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 8 previous similar messages Lustre: MGS: Client d0e2946c-e12b-84ad-283d-67f5cbf08e9b (at 10.7.104.33@o2ib) reconnecting Lustre: Skipped 236 previous similar messages Lustre: meerkat-OST0012-osc: Connection restored to meerkat-OST0012 (at 172.25.32.248@tcp) Lustre: Skipped 8 previous similar messages Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415020533/real 1415020538] req@ffff8802ebd38000 x1483597766942660/t0(0) o13->meerkat-OST000e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415020542 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Lustre: meerkat-OST000e-osc: Connection to meerkat-OST000e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages LustreError: 3568:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST002c-osc: can't precreate: rc = -11 LustreError: 3568:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST002c-osc: cannot precreate objects: rc = -11 LustreError: 3519:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0016-osc: can't precreate: rc = -11 LustreError: 3519:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3519:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0016-osc: cannot precreate objects: rc = -11 LustreError: 3519:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415020552/real 1415020557] req@ffff880624ce8400 x1483597766946736/t0(0) o8->meerkat-OST0036-osc@172.25.32.243@tcp:28/4 lens 400/544 e 0 to 1 dl 1415020561 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 29 previous similar messages LustreError: 11-0: meerkat-OST0006-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 3 previous similar messages Lustre: meerkat-OST0014-osc: Connection restored to meerkat-OST0014 (at 172.25.32.115@tcp) Lustre: Skipped 3 previous similar messages Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415020652/real 1415020654] req@ffff880317f90000 x1483597766966308/t0(0) o13->meerkat-OST001a-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415020663 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Lustre: meerkat-OST001a-osc: Connection to meerkat-OST001a (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST001a-osc: Connection restored to meerkat-OST001a (at 172.25.32.248@tcp) Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST003e-osc: Connection to meerkat-OST003e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages LustreError: 3611:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST003e-osc: can't precreate: rc = -11 LustreError: 3611:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3611:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST003e-osc: cannot precreate objects: rc = -11 LustreError: 3611:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message LustreError: 3555:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0026-osc: can't precreate: rc = -11 LustreError: 3555:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3555:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0026-osc: cannot precreate objects: rc = -11 LustreError: 3555:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message Lustre: meerkat-OST003e-osc: Connection restored to meerkat-OST003e (at 172.25.32.243@tcp) Lustre: Skipped 2 previous similar messages LustreError: 11-0: meerkat-OST000e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415020734/real 1415020736] req@ffff8806283be800 x1483597766987556/t0(0) o13->meerkat-OST0012-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415020745 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Lustre: meerkat-OST0016-osc: Connection to meerkat-OST0016 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages LustreError: 3519:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0016-osc: can't precreate: rc = -11 LustreError: 3519:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0016-osc: cannot precreate objects: rc = -11 Lustre: meerkat-OST0016-osc: Connection restored to meerkat-OST0016 (at 172.25.32.243@tcp) Lustre: Skipped 6 previous similar messages Lustre: 3519:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415020910/real 1415020914] req@ffff8805b8a02c00 x1483597767142832/t0(0) o5->meerkat-OST0016-osc@172.25.32.243@tcp:28/4 lens 432/432 e 0 to 1 dl 1415020924 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3519:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages LustreError: 3519:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0016-osc: can't precreate: rc = -11 LustreError: 3519:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0016-osc: cannot precreate objects: rc = -11 Lustre: MGS: Client 78475e91-2118-5335-4ff6-c225b96f7b22 (at 10.7.103.242@o2ib) reconnecting Lustre: Skipped 479 previous similar messages Lustre: 3593:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415021299/real 1415021303] req@ffff8806159a3800 x1483597767802092/t0(0) o5->meerkat-OST0036-osc@172.25.32.243@tcp:28/4 lens 432/432 e 0 to 1 dl 1415021317 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3593:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Lustre: meerkat-OST0036-osc: Connection to meerkat-OST0036 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 5 previous similar messages LustreError: 3593:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0036-osc: can't precreate: rc = -11 LustreError: 3593:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3593:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0036-osc: cannot precreate objects: rc = -11 LustreError: 3593:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages Lustre: meerkat-OST0036-osc: Connection restored to meerkat-OST0036 (at 172.25.32.243@tcp) Lustre: Skipped 5 previous similar messages LustreError: 3601:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST003a-osc: can't precreate: rc = -11 LustreError: 3601:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST003a-osc: cannot precreate objects: rc = -11 Lustre: MGS: Client 0d2d22b3-b24d-8f70-4232-92dc4e356009 (at 10.7.103.122@o2ib) reconnecting Lustre: Skipped 83 previous similar messages Lustre: 3331:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415022261/real 1415022262] req@ffff8806390a3400 x1483597768391012/t0(0) o13->meerkat-OST0012-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415022276 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3331:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Lustre: meerkat-OST0012-osc: Connection to meerkat-OST0012 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 8 previous similar messages Lustre: meerkat-OST0012-osc: Connection restored to meerkat-OST0012 (at 172.25.32.248@tcp) Lustre: Skipped 8 previous similar messages Lustre: meerkat-MDT0000: Client 626f7bc2-b5be-dba8-b3b6-09bc7cab6bef (at 10.7.103.117@o2ib) reconnecting Lustre: Skipped 200 previous similar messages Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs LustreError: 12073:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880336016400 x1482141742520424/t0(0) o37->8c5b20e1-4e92-4313-0ab5-a06d9154e563@198.202.118.30@tcp:0/0 lens 448/440 e 3 to 0 dl 1415022733 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: 3331:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415022947/real 1415022955] req@ffff88062f6da800 x1483597768519616/t0(0) o13->meerkat-OST0006-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415022964 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: meerkat-OST0006-osc: Connection to meerkat-OST0006 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete LustreError: 3519:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0016-osc: can't precreate: rc = -11 LustreError: 3519:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 3 previous similar messages LustreError: 3519:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0016-osc: cannot precreate objects: rc = -11 LustreError: 3519:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 3 previous similar messages Lustre: meerkat-OST000e-osc: Connection to meerkat-OST000e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: meerkat-OST0014-osc: Connection to meerkat-OST0014 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: meerkat-OST0014-osc: Connection restored to meerkat-OST0014 (at 172.25.32.115@tcp) Lustre: Skipped 3 previous similar messages Lustre: meerkat-OST003c-osc: Connection to meerkat-OST003c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: meerkat-OST003c-osc: Connection restored to meerkat-OST003c (at 172.25.32.115@tcp) Lustre: meerkat-MDT0000: Client 5e35aead-29b9-2bc2-275b-7fb0ed868b9d (at 198.202.119.81@tcp) reconnecting Lustre: Skipped 264 previous similar messages Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs LustreError: 4733:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff8802f1246400 x1482141742614276/t0(0) o37->8c5b20e1-4e92-4313-0ab5-a06d9154e563@198.202.118.30@tcp:0/0 lens 448/440 e 3 to 0 dl 1415023538 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: 3331:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415024057/real 0] req@ffff880421c5e000 x1483597769278504/t0(0) o6->meerkat-OST0036-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415024064 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3331:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Lustre: meerkat-OST0036-osc: Connection to meerkat-OST0036 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 138-a: meerkat-MDT0000: A client on nid 198.202.118.135@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3395:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415024074 with bad export cookie 459840026484524862 LustreError: 3395:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) Skipped 1 previous similar message LustreError: 11-0: meerkat-OST0006-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST001c-osc: Connection restored to meerkat-OST001c (at 172.25.32.115@tcp) Lustre: MGS: Client 0c823b03-f9b4-bd9a-49da-7a6eceb1b8da (at 10.7.104.20@o2ib) reconnecting Lustre: Skipped 7 previous similar messages Lustre: meerkat-OST0026-osc: Connection to meerkat-OST0026 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 3575:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST002e-osc: can't precreate: rc = -11 LustreError: 3575:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST002e-osc: cannot precreate objects: rc = -11 LustreError: 11-0: meerkat-OST000c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: meerkat-OST0034-osc: Connection restored to meerkat-OST0034 (at 172.25.32.115@tcp) Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST003e-osc: Connection restored to meerkat-OST003e (at 172.25.32.243@tcp) Lustre: Skipped 4 previous similar messages LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff880094993480/0x661ae115e113ca0 lrc: 3/0,0 mode: PR/PR res: [0x2000060aa:0x8f03:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0bb4934e2 expref: 38 pid: 13049 timeout: 4450601601 lvb_type: 0 LustreError: 13028:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff88008b7da800 x1474469779536532/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 2 to 0 dl 1415024739 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415024657/real 1415024663] req@ffff8804272d5c00 x1483597773001596/t0(0) o13->meerkat-OST001c-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415024667 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 257 previous similar messages Lustre: meerkat-OST001c-osc: Connection to meerkat-OST001c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 11 previous similar messages LustreError: 3575:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST002e-osc: can't precreate: rc = -11 LustreError: 3575:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST002e-osc: cannot precreate objects: rc = -11 LustreError: 3537:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST001e-osc: can't precreate: rc = -11 LustreError: 3537:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST001e-osc: cannot precreate objects: rc = -11 LustreError: 3501:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000e-osc: can't precreate: rc = -11 LustreError: 3501:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000e-osc: cannot precreate objects: rc = -11 Lustre: meerkat-OST0034-osc: Connection restored to meerkat-OST0034 (at 172.25.32.115@tcp) Lustre: Skipped 6 previous similar messages LustreError: 11-0: meerkat-OST0004-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 6 previous similar messages Lustre: meerkat-OST0026-osc: Connection restored to meerkat-OST0026 (at 172.25.32.243@tcp) Lustre: Skipped 3 previous similar messages Lustre: meerkat-OST000e-osc: Connection to meerkat-OST000e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 12 previous similar messages LustreError: 3501:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000e-osc: can't precreate: rc = -11 LustreError: 3501:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000e-osc: cannot precreate objects: rc = -11 LustreError: 3593:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0036-osc: can't precreate: rc = -11 LustreError: 3593:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 7 previous similar messages LustreError: 3593:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0036-osc: cannot precreate objects: rc = -11 LustreError: 3593:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 7 previous similar messages LustreError: 11-0: meerkat-OST0006-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: meerkat-OST0036-osc: Connection restored to meerkat-OST0036 (at 172.25.32.243@tcp) Lustre: Skipped 8 previous similar messages Lustre: meerkat-OST000e-osc: Connection to meerkat-OST000e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 3501:0:(osp_precreate.c:737:osp_precreate_cleanup_orphans()) meerkat-OST000e-osc: cannot cleanup orphans: rc = -11 LustreError: 3501:0:(osp_precreate.c:737:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message LustreError: 3515:0:(osp_precreate.c:737:osp_precreate_cleanup_orphans()) meerkat-OST0014-osc: cannot cleanup orphans: rc = -11 Lustre: meerkat-MDT0000: Client 08f6e3ea-5ac6-8cca-cdf6-fd62e34dd220 (at 10.7.102.124@o2ib) reconnecting Lustre: Skipped 134 previous similar messages Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415025845/real 1415025850] req@ffff8806391f8800 x1483597773866264/t0(0) o13->meerkat-OST001c-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415025852 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 331 previous similar messages Lustre: meerkat-OST001c-osc: Connection to meerkat-OST001c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 5 previous similar messages Lustre: MGS: Client 4da277d2-a8b3-34b8-d086-948614162c14 (at 10.7.103.129@o2ib) reconnecting Lustre: Skipped 1 previous similar message Lustre: meerkat-OST0036-osc: Connection restored to meerkat-OST0036 (at 172.25.32.243@tcp) Lustre: Skipped 21 previous similar messages Lustre: meerkat-OST000e-osc: Connection restored to meerkat-OST000e (at 172.25.32.243@tcp) Lustre: meerkat-OST001e-osc: Connection to meerkat-OST001e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 5 previous similar messages LustreError: 3550:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0024-osc: can't precreate: rc = -11 LustreError: 3550:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3550:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0024-osc: cannot precreate objects: rc = -11 LustreError: 3550:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message LustreError: 3519:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0016-osc: can't precreate: rc = -11 LustreError: 3519:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3519:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0016-osc: cannot precreate objects: rc = -11 LustreError: 3519:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages LustreError: 3611:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST003e-osc: can't precreate: rc = -11 LustreError: 3611:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3611:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST003e-osc: cannot precreate objects: rc = -11 LustreError: 3611:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message Lustre: meerkat-OST002c-osc: Connection to meerkat-OST002c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 13 previous similar messages Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 4 previous similar messages Lustre: meerkat-OST002e-osc: Connection to meerkat-OST002e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 3588:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0034-osc: can't precreate: rc = -11 LustreError: 3588:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0034-osc: cannot precreate objects: rc = -11 Lustre: meerkat-OST002e-osc: Connection restored to meerkat-OST002e (at 172.25.32.243@tcp) Lustre: Skipped 15 previous similar messages LustreError: 3493:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000a-osc: can't precreate: rc = -11 LustreError: 3493:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3493:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000a-osc: cannot precreate objects: rc = -11 LustreError: 3493:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message Lustre: meerkat-OST001a-osc: Connection restored to meerkat-OST001a (at 172.25.32.248@tcp) Lustre: Skipped 3 previous similar messages Lustre: MGS: Client a9eb755d-491f-3091-2455-8721d0a45bcb (at 10.7.103.205@o2ib) reconnecting Lustre: Skipped 521 previous similar messages Lustre: 3593:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415026451/real 1415026453] req@ffff8806383ed800 x1483597774023952/t0(0) o5->meerkat-OST0036-osc@172.25.32.243@tcp:28/4 lens 432/432 e 0 to 1 dl 1415026472 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3593:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 64 previous similar messages Lustre: meerkat-OST0036-osc: Connection to meerkat-OST0036 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 10 previous similar messages LustreError: 3593:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0036-osc: can't precreate: rc = -11 LustreError: 3593:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3593:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0036-osc: cannot precreate objects: rc = -11 LustreError: 3593:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 12 previous similar messages Lustre: meerkat-OST001e-osc: Connection to meerkat-OST001e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 10 previous similar messages LustreError: 3537:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST001e-osc: can't precreate: rc = -11 LustreError: 3537:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST001e-osc: cannot precreate objects: rc = -11 Lustre: 3330:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415027262/real 1415027263] req@ffff880316a91800 x1483597774402588/t0(0) o13->meerkat-OST000a-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415027271 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3330:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 37 previous similar messages Lustre: meerkat-OST000a-osc: Connection to meerkat-OST000a (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 9 previous similar messages Lustre: meerkat-OST000a-osc: Connection restored to meerkat-OST000a (at 172.25.32.248@tcp) Lustre: Skipped 14 previous similar messages Lustre: meerkat-MDT0000: Client a9a37b3e-0794-bebc-95da-cc40953d0062 (at 10.7.103.145@o2ib) reconnecting Lustre: Skipped 109 previous similar messages Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs LNet: Service thread pid 6021 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 6021, comm: mdt_rdpg01_004 Call Trace: [] ? try_to_free_buffers+0x51/0xc0 [] ? jbd2_journal_try_to_free_buffers+0x48/0x150 [jbd2] [] ? bdev_try_to_free_page+0x48/0x90 [ldiskfs] [] ? shrink_page_list.clone.3+0xd0/0x650 [] ? mem_cgroup_lru_del_list+0x2b/0xb0 [] ? isolate_lru_pages.clone.0+0xd7/0x170 [] ? mem_cgroup_lru_add_list+0x37/0xc0 [] ? shrink_inactive_list+0x726/0x830 [] ? mem_cgroup_lru_del_list+0x2b/0xb0 [] ? shrink_active_list+0x297/0x370 [] ? shrink_mem_cgroup_zone+0x3ae/0x610 [] ? mem_cgroup_iter+0xfd/0x280 [] ? shrink_zone+0x63/0xb0 [] ? zone_reclaim+0x349/0x400 [] ? mempool_alloc_slab+0x15/0x20 [] ? get_page_from_freelist+0x69c/0x830 [] ? native_sched_clock+0x13/0x80 [] ? __alloc_pages_nodemask+0x113/0x8d0 [] ? blk_queue_bio+0x121/0x5d0 [] ? transfer_objects+0x5c/0x80 [] ? mempool_alloc_slab+0x15/0x20 [] ? alloc_pages_current+0xaa/0x110 [] ? __page_cache_alloc+0x87/0x90 [] ? find_or_create_page+0x4f/0xb0 [] ? __getblk+0xed/0x2a0 [] ? __breadahead+0x12/0x40 [] ? __ldiskfs_get_inode_loc+0x33e/0x3b0 [ldiskfs] [] ? ldiskfs_iget+0x86/0x800 [ldiskfs] [] ? osd_iget+0x2e/0x2c0 [osd_ldiskfs] [] ? osd_ea_fid_get+0x176/0x2c0 [osd_ldiskfs] [] ? osd_it_ea_rec+0xb45/0x1470 [osd_ldiskfs] [] ? call_filldir+0xb5/0x150 [ldiskfs] [] ? ldiskfs_readdir+0xd0/0x730 [ldiskfs] [] ? osd_ldiskfs_filldir+0x0/0x480 [osd_ldiskfs] [] ? lod_it_rec+0x21/0x90 [lod] [] ? mdd_dir_page_build+0xfc/0x210 [mdd] [] ? dt_index_walk+0x162/0x3d0 [obdclass] [] ? mdd_dir_page_build+0x0/0x210 [mdd] [] ? mdd_readpage+0x38b/0x5a0 [mdd] [] ? mdt_readpage+0x47f/0x960 [mdt] [] ? mdt_handle_common+0x647/0x16d0 [mdt] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ? mds_readpage_handle+0x15/0x20 [mdt] [] ? ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ? ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415027588.6021 LustreError: 6021:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880318035850 x1482141744205940/t0(0) o37->8c5b20e1-4e92-4313-0ab5-a06d9154e563@198.202.118.30@tcp:0/0 lens 448/440 e 2 to 0 dl 1415027620 ref 1 fl Interpret:/0/0 rc 0/0 LNet: Service thread pid 6021 completed after 200.06s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415027918/real 1415027925] req@ffff88059236cc00 x1483597775310280/t0(0) o13->meerkat-OST0036-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415027926 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Lustre: meerkat-OST0036-osc: Connection to meerkat-OST0036 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: meerkat-OST0036-osc: Connection restored to meerkat-OST0036 (at 172.25.32.243@tcp) Lustre: Skipped 3 previous similar messages Lustre: meerkat-MDT0000: Client 5e6cdb3e-b53d-8452-b1f5-c464b4dcf978 (at 10.7.103.207@o2ib) reconnecting Lustre: Skipped 40 previous similar messages LustreError: 3601:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST003a-osc: can't precreate: rc = -11 LustreError: 3601:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 8 previous similar messages LustreError: 3601:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST003a-osc: cannot precreate objects: rc = -11 LustreError: 3601:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 8 previous similar messages LustreError: 3575:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST002e-osc: can't precreate: rc = -11 LustreError: 3575:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST002e-osc: cannot precreate objects: rc = -11 LustreError: 11-0: meerkat-OST003e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: 11-0: meerkat-OST0006-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 7 previous similar messages Lustre: 3329:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415028515/real 0] req@ffff880639493c00 x1483597775734292/t0(0) o6->meerkat-OST0002-osc@172.25.32.248@tcp:28/4 lens 664/432 e 0 to 1 dl 1415028532 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3329:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 89 previous similar messages Lustre: meerkat-OST0002-osc: Connection to meerkat-OST0002 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 45 previous similar messages LustreError: 3583:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0032-osc: can't precreate: rc = -11 LustreError: 3583:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3583:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0032-osc: cannot precreate objects: rc = -11 LustreError: 3583:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message LustreError: 11-0: meerkat-OST0002-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: Skipped 5 previous similar messages Lustre: meerkat-OST001a-osc: Connection restored to meerkat-OST001a (at 172.25.32.248@tcp) Lustre: Skipped 45 previous similar messages LustreError: 11-0: meerkat-OST002e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 6 previous similar messages Lustre: MGS: Client 9e3bce36-9108-8552-1ad3-889b3db04fde (at 10.7.104.25@o2ib) reconnecting Lustre: Skipped 335 previous similar messages LustreError: 11-0: meerkat-OST0032-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: Skipped 15 previous similar messages LustreError: 11-0: meerkat-OST0016-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 2 previous similar messages LustreError: 3575:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST002e-osc: can't precreate: rc = -11 LustreError: 3575:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3575:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST002e-osc: cannot precreate objects: rc = -11 LustreError: 3575:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415029063/real 1415029124] req@ffff8806382bac00 x1483597777040564/t0(0) o6->meerkat-OST0024-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415029132 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 360 previous similar messages Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 26 previous similar messages LustreError: 11-0: meerkat-OST0004-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -114. Lustre: MGS: Client 6f83ea71-b783-c7ce-74f5-8d586c8ae0bd (at 10.7.103.117@o2ib) refused reconnection, still busy with 1 active RPCs LustreError: 13232:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880638ea5050 x1482785620421176/t0(0) o256->6f83ea71-b783-c7ce-74f5-8d586c8ae0bd@10.7.103.117@o2ib:0/0 lens 304/240 e 1 to 0 dl 1415029239 ref 1 fl Interpret:/0/0 rc 0/0 LNet: Service thread pid 3623 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 3623, comm: ll_mgs_0009 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] cfs_waitq_timedwait+0x11/0x20 [libcfs] [] target_bulk_io+0x3b8/0x910 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_prep_bulk_exp+0x6f/0x180 [ptlrpc] [] mgs_get_ir_logs+0x93d/0x130a [mgs] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [] ? lustre_swab_mgs_config_body+0x0/0x30 [ptlrpc] [] mgs_handle+0x4ea/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415029324.3623 LNet: Service thread pid 3622 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 3622, comm: ll_mgs_0008 Call Trace: [] ? lock_timer_base+0x3c/0x70 [] schedule_timeout+0x192/0x2e0 [] ? process_timeout+0x0/0x10 [] cfs_waitq_timedwait+0x11/0x20 [libcfs] [] target_bulk_io+0x3b8/0x910 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_prep_bulk_exp+0x6f/0x180 [ptlrpc] [] mgs_get_ir_logs+0x93d/0x130a [mgs] [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [] ? lustre_swab_mgs_config_body+0x0/0x30 [ptlrpc] [] mgs_handle+0x4ea/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415029326.3622 Lustre: meerkat-MDT0000: Client 554f947e-3aac-c35e-1574-5922f7741070 (at 10.7.103.227@o2ib) reconnecting Lustre: Skipped 573 previous similar messages Lustre: 12425:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-383), not sending early reply req@ffff880636c84850 x1462440074340260/t0(0) o256->391f98f8-f601-2680-6640-6c7e17b6740c@10.7.104.38@o2ib:0/0 lens 304/240 e 4 to 0 dl 1415030112 ref 2 fl Interpret:/0/0 rc 0/0 Lustre: 12425:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-383), not sending early reply req@ffff8800aed80050 x1474605177627176/t0(0) o256->f6648022-2c6c-28cc-234f-f9881317da53@10.7.103.88@o2ib:0/0 lens 304/240 e 4 to 0 dl 1415030114 ref 2 fl Interpret:/0/0 rc 0/0 LustreError: 3623:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 988+0s req@ffff880636c84850 x1462440074340260/t0(0) o256->391f98f8-f601-2680-6640-6c7e17b6740c@10.7.104.38@o2ib:0/0 lens 304/240 e 4 to 0 dl 1415030112 ref 1 fl Interpret:/0/0 rc 0/0 LNet: Service thread pid 3623 completed after 988.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). LustreError: 3622:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 988+0s req@ffff8800aed80050 x1474605177627176/t0(0) o256->f6648022-2c6c-28cc-234f-f9881317da53@10.7.103.88@o2ib:0/0 lens 304/240 e 4 to 0 dl 1415030114 ref 1 fl Interpret:/0/0 rc 0/0 LNet: Service thread pid 3622 completed after 988.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: MGS: Client 8919034e-7808-4bea-0fa3-623d5abe5ed8 (at 10.7.103.230@o2ib) reconnecting Lustre: Skipped 65 previous similar messages Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415031058/real 0] req@ffff8800ba665c00 x1483597779577568/t0(0) o6->meerkat-OST000e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415031070 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Lustre: meerkat-OST000e-osc: Connection to meerkat-OST000e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 42 previous similar messages LustreError: 3593:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0036-osc: can't precreate: rc = -11 LustreError: 3593:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 6 previous similar messages LustreError: 3593:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0036-osc: cannot precreate objects: rc = -11 LustreError: 3593:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 6 previous similar messages Lustre: meerkat-OST0024-osc: Connection restored to meerkat-OST0024 (at 172.25.32.115@tcp) Lustre: Skipped 15 previous similar messages LustreError: 11-0: meerkat-OST0036-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. Lustre: 3336:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415031349/real 0] req@ffff88009c46fc00 x1483597781020780/t0(0) o13->meerkat-OST002e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415031365 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3336:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 143 previous similar messages Lustre: meerkat-OST002e-osc: Connection to meerkat-OST002e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 3480:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0004-osc: can't precreate: rc = -11 LustreError: 3480:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3480:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0004-osc: cannot precreate objects: rc = -11 LustreError: 3480:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3395:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415031369 with bad export cookie 459840026585284451 LustreError: 11-0: meerkat-OST0004-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 4 previous similar messages Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 15 previous similar messages LustreError: 3550:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0024-osc: can't precreate: rc = -11 LustreError: 3550:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 8 previous similar messages LustreError: 3550:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0024-osc: cannot precreate objects: rc = -11 LustreError: 3550:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 8 previous similar messages Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415031510/real 0] req@ffff880337ec9000 x1483597781083732/t0(0) o6->meerkat-OST0026-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415031536 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 189 previous similar messages Lustre: meerkat-OST0026-osc: Connection to meerkat-OST0026 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 21 previous similar messages LustreError: 11-0: meerkat-OST0026-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 6 previous similar messages LustreError: 3497:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000c-osc: can't precreate: rc = -11 LustreError: 3497:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 11 previous similar messages LustreError: 3497:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000c-osc: cannot precreate objects: rc = -11 LustreError: 3497:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 11 previous similar messages LustreError: 11-0: meerkat-OST003e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 13 previous similar messages Lustre: meerkat-OST0024-osc: Connection restored to meerkat-OST0024 (at 172.25.32.115@tcp) Lustre: Skipped 37 previous similar messages LNet: Service thread pid 13034 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 13034, comm: mdt_rdpg00_019 Call Trace: [] ? try_to_free_buffers+0x51/0xc0 [] ? jbd2_journal_try_to_free_buffers+0xa7/0x150 [jbd2] [] ? bdev_try_to_free_page+0x48/0x90 [ldiskfs] [] ? blkdev_releasepage+0x20/0x50 [] ? shrink_page_list.clone.3+0xd0/0x650 [] ? mem_cgroup_lru_del_list+0x2b/0xb0 [] ? mem_cgroup_lru_del+0x39/0x40 [] ? isolate_lru_pages.clone.0+0xd7/0x170 [] ? shrink_inactive_list+0x343/0x830 [] ? shrink_active_list+0x297/0x370 [] ? shrink_mem_cgroup_zone+0x3ae/0x610 [] ? mem_cgroup_iter+0xfd/0x280 [] ? shrink_zone+0x63/0xb0 [] ? zone_reclaim+0x349/0x400 [] ? __zone_watermark_ok+0x0/0xb0 [] ? get_page_from_freelist+0x69c/0x830 [] ? __alloc_pages_nodemask+0x113/0x8d0 [] ? sptlrpc_svc_alloc_rs+0x74/0x2a0 [ptlrpc] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] [] ? reschedule_interrupt+0xe/0x20 [] ? alloc_pages_current+0xaa/0x110 [] ? cfs_alloc_page+0x17/0x20 [libcfs] [] ? mdt_readpage+0x1de/0x960 [mdt] [] ? mdt_handle_common+0x647/0x16d0 [mdt] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ? mds_readpage_handle+0x15/0x20 [mdt] [] ? ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ? ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415031836.13034 LNet: Service thread pid 13034 completed after 278.91s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: meerkat-MDT0000: Client ca5f52b2-308e-1ca0-dfd2-f55788202d8f (at 10.7.103.236@o2ib) reconnecting Lustre: Skipped 330 previous similar messages Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415031932/real 1415031948] req@ffff880628cb0400 x1483597781306524/t0(0) o6->meerkat-OST000c-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415031957 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 233 previous similar messages Lustre: meerkat-OST000c-osc: Connection to meerkat-OST000c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 28 previous similar messages LustreError: 3497:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000c-osc: can't precreate: rc = -11 LustreError: 3497:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3497:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000c-osc: cannot precreate objects: rc = -11 LustreError: 3497:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message LustreError: 11-0: meerkat-OST0026-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 10 previous similar messages Lustre: meerkat-OST001e-osc: Connection restored to meerkat-OST001e (at 172.25.32.243@tcp) Lustre: Skipped 12 previous similar messages Lustre: meerkat-MDT0000: haven't heard from client 9f01afca-fb0c-654d-79e1-c083d869f3b3 (at 198.202.118.88@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806302cf800, cur 1415032071 expire 1415031921 last 1415031844 Lustre: Skipped 36 previous similar messages Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415034213/real 0] req@ffff8804e8822000 x1483597784178796/t0(0) o6->meerkat-OST0026-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415034223 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 74 previous similar messages Lustre: meerkat-OST0026-osc: Connection to meerkat-OST0026 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 14 previous similar messages Lustre: MGS: Client 4d139ba0-d0f2-60a5-7db5-b5ff65514b2f (at 10.7.103.177@o2ib) reconnecting Lustre: Skipped 17 previous similar messages Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: Skipped 14 previous similar messages LustreError: 11-0: meerkat-OST0034-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 8 previous similar messages Lustre: MGS: Client 26c89916-1f29-398e-8f2b-8dc7ab550eff (at 10.7.103.153@o2ib) reconnecting Lustre: Skipped 4 previous similar messages Lustre: meerkat-MDT0000: Client 25f20950-7528-2c17-4c69-20c4749a0f7f (at 198.202.119.80@tcp) reconnecting Lustre: Skipped 6 previous similar messages Lustre: meerkat-MDT0000: Client 67ec6304-f534-67dc-3616-c1b0ed014964 (at 10.7.103.125@o2ib) reconnecting Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415035150/real 0] req@ffff8803cb7dd800 x1483597787704804/t0(0) o6->meerkat-OST000e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415035162 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 102 previous similar messages Lustre: meerkat-OST000e-osc: Connection to meerkat-OST000e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 13 previous similar messages LustreError: 3485:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0006-osc: can't precreate: rc = -11 LustreError: 3485:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3485:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0006-osc: cannot precreate objects: rc = -11 LustreError: 3485:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages Lustre: meerkat-MDT0000: Client d1fdcea8-c2ec-897d-75dc-7b3fe95da5a3 (at 10.7.102.119@o2ib) refused reconnection, still busy with 1 active RPCs LustreError: 11-0: meerkat-OST0014-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 4 previous similar messages Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: Skipped 13 previous similar messages Lustre: meerkat-OST0014-osc: Connection restored to meerkat-OST0014 (at 172.25.32.115@tcp) Lustre: Skipped 6 previous similar messages LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff88004951e900/0x661ae116d85d72a lrc: 3/0,0 mode: PR/PR res: [0x2000060b4:0x1b103:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0bc49e6ff expref: 6 pid: 3419 timeout: 4461390606 lvb_type: 0 LustreError: 13042:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff8803365dd850 x1474469811709528/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 2 to 0 dl 1415035556 ref 1 fl Interpret:/0/0 rc 0/0 LustreError: 13034:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff88007a1ceb40/0x661ae116dd42818 lrc: 3/0,0 mode: PR/PR res: [0x2000060b4:0x1b19e:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0bc4c0a41 expref: 6 pid: 13057 timeout: 4461572500 lvb_type: 0 Lustre: MGS: Client 1b36b0b5-dda1-1aa6-43f2-9c82e50838ab (at 10.7.102.113@o2ib) reconnecting Lustre: Skipped 20 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415037609/real 1415037616] req@ffff880314764c00 x1483597789102400/t0(0) o13->meerkat-OST0026-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415037618 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 142 previous similar messages Lustre: meerkat-OST0026-osc: Connection to meerkat-OST0026 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415037609/real 1415037618] req@ffff88009d135c00 x1483597789102420/t0(0) o13->meerkat-OST000c-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415037619 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Lustre: meerkat-OST000c-osc: Connection to meerkat-OST000c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: 3334:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415037609/real 1415037620] req@ffff88062537bc00 x1483597789102428/t0(0) o13->meerkat-OST001e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415037623 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3334:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Lustre: meerkat-OST001e-osc: Connection to meerkat-OST001e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: meerkat-OST0026-osc: Connection restored to meerkat-OST0026 (at 172.25.32.243@tcp) Lustre: Skipped 8 previous similar messages Lustre: MGS: Client 7b773074-55fc-138b-3ab5-daf011387e37 (at 10.7.103.150@o2ib) reconnecting Lustre: Skipped 39 previous similar messages Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415037778/real 1415037779] req@ffff8805b0096400 x1483597789131848/t0(0) o13->meerkat-OST000a-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415037786 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Lustre: meerkat-OST000a-osc: Connection to meerkat-OST000a (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: meerkat-OST000a-osc: Connection restored to meerkat-OST000a (at 172.25.32.248@tcp) Lustre: Skipped 8 previous similar messages Lustre: 3511:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415037805/real 1415037809] req@ffff88061fb26800 x1483597789139748/t0(0) o5->meerkat-OST0012-osc@172.25.32.248@tcp:28/4 lens 432/432 e 0 to 1 dl 1415037814 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3511:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Lustre: meerkat-OST0012-osc: Connection to meerkat-OST0012 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages LustreError: 3511:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0012-osc: can't precreate: rc = -11 LustreError: 3511:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3511:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0012-osc: cannot precreate objects: rc = -11 LustreError: 3511:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message Lustre: meerkat-OST0012-osc: Connection restored to meerkat-OST0012 (at 172.25.32.248@tcp) Lustre: Skipped 6 previous similar messages LustreError: 11-0: meerkat-OST0032-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: Skipped 8 previous similar messages Lustre: meerkat-OST000a-osc: Connection restored to meerkat-OST000a (at 172.25.32.248@tcp) Lustre: Skipped 1 previous similar message Lustre: meerkat-MDT0000: Client d0ef9f68-e5bb-e83c-70bd-911dbc0a41d0 (at 10.7.104.10@o2ib) reconnecting Lustre: Skipped 20 previous similar messages Lustre: 3330:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415038223/real 1415038227] req@ffff880639029800 x1483597789400256/t0(0) o13->meerkat-OST0004-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415038230 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3330:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Lustre: meerkat-OST0004-osc: Connection to meerkat-OST0004 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: meerkat-OST000e-osc: Connection restored to meerkat-OST000e (at 172.25.32.243@tcp) Lustre: Skipped 1 previous similar message LustreError: 11-0: meerkat-OST0006-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415038347/real 1415038354] req@ffff88006fdda800 x1483597789429896/t0(0) o13->meerkat-OST003c-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415038356 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Lustre: meerkat-OST003c-osc: Connection to meerkat-OST003c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 12 previous similar messages Lustre: meerkat-OST003c-osc: Connection restored to meerkat-OST003c (at 172.25.32.115@tcp) Lustre: Skipped 12 previous similar messages Lustre: meerkat-MDT0000: Client d51b7bc1-cc6b-7501-a15a-b83feb3402ca (at 10.7.103.178@o2ib) reconnecting Lustre: Skipped 53 previous similar messages Lustre: MGS: Client 03283624-9881-7b2e-fb7d-d18bdf93c403 (at 10.7.103.178@o2ib) reconnecting Lustre: Skipped 3 previous similar messages Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415039173/real 1415039181] req@ffff8806390be400 x1483597789621432/t0(0) o13->meerkat-OST001c-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415039183 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Lustre: meerkat-OST001c-osc: Connection to meerkat-OST001c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages LustreError: 11-0: meerkat-OST001c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST002e-osc: Connection restored to meerkat-OST002e (at 172.25.32.243@tcp) Lustre: Skipped 3 previous similar messages Lustre: meerkat-OST001c-osc: Connection restored to meerkat-OST001c (at 172.25.32.115@tcp) Lustre: Skipped 3 previous similar messages Lustre: 13059:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415040614/real 1415040617] req@ffff880094c97c00 x1483597791221564/t0(0) o104->meerkat-MDT0000@192.168.230.53@tcp:15/16 lens 296/224 e 0 to 1 dl 1415040621 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 13059:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Lustre: MGS: Client 257e611b-ab9d-f79f-e12a-68d2d6c0b1ab (at 10.7.103.253@o2ib) reconnecting Lustre: Skipped 18 previous similar messages Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415041028/real 1415041036] req@ffff88009cef5400 x1483597792538856/t0(0) o13->meerkat-OST0024-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415041039 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: meerkat-OST0024-osc: Connection to meerkat-OST0024 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: meerkat-OST002e-osc: Connection restored to meerkat-OST002e (at 172.25.32.243@tcp) LustreError: 11-0: meerkat-OST0036-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST0036-osc: Connection restored to meerkat-OST0036 (at 172.25.32.243@tcp) Lustre: Skipped 12 previous similar messages Lustre: MGS: Client 6c923fb5-95df-59c8-b335-b2a058cf277a (at 10.7.104.30@o2ib) reconnecting Lustre: Skipped 77 previous similar messages Lustre: MGS: Client 7736135f-e875-47f2-48ca-2841ea9fa424 (at 10.7.103.92@o2ib) reconnecting Lustre: Skipped 4 previous similar messages Lustre: meerkat-MDT0000: Client f8ee92d4-3bc1-7fca-234a-43af75dc6463 (at 10.7.101.153@o2ib) reconnecting Lustre: Skipped 45 previous similar messages Lustre: meerkat-MDT0000: Client b0de5975-d7e7-33f7-66b4-296c80ca90e9 (at 10.7.104.21@o2ib) reconnecting Lustre: Skipped 15 previous similar messages Lustre: meerkat-MDT0000: Client df14d7af-da48-08dc-6e9a-fdf12c634f07 (at 10.7.103.212@o2ib) reconnecting Lustre: Skipped 7 previous similar messages Lustre: MGS: Client 129d2a15-cea6-5690-7239-ee557bbfe5a2 (at 10.7.104.31@o2ib) reconnecting Lustre: Skipped 5 previous similar messages Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415045714/real 0] req@ffff880628103800 x1483597796903892/t0(0) o13->meerkat-OST0034-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415045721 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 13 previous similar messages Lustre: 3336:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415045714/real 1415045720] req@ffff880639271c00 x1483597796903880/t0(0) o13->meerkat-OST0014-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415045721 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3336:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Lustre: meerkat-OST0014-osc: Connection to meerkat-OST0014 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages Lustre: meerkat-MDT0000: Client 801b6f4b-9590-93a2-a367-05c2b1ee007d (at 10.7.103.90@o2ib) reconnecting Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415045714/real 0] req@ffff88063a271000 x1483597796903884/t0(0) o13->meerkat-OST002c-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415045721 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Lustre: meerkat-OST002c-osc: Connection to meerkat-OST002c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: MGS: Client 2d5a94d1-9feb-ce7e-4ca1-f5830ab6a290 (at 10.7.103.128@o2ib) reconnecting Lustre: Skipped 4 previous similar messages Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415045721/real 0] req@ffff880308b04400 x1483597796905248/t0(0) o8->meerkat-OST0016-osc@172.25.32.243@tcp:28/4 lens 400/544 e 0 to 1 dl 1415045727 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Lustre: meerkat-OST0004-osc: Connection to meerkat-OST0004 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: meerkat-OST002e-osc: Connection restored to meerkat-OST002e (at 172.25.32.243@tcp) LustreError: 11-0: meerkat-OST0004-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST0026-osc: Connection restored to meerkat-OST0026 (at 172.25.32.243@tcp) Lustre: Skipped 4 previous similar messages Lustre: MGS: Client 50c8f841-fe78-dc62-ac9c-ee35b284b6fb (at 10.7.102.116@o2ib) reconnecting Lustre: Skipped 137 previous similar messages Lustre: meerkat-MDT0000: Client f81c7a48-3468-23e3-7c19-73a88f2b85a5 (at 10.7.103.120@o2ib) reconnecting Lustre: Skipped 25 previous similar messages Lustre: MGS: Client e3048a90-2d03-0c63-6492-695b16b9312e (at 10.7.103.229@o2ib) reconnecting Lustre: MGS: Client b0201db9-625e-d627-0e51-d3890f025be0 (at 198.202.118.109@tcp) reconnecting Lustre: MGS: Client 6aafa7af-37d4-3ea6-9507-bcc84c493305 (at 10.7.101.213@o2ib) reconnecting Lustre: Skipped 3 previous similar messages Lustre: 13056:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415048404/real 1415048406] req@ffff88062270b800 x1483597800111036/t0(0) o104->meerkat-MDT0000@192.168.230.53@tcp:15/16 lens 296/224 e 0 to 1 dl 1415048412 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 13056:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 Lustre: meerkat-MDT0000: Client 78edccf5-1559-fbf5-cf55-e599ac7354a4 (at 10.7.101.135@o2ib) reconnecting Lustre: Skipped 6 previous similar messages Lustre: meerkat-MDT0000: Client f81c7a48-3468-23e3-7c19-73a88f2b85a5 (at 10.7.103.120@o2ib) reconnecting Lustre: Skipped 2 previous similar messages Lustre: 3533:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415048407/real 1415048414] req@ffff8803866a3400 x1483597800111536/t0(0) o5->meerkat-OST001c-osc@172.25.32.115@tcp:28/4 lens 432/432 e 0 to 1 dl 1415048418 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: meerkat-OST001c-osc: Connection to meerkat-OST001c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete LustreError: 3533:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST001c-osc: can't precreate: rc = -11 LustreError: 3533:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 3 previous similar messages LustreError: 3533:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST001c-osc: cannot precreate objects: rc = -11 LustreError: 3533:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 3 previous similar messages Lustre: MGS: Client 9c5220fa-99f8-5f3d-8e04-34c5a450a101 (at 198.202.119.94@tcp) refused reconnection, still busy with 1 active RPCs LustreError: 3397:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415048418 with bad export cookie 459840026851165444 LustreError: 3397:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415048418 with bad export cookie 459840026851165444 LustreError: 11-0: meerkat-OST001c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. Lustre: MGS: Client 09c38d0d-de46-8de0-2391-3b45d1e7e6fb (at 10.7.101.113@o2ib) reconnecting Lustre: Skipped 66 previous similar messages Lustre: MGS: Client a5035aa2-1862-18de-4571-614b90431111 (at 198.202.119.76@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-OST001c-osc: Connection restored to meerkat-OST001c (at 172.25.32.115@tcp) Lustre: Skipped 9 previous similar messages Lustre: MGS: Client a5035aa2-1862-18de-4571-614b90431111 (at 198.202.119.76@tcp) reconnecting Lustre: Skipped 29 previous similar messages Lustre: 13057:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415048649/real 1415048649] req@ffff88039a7f5400 x1483597800427468/t0(0) o104->meerkat-MDT0000@198.202.118.54@tcp:15/16 lens 296/224 e 0 to 1 dl 1415048656 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 LustreError: 138-a: meerkat-MDT0000: A client on nid 198.202.118.54@tcp was evicted due to a lock blocking callback time out: rc -107 Lustre: meerkat-MDT0000: Client bb2497e4-25eb-0d39-bd91-8fce8e6ac800 (at 10.7.102.235@o2ib) reconnecting Lustre: Skipped 1 previous similar message LustreError: 3630:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 198.202.118.54@tcp arrived at 1415048658 with bad export cookie 459840023463757574 Lustre: meerkat-MDT0000: Client 2580820c-ce4a-218c-fa18-544aea2f5059 (at 192.168.230.51@tcp) reconnecting Lustre: meerkat-MDT0000: Client 2580820c-ce4a-218c-fa18-544aea2f5059 (at 192.168.230.51@tcp) refused reconnection, still busy with 2 active RPCs Lustre: meerkat-MDT0000: Client 2580820c-ce4a-218c-fa18-544aea2f5059 (at 192.168.230.51@tcp) reconnecting Lustre: 3334:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415058582/real 1415058587] req@ffff880639332000 x1483597810291784/t0(0) o6->meerkat-OST0034-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415058590 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415058582/real 1415058587] req@ffff8806394b3000 x1483597810291752/t0(0) o6->meerkat-OST0034-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415058590 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Lustre: meerkat-OST000c-osc: Connection to meerkat-OST000c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete LustreError: 3497:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000c-osc: can't precreate: rc = -11 LustreError: 3497:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000c-osc: cannot precreate objects: rc = -11 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415058582/real 1415058587] req@ffff88007ecb6c00 x1483597810294452/t0(0) o6->meerkat-OST001e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415058591 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Lustre: meerkat-OST0014-osc: Connection to meerkat-OST0014 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: 3537:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415058583/real 1415058587] req@ffff8800b0057800 x1483597810299896/t0(0) o5->meerkat-OST001e-osc@172.25.32.243@tcp:28/4 lens 432/432 e 0 to 1 dl 1415058592 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3537:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages LustreError: 3537:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST001e-osc: can't precreate: rc = -11 LustreError: 3537:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST001e-osc: cannot precreate objects: rc = -11 Lustre: meerkat-OST0016-osc: Connection to meerkat-OST0016 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: meerkat-OST001e-osc: Connection restored to meerkat-OST001e (at 172.25.32.243@tcp) Lustre: meerkat-MDT0000: Client f8a37406-5da9-98a2-8c68-a139bd1b8714 (at 10.7.103.238@o2ib) reconnecting Lustre: MGS: Client ed45086d-2338-73b9-f3b8-cdba30566ffb (at 10.7.104.44@o2ib) reconnecting Lustre: meerkat-MDT0000: Client bc9820fe-c363-1407-1559-af9a0644c64f (at 10.7.103.217@o2ib) reconnecting Lustre: Skipped 66 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415059335/real 0] req@ffff88033810a800 x1483597812654228/t0(0) o6->meerkat-OST003e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415059348 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Lustre: meerkat-OST003e-osc: Connection to meerkat-OST003e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415059335/real 1415059342] req@ffff880639049c00 x1483597812653652/t0(0) o6->meerkat-OST002c-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415059348 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415059335/real 1415059348] req@ffff880312baf800 x1483597812653880/t0(0) o6->meerkat-OST003e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415059348 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415059335/real 0] req@ffff88042aca0800 x1483597812654288/t0(0) o6->meerkat-OST0034-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415059350 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: MGS: Client 5e53eda6-ba3d-2c82-177a-72218afd0fa8 (at 10.7.103.93@o2ib) reconnecting Lustre: Skipped 3 previous similar messages Lustre: meerkat-OST0006-osc: Connection to meerkat-OST0006 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: meerkat-OST002c-osc: Connection restored to meerkat-OST002c (at 172.25.32.115@tcp) Lustre: Skipped 4 previous similar messages LustreError: 11-0: meerkat-OST000c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: Skipped 2 previous similar messages Lustre: meerkat-MDT0000: Client 153f7fde-699f-946d-90c0-af8231c37b4b (at 10.7.103.94@o2ib) reconnecting Lustre: Skipped 20 previous similar messages Lustre: 6375:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415059757/real 1415059758] req@ffff880317760400 x1483597812703028/t0(0) o104->meerkat-MDT0000@192.168.230.53@tcp:15/16 lens 296/224 e 0 to 1 dl 1415059768 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 6375:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3630:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415059768 with bad export cookie 459840027438824761 Lustre: meerkat-MDT0000: Client e016f72b-cc4a-cee3-5faa-cdb0f5a24764 (at 10.7.102.192@o2ib) reconnecting Lustre: Skipped 11 previous similar messages Lustre: meerkat-MDT0000: Client d1fdcea8-c2ec-897d-75dc-7b3fe95da5a3 (at 10.7.102.119@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 677d7d3a-37ea-920c-c096-20a623186fa9 (at 10.7.103.114@o2ib) reconnecting Lustre: Skipped 21 previous similar messages LustreError: 13042:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff8800acc2f480/0x661ae11b70ceedf lrc: 3/0,0 mode: PR/PR res: [0x2000060bc:0x5508:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0be13beea expref: 36 pid: 3419 timeout: 4486388246 lvb_type: 0 LustreError: 13042:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff8800a3513800 x1474469876758132/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 0 to 0 dl 1415060499 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415060429/real 1415060430] req@ffff88031321c800 x1483597813199484/t0(0) o13->meerkat-OST001c-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415060438 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: meerkat-OST001c-osc: Connection to meerkat-OST001c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415060429/real 1415060430] req@ffff880636df9400 x1483597813199476/t0(0) o13->meerkat-OST0026-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415060438 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: meerkat-OST0026-osc: Connection to meerkat-OST0026 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: 3336:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415060432/real 0] req@ffff8800b8211400 x1483597813201928/t0(0) o6->meerkat-OST0034-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415060440 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: MGS: Client 340f1fa1-9370-bc71-a6e3-834f520374a2 (at 10.7.103.181@o2ib) reconnecting Lustre: Skipped 7 previous similar messages Lustre: meerkat-OST0014-osc: Connection to meerkat-OST0014 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 5 previous similar messages Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415060432/real 0] req@ffff8800aa611400 x1483597813202612/t0(0) o6->meerkat-OST000e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415060443 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 82 previous similar messages Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415060440/real 0] req@ffff8801efc24800 x1483597813206052/t0(0) o8->meerkat-OST0036-osc@172.25.32.243@tcp:28/4 lens 400/544 e 0 to 1 dl 1415060447 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 52 previous similar messages Lustre: meerkat-MDT0000: Client 06bc4379-10ca-76ad-cd98-1d1013f1b911 (at 10.7.103.252@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: 18052:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415060449/real 1415060451] req@ffff8802ebaad800 x1483597813206540/t0(0) o104->meerkat-MDT0000@10.7.103.252@o2ib:15/16 lens 296/224 e 0 to 1 dl 1415060456 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 18052:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 2 previous similar messages Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: meerkat-MDT0000: Client 7a7ab9a5-c8e6-abb6-2f14-1ebe9b1fdab3 (at 10.7.104.32@o2ib) reconnecting Lustre: Skipped 218 previous similar messages Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415060911/real 1415060921] req@ffff8800a7a9fc00 x1483597814687272/t0(0) o6->meerkat-OST000c-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415060925 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: meerkat-OST000c-osc: Connection to meerkat-OST000c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 7 previous similar messages Lustre: meerkat-OST003c-osc: Connection to meerkat-OST003c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: Skipped 13 previous similar messages LustreError: 11-0: meerkat-OST0006-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 2 previous similar messages Lustre: meerkat-OST0024-osc: Connection restored to meerkat-OST0024 (at 172.25.32.115@tcp) Lustre: Skipped 6 previous similar messages Lustre: meerkat-MDT0000: Client 22670471-b57b-0d1a-cd38-f4f39735b005 (at 10.7.103.146@o2ib) reconnecting Lustre: Skipped 34 previous similar messages Lustre: meerkat-MDT0000: Client ddd3605f-043b-06f5-bf38-57f0e4664f5a (at 10.7.104.50@o2ib) reconnecting Lustre: Skipped 8 previous similar messages Lustre: 3334:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415061740/real 0] req@ffff8802f358ec00 x1483597815529932/t0(0) o13->meerkat-OST002e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415061765 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3334:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Lustre: meerkat-OST002e-osc: Connection to meerkat-OST002e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 7 previous similar messages Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: Skipped 1 previous similar message LustreError: 11-0: meerkat-OST0004-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 5 previous similar messages LustreError: 13039:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff8801d4577b40/0x661ae11b99d8bc9 lrc: 3/0,0 mode: PR/PR res: [0x20000610c:0x1f503:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0be51fa34 expref: 81 pid: 13055 timeout: 4488011128 lvb_type: 0 LustreError: 13055:0:(client.c:1048:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880012cae000 x1483597815620664/t0(0) o104->meerkat-MDT0000@192.168.230.53@tcp:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 LustreError: 13055:0:(ldlm_lockd.c:709:ldlm_handle_ast_error()) ### client (nid 192.168.230.53@tcp) returned 0 from blocking AST ns: mdt-meerkat-MDT0000_UUID lock: ffff88008b0e2b40/0x661ae11b99d8341 lrc: 1/0,0 mode: --/PR res: [0x20000649f:0xb6a1:0x0].0 bits 0x1b rrc: 2 type: IBT flags: 0xa01000000020 nid: 192.168.230.53@tcp remote: 0x7185dbd0be51f87b expref: 7 pid: 16128 timeout: 4488112000 lvb_type: 0 LustreError: 13039:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880013e22c00 x1474469883597552/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 0 to 0 dl 1415062090 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415062066/real 0] req@ffff880624f77c00 x1483597815688932/t0(0) o6->meerkat-OST0016-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415062081 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 111 previous similar messages Lustre: meerkat-OST0016-osc: Connection to meerkat-OST0016 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST002e-osc: Connection to meerkat-OST002e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: meerkat-OST0024-osc: Connection restored to meerkat-OST0024 (at 172.25.32.115@tcp) Lustre: Skipped 9 previous similar messages Lustre: MGS: Client 205ae146-0f5e-d098-32de-06175bb2025c (at 10.7.103.146@o2ib) refused reconnection, still busy with 1 active RPCs LustreError: 13957:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8800a6ac1050 x1474741105170980/t0(0) o256->205ae146-0f5e-d098-32de-06175bb2025c@10.7.103.146@o2ib:0/0 lens 304/240 e 0 to 0 dl 1415062106 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415062091/real 1415062103] req@ffff880625ef2c00 x1483597815699616/t0(0) o6->meerkat-OST0016-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415062106 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Lustre: meerkat-OST0016-osc: Connection to meerkat-OST0016 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: meerkat-OST0016-osc: Connection restored to meerkat-OST0016 (at 172.25.32.243@tcp) Lustre: Skipped 2 previous similar messages Lustre: MGS: Client 6c6b7be8-a065-b42c-deb7-552637a6440d (at 10.7.103.141@o2ib) reconnecting Lustre: Skipped 136 previous similar messages Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415062853/real 0] req@ffff880624f79000 x1483597815917792/t0(0) o6->meerkat-OST0024-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415062865 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Lustre: meerkat-OST0024-osc: Connection to meerkat-OST0024 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: meerkat-OST0004-osc: Connection to meerkat-OST0004 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 13 previous similar messages Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415062853/real 0] req@ffff880625804c00 x1483597815917436/t0(0) o6->meerkat-OST002e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415062867 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 100 previous similar messages LustreError: 11-0: meerkat-OST000e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 9 previous similar messages Lustre: meerkat-OST002e-osc: Connection restored to meerkat-OST002e (at 172.25.32.243@tcp) Lustre: Skipped 1 previous similar message LustreError: 11-0: meerkat-OST000c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 2 previous similar messages Lustre: meerkat-OST0026-osc: Connection restored to meerkat-OST0026 (at 172.25.32.243@tcp) Lustre: Skipped 11 previous similar messages Lustre: 3329:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415063040/real 1415063043] req@ffff8803c7993800 x1483597815949468/t0(0) o13->meerkat-OST0032-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415063047 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3329:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 19 previous similar messages Lustre: meerkat-OST0032-osc: Connection to meerkat-OST0032 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 11-0: meerkat-OST0002-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST003a-osc: Connection restored to meerkat-OST003a (at 172.25.32.248@tcp) Lustre: Skipped 3 previous similar messages Lustre: meerkat-OST0002-osc: Connection restored to meerkat-OST0002 (at 172.25.32.248@tcp) Lustre: Skipped 1 previous similar message Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415063653/real 1415063656] req@ffff880636d5b000 x1483597816061004/t0(0) o13->meerkat-OST0034-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415063660 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 70 previous similar messages Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 7 previous similar messages Lustre: meerkat-MDT0000: Client fd825eb2-1223-8979-93ea-5a1cfbdf1776 (at 10.7.103.251@o2ib) reconnecting Lustre: Skipped 87 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 Lustre: 3427:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415063684/real 1415063687] req@ffff88063824f800 x1483597816063460/t0(0) o104->meerkat-MDT0000@198.202.118.83@tcp:15/16 lens 296/224 e 0 to 1 dl 1415063691 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3427:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 110 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 198.202.118.83@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 5217:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 198.202.118.83@tcp arrived at 1415063699 with bad export cookie 459840026370725358 LustreError: 5217:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415063705 with bad export cookie 459840028121764991 Lustre: MGS: Client 04d56312-4342-e8a8-7f89-27ad319f3f1d (at 10.7.104.17@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: Skipped 5 previous similar messages LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff880097477480/0x661ae11bb3966be lrc: 3/0,0 mode: PR/PR res: [0x20000610c:0x1f744:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0be652e4f expref: 40 pid: 3726 timeout: 4489904438 lvb_type: 0 LustreError: 13042:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff880311ef0000 x1474469886634120/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 0 to 0 dl 1415063955 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: 13042:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (113:16s); client may timeout. req@ffff880311ef0000 x1474469886634120/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/408 e 0 to 0 dl 1415063955 ref 1 fl Complete:/0/0 rc -107/-107 Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) reconnecting Lustre: Skipped 708 previous similar messages Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs LNet: Service thread pid 12073 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 12073, comm: mdt_rdpg01_007 Call Trace: [] ? try_to_free_buffers+0x51/0xc0 [] ? jbd2_journal_try_to_free_buffers+0xa7/0x150 [jbd2] [] ? apic_timer_interrupt+0xe/0x20 [] ? bdev_try_to_free_page+0x48/0x90 [ldiskfs] [] ? unlock_page+0x27/0x30 [] ? shrink_page_list.clone.3+0xd0/0x650 [] ? mem_cgroup_lru_del_list+0x2b/0xb0 [] ? isolate_lru_pages.clone.0+0xd7/0x170 [] ? apic_timer_interrupt+0xe/0x20 [] ? shrink_inactive_list+0x3dc/0x830 [] ? mem_cgroup_lru_del_list+0x2b/0xb0 [] ? shrink_active_list+0x1bd/0x370 [] ? shrink_mem_cgroup_zone+0x3ae/0x610 [] ? mem_cgroup_iter+0xfd/0x280 [] ? shrink_zone+0x63/0xb0 [] ? zone_reclaim+0x349/0x400 [] ? mempool_alloc_slab+0x15/0x20 [] ? get_page_from_freelist+0x69c/0x830 [] ? native_sched_clock+0x13/0x80 [] ? __alloc_pages_nodemask+0x113/0x8d0 [] ? blk_queue_bio+0x121/0x5d0 [] ? mempool_alloc_slab+0x15/0x20 [] ? alloc_pages_current+0xaa/0x110 [] ? __page_cache_alloc+0x87/0x90 [] ? find_or_create_page+0x4f/0xb0 [] ? __getblk+0xed/0x2a0 [] ? __breadahead+0x12/0x40 [] ? __ldiskfs_get_inode_loc+0x33e/0x3b0 [ldiskfs] [] ? ldiskfs_get_inode_loc+0x1c/0x20 [ldiskfs] [] ? ldiskfs_xattr_get+0x7c/0x330 [ldiskfs] [] ? iget_locked+0x49/0x170 [] ? ldiskfs_iget+0x37/0x800 [ldiskfs] [] ? ldiskfs_xattr_trusted_get+0x2b/0x30 [ldiskfs] [] ? generic_getxattr+0x87/0x90 [] ? osd_get_lma+0x5e/0x160 [osd_ldiskfs] [] ? osd_iget+0x178/0x2c0 [osd_ldiskfs] [] ? osd_ea_fid_get+0x19b/0x2c0 [osd_ldiskfs] [] ? osd_remote_fid+0x9a/0x280 [osd_ldiskfs] [] ? zone_statistics+0x70/0xc0 [] ? osd_it_ea_rec+0xb45/0x1470 [osd_ldiskfs] [] ? call_filldir+0xb5/0x150 [ldiskfs] [] ? ldiskfs_readdir+0xd0/0x730 [ldiskfs] [] ? osd_ldiskfs_filldir+0x0/0x480 [osd_ldiskfs] [] ? lod_it_rec+0x21/0x90 [lod] [] ? mdd_dir_page_build+0xfc/0x210 [mdd] [] ? dt_index_walk+0x162/0x3d0 [obdclass] [] ? mdd_dir_page_build+0x0/0x210 [mdd] [] ? mdd_readpage+0x38b/0x5a0 [mdd] [] ? mdt_readpage+0x47f/0x960 [mdt] [] ? mdt_handle_common+0x647/0x16d0 [mdt] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ? mds_readpage_handle+0x15/0x20 [mdt] [] ? ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ? ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415064926.12073 LustreError: 12073:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff8800a802c800 x1482141757710840/t0(0) o37->8c5b20e1-4e92-4313-0ab5-a06d9154e563@198.202.118.30@tcp:0/0 lens 448/440 e 2 to 0 dl 1415064958 ref 1 fl Interpret:/0/0 rc 0/0 LNet: Service thread pid 12073 completed after 207.57s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415064944/real 1415064945] req@ffff880624ea7400 x1483597817084984/t0(0) o13->meerkat-OST0016-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415064954 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Lustre: meerkat-OST0016-osc: Connection to meerkat-OST0016 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST001e-osc: Connection to meerkat-OST001e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415064949/real 1415064949] req@ffff8805cb029400 x1483597817085440/t0(0) o13->meerkat-OST000e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415064959 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Lustre: meerkat-OST0006-osc: Connection to meerkat-OST0006 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: Skipped 15 previous similar messages LustreError: 11-0: meerkat-OST002c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 4 previous similar messages Lustre: meerkat-OST002c-osc: Connection restored to meerkat-OST002c (at 172.25.32.115@tcp) Lustre: Skipped 8 previous similar messages Lustre: 6377:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415065084/real 1415065084] req@ffff88000f7a0c00 x1483597817414456/t0(0) o104->meerkat-MDT0000@10.7.103.252@o2ib:15/16 lens 296/224 e 0 to 1 dl 1415065091 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 6377:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Lustre: meerkat-OST001c-osc: Connection to meerkat-OST001c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415065094/real 0] req@ffff8803c793b400 x1483597817417976/t0(0) o6->meerkat-OST000e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415065107 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 119 previous similar messages Lustre: meerkat-MDT0000: Client 426a4ee4-22a1-0043-48af-90853c7ea3f9 (at 10.7.101.20@o2ib) refused reconnection, still busy with 1 active RPCs LustreError: 11-0: meerkat-OST003e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 3 previous similar messages Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: Skipped 3 previous similar messages Lustre: meerkat-OST001c-osc: Connection restored to meerkat-OST001c (at 172.25.32.115@tcp) Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415065137/real 0] req@ffff8800afe25c00 x1483597817457628/t0(0) o6->meerkat-OST0014-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415065153 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Lustre: meerkat-OST0014-osc: Connection to meerkat-OST0014 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 11-0: meerkat-OST0004-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 14 previous similar messages Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: Skipped 14 previous similar messages Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 6 previous similar messages Lustre: meerkat-MDT0000: Client ddd3605f-043b-06f5-bf38-57f0e4664f5a (at 10.7.104.50@o2ib) reconnecting Lustre: Skipped 310 previous similar messages Lustre: 3331:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415065484/real 1415065492] req@ffff88009e1c6400 x1483597817600428/t0(0) o13->meerkat-OST001c-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415065502 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3331:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 142 previous similar messages Lustre: meerkat-OST001c-osc: Connection to meerkat-OST001c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 11-0: meerkat-OST001c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 8 previous similar messages Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: Skipped 8 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.61@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 17494:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.61@tcp arrived at 1415065535 with bad export cookie 459840026102189265 Lustre: meerkat-OST000c-osc: Connection to meerkat-OST000c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 12 previous similar messages Lustre: meerkat-OST0014-osc: Connection restored to meerkat-OST0014 (at 172.25.32.115@tcp) LustreError: 11-0: meerkat-OST0034-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 11 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 198.202.118.30@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3632:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 198.202.118.30@tcp arrived at 1415065590 with bad export cookie 459840025472430037 Lustre: meerkat-OST0002-osc: Connection restored to meerkat-OST0002 (at 172.25.32.248@tcp) Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST000c-osc: Connection to meerkat-OST000c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 11 previous similar messages LustreError: 11-0: meerkat-OST000c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 2 previous similar messages Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415066012/real 1415066021] req@ffff880017214c00 x1483597818275868/t0(0) o6->meerkat-OST0016-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415066035 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 190 previous similar messages Lustre: meerkat-OST0016-osc: Connection to meerkat-OST0016 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 12 previous similar messages LustreError: 11-0: meerkat-OST001c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 10 previous similar messages Lustre: meerkat-OST0024-osc: Connection restored to meerkat-OST0024 (at 172.25.32.115@tcp) Lustre: Skipped 20 previous similar messages Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) reconnecting Lustre: Skipped 110 previous similar messages Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: MGS: Client 91054467-dd7c-153e-d3c9-b514bed2c0dc (at 10.7.104.46@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: Skipped 4 previous similar messages LustreError: 3411:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88031510d050 x1474566791464252/t0(0) o256->91054467-dd7c-153e-d3c9-b514bed2c0dc@10.7.104.46@o2ib:0/0 lens 304/240 e 1 to 0 dl 1415066280 ref 1 fl Interpret:/0/0 rc 0/0 LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 106s: evicting client at 198.202.118.30@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff880012a47d80/0x661ae11bef2dcc7 lrc: 3/0,0 mode: PR/PR res: [0x200001cbd:0x4bf7:0x0].0 bits 0x1b rrc: 2 type: IBT flags: 0x200000000020 nid: 198.202.118.30@tcp remote: 0xeb38bc19267d595f expref: 68 pid: 3423 timeout: 4492237293 lvb_type: 0 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415067264/real 1415067272] req@ffff8803e6c77400 x1483597820377688/t0(0) o13->meerkat-OST0024-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415067272 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Lustre: meerkat-OST0024-osc: Connection to meerkat-OST0024 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 5 previous similar messages LustreError: 3485:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0006-osc: can't precreate: rc = -11 LustreError: 3485:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0006-osc: cannot precreate objects: rc = -11 Lustre: meerkat-OST0024-osc: Connection restored to meerkat-OST0024 (at 172.25.32.115@tcp) LustreError: 11-0: meerkat-OST0026-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 4 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415067392/real 1415067397] req@ffff880639434000 x1483597820464152/t0(0) o13->meerkat-OST000e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415067400 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Lustre: meerkat-OST000e-osc: Connection to meerkat-OST000e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: meerkat-MDT0000: Client 6c211c26-a1f8-457a-bb56-e9fdfe251154 (at 10.7.103.123@o2ib) reconnecting Lustre: Skipped 105 previous similar messages LustreError: 3611:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST003e-osc: can't precreate: rc = -11 LustreError: 3611:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST003e-osc: cannot precreate objects: rc = -11 LustreError: 11-0: meerkat-OST000e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST0016-osc: Connection restored to meerkat-OST0016 (at 172.25.32.243@tcp) Lustre: Skipped 15 previous similar messages Lustre: meerkat-MDT0000: Client 0d91792a-103f-eb01-3cbd-8a27ee8d05e1 (at 10.7.100.177@o2ib) refused reconnection, still busy with 1 active RPCs LustreError: 11-0: meerkat-OST000a-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: Skipped 3 previous similar messages Lustre: MGS: Client 5531fea2-771f-bf20-2d4d-1c8dd91f22a5 (at 10.7.100.180@o2ib) reconnecting Lustre: Skipped 286 previous similar messages Lustre: meerkat-OST000a-osc: Connection restored to meerkat-OST000a (at 172.25.32.248@tcp) Lustre: Skipped 20 previous similar messages Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415067498/real 0] req@ffff88062515f400 x1483597820527416/t0(0) o6->meerkat-OST002e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415067508 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 106 previous similar messages Lustre: meerkat-OST002e-osc: Connection to meerkat-OST002e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 21 previous similar messages LustreError: 11-0: meerkat-OST003c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. Lustre: 3334:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415067733/real 1415067735] req@ffff880629f5a800 x1483597820658512/t0(0) o13->meerkat-OST002a-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415067745 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3334:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Lustre: meerkat-OST002a-osc: Connection to meerkat-OST002a (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: meerkat-MDT0000: Client 3ca2cd97-c452-fe95-a76c-42b7ef700df0 (at 10.7.103.102@o2ib) reconnecting Lustre: Skipped 123 previous similar messages Lustre: meerkat-OST002a-osc: Connection restored to meerkat-OST002a (at 172.25.32.248@tcp) Lustre: Skipped 2 previous similar messages LustreError: 11-0: meerkat-OST0036-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. Lustre: 6375:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415068423/real 1415068425] req@ffff8803380ef400 x1483597820896776/t0(0) o104->meerkat-MDT0000@192.168.230.53@tcp:15/16 lens 296/224 e 0 to 1 dl 1415068431 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 6375:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 Lustre: MGS: Client 3e40fb73-466b-b7fd-63d7-46c10dc13c02 (at 10.7.103.210@o2ib) reconnecting Lustre: Skipped 8 previous similar messages Lustre: meerkat-OST0024-osc: Connection to meerkat-OST0024 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 12 previous similar messages LustreError: 3397:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415068435 with bad export cookie 459840028148044412 Lustre: meerkat-OST0014-osc: Connection restored to meerkat-OST0014 (at 172.25.32.115@tcp) Lustre: Skipped 12 previous similar messages LustreError: 11-0: meerkat-OST002c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 4 previous similar messages LustreError: 3550:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0024-osc: can't precreate: rc = -11 LustreError: 3550:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0024-osc: cannot precreate objects: rc = -11 LustreError: 3606:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST003c-osc: can't precreate: rc = -11 LustreError: 3606:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST003c-osc: cannot precreate objects: rc = -11 LustreError: 3515:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0014-osc: can't precreate: rc = -11 LustreError: 3515:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0014-osc: cannot precreate objects: rc = -11 LustreError: 11-0: meerkat-OST0014-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: meerkat-MDT0000: Client f8ee92d4-3bc1-7fca-234a-43af75dc6463 (at 10.7.101.153@o2ib) refused reconnection, still busy with 2 active RPCs LustreError: 3568:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST002c-osc: can't precreate: rc = -11 LustreError: 3568:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST002c-osc: cannot precreate objects: rc = -11 LustreError: 3497:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000c-osc: can't precreate: rc = -11 LustreError: 3497:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000c-osc: cannot precreate objects: rc = -11 LustreError: 138-a: meerkat-MDT0000: A client on nid 198.202.118.109@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3480:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0004-osc: can't precreate: rc = -11 LustreError: 3480:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3480:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0004-osc: cannot precreate objects: rc = -11 LustreError: 3480:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages LustreError: 17506:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 198.202.118.109@tcp arrived at 1415068720 with bad export cookie 459840023463757056 LustreError: 11-0: meerkat-OST0034-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 5 previous similar messages Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415069172/real 1415069174] req@ffff8800b598cc00 x1483597821344280/t0(0) o13->meerkat-OST0012-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415069179 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 182 previous similar messages Lustre: meerkat-OST0012-osc: Connection to meerkat-OST0012 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 43 previous similar messages LustreError: 3528:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST001a-osc: can't precreate: rc = -11 LustreError: 3528:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3528:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST001a-osc: cannot precreate objects: rc = -11 LustreError: 3528:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages Lustre: MGS: Client 5784ef13-ba8c-9aa2-6f0e-865460a8279a (at 10.7.104.36@o2ib) reconnecting Lustre: Skipped 247 previous similar messages Lustre: meerkat-MDT0000: Client 6249e52a-f753-9378-7db9-6a4f5d439ce4 (at 198.202.119.70@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-OST0002-osc: Connection restored to meerkat-OST0002 (at 172.25.32.248@tcp) Lustre: Skipped 43 previous similar messages Lustre: meerkat-MDT0000: Client 6249e52a-f753-9378-7db9-6a4f5d439ce4 (at 198.202.119.70@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 6249e52a-f753-9378-7db9-6a4f5d439ce4 (at 198.202.119.70@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 6249e52a-f753-9378-7db9-6a4f5d439ce4 (at 198.202.119.70@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 6249e52a-f753-9378-7db9-6a4f5d439ce4 (at 198.202.119.70@tcp) refused reconnection, still busy with 1 active RPCs LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 120s: evicting client at 198.202.119.70@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff8800961dc240/0x661ae11d096b08e lrc: 3/0,0 mode: PR/PR res: [0x200007af4:0x14fe:0x0].0 bits 0x1b rrc: 2 type: IBT flags: 0x200000000020 nid: 198.202.119.70@tcp remote: 0x5997e784a19cb00d expref: 10 pid: 18063 timeout: 4495260123 lvb_type: 0 LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff8800a778fd80/0x661ae11d319bee4 lrc: 3/0,0 mode: PR/PR res: [0x20000628a:0x1a097:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0beeddf72 expref: 41 pid: 3419 timeout: 4495760579 lvb_type: 0 LustreError: 13032:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff8803365c8800 x1474469902721964/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 0 to 0 dl 1415069859 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415070130/real 0] req@ffff880637cba800 x1483597821731476/t0(0) o6->meerkat-OST0032-osc@172.25.32.248@tcp:28/4 lens 664/432 e 0 to 1 dl 1415070138 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 58 previous similar messages Lustre: meerkat-OST0032-osc: Connection to meerkat-OST0032 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 7 previous similar messages Lustre: meerkat-MDT0000: Client ec5f24f5-ada8-4b96-7332-dcad165b6f3e (at 10.7.103.104@o2ib) reconnecting Lustre: Skipped 91 previous similar messages LustreError: 11-0: meerkat-OST001a-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST0012-osc: Connection restored to meerkat-OST0012 (at 172.25.32.248@tcp) Lustre: Skipped 7 previous similar messages LustreError: 3601:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST003a-osc: can't precreate: rc = -11 LustreError: 3601:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST003a-osc: cannot precreate objects: rc = -11 LustreError: 11-0: meerkat-OST0002-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3497:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000c-osc: can't precreate: rc = -11 LustreError: 3497:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000c-osc: cannot precreate objects: rc = -11 LustreError: 17490:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415070433 with bad export cookie 459840028550703417 LustreError: 11-0: meerkat-OST0036-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 6 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 17490:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415070487 with bad export cookie 459840028581664627 Lustre: MGS: Client c5b3e797-4442-c473-f513-0c396d194c53 (at 10.7.103.232@o2ib) reconnecting Lustre: Skipped 79 previous similar messages Lustre: meerkat-MDT0000: Client 0e5d00cf-c439-16f1-4be8-6aea02a219d9 (at 10.7.104.11@o2ib) reconnecting Lustre: Skipped 52 previous similar messages LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff8800aac65480/0x661ae11d7d7f334 lrc: 3/0,0 mode: PR/PR res: [0x200006136:0x1eb70:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0bf0a93ff expref: 104 pid: 13056 timeout: 4497591781 lvb_type: 0 LustreError: 13032:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff8803d138c800 x1474469906574956/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 2 to 0 dl 1415071749 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415071719/real 1415071726] req@ffff8803168f2800 x1483597823019416/t0(0) o6->meerkat-OST0036-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415071751 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 299 previous similar messages Lustre: meerkat-OST0036-osc: Connection to meerkat-OST0036 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 41 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3575:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST002e-osc: can't precreate: rc = -11 LustreError: 3575:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3575:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST002e-osc: cannot precreate objects: rc = -11 LustreError: 3575:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages LustreError: 3497:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000c-osc: can't precreate: rc = -11 LustreError: 3497:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000c-osc: cannot precreate objects: rc = -11 LustreError: 3519:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0016-osc: can't precreate: rc = -11 LustreError: 3519:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0016-osc: cannot precreate objects: rc = -11 LustreError: 5217:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415071770 with bad export cookie 459840028630586584 Lustre: meerkat-OST002c-osc: Connection restored to meerkat-OST002c (at 172.25.32.115@tcp) Lustre: Skipped 41 previous similar messages LustreError: 11-0: meerkat-OST0016-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 8 previous similar messages Lustre: 3330:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415071891/real 0] req@ffff88033755cc00 x1483597823326856/t0(0) o6->meerkat-OST0012-osc@172.25.32.248@tcp:28/4 lens 664/432 e 0 to 1 dl 1415071900 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3330:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 25 previous similar messages Lustre: meerkat-OST0012-osc: Connection to meerkat-OST0012 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST0012-osc: Connection restored to meerkat-OST0012 (at 172.25.32.248@tcp) Lustre: Skipped 15 previous similar messages LustreError: 11-0: meerkat-OST000a-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. Lustre: MGS: Client 1db79124-d61e-fd93-ce2b-f11a78924a97 (at 10.7.103.149@o2ib) reconnecting Lustre: Skipped 82 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3588:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0034-osc: can't precreate: rc = -11 LustreError: 3588:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0034-osc: cannot precreate objects: rc = -11 LustreError: 5217:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415072016 with bad export cookie 459840028635743225 LustreError: 3537:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST001e-osc: can't precreate: rc = -11 LustreError: 3537:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3537:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST001e-osc: cannot precreate objects: rc = -11 LustreError: 3537:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message Lustre: 10982:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415072069/real 1415072071] req@ffff8800b35c5800 x1483597823423280/t0(0) o104->meerkat-MDT0000@192.168.230.53@tcp:15/16 lens 296/224 e 0 to 1 dl 1415072076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 10982:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 81 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3397:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415072079 with bad export cookie 459840028653163621 Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 20 previous similar messages LustreError: 11-0: meerkat-OST0034-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: meerkat-OST0034-osc: Connection restored to meerkat-OST0034 (at 172.25.32.115@tcp) Lustre: Skipped 20 previous similar messages LNet: Service thread pid 13030 was inactive for 250.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 13030, comm: mdt_rdpg00_015 Call Trace: [] ? shrink_inactive_list+0x343/0x830 [] shrink_mem_cgroup_zone+0x3ae/0x610 [] ? mem_cgroup_iter+0xfd/0x280 [] shrink_zone+0x63/0xb0 [] zone_reclaim+0x349/0x400 [] get_page_from_freelist+0x69c/0x830 [] __alloc_pages_nodemask+0x113/0x8d0 [] ? sptlrpc_svc_alloc_rs+0x74/0x2a0 [ptlrpc] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] [] ? __req_capsule_get+0x166/0x700 [ptlrpc] [] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] [] alloc_pages_current+0xaa/0x110 [] cfs_alloc_page+0x17/0x20 [libcfs] [] mdt_readpage+0x1de/0x960 [mdt] [] mdt_handle_common+0x647/0x16d0 [mdt] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] mds_readpage_handle+0x15/0x20 [mdt] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415072412.13030 LNet: Service thread pid 13030 completed after 268.74s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415072505/real 1415072511] req@ffff880312019400 x1483597823557916/t0(0) o6->meerkat-OST0032-osc@172.25.32.248@tcp:28/4 lens 664/432 e 0 to 1 dl 1415072516 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Lustre: meerkat-OST0032-osc: Connection to meerkat-OST0032 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete LustreError: 11-0: meerkat-OST000a-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST0032-osc: Connection restored to meerkat-OST0032 (at 172.25.32.248@tcp) Lustre: meerkat-MDT0000: Client 754c5178-5f50-b603-98ca-943e891325ba (at 10.7.103.219@o2ib) reconnecting Lustre: Skipped 109 previous similar messages LustreError: 11-0: meerkat-OST0032-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: Skipped 2 previous similar messages Lustre: MGS: Client ff5d1d61-547b-ebf8-28c9-e770c38da4e5 (at 10.7.101.123@o2ib) reconnecting Lustre: Skipped 98 previous similar messages Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415073263/real 1415073268] req@ffff88055004c800 x1483597823898792/t0(0) o13->meerkat-OST0026-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415073270 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 35 previous similar messages Lustre: meerkat-OST0026-osc: Connection to meerkat-OST0026 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages LustreError: 3537:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST001e-osc: can't precreate: rc = -11 LustreError: 3537:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST001e-osc: cannot precreate objects: rc = -11 Lustre: meerkat-OST0014-osc: Connection restored to meerkat-OST0014 (at 172.25.32.115@tcp) Lustre: Skipped 6 previous similar messages LustreError: 3485:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0006-osc: can't precreate: rc = -11 LustreError: 3485:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0006-osc: cannot precreate objects: rc = -11 LustreError: 3533:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST001c-osc: can't precreate: rc = -11 LustreError: 3533:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3533:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST001c-osc: cannot precreate objects: rc = -11 LustreError: 3533:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages LustreError: 11-0: meerkat-OST000c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: 3515:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0014-osc: can't precreate: rc = -11 LustreError: 3515:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3515:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0014-osc: cannot precreate objects: rc = -11 LustreError: 3515:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff8800b7858480/0x661ae11dd94cddc lrc: 3/0,0 mode: PR/PR res: [0x200006136:0x1ed6c:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0bf344729 expref: 41 pid: 14332 timeout: 4500309504 lvb_type: 0 Lustre: MGS: Client 925f07f8-c15e-e75b-e6cb-5ba79a45fe54 (at 10.7.104.28@o2ib) reconnecting Lustre: Skipped 361 previous similar messages Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415074375/real 1415074382] req@ffff88028dfd5800 x1483597825159176/t0(0) o13->meerkat-OST001a-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415074389 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 134 previous similar messages Lustre: meerkat-OST003a-osc: Connection to meerkat-OST003a (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 57 previous similar messages LustreError: 3583:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0032-osc: can't precreate: rc = -11 LustreError: 3583:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3583:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0032-osc: cannot precreate objects: rc = -11 LustreError: 3583:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message Lustre: meerkat-OST001a-osc: Connection restored to meerkat-OST001a (at 172.25.32.248@tcp) Lustre: Skipped 56 previous similar messages LustreError: 11-0: meerkat-OST0032-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: Skipped 14 previous similar messages LustreError: 3568:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST002c-osc: can't precreate: rc = -11 LustreError: 3568:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST002c-osc: cannot precreate objects: rc = -11 Lustre: meerkat-MDT0000: Client 21754531-e3fa-348b-5436-fe3926449139 (at 10.7.103.188@o2ib) reconnecting Lustre: Skipped 611 previous similar messages Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415075616/real 1415075625] req@ffff880638cdd400 x1483597828311008/t0(0) o13->meerkat-OST0026-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415075634 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 208 previous similar messages Lustre: meerkat-OST0026-osc: Connection to meerkat-OST0026 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 22 previous similar messages LustreError: 3515:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0014-osc: can't precreate: rc = -11 LustreError: 3515:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 4 previous similar messages LustreError: 3515:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0014-osc: cannot precreate objects: rc = -11 LustreError: 3515:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 4 previous similar messages LustreError: 3485:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0006-osc: can't precreate: rc = -11 LustreError: 3485:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 4 previous similar messages LustreError: 3485:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0006-osc: cannot precreate objects: rc = -11 LustreError: 3485:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 4 previous similar messages Lustre: meerkat-OST0014-osc: Connection restored to meerkat-OST0014 (at 172.25.32.115@tcp) Lustre: Skipped 23 previous similar messages Lustre: MGS: Client f35e308d-5ebc-dd32-ed06-c912f0614834 (at 10.7.101.72@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: 3575:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415075725/real 1415075736] req@ffff8806391f2800 x1483597828447912/t0(0) o5->meerkat-OST002e-osc@172.25.32.243@tcp:28/4 lens 432/432 e 0 to 1 dl 1415075746 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3575:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Lustre: meerkat-OST002e-osc: Connection to meerkat-OST002e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 14 previous similar messages LustreError: 3575:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST002e-osc: can't precreate: rc = -11 LustreError: 3575:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3575:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST002e-osc: cannot precreate objects: rc = -11 LustreError: 3575:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message Lustre: meerkat-MDT0000: Client 426a4ee4-22a1-0043-48af-90853c7ea3f9 (at 10.7.101.20@o2ib) refused reconnection, still busy with 1 active RPCs LustreError: 11-0: meerkat-OST000c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 8 previous similar messages Lustre: meerkat-OST0024-osc: Connection restored to meerkat-OST0024 (at 172.25.32.115@tcp) Lustre: Skipped 14 previous similar messages Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415075966/real 0] req@ffff8803141ab800 x1483597828881624/t0(0) o6->meerkat-OST000e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415075987 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 163 previous similar messages Lustre: meerkat-OST000e-osc: Connection to meerkat-OST000e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 23 previous similar messages LustreError: 3588:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0034-osc: can't precreate: rc = -11 LustreError: 3588:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3588:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0034-osc: cannot precreate objects: rc = -11 LustreError: 3588:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages Lustre: meerkat-MDT0000: Client f8ee92d4-3bc1-7fca-234a-43af75dc6463 (at 10.7.101.153@o2ib) refused reconnection, still busy with 3 active RPCs LustreError: 11-0: meerkat-OST0034-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 14 previous similar messages Lustre: meerkat-OST003e-osc: Connection restored to meerkat-OST003e (at 172.25.32.243@tcp) Lustre: Skipped 23 previous similar messages Lustre: meerkat-MDT0000: Client f8ee92d4-3bc1-7fca-234a-43af75dc6463 (at 10.7.101.153@o2ib) reconnecting Lustre: Skipped 265 previous similar messages Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415076618/real 1415076618] req@ffff880625232400 x1483597829010820/t0(0) o13->meerkat-OST0032-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415076628 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 138 previous similar messages Lustre: meerkat-OST0032-osc: Connection to meerkat-OST0032 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 3511:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0012-osc: can't precreate: rc = -11 LustreError: 3511:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 3511:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0012-osc: cannot precreate objects: rc = -11 LustreError: 3511:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 2 previous similar messages Lustre: meerkat-OST0032-osc: Connection restored to meerkat-OST0032 (at 172.25.32.248@tcp) Lustre: Skipped 15 previous similar messages Lustre: meerkat-MDT0000: Client f1182879-c11c-eaf1-3223-bf4141822b94 (at 10.7.104.36@o2ib) reconnecting Lustre: Skipped 11 previous similar messages LustreError: 3515:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0014-osc: can't precreate: rc = -11 LustreError: 3515:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0014-osc: cannot precreate objects: rc = -11 LustreError: 11-0: meerkat-OST0036-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 6 previous similar messages LustreError: 11-0: meerkat-OST0006-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: MGS: Client 925f07f8-c15e-e75b-e6cb-5ba79a45fe54 (at 10.7.104.28@o2ib) reconnecting Lustre: Skipped 41 previous similar messages Lustre: 3334:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415077408/real 1415077412] req@ffff880093cbd400 x1483597830037904/t0(0) o6->meerkat-OST0014-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415077426 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3334:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Lustre: meerkat-OST0014-osc: Connection to meerkat-OST0014 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 9 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 4270:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415077428 with bad export cookie 459840028724803882 Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 9 previous similar messages LustreError: 11-0: meerkat-OST002a-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 5217:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415077734 with bad export cookie 459840028831826385 LustreError: 11-0: meerkat-OST0014-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 2 previous similar messages LustreError: 3583:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0032-osc: can't precreate: rc = -11 LustreError: 3583:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3583:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0032-osc: cannot precreate objects: rc = -11 LustreError: 3583:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message Lustre: MGS: Client be660024-e673-8ec8-3d9e-2ed734bb478f (at 10.7.100.199@o2ib) reconnecting Lustre: Skipped 34 previous similar messages Lustre: 13050:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415078207/real 0] req@ffff880313f7f000 x1483597830707812/t0(0) o104->meerkat-MDT0000@192.168.230.53@tcp:15/16 lens 296/224 e 0 to 1 dl 1415078217 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 13050:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 152 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 17506:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415078222 with bad export cookie 459840028838029596 Lustre: MGS: Client b93e6e57-621d-32a5-56e2-9afd155c2f6a (at 10.7.103.91@o2ib) reconnecting Lustre: Skipped 30 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415078917/real 1415078928] req@ffff8800a7bb5c00 x1483597831708220/t0(0) o13->meerkat-OST0034-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415078937 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 35 previous similar messages Lustre: meerkat-OST0034-osc: Connection restored to meerkat-OST0034 (at 172.25.32.115@tcp) Lustre: Skipped 35 previous similar messages Lustre: MGS: Client 3fccb4b9-5151-3767-7195-d50e18563e93 (at 10.7.100.215@o2ib) refused reconnection, still busy with 1 active RPCs LustreError: 3615:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88031556b850 x1474770478253916/t0(0) o256->08b0b2ff-7cef-f1cf-d9b8-9208272ff033@10.7.101.155@o2ib:0/0 lens 304/240 e 0 to 0 dl 1415079183 ref 1 fl Interpret:/0/0 rc 0/0 LustreError: 13957:0:(ldlm_lib.c:2711:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88031577e850 x1474629358805508/t0(0) o256->0f0ec231-9987-ba49-f697-111e832e7849@10.7.101.232@o2ib:0/0 lens 304/240 e 0 to 0 dl 1415079183 ref 1 fl Interpret:/0/0 rc 0/0 LustreError: 13957:0:(ldlm_lib.c:2711:target_bulk_io()) Skipped 1 previous similar message Lustre: meerkat-OST001c-osc: Connection to meerkat-OST001c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete LustreError: 11-0: meerkat-OST0026-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 7 previous similar messages Lustre: meerkat-OST0014-osc: Connection restored to meerkat-OST0014 (at 172.25.32.115@tcp) Lustre: meerkat-MDT0000: Client 90f47241-e712-700e-d0e8-8575e99459a9 (at 198.202.118.83@tcp) refused reconnection, still busy with 1 active RPCs Lustre: Skipped 4 previous similar messages Lustre: meerkat-MDT0000: Client 3d3270bd-41c8-4d1e-df97-8f5b53ba3ed5 (at 10.7.103.198@o2ib) reconnecting Lustre: Skipped 204 previous similar messages Lustre: meerkat-MDT0000: Client d2853a1d-e0b5-38b9-b8ba-c3d39bdfb60a (at 10.7.103.136@o2ib) reconnecting Lustre: Skipped 2 previous similar messages Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415080172/real 0] req@ffff880312afc000 x1483597833668016/t0(0) o13->meerkat-OST002e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415080185 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 139 previous similar messages Lustre: meerkat-OST002e-osc: Connection to meerkat-OST002e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 15 previous similar messages Lustre: MGS: haven't heard from client c947dcfe-9c6f-68a0-8e34-9e4355716838 (at 10.7.100.86@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880625ef7c00, cur 1415080916 expire 1415080766 last 1415080689 Lustre: Skipped 1 previous similar message Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415081349/real 0] req@ffff880311db1800 x1483597833999788/t0(0) o13->meerkat-OST0014-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415081356 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Lustre: meerkat-OST0014-osc: Connection to meerkat-OST0014 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 11 previous similar messages LustreError: 3519:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0016-osc: can't precreate: rc = -11 LustreError: 3519:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3519:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0016-osc: cannot precreate objects: rc = -11 LustreError: 3519:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message Lustre: meerkat-OST0014-osc: Connection restored to meerkat-OST0014 (at 172.25.32.115@tcp) Lustre: Skipped 11 previous similar messages LustreError: 11-0: meerkat-OST003e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 2 previous similar messages LustreError: 3533:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST001c-osc: can't precreate: rc = -107 LustreError: 3588:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0034-osc: can't precreate: rc = -107 Lustre: MGS: Client c6f0ad96-6f3b-0e22-7eb0-32d00a512479 (at 10.7.100.171@o2ib) reconnecting Lustre: Skipped 125 previous similar messages Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415082405/real 0] req@ffff88056ccdcc00 x1483597837089344/t0(0) o13->meerkat-OST0036-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415082414 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Lustre: meerkat-OST0036-osc: Connection to meerkat-OST0036 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 13 previous similar messages Lustre: 3330:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415082405/real 0] req@ffff880386088000 x1483597837089364/t0(0) o13->meerkat-OST002e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415082415 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3330:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Lustre: meerkat-MDT0000: Client 939d1493-6d09-b7ef-dee6-96cab91a5b5f (at 10.7.104.19@o2ib) reconnecting Lustre: Skipped 13 previous similar messages Lustre: 3336:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415082409/real 0] req@ffff88062409c400 x1483597837090612/t0(0) o6->meerkat-OST0036-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415082418 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3336:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Lustre: meerkat-OST0034-osc: Connection restored to meerkat-OST0034 (at 172.25.32.115@tcp) Lustre: Skipped 13 previous similar messages LustreError: 11-0: meerkat-OST001c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 2 previous similar messages Lustre: meerkat-MDT0000: Client 453cd0d9-c8e3-0e50-da9e-7953a9c89205 (at 192.168.230.53@tcp) refused reconnection, still busy with 2 active RPCs Lustre: meerkat-OST002c-osc: Connection restored to meerkat-OST002c (at 172.25.32.115@tcp) Lustre: Skipped 8 previous similar messages Lustre: 3546:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415083789/real 1415083792] req@ffff880388b4e400 x1483597839323468/t0(0) o5->meerkat-OST0022-osc@172.25.32.248@tcp:28/4 lens 432/432 e 0 to 1 dl 1415083796 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3546:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Lustre: meerkat-OST0022-osc: Connection to meerkat-OST0022 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 3546:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0022-osc: can't precreate: rc = -11 LustreError: 3546:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0022-osc: cannot precreate objects: rc = -11 LustreError: 3528:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST001a-osc: cannot precreate objects: rc = -11 Lustre: 3583:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415083790/real 1415083794] req@ffff880624acb400 x1483597839325212/t0(0) o5->meerkat-OST0032-osc@172.25.32.248@tcp:28/4 lens 432/432 e 0 to 1 dl 1415083797 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3583:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Lustre: meerkat-OST0032-osc: Connection to meerkat-OST0032 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 3583:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0032-osc: can't precreate: rc = -11 LustreError: 3583:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message LustreError: 3583:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0032-osc: cannot precreate objects: rc = -11 LustreError: 3511:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0012-osc: cannot precreate objects: rc = -11 Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415083792/real 1415083795] req@ffff88063829ac00 x1483597839326888/t0(0) o13->meerkat-OST0002-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415083799 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Lustre: meerkat-OST0002-osc: Connection to meerkat-OST0002 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 3493:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000a-osc: can't precreate: rc = -11 LustreError: 3493:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 1 previous similar message Lustre: meerkat-MDT0000: Client 22c2240a-4654-8283-084d-9d766404d3d0 (at 10.7.103.76@o2ib) reconnecting Lustre: Skipped 8 previous similar messages Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415083796/real 1415083799] req@ffff8803873fbc00 x1483597839328304/t0(0) o8->meerkat-OST001a-osc@172.25.32.248@tcp:28/4 lens 400/544 e 0 to 1 dl 1415083802 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 198.202.118.30@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 6402:0:(ldlm_lockd.c:709:ldlm_handle_ast_error()) ### client (nid 198.202.118.30@tcp) returned 0 from blocking AST ns: mdt-meerkat-MDT0000_UUID lock: ffff88003749c900/0x661ae11e9204432 lrc: 4/0,0 mode: PR/PR res: [0x200007b3f:0x3a3:0x0].0 bits 0x13 rrc: 3 type: IBT flags: 0x200000000020 nid: 198.202.118.30@tcp remote: 0xeb38bc192684c656 expref: 12066 pid: 3702 timeout: 4509862212 lvb_type: 0 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415083796/real 1415083799] req@ffff88003799e400 x1483597839328292/t0(0) o13->meerkat-OST003a-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415083808 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Lustre: meerkat-OST003a-osc: Connection to meerkat-OST003a (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415083808/real 1415083812] req@ffff8804e7df4400 x1483597839328976/t0(0) o8->meerkat-OST003a-osc@172.25.32.248@tcp:28/4 lens 400/544 e 0 to 1 dl 1415083819 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: MGS: Client a87a4daa-1f06-851c-0354-ca42ee5b6cd9 (at 10.7.104.48@o2ib) reconnecting Lustre: Skipped 16 previous similar messages LustreError: 3398:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 198.202.118.30@tcp arrived at 1415083825 with bad export cookie 459840028218155166 LustreError: 3398:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 198.202.118.30@tcp arrived at 1415083835 with bad export cookie 459840028218155166 Lustre: meerkat-OST000a-osc: Connection restored to meerkat-OST000a (at 172.25.32.248@tcp) Lustre: Skipped 6 previous similar messages Lustre: MGS: Client 1825ec1f-038f-2351-5846-6ae2e7896cde (at 172.25.32.248@tcp) reconnecting Lustre: Skipped 182 previous similar messages Lustre: 3501:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415083866/real 0] req@ffff88062e4a1000 x1483597839364484/t0(0) o5->meerkat-OST000e-osc@172.25.32.243@tcp:28/4 lens 432/432 e 0 to 1 dl 1415083873 ref 3 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: meerkat-OST000e-osc: Connection to meerkat-OST000e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete LustreError: 3501:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST000e-osc: can't precreate: rc = -11 LustreError: 3501:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST000e-osc: cannot precreate objects: rc = -11 LustreError: 3501:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 1 previous similar message LustreError: 3537:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST001e-osc: cannot precreate objects: rc = -11 LustreError: 3537:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 4 previous similar messages Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 7 previous similar messages Lustre: MGS: Client b999c2a5-c524-f344-1fbb-d023f0fefc35 (at 10.7.101.204@o2ib) reconnecting Lustre: Skipped 61 previous similar messages Lustre: MGS: Client 2bf0062f-3663-81f0-a48f-29fa118f627d (at 10.7.103.134@o2ib) reconnecting Lustre: Skipped 25 previous similar messages Lustre: meerkat-MDT0000: Client d30c0bc1-3d70-1e41-bdd7-595ae8e0af70 (at 10.7.101.101@o2ib) reconnecting Lustre: Skipped 2 previous similar messages Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415087643/real 0] req@ffff88050f8a9400 x1483597845363180/t0(0) o13->meerkat-OST000c-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415087652 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 33 previous similar messages Lustre: meerkat-OST000c-osc: Connection to meerkat-OST000c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST002e-osc: Connection to meerkat-OST002e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages LustreError: 3575:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST002e-osc: can't precreate: rc = -11 LustreError: 3575:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 5 previous similar messages LustreError: 3575:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST002e-osc: cannot precreate objects: rc = -11 Lustre: meerkat-MDT0000: Client 15e1bcf6-a618-80ca-a17a-70ccf8d686ab (at 10.7.101.234@o2ib) reconnecting Lustre: Skipped 1 previous similar message LustreError: 11-0: meerkat-OST000c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 6 previous similar messages Lustre: meerkat-OST0014-osc: Connection restored to meerkat-OST0014 (at 172.25.32.115@tcp) Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 5 previous similar messages Lustre: 13056:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415087764/real 0] req@ffff8803151b5400 x1483597846036784/t0(0) o104->meerkat-MDT0000@198.202.118.250@tcp:15/16 lens 296/224 e 0 to 1 dl 1415087771 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 13056:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 198.202.118.250@tcp was evicted due to a lock blocking callback time out: rc -107 Lustre: MGS: haven't heard from client da599621-f5fd-4721-c5bc-a90e61c9b48f (at 198.202.118.250@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802f144c000, cur 1415087876 expire 1415087726 last 1415087649 Lustre: Skipped 1 previous similar message Lustre: meerkat-MDT0000: Client d566c21c-e2f0-d76c-7091-ba5d08d31923 (at 10.7.103.230@o2ib) reconnecting Lustre: Skipped 8 previous similar messages Lustre: MGS: Client 13889f7c-6e6f-8162-0b68-2ea4a0113e6e (at 10.7.103.85@o2ib) reconnecting Lustre: meerkat-MDT0000: Client 9f47e6bc-36d9-460d-66fd-5823ee4b4830 (at 10.7.103.110@o2ib) reconnecting Lustre: 3330:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415089092/real 1415089099] req@ffff880315036400 x1483597849779224/t0(0) o13->meerkat-OST0002-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415089103 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: meerkat-OST0002-osc: Connection to meerkat-OST0002 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 7 previous similar messages Lustre: meerkat-OST0002-osc: Connection restored to meerkat-OST0002 (at 172.25.32.248@tcp) Lustre: Skipped 8 previous similar messages Lustre: meerkat-MDT0000: Client a820882d-3cad-935d-55b4-e10fcd66da5e (at 10.7.103.91@o2ib) reconnecting Lustre: Skipped 3 previous similar messages Lustre: meerkat-MDT0000: Client 7d1f7ac4-4d34-dbcc-745c-8d538331bf7c (at 10.7.104.26@o2ib) reconnecting Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415089682/real 1415089684] req@ffff8802b0f71000 x1483597849874000/t0(0) o13->meerkat-OST003e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415089693 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: meerkat-OST003e-osc: Connection to meerkat-OST003e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: meerkat-MDT0000: Client 7762a2df-1367-dbf5-af56-076affd65204 (at 10.7.101.11@o2ib) reconnecting Lustre: Skipped 1 previous similar message Lustre: meerkat-MDT0000: Client 604340f3-490f-c04f-1cf3-5dc9d71f90e8 (at 10.7.103.172@o2ib) reconnecting Lustre: Skipped 17 previous similar messages Lustre: meerkat-MDT0000: Client d4dfa488-e893-2b1b-28df-dad87777399c (at 10.7.104.23@o2ib) reconnecting Lustre: 18063:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415089831/real 1415089831] req@ffff880336165c00 x1483597849882396/t0(0) o104->meerkat-MDT0000@198.202.118.30@tcp:15/16 lens 296/224 e 0 to 1 dl 1415089838 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 18063:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 198.202.118.30@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 18063:0:(client.c:1048:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880336165c00 x1483597849882876/t0(0) o104->meerkat-MDT0000@198.202.118.30@tcp:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 LustreError: 18063:0:(ldlm_lockd.c:709:ldlm_handle_ast_error()) ### client (nid 198.202.118.30@tcp) returned 0 from blocking AST ns: mdt-meerkat-MDT0000_UUID lock: ffff88005d4d6d80/0x661ae11f4c75bad lrc: 4/0,0 mode: PR/PR res: [0x200007c53:0x73:0x0].0 bits 0x1b rrc: 2 type: IBT flags: 0x200000000020 nid: 198.202.118.30@tcp remote: 0xeb38bc19269846d0 expref: 29291 pid: 3423 timeout: 4515900208 lvb_type: 0 LustreError: 11191:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 198.202.118.30@tcp arrived at 1415089840 with bad export cookie 459840029014630244 LustreError: 11191:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 198.202.118.30@tcp arrived at 1415089841 with bad export cookie 459840029014630244 LustreError: 11191:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) Skipped 1 previous similar message Lustre: MGS: Client 293606e9-3b88-767f-b475-0649efe731b5 (at 10.7.104.21@o2ib) reconnecting Lustre: Skipped 14 previous similar messages Lustre: 3331:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415090173/real 0] req@ffff880406422000 x1483597850562696/t0(0) o6->meerkat-OST003e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415090186 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: meerkat-OST003e-osc: Connection to meerkat-OST003e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: meerkat-MDT0000: Client 16b51aa4-4601-0ff5-ec09-a4cab24829b5 (at 10.7.104.15@o2ib) reconnecting Lustre: Skipped 7 previous similar messages LustreError: 11-0: meerkat-OST003e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 8 previous similar messages Lustre: meerkat-OST003e-osc: Connection restored to meerkat-OST003e (at 172.25.32.243@tcp) Lustre: Skipped 2 previous similar messages Lustre: meerkat-MDT0000: Client 554f947e-3aac-c35e-1574-5922f7741070 (at 10.7.103.227@o2ib) reconnecting Lustre: meerkat-MDT0000: Client 60e75580-4a61-8b60-534d-5fdd685cf9f8 (at 10.7.102.112@o2ib) reconnecting Lustre: Skipped 11 previous similar messages Lustre: meerkat-MDT0000: Client 455d8348-5c81-e5b3-9b85-958e5ce362de (at 10.7.101.132@o2ib) reconnecting Lustre: meerkat-MDT0000: Client 6874f841-0b73-0db5-59aa-35201b2e50d6 (at 10.7.102.75@o2ib) reconnecting Lustre: Skipped 4 previous similar messages Lustre: meerkat-MDT0000: Client 793ff973-829e-ab76-f860-fb973fe841b1 (at 10.7.100.218@o2ib) reconnecting Lustre: Skipped 1 previous similar message Lustre: MGS: Client ab48772f-2ad8-4499-0d18-56c528692be8 (at 10.7.103.112@o2ib) reconnecting Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415092320/real 0] req@ffff880315146000 x1483597855282772/t0(0) o6->meerkat-OST000a-osc@172.25.32.248@tcp:28/4 lens 664/432 e 0 to 1 dl 1415092329 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Lustre: meerkat-OST000a-osc: Connection to meerkat-OST000a (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: meerkat-OST0002-osc: Connection restored to meerkat-OST0002 (at 172.25.32.248@tcp) LustreError: 11-0: meerkat-OST0022-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST0022-osc: Connection restored to meerkat-OST0022 (at 172.25.32.248@tcp) Lustre: Skipped 3 previous similar messages Lustre: MGS: Client ed45086d-2338-73b9-f3b8-cdba30566ffb (at 10.7.104.44@o2ib) reconnecting Lustre: Skipped 18 previous similar messages Lustre: meerkat-MDT0000: Client 51058832-a631-50b7-dd44-4d95263441e7 (at 10.7.103.204@o2ib) reconnecting Lustre: Skipped 1 previous similar message Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415092804/real 1415092814] req@ffff88007ef39c00 x1483597856345952/t0(0) o6->meerkat-OST0006-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415092821 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Lustre: meerkat-OST0006-osc: Connection to meerkat-OST0006 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) LustreError: 11-0: meerkat-OST0026-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST0026-osc: Connection restored to meerkat-OST0026 (at 172.25.32.243@tcp) Lustre: Skipped 13 previous similar messages Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415093553/real 0] req@ffff8803362c8800 x1483597857927700/t0(0) o6->meerkat-OST000c-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415093562 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 25 previous similar messages Lustre: meerkat-OST000c-osc: Connection to meerkat-OST000c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 8 previous similar messages Lustre: 3334:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415093553/real 0] req@ffff88034d867000 x1483597857927636/t0(0) o6->meerkat-OST002e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415093562 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: meerkat-OST002e-osc: Connection to meerkat-OST002e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: meerkat-OST000e-osc: Connection to meerkat-OST000e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: meerkat-MDT0000: Client 06bc4379-10ca-76ad-cd98-1d1013f1b911 (at 10.7.103.252@o2ib) reconnecting Lustre: Skipped 109 previous similar messages Lustre: 3331:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415093553/real 0] req@ffff8800957b0400 x1483597857927500/t0(0) o6->meerkat-OST001e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415093563 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3331:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415093553/real 1415093555] req@ffff880315933000 x1483597857927436/t0(0) o13->meerkat-OST001e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415093563 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 19 previous similar messages LustreError: 11-0: meerkat-OST002e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: meerkat-OST0036-osc: Connection restored to meerkat-OST0036 (at 172.25.32.243@tcp) Lustre: Skipped 1 previous similar message Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 5 previous similar messages Lustre: meerkat-MDT0000: Client 99f8dd87-5cc2-3715-1d6a-9c5adcba375d (at 10.7.103.81@o2ib) reconnecting Lustre: Skipped 48 previous similar messages Lustre: meerkat-MDT0000: Client b6bd3474-af36-9e6a-ad8e-971c4ac406f0 (at 10.7.103.138@o2ib) reconnecting Lustre: Skipped 3 previous similar messages Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415095458/real 0] req@ffff8803c897ec00 x1483597859807372/t0(0) o6->meerkat-OST003e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415095472 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: meerkat-OST003e-osc: Connection to meerkat-OST003e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: 3329:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415095458/real 0] req@ffff880368c1c800 x1483597859806772/t0(0) o6->meerkat-OST003e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415095472 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3329:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Lustre: meerkat-OST0006-osc: Connection to meerkat-OST0006 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 11 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415095458/real 0] req@ffff880585927800 x1483597859806408/t0(0) o6->meerkat-OST0034-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415095472 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 77 previous similar messages Lustre: meerkat-OST000e-osc: Connection to meerkat-OST000e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415095461/real 0] req@ffff880317427400 x1483597859814448/t0(0) o13->meerkat-OST0024-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415095475 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 30 previous similar messages LustreError: 11-0: meerkat-OST0026-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 9 previous similar messages Lustre: meerkat-OST003c-osc: Connection restored to meerkat-OST003c (at 172.25.32.115@tcp) Lustre: Skipped 9 previous similar messages Lustre: meerkat-OST0014-osc: Connection restored to meerkat-OST0014 (at 172.25.32.115@tcp) Lustre: Skipped 5 previous similar messages Lustre: meerkat-MDT0000: Client 878a6d85-3d31-0831-dba8-bc6c7ff77670 (at 10.7.100.202@o2ib) reconnecting Lustre: Skipped 97 previous similar messages Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415096056/real 0] req@ffff8805823eac00 x1483597860826128/t0(0) o13->meerkat-OST0022-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415096068 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Lustre: meerkat-OST0022-osc: Connection to meerkat-OST0022 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete LustreError: 11-0: meerkat-OST0022-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: Skipped 9 previous similar messages Lustre: meerkat-OST0002-osc: Connection restored to meerkat-OST0002 (at 172.25.32.248@tcp) Lustre: Skipped 9 previous similar messages Lustre: meerkat-OST001a-osc: Connection restored to meerkat-OST001a (at 172.25.32.248@tcp) Lustre: Skipped 1 previous similar message Lustre: meerkat-MDT0000: Client 390354a3-cbcf-ff9a-7193-d87216b1f2be (at 10.7.104.13@o2ib) reconnecting Lustre: Skipped 30 previous similar messages Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415098287/real 0] req@ffff880409dfc800 x1483597866169532/t0(0) o13->meerkat-OST0022-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415098294 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Lustre: meerkat-OST0022-osc: Connection to meerkat-OST0022 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 7 previous similar messages Lustre: meerkat-OST0012-osc: Connection to meerkat-OST0012 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: meerkat-MDT0000: Client 73e79cb3-b4cc-0602-7382-c94649e75f37 (at 10.7.100.233@o2ib) reconnecting Lustre: Skipped 17 previous similar messages LustreError: 11-0: meerkat-OST000a-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: Skipped 5 previous similar messages Lustre: meerkat-OST0022-osc: Connection restored to meerkat-OST0022 (at 172.25.32.248@tcp) Lustre: Skipped 5 previous similar messages Lustre: meerkat-OST0032-osc: Connection restored to meerkat-OST0032 (at 172.25.32.248@tcp) Lustre: Skipped 2 previous similar messages Lustre: meerkat-MDT0000: Client 0e5d00cf-c439-16f1-4be8-6aea02a219d9 (at 10.7.104.11@o2ib) reconnecting Lustre: Skipped 3 previous similar messages Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415098402/real 1415098407] req@ffff88061eab9400 x1483597866413348/t0(0) o6->meerkat-OST0032-osc@172.25.32.248@tcp:28/4 lens 664/432 e 0 to 1 dl 1415098413 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3339:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Lustre: meerkat-OST0032-osc: Connection to meerkat-OST0032 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415098404/real 1415098409] req@ffff880428f51000 x1483597866413716/t0(0) o13->meerkat-OST0002-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415098415 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Lustre: meerkat-OST0002-osc: Connection to meerkat-OST0002 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415098404/real 1415098407] req@ffff880308a60c00 x1483597866413712/t0(0) o13->meerkat-OST003a-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415098417 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Lustre: meerkat-OST003a-osc: Connection to meerkat-OST003a (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415098415/real 0] req@ffff8800131cd400 x1483597866415172/t0(0) o8->meerkat-OST001a-osc@172.25.32.248@tcp:28/4 lens 400/544 e 0 to 1 dl 1415098425 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415098443/real 0] req@ffff88062478e800 x1483597866422800/t0(0) o8->meerkat-OST0022-osc@172.25.32.248@tcp:28/4 lens 400/544 e 0 to 1 dl 1415098458 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Lustre: meerkat-OST0002-osc: Connection restored to meerkat-OST0002 (at 172.25.32.248@tcp) Lustre: Skipped 2 previous similar messages Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415098491/real 1415098497] req@ffff88062e415c00 x1483597866432140/t0(0) o13->meerkat-OST001e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415098506 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Lustre: meerkat-OST001e-osc: Connection to meerkat-OST001e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: meerkat-OST0024-osc: Connection to meerkat-OST0024 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 10 previous similar messages Lustre: meerkat-OST001e-osc: Connection restored to meerkat-OST001e (at 172.25.32.243@tcp) Lustre: Skipped 7 previous similar messages Lustre: meerkat-MDT0000: Client b0739527-b271-1b52-ad61-e87523d12868 (at 10.7.102.95@o2ib) reconnecting Lustre: Skipped 341 previous similar messages Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415098566/real 0] req@ffff8800aec34c00 x1483597866437988/t0(0) o13->meerkat-OST001c-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415098586 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Lustre: meerkat-OST001c-osc: Connection to meerkat-OST001c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: meerkat-OST000e-osc: Connection restored to meerkat-OST000e (at 172.25.32.243@tcp) Lustre: Skipped 11 previous similar messages Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415098824/real 1415098837] req@ffff8805c5e39400 x1483597866632732/t0(0) o13->meerkat-OST0026-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415098844 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Lustre: meerkat-OST0026-osc: Connection to meerkat-OST0026 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 10 previous similar messages Lustre: meerkat-OST001c-osc: Connection restored to meerkat-OST001c (at 172.25.32.115@tcp) Lustre: Skipped 10 previous similar messages Lustre: meerkat-MDT0000: Client f8ee92d4-3bc1-7fca-234a-43af75dc6463 (at 10.7.101.153@o2ib) refused reconnection, still busy with 2 active RPCs Lustre: meerkat-MDT0000: Client de0a2eff-47b1-9602-3e15-0f323ce91cee (at 10.7.103.78@o2ib) reconnecting Lustre: Skipped 8 previous similar messages Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415099294/real 1415099302] req@ffff880098fd8000 x1483597866817776/t0(0) o13->meerkat-OST0012-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415099309 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Lustre: meerkat-OST0012-osc: Connection to meerkat-OST0012 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 9 previous similar messages Lustre: meerkat-OST0012-osc: Connection restored to meerkat-OST0012 (at 172.25.32.248@tcp) Lustre: Skipped 9 previous similar messages LustreError: 11-0: meerkat-OST000a-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: Skipped 2 previous similar messages Lustre: meerkat-OST0022-osc: Connection restored to meerkat-OST0022 (at 172.25.32.248@tcp) Lustre: Skipped 4 previous similar messages LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 934s: evicting client at 198.202.118.242@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff88009cbc8900/0x661ae120c922f9e lrc: 3/0,0 mode: CR/CR res: [0x20000600d:0x1c70:0x0].0 bits 0x8 rrc: 3 type: IBT flags: 0x20 nid: 198.202.118.242@tcp remote: 0x4b199c37ff719adf expref: 63 pid: 6387 timeout: 4525346305 lvb_type: 3 Lustre: meerkat-OST000e-osc: Connection to meerkat-OST000e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages Lustre: meerkat-MDT0000: Client d78661eb-db5a-5a13-ba5b-e94235960c48 (at 10.7.103.248@o2ib) reconnecting Lustre: Skipped 61 previous similar messages Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 1 previous similar message LustreError: 11-0: meerkat-OST000e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message LustreError: 138-a: meerkat-MDT0000: A client on nid 198.202.118.51@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 4891:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 198.202.118.51@tcp arrived at 1415099589 with bad export cookie 459840023463757658 Lustre: MGS: Client cad2c263-5bd5-7f84-5d90-d7d24b1479c0 (at 10.7.103.208@o2ib) reconnecting Lustre: Skipped 66 previous similar messages Lustre: 3334:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415100260/real 1415100267] req@ffff880603056400 x1483597867653772/t0(0) o13->meerkat-OST0026-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415100273 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3334:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 153 previous similar messages Lustre: meerkat-OST0026-osc: Connection to meerkat-OST0026 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: meerkat-MDT0000: Client eab6da10-805f-6cc4-ba00-2afc127fd6a7 (at 10.7.100.135@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-OST000e-osc: Connection restored to meerkat-OST000e (at 172.25.32.243@tcp) Lustre: Skipped 15 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415100350/real 1415100354] req@ffff8805c372f800 x1483597867688020/t0(0) o13->meerkat-OST0032-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415100357 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 40 previous similar messages Lustre: meerkat-OST000a-osc: Connection restored to meerkat-OST000a (at 172.25.32.248@tcp) Lustre: Skipped 15 previous similar messages Lustre: 13051:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415100465/real 0] req@ffff880090c16800 x1483597867730512/t0(0) o104->meerkat-MDT0000@192.168.230.53@tcp:15/16 lens 296/224 e 0 to 1 dl 1415100474 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 13051:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 78 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 17488:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415100483 with bad export cookie 459840028849017363 LustreError: 11-0: meerkat-OST000e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: meerkat-OST0034-osc: Connection restored to meerkat-OST0034 (at 172.25.32.115@tcp) Lustre: Skipped 7 previous similar messages Lustre: meerkat-OST0016-osc: Connection restored to meerkat-OST0016 (at 172.25.32.243@tcp) Lustre: Skipped 2 previous similar messages Lustre: meerkat-OST000e-osc: Connection restored to meerkat-OST000e (at 172.25.32.243@tcp) Lustre: Skipped 11 previous similar messages Lustre: 3330:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415100744/real 1415100747] req@ffff88062c3d8400 x1483597867828508/t0(0) o13->meerkat-OST000a-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415100756 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3330:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 48 previous similar messages LustreError: 11-0: meerkat-OST000a-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: Skipped 12 previous similar messages Lustre: meerkat-OST003a-osc: Connection restored to meerkat-OST003a (at 172.25.32.248@tcp) Lustre: Skipped 2 previous similar messages Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 47 previous similar messages Lustre: meerkat-MDT0000: Client fd825eb2-1223-8979-93ea-5a1cfbdf1776 (at 10.7.103.251@o2ib) reconnecting Lustre: Skipped 923 previous similar messages LustreError: 11-0: meerkat-OST000e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 2 previous similar messages Lustre: meerkat-MDT0000: Client 604340f3-490f-c04f-1cf3-5dc9d71f90e8 (at 10.7.103.172@o2ib) reconnecting Lustre: Skipped 159 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415101655/real 1415101665] req@ffff88062b4bd800 x1483597868850024/t0(0) o13->meerkat-OST003c-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415101671 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 144 previous similar messages Lustre: meerkat-OST003c-osc: Connection to meerkat-OST003c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 18 previous similar messages Lustre: meerkat-OST003c-osc: Connection restored to meerkat-OST003c (at 172.25.32.115@tcp) Lustre: Skipped 24 previous similar messages Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415101714/real 0] req@ffff880636db9c00 x1483597868891340/t0(0) o6->meerkat-OST003c-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415101730 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3335:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 31 previous similar messages Lustre: meerkat-MDT0000: Client 4b4293da-929d-4ecd-6e11-849ae79785d1 (at 10.7.102.127@o2ib) refused reconnection, still busy with 1 active RPCs LustreError: 11-0: meerkat-OST0036-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 4 previous similar messages Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 15 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415101840/real 1415101849] req@ffff8802b0f71400 x1483597868981116/t0(0) o13->meerkat-OST000e-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415101855 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 132 previous similar messages Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: Skipped 19 previous similar messages LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff88001315c900/0x661ae12110e8294 lrc: 3/0,0 mode: PR/PR res: [0x2000061e0:0x151c4:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 192.168.230.53@tcp remote: 0x7185dbd0c2e5253a expref: 103 pid: 16128 timeout: 4527966804 lvb_type: 0 LustreError: 13039:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff8802f1c7b400 x1474470017204276/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 0 to 0 dl 1415102024 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: 13039:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (120:14s); client may timeout. req@ffff8802f1c7b400 x1474470017204276/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/408 e 0 to 0 dl 1415102024 ref 1 fl Complete:/0/0 rc -107/-107 Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415102091/real 1415102100] req@ffff8802e0d67c00 x1483597869086436/t0(0) o13->meerkat-OST0036-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415102110 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3332:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 20 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3396:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415102168 with bad export cookie 459840029589164582 Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 12 previous similar messages Lustre: meerkat-MDT0000: Client 643ef51d-da34-406c-42d1-6153042f93f5 (at 10.7.101.164@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 240a5506-2bb5-1803-8773-4c7c570fa8b2 (at 10.7.104.47@o2ib) reconnecting Lustre: Skipped 825 previous similar messages Lustre: meerkat-OST000c-osc: Connection to meerkat-OST000c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 64 previous similar messages LustreError: 11-0: meerkat-OST002c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. Lustre: meerkat-MDT0000: Client f8ee92d4-3bc1-7fca-234a-43af75dc6463 (at 10.7.101.153@o2ib) refused reconnection, still busy with 2 active RPCs LustreError: Skipped 6 previous similar messages Lustre: 13056:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415102440/real 1415102444] req@ffff8800b020c800 x1483597869399016/t0(0) o104->meerkat-MDT0000@192.168.230.53@tcp:15/16 lens 296/224 e 0 to 1 dl 1415102455 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 13056:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 184 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3397:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415102462 with bad export cookie 459840029592549243 LustreError: 11-0: meerkat-OST000c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 2 previous similar messages Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 31 previous similar messages LustreError: 11-0: meerkat-OST0032-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: Skipped 4 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3396:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415102579 with bad export cookie 459840029599760209 LustreError: 11-0: meerkat-OST002c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 7 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3396:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415102630 with bad export cookie 459840029602762208 LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3397:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415102676 with bad export cookie 459840029604847445 LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3396:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415102739 with bad export cookie 459840029607583773 LustreError: 11-0: meerkat-OST0024-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. Lustre: meerkat-MDT0000: Client 8fd7208f-6750-22cf-5fcb-66be7c5417f9 (at 10.7.102.254@o2ib) reconnecting Lustre: Skipped 288 previous similar messages Lustre: meerkat-MDT0000: Client 0d91792a-103f-eb01-3cbd-8a27ee8d05e1 (at 10.7.100.177@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-OST000a-osc: Connection restored to meerkat-OST000a (at 172.25.32.248@tcp) Lustre: Skipped 51 previous similar messages LustreError: 11-0: meerkat-OST002a-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415103396/real 1415103406] req@ffff8800056ef000 x1483597870320632/t0(0) o6->meerkat-OST000e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415103407 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 334 previous similar messages Lustre: meerkat-OST000e-osc: Connection to meerkat-OST000e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 75 previous similar messages LustreError: 11-0: meerkat-OST000e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. Lustre: meerkat-MDT0000: Client b6bd3474-af36-9e6a-ad8e-971c4ac406f0 (at 10.7.103.138@o2ib) reconnecting Lustre: Skipped 80 previous similar messages LustreError: 11-0: meerkat-OST003a-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST0002-osc: Connection restored to meerkat-OST0002 (at 172.25.32.248@tcp) Lustre: Skipped 14 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 17508:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415103802 with bad export cookie 459840029610384109 Lustre: meerkat-MDT0000: Client 5e6cdb3e-b53d-8452-b1f5-c464b4dcf978 (at 10.7.103.207@o2ib) reconnecting Lustre: Skipped 16 previous similar messages Lustre: 10982:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415104252/real 1415104253] req@ffff880314a52000 x1483597871166388/t0(0) o104->meerkat-MDT0000@192.168.230.53@tcp:15/16 lens 296/224 e 0 to 1 dl 1415104263 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 10982:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 41 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 17508:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415104265 with bad export cookie 459840029731605566 Lustre: meerkat-OST000a-osc: Connection to meerkat-OST000a (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 11-0: meerkat-OST000a-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: Skipped 7 previous similar messages Lustre: meerkat-MDT0000: Client 551d66d1-c308-5d9d-d3c2-baa50f7371c0 (at 198.202.119.67@tcp) refused reconnection, still busy with 1 active RPCs Lustre: MGS: Client 90a4b178-38c3-539b-9e40-1729d8abcc72 (at 10.7.103.139@o2ib) reconnecting Lustre: Skipped 95 previous similar messages LNet: Service thread pid 4738 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 4738, comm: mdt_rdpg00_004 Call Trace: [] ? shrink_inactive_list+0x343/0x830 [] ? shrink_active_list+0x297/0x370 [] shrink_mem_cgroup_zone+0x3ae/0x610 [] ? mem_cgroup_iter+0xfd/0x280 [] shrink_zone+0x63/0xb0 [] zone_reclaim+0x349/0x400 [] ? mempool_alloc_slab+0x15/0x20 [] get_page_from_freelist+0x69c/0x830 [] ? native_sched_clock+0x13/0x80 [] __alloc_pages_nodemask+0x113/0x8d0 [] ? blk_queue_bio+0x121/0x5d0 [] ? perf_event_task_sched_out+0x33/0x80 [] ? mempool_alloc_slab+0x15/0x20 [] alloc_pages_current+0xaa/0x110 [] __page_cache_alloc+0x87/0x90 [] find_or_create_page+0x4f/0xb0 [] __getblk+0xed/0x2a0 [] __breadahead+0x12/0x40 [] __ldiskfs_get_inode_loc+0x33e/0x3b0 [ldiskfs] [] ldiskfs_iget+0x86/0x800 [ldiskfs] [] osd_iget+0x2e/0x2c0 [osd_ldiskfs] [] osd_ea_fid_get+0x176/0x2c0 [osd_ldiskfs] [] ? zone_statistics+0x99/0xc0 [] osd_it_ea_rec+0xb45/0x1470 [osd_ldiskfs] [] ? call_filldir+0xb5/0x150 [ldiskfs] [] ? ldiskfs_readdir+0x5a9/0x730 [ldiskfs] [] ? osd_ldiskfs_filldir+0x0/0x480 [osd_ldiskfs] [] lod_it_rec+0x21/0x90 [lod] [] mdd_dir_page_build+0xfc/0x210 [mdd] [] dt_index_walk+0x162/0x3d0 [obdclass] [] ? mdd_dir_page_build+0x0/0x210 [mdd] [] mdd_readpage+0x38b/0x5a0 [mdd] [] mdt_readpage+0x47f/0x960 [mdt] [] mdt_handle_common+0x647/0x16d0 [mdt] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] mds_readpage_handle+0x15/0x20 [mdt] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415105117.4738 LNet: Service thread pid 4738 completed after 205.16s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415105321/real 0] req@ffff8803cb25e400 x1483597873225340/t0(0) o13->meerkat-OST0026-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415105335 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 19 previous similar messages Lustre: meerkat-OST0026-osc: Connection to meerkat-OST0026 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages Lustre: meerkat-OST0026-osc: Connection restored to meerkat-OST0026 (at 172.25.32.243@tcp) Lustre: Skipped 15 previous similar messages Lustre: meerkat-MDT0000: Client 353c4913-a792-976e-a324-cfe5bf8d2c6b (at 10.7.103.88@o2ib) reconnecting Lustre: Skipped 202 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 17508:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415105667 with bad export cookie 459840029792694713 LustreError: 11-0: meerkat-OST001e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: meerkat-OST0026-osc: Connection restored to meerkat-OST0026 (at 172.25.32.243@tcp) Lustre: MGS: Client 4391f90b-76ce-dbc4-a76f-6290568d58dd (at 10.7.101.159@o2ib) reconnecting Lustre: Skipped 87 previous similar messages Lustre: 3329:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415106939/real 1415106951] req@ffff8803ff126000 x1483597876271412/t0(0) o13->meerkat-OST0014-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415106954 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3329:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 44 previous similar messages Lustre: meerkat-OST0014-osc: Connection to meerkat-OST0014 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 7 previous similar messages LustreError: 11-0: meerkat-OST0004-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST0014-osc: Connection restored to meerkat-OST0014 (at 172.25.32.115@tcp) Lustre: Skipped 6 previous similar messages Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 5 previous similar messages Lustre: 3334:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415107108/real 0] req@ffff8804e7fde000 x1483597876460844/t0(0) o6->meerkat-OST0032-osc@172.25.32.248@tcp:28/4 lens 664/432 e 0 to 1 dl 1415107117 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3334:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Lustre: meerkat-OST0032-osc: Connection to meerkat-OST0032 (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 7 previous similar messages Lustre: meerkat-OST0032-osc: Connection restored to meerkat-OST0032 (at 172.25.32.248@tcp) Lustre: Skipped 1 previous similar message LustreError: 11-0: meerkat-OST0022-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: meerkat-MDT0000: Client 5a590ecc-e12f-428a-22b6-35ae3614b1e5 (at 10.7.104.37@o2ib) reconnecting Lustre: Skipped 76 previous similar messages Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415107846/real 0] req@ffff880316987c00 x1483597877733116/t0(0) o6->meerkat-OST003c-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415107867 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3321:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Lustre: meerkat-OST003c-osc: Connection to meerkat-OST003c (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages LustreError: 11-0: meerkat-OST0016-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 2 previous similar messages Lustre: meerkat-OST000e-osc: Connection restored to meerkat-OST000e (at 172.25.32.243@tcp) Lustre: Skipped 6 previous similar messages Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 4 previous similar messages Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs LustreError: 178:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 116s: evicting client at 198.202.118.30@tcp ns: mdt-meerkat-MDT0000_UUID lock: ffff880290fa1d80/0x661ae1239f3f636 lrc: 3/0,0 mode: PR/PR res: [0x200001cb3:0x4794:0x0].0 bits 0x1b rrc: 2 type: IBT flags: 0x200000000020 nid: 198.202.118.30@tcp remote: 0xeb38bc1926f03ce2 expref: 21864 pid: 3421 timeout: 4534017815 lvb_type: 0 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415108852/real 0] req@ffff8803c7d14800 x1483597879483188/t0(0) o6->meerkat-OST0016-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415108861 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3324:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 113 previous similar messages Lustre: meerkat-OST0016-osc: Connection to meerkat-OST0016 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 11-0: meerkat-OST000c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 10 previous similar messages Lustre: meerkat-OST0016-osc: Connection restored to meerkat-OST0016 (at 172.25.32.243@tcp) Lustre: Skipped 10 previous similar messages Lustre: meerkat-OST000c-osc: Connection restored to meerkat-OST000c (at 172.25.32.115@tcp) Lustre: MGS: Client 74b87a81-45c4-2896-d16e-dfa539ef3f37 (at 10.7.103.89@o2ib) reconnecting Lustre: Skipped 98 previous similar messages Lustre: 3334:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415109321/real 1415109325] req@ffff880437902000 x1483597880025712/t0(0) o13->meerkat-OST0016-osc@172.25.32.243@tcp:7/4 lens 224/368 e 0 to 1 dl 1415109336 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3334:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Lustre: meerkat-OST0016-osc: Connection to meerkat-OST0016 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 11-0: meerkat-OST0016-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST0036-osc: Connection restored to meerkat-OST0036 (at 172.25.32.243@tcp) Lustre: meerkat-OST0016-osc: Connection restored to meerkat-OST0016 (at 172.25.32.243@tcp) Lustre: Skipped 6 previous similar messages LustreError: 11-0: meerkat-OST0012-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST0012-osc: Connection restored to meerkat-OST0012 (at 172.25.32.248@tcp) Lustre: Skipped 4 previous similar messages Lustre: meerkat-MDT0000: Client 4585fba1-34eb-db82-2ec4-112de0e24605 (at 10.7.101.32@o2ib) reconnecting Lustre: Skipped 72 previous similar messages Lustre: 3331:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415109878/real 1415109883] req@ffff880639638800 x1483597880732996/t0(0) o6->meerkat-OST0034-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415109896 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3331:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST0034-osc: Connection restored to meerkat-OST0034 (at 172.25.32.115@tcp) Lustre: Skipped 3 previous similar messages Lustre: MGS: Client 129d2a15-cea6-5690-7239-ee557bbfe5a2 (at 10.7.104.31@o2ib) reconnecting Lustre: Skipped 32 previous similar messages Lustre: meerkat-MDT0000: Client 00a8781a-60e9-4cbc-7903-4e800761e303 (at 10.7.103.206@o2ib) reconnecting Lustre: Skipped 9 previous similar messages Lustre: 3336:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415110543/real 0] req@ffff8806391d6400 x1483597881654080/t0(0) o6->meerkat-OST0014-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415110555 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3336:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Lustre: meerkat-OST0014-osc: Connection to meerkat-OST0014 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 10 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 3395:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415110557 with bad export cookie 459840029944776979 Lustre: meerkat-OST0024-osc: Connection restored to meerkat-OST0024 (at 172.25.32.115@tcp) Lustre: Skipped 10 previous similar messages LustreError: 11-0: meerkat-OST000e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 3 previous similar messages Lustre: 13050:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415110886/real 1415110886] req@ffff88006a628000 x1483597882268176/t0(0) o104->meerkat-MDT0000@192.168.230.53@tcp:15/16 lens 296/224 e 0 to 1 dl 1415110897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 13050:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 60 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 17508:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 198.202.118.101@tcp arrived at 1415110902 with bad export cookie 459840023463754466 LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: Skipped 2 previous similar messages LustreError: 17508:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415110937 with bad export cookie 459840030661506202 LustreError: 17508:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) Skipped 2 previous similar messages Lustre: meerkat-OST003a-osc: Connection to meerkat-OST003a (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 11-0: meerkat-OST000a-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST0032-osc: Connection restored to meerkat-OST0032 (at 172.25.32.248@tcp) Lustre: Skipped 15 previous similar messages LustreError: 11-0: meerkat-OST0016-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: meerkat-MDT0000: Client b8523ede-3cb9-6546-a498-b71829db9de7 (at 10.7.103.233@o2ib) reconnecting Lustre: Skipped 241 previous similar messages Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415111775/real 0] req@ffff88039c8b1400 x1483597884024780/t0(0) o13->meerkat-OST0024-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415111796 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3337:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 163 previous similar messages Lustre: meerkat-OST0024-osc: Connection to meerkat-OST0024 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 23 previous similar messages LustreError: 11-0: meerkat-OST002c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 2 previous similar messages Lustre: meerkat-OST003c-osc: Connection restored to meerkat-OST003c (at 172.25.32.115@tcp) Lustre: Skipped 23 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 198.202.118.30@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 11192:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 198.202.118.30@tcp arrived at 1415111991 with bad export cookie 459840030291557266 LustreError: 11-0: meerkat-OST002c-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 14 previous similar messages LustreError: 3475:0:(osp_precreate.c:484:osp_precreate_send()) meerkat-OST0002-osc: can't precreate: rc = -11 LustreError: 3475:0:(osp_precreate.c:989:osp_precreate_thread()) meerkat-OST0002-osc: cannot precreate objects: rc = -11 LustreError: 11-0: meerkat-OST0002-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: Skipped 9 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 10.7.102.127@o2ib was evicted due to a lock blocking callback time out: rc -107 LustreError: 6155:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 10.7.102.127@o2ib arrived at 1415112208 with bad export cookie 459840026290041867 Lustre: MGS: Client 27328d65-fd9a-d7d6-2c62-a94cb9ce1f44 (at 10.7.103.189@o2ib) reconnecting Lustre: Skipped 71 previous similar messages LustreError: 11-0: meerkat-OST001e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415112873/real 0] req@ffff88041d1ae000 x1483597886580096/t0(0) o6->meerkat-OST0036-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415112891 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 382 previous similar messages Lustre: meerkat-OST0036-osc: Connection to meerkat-OST0036 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 48 previous similar messages Lustre: meerkat-OST0016-osc: Connection restored to meerkat-OST0016 (at 172.25.32.243@tcp) Lustre: Skipped 48 previous similar messages Lustre: MGS: Client c6f0ad96-6f3b-0e22-7eb0-32d00a512479 (at 10.7.100.171@o2ib) refused reconnection, still busy with 1 active RPCs Lustre: MGS: Client c6f0ad96-6f3b-0e22-7eb0-32d00a512479 (at 10.7.100.171@o2ib) reconnecting Lustre: Skipped 201 previous similar messages Lustre: MGS: Client 0c823b03-f9b4-bd9a-49da-7a6eceb1b8da (at 10.7.104.20@o2ib) reconnecting Lustre: Skipped 6 previous similar messages Lustre: MGS: Client 7a5fb360-063f-24c6-d8f7-a9cef7c94c93 (at 10.7.103.99@o2ib) reconnecting Lustre: Skipped 98 previous similar messages Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415114897/real 1415114909] req@ffff8806390fd400 x1483597889666232/t0(0) o6->meerkat-OST002e-osc@172.25.32.243@tcp:28/4 lens 664/432 e 0 to 1 dl 1415114916 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3333:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 39 previous similar messages Lustre: meerkat-OST002e-osc: Connection to meerkat-OST002e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 4 previous similar messages Lustre: meerkat-MDT0000: Client 1030e9db-653a-eaea-5985-abb781a6c220 (at 10.7.104.24@o2ib) reconnecting Lustre: Skipped 13 previous similar messages Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415115082/real 1415115086] req@ffff880313d29000 x1483597890054408/t0(0) o13->meerkat-OST003a-osc@172.25.32.248@tcp:7/4 lens 224/368 e 0 to 1 dl 1415115093 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Lustre: meerkat-OST003a-osc: Connection to meerkat-OST003a (at 172.25.32.248@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 11 previous similar messages Lustre: meerkat-OST0002-osc: Connection restored to meerkat-OST0002 (at 172.25.32.248@tcp) Lustre: Skipped 11 previous similar messages LustreError: 11-0: meerkat-OST002a-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. LustreError: Skipped 14 previous similar messages Lustre: 3329:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415115319/real 0] req@ffff880637c92000 x1483597890644440/t0(0) o6->meerkat-OST0004-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415115338 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3329:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 72 previous similar messages Lustre: meerkat-OST0004-osc: Connection to meerkat-OST0004 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 7 previous similar messages LustreError: 11-0: meerkat-OST0004-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. LustreError: Skipped 1 previous similar message Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 7 previous similar messages LustreError: 11-0: meerkat-OST0002-osc: Communicating with 172.25.32.248@tcp, operation ost_connect failed with -16. Lustre: MGS: Client f7770193-019d-1ec0-9bc8-dfc088e62217 (at 10.7.103.119@o2ib) reconnecting Lustre: Skipped 69 previous similar messages Lustre: meerkat-MDT0000: Client 0bf744b3-2fe8-8236-cd27-d04e85730316 (at 10.7.104.45@o2ib) reconnecting Lustre: Skipped 135 previous similar messages Lustre: 8922:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415116758/real 1415116768] req@ffff8803162d8800 x1483597892313980/t0(0) o104->meerkat-MDT0000@192.168.230.53@tcp:15/16 lens 296/224 e 0 to 1 dl 1415116770 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 8922:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 Lustre: meerkat-OST0024-osc: Connection to meerkat-OST0024 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 3396:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415116787 with bad export cookie 459840030667912196 LustreError: 11-0: meerkat-OST0024-osc: Communicating with 172.25.32.115@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST0016-osc: Connection restored to meerkat-OST0016 (at 172.25.32.243@tcp) Lustre: Skipped 1 previous similar message Lustre: 13056:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415117050/real 1415117057] req@ffff8800b4c44400 x1483597892742584/t0(0) o104->meerkat-MDT0000@192.168.230.53@tcp:15/16 lens 296/224 e 0 to 1 dl 1415117060 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 13056:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 142 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 13023:0:(ldlm_lib.c:2706:target_bulk_io()) @@@ Eviction on bulk PUT req@ffff880313aa8800 x1474470084818608/t0(0) o37->453cd0d9-c8e3-0e50-da9e-7953a9c89205@192.168.230.53@tcp:0/0 lens 448/440 e 0 to 0 dl 1415117107 ref 1 fl Interpret:/0/0 rc 0/0 Lustre: meerkat-OST002e-osc: Connection to meerkat-OST002e (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 17500:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415117069 with bad export cookie 459840031467710300 LustreError: 11-0: meerkat-OST002e-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. LustreError: Skipped 11 previous similar messages Lustre: meerkat-OST0024-osc: Connection restored to meerkat-OST0024 (at 172.25.32.115@tcp) Lustre: Skipped 15 previous similar messages LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 LustreError: 17500:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415117106 with bad export cookie 459840031491800688 Lustre: meerkat-MDT0000: Client f8ee92d4-3bc1-7fca-234a-43af75dc6463 (at 10.7.101.153@o2ib) refused reconnection, still busy with 2 active RPCs Lustre: MGS: Client 39f5069f-2c6f-d283-1258-f61c7bc7ffaf (at 10.7.103.224@o2ib) reconnecting Lustre: Skipped 62 previous similar messages Lustre: MGS: Client 3a2bff1d-ee2b-0ba9-fcab-e7935f24ce5d (at 10.7.103.171@o2ib) reconnecting Lustre: Skipped 9 previous similar messages Lustre: 3330:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415117945/real 0] req@ffff88062522fc00 x1483597894420732/t0(0) o6->meerkat-OST0034-osc@172.25.32.115@tcp:28/4 lens 664/432 e 0 to 1 dl 1415117958 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3330:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Lustre: meerkat-OST0034-osc: Connection to meerkat-OST0034 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 138-a: meerkat-MDT0000: A client on nid 192.168.230.53@tcp was evicted due to a lock blocking callback time out: rc -107 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415117958/real 0] req@ffff880624123c00 x1483597894426108/t0(0) o8->meerkat-OST002e-osc@172.25.32.243@tcp:28/4 lens 400/544 e 0 to 1 dl 1415117970 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 144 previous similar messages LustreError: 3396:0:(ldlm_lockd.c:2348:ldlm_cancel_handler()) ldlm_cancel from 192.168.230.53@tcp arrived at 1415117971 with bad export cookie 459840031497161050 Lustre: meerkat-OST0004-osc: Connection restored to meerkat-OST0004 (at 172.25.32.115@tcp) Lustre: Skipped 1 previous similar message Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) reconnecting Lustre: Skipped 162 previous similar messages Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8c5b20e1-4e92-4313-0ab5-a06d9154e563 (at 198.202.118.30@tcp) refused reconnection, still busy with 1 active RPCs LNet: Service thread pid 6021 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 6021, comm: mdt_rdpg01_004 Call Trace: [] ? try_to_free_buffers+0x51/0xc0 [] ? jbd2_journal_try_to_free_buffers+0xa7/0x150 [jbd2] [] ? bdev_try_to_free_page+0x48/0x90 [ldiskfs] [] ? shrink_page_list.clone.3+0xd0/0x650 [] ? mem_cgroup_lru_del_list+0x2b/0xb0 [] ? isolate_lru_pages.clone.0+0xd7/0x170 [] ? shrink_inactive_list+0x191/0x830 [] ? shrink_active_list+0x297/0x370 [] ? shrink_mem_cgroup_zone+0x3ae/0x610 [] ? mem_cgroup_iter+0xfd/0x280 [] ? shrink_zone+0x63/0xb0 [] ? zone_reclaim+0x349/0x400 [] ? get_page_from_freelist+0x69c/0x830 [] ? __alloc_pages_nodemask+0x113/0x8d0 [] ? cfs_alloc+0x30/0x60 [libcfs] [] ? alloc_pages_current+0xaa/0x110 [] ? cfs_alloc_page+0x17/0x20 [libcfs] [] ? mdt_readpage+0x1de/0x960 [mdt] [] ? mdt_handle_common+0x647/0x16d0 [mdt] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ? mds_readpage_handle+0x15/0x20 [mdt] [] ? ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? default_wake_function+0x0/0x20 [] ? ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415118639.6021 LustreError: 6021:0:(ldlm_lib.c:2730:target_bulk_io()) @@@ bulk PUT failed: rc -107 req@ffff8803187a4850 x1482141770819132/t0(0) o37->8c5b20e1-4e92-4313-0ab5-a06d9154e563@198.202.118.30@tcp:0/0 lens 448/440 e 2 to 0 dl 1415118847 ref 1 fl Interpret:/0/0 rc 0/0 LNet: Service thread pid 6021 completed after 204.99s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: meerkat-MDT0000: Client 3368e781-3fb3-27f7-53cb-7ad4aae5a7fa (at 198.202.118.50@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 8eb77cae-811b-e0a8-2317-191a1ef75f05 (at 198.202.119.12@tcp) refused reconnection, still busy with 1 active RPCs Lustre: meerkat-MDT0000: Client 3368e781-3fb3-27f7-53cb-7ad4aae5a7fa (at 198.202.118.50@tcp) refused reconnection, still busy with 1 active RPCs INFO: task kswapd0:178 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kswapd0 D 000000000000000a 0 178 2 0x00000000 ffff880338329a80 0000000000000046 0000000000000000 ffff880338329a50 ffffea00025a7ab8 ffff880338329b50 0000000000000020 000000000000001f ffff880338327ab8 ffff880338329fd8 000000000000fb88 ffff880338327ab8 Call Trace: [] start_this_handle+0x27a/0x4a0 [jbd2] [] ? autoremove_wake_function+0x0/0x40 [] jbd2_journal_start+0xd0/0x110 [jbd2] [] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs] [] ldiskfs_dquot_drop+0x34/0x80 [ldiskfs] [] vfs_dq_drop+0x52/0x60 [] clear_inode+0x93/0x140 [] dispose_list+0x40/0x120 [] shrink_icache_memory+0x274/0x2e0 [] shrink_slab+0x12a/0x1a0 [] balance_pgdat+0x59a/0x820 [] kswapd+0x134/0x3c0 [] ? autoremove_wake_function+0x0/0x40 [] ? kswapd+0x0/0x3c0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 INFO: task kswapd1:179 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kswapd1 D 0000000000000004 0 179 2 0x00000000 ffff88033832da80 0000000000000046 0000000000000000 ffff88033832da50 ffffea000d70b090 ffff88033832db50 0000000000000020 0000000000000010 ffff880338327058 ffff88033832dfd8 000000000000fb88 ffff880338327058 Call Trace: [] start_this_handle+0x27a/0x4a0 [jbd2] [] ? autoremove_wake_function+0x0/0x40 [] jbd2_journal_start+0xd0/0x110 [jbd2] [] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs] [] ldiskfs_dquot_drop+0x34/0x80 [ldiskfs] [] vfs_dq_drop+0x52/0x60 [] clear_inode+0x93/0x140 [] dispose_list+0x40/0x120 [] shrink_icache_memory+0x274/0x2e0 [] shrink_slab+0x12a/0x1a0 [] balance_pgdat+0x59a/0x820 [] kswapd+0x134/0x3c0 [] ? autoremove_wake_function+0x0/0x40 [] ? kswapd+0x0/0x3c0 [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 INFO: task jbd2/md0-8:3363 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. jbd2/md0-8 D 0000000000000000 0 3363 2 0x00000080 ffff880637cedd20 0000000000000046 0000000000000000 ffff880339fba000 ffff880548a52800 ffff880548a52800 ffff880637cedca0 ffff88063cd67540 ffff88063cd67af8 ffff880637cedfd8 000000000000fb88 ffff88063cd67af8 Call Trace: [] jbd2_journal_commit_transaction+0x19f/0x15a0 [jbd2] [] ? __switch_to+0xd0/0x320 [] ? lock_timer_base+0x3c/0x70 [] ? autoremove_wake_function+0x0/0x40 [] kjournald2+0xb8/0x220 [jbd2] [] ? autoremove_wake_function+0x0/0x40 [] ? kjournald2+0x0/0x220 [jbd2] [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 INFO: task mdt03_002:3429 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. mdt03_002 D 000000000000000e 0 3429 2 0x00000080 ffff88031872b9c0 0000000000000046 0000000000000000 ffff88063bd85ac0 0000000000000000 ffff8804044e4240 ffff88031872b990 ffff880632061aa0 ffff880318725098 ffff88031872bfd8 000000000000fb88 ffff880318725098 Call Trace: [] start_this_handle+0x27a/0x4a0 [jbd2] [] ? cache_alloc_refill+0x15b/0x240 [] ? autoremove_wake_function+0x0/0x40 [] jbd2_journal_start+0xd0/0x110 [jbd2] [] ? mdt_txn_start_cb+0xf9/0x340 [mdt] [] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs] [] osd_trans_start+0x1df/0x680 [osd_ldiskfs] [] lod_trans_start+0x1b9/0x250 [lod] [] mdd_trans_start+0x17/0x20 [mdd] [] mdd_attr_set+0x4a3/0x1390 [mdd] [] mdt_attr_set+0x268/0x560 [mdt] [] mdt_reint_setattr+0x5bd/0xcf0 [mdt] [] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] [] mdt_reint_rec+0x41/0xe0 [mdt] [] mdt_reint_internal+0x4c3/0x780 [mdt] [] mdt_reint+0x44/0xe0 [mdt] [] mdt_handle_common+0x647/0x16d0 [mdt] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] mds_regular_handle+0x15/0x20 [mdt] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 INFO: task mdt_rdpg01_000:3432 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. mdt_rdpg01_00 D 000000000000000b 0 3432 2 0x00000080 ffff8803180a7a60 0000000000000046 0000000000000000 ffff8803388d3700 0000000000000002 ffff880336368000 ffff88028b6b4278 ffff88028b6b42d0 ffff88031809d058 ffff8803180a7fd8 000000000000fb88 ffff88031809d058 Call Trace: [] start_this_handle+0x27a/0x4a0 [jbd2] [] ? cache_alloc_refill+0x15b/0x240 [] ? autoremove_wake_function+0x0/0x40 [] jbd2_journal_start+0xd0/0x110 [jbd2] [] ? mdt_txn_start_cb+0xf9/0x340 [mdt] [] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs] [] osd_trans_start+0x1df/0x680 [osd_ldiskfs] [] lod_trans_start+0x1b9/0x250 [lod] [] mdd_trans_start+0x17/0x20 [mdd] [] mdd_close+0x6be/0xb80 [mdd] [] mdt_mfd_close+0x129/0x6e0 [mdt] [] mdt_close+0x67a/0xab0 [mdt] [] mdt_handle_common+0x647/0x16d0 [mdt] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] mds_readpage_handle+0x15/0x20 [mdt] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 INFO: task mdt_rdpg01_001:3433 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. mdt_rdpg01_00 D 000000000000000a 0 3433 2 0x00000080 ffff8803180b5a50 0000000000000046 0000000000000000 ffff8803388d3500 0000000000000000 ffff88033fd10440 ffff880315ea7840 ffff88033fc217c0 ffff88031809c5f8 ffff8803180b5fd8 000000000000fb88 ffff88031809c5f8 Call Trace: [] start_this_handle+0x27a/0x4a0 [jbd2] [] ? cache_alloc_refill+0x15b/0x240 [] ? autoremove_wake_function+0x0/0x40 [] jbd2_journal_start+0xd0/0x110 [jbd2] [] ? mdt_txn_start_cb+0xf9/0x340 [mdt] [] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs] [] osd_trans_start+0x1df/0x680 [osd_ldiskfs] [] lod_trans_start+0x1b9/0x250 [lod] [] mdd_trans_start+0x17/0x20 [mdd] [] mdd_attr_set+0x4a3/0x1390 [mdd] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] [] mdt_mfd_close+0x502/0x6e0 [mdt] [] mdt_close+0x67a/0xab0 [mdt] [] mdt_handle_common+0x647/0x16d0 [mdt] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] mds_readpage_handle+0x15/0x20 [mdt] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 INFO: task mdt_rdpg02_000:3434 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. mdt_rdpg02_00 D 000000000000000c 0 3434 2 0x00000080 ffff8803180c3a50 0000000000000046 0000000000000000 ffff880632fc5e00 ffff88062f1dcd80 ffff8806382309c0 0000000000000000 00000000000001a8 ffff8803180c1af8 ffff8803180c3fd8 000000000000fb88 ffff8803180c1af8 Call Trace: [] start_this_handle+0x27a/0x4a0 [jbd2] [] ? cache_alloc_refill+0x15b/0x240 [] ? autoremove_wake_function+0x0/0x40 [] jbd2_journal_start+0xd0/0x110 [jbd2] [] ? mdt_txn_start_cb+0xf9/0x340 [mdt] [] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs] [] osd_trans_start+0x1df/0x680 [osd_ldiskfs] [] lod_trans_start+0x1b9/0x250 [lod] [] mdd_trans_start+0x17/0x20 [mdd] [] mdd_attr_set+0x4a3/0x1390 [mdd] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] [] mdt_mfd_close+0x502/0x6e0 [mdt] [] mdt_close+0x67a/0xab0 [mdt] [] mdt_handle_common+0x647/0x16d0 [mdt] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] mds_readpage_handle+0x15/0x20 [mdt] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 INFO: task osp-syn-4:3481 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. osp-syn-4 D 0000000000000005 0 3481 2 0x00000080 ffff88062f469700 0000000000000046 0000000000000000 0002022000000000 0000000000000282 0000000000000010 ffff88008a5aab50 ffff88062f469e80 ffff88063a2665f8 ffff88062f469fd8 000000000000fb88 ffff88063a2665f8 Call Trace: [] start_this_handle+0x27a/0x4a0 [jbd2] [] ? autoremove_wake_function+0x0/0x40 [] jbd2_journal_start+0xd0/0x110 [jbd2] [] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs] [] osd_trans_start+0x1df/0x680 [osd_ldiskfs] [] llog_write+0x22c/0x440 [obdclass] [] llog_cancel_rec+0xbc/0x7c0 [obdclass] [] llog_cat_cancel_records+0x107/0x340 [obdclass] [] osp_sync_process_committed+0x231/0x750 [osp] [] osp_sync_process_queues+0x94/0x15e0 [osp] [] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs] [] ? default_wake_function+0x0/0x20 [] llog_process_thread+0x8fb/0xe00 [obdclass] [] ? osp_sync_process_queues+0x0/0x15e0 [osp] [] llog_process_or_fork+0x12d/0x660 [obdclass] [] llog_cat_process_cb+0x56a/0x620 [obdclass] [] llog_process_thread+0x8fb/0xe00 [obdclass] [] ? llog_cat_process_cb+0x0/0x620 [obdclass] [] llog_process_or_fork+0x12d/0x660 [obdclass] [] llog_cat_process_or_fork+0x89/0x350 [obdclass] [] ? __wake_up_common+0x59/0x90 [] ? osp_sync_process_queues+0x0/0x15e0 [osp] [] llog_cat_process+0x19/0x20 [obdclass] [] ? cfs_waitq_signal+0x1a/0x20 [libcfs] [] osp_sync_thread+0x240/0x7e0 [osp] [] ? osp_sync_thread+0x0/0x7e0 [osp] [] child_rip+0xa/0x20 [] ? osp_sync_thread+0x0/0x7e0 [osp] [] ? osp_sync_thread+0x0/0x7e0 [osp] [] ? child_rip+0x0/0x20 INFO: task osp-syn-12:3498 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. osp-syn-12 D 0000000000000000 0 3498 2 0x00000080 ffff88062f0ed700 0000000000000046 00000010000494a8 0000000000061250 ffff8800000494a8 0000000000000000 ffff88062f0ed6d0 ffff88062f0ede80 ffff88062f0ebab8 ffff88062f0edfd8 000000000000fb88 ffff88062f0ebab8 Call Trace: [] ? prepare_to_wait+0x4e/0x80 [] start_this_handle+0x27a/0x4a0 [jbd2] [] ? autoremove_wake_function+0x0/0x40 [] jbd2_journal_start+0xd0/0x110 [jbd2] [] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs] [] osd_trans_start+0x1df/0x680 [osd_ldiskfs] [] llog_write+0x22c/0x440 [obdclass] [] ? load_balance_fair+0x208/0x2f0 [] llog_cancel_rec+0xbc/0x7c0 [obdclass] [] llog_cat_cancel_records+0x107/0x340 [obdclass] [] osp_sync_process_committed+0x231/0x750 [osp] [] osp_sync_process_queues+0x94/0x15e0 [osp] [] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs] [] ? default_wake_function+0x0/0x20 [] llog_process_thread+0x8fb/0xe00 [obdclass] [] ? osp_sync_process_queues+0x0/0x15e0 [osp] [] llog_process_or_fork+0x12d/0x660 [obdclass] [] llog_cat_process_cb+0x56a/0x620 [obdclass] [] llog_process_thread+0x8fb/0xe00 [obdclass] [] ? llog_cat_process_cb+0x0/0x620 [obdclass] [] llog_process_or_fork+0x12d/0x660 [obdclass] [] llog_cat_process_or_fork+0x89/0x350 [obdclass] [] ? __wake_up_common+0x59/0x90 [] ? osp_sync_process_queues+0x0/0x15e0 [osp] [] llog_cat_process+0x19/0x20 [obdclass] [] ? cfs_waitq_signal+0x1a/0x20 [libcfs] [] osp_sync_thread+0x240/0x7e0 [osp] [] ? osp_sync_thread+0x0/0x7e0 [osp] [] child_rip+0xa/0x20 [] ? osp_sync_thread+0x0/0x7e0 [osp] [] ? osp_sync_thread+0x0/0x7e0 [osp] [] ? child_rip+0x0/0x20 INFO: task osp-syn-19:3514 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. osp-syn-19 D 0000000000000009 0 3514 2 0x00000080 ffff88062e4a7700 0000000000000046 0000000000000000 0002022000000000 0000000000000286 0000000000000010 ffff8801a44405f0 ffff88062e4a7e80 ffff88062e445098 ffff88062e4a7fd8 000000000000fb88 ffff88062e445098 Call Trace: [] start_this_handle+0x27a/0x4a0 [jbd2] [] ? autoremove_wake_function+0x0/0x40 [] jbd2_journal_start+0xd0/0x110 [jbd2] [] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs] [] osd_trans_start+0x1df/0x680 [osd_ldiskfs] [] llog_write+0x22c/0x440 [obdclass] [] llog_cancel_rec+0xbc/0x7c0 [obdclass] [] llog_cat_cancel_records+0x107/0x340 [obdclass] [] osp_sync_process_committed+0x231/0x750 [osp] [] osp_sync_process_queues+0x94/0x15e0 [osp] [] ? osd_object_read_unlock+0x8b/0xd0 [osd_ldiskfs] [] ? default_wake_function+0x0/0x20 [] llog_process_thread+0x8fb/0xe00 [obdclass] [] ? osp_sync_process_queues+0x0/0x15e0 [osp] [] llog_process_or_fork+0x12d/0x660 [obdclass] [] llog_cat_process_cb+0x56a/0x620 [obdclass] [] llog_process_thread+0x8fb/0xe00 [obdclass] [] ? llog_cat_process_cb+0x0/0x620 [obdclass] [] llog_process_or_fork+0x12d/0x660 [obdclass] [] llog_cat_process_or_fork+0x89/0x350 [obdclass] [] ? __wake_up_common+0x59/0x90 [] ? osp_sync_process_queues+0x0/0x15e0 [osp] [] llog_cat_process+0x19/0x20 [obdclass] [] ? cfs_waitq_signal+0x1a/0x20 [libcfs] [] osp_sync_thread+0x240/0x7e0 [osp] [] ? osp_sync_thread+0x0/0x7e0 [osp] [] child_rip+0xa/0x20 [] ? osp_sync_thread+0x0/0x7e0 [osp] [] ? osp_sync_thread+0x0/0x7e0 [osp] [] ? child_rip+0x0/0x20 Lustre: meerkat-MDT0000: Client 112aee85-df01-240f-a2da-abf3a40edd58 (at 198.202.118.93@tcp) refused reconnection, still busy with 1 active RPCs Lustre: Skipped 1 previous similar message LNet: Service thread pid 14332 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 14332, comm: mdt00_029 Call Trace: [] ? try_to_free_buffers+0x51/0xc0 [] ? jbd2_journal_try_to_free_buffers+0xa7/0x150 [jbd2] [] ? bdev_try_to_free_page+0x48/0x90 [ldiskfs] [] ? blkdev_releasepage+0x36/0x50 [] ? shrink_page_list.clone.3+0x517/0x650 [] ? mem_cgroup_lru_del_list+0x2b/0xb0 [] ? isolate_lru_pages.clone.0+0xd7/0x170 [] ? __pagevec_release+0x26/0x40 [] ? shrink_inactive_list+0x191/0x830 [] ? shrink_active_list+0x297/0x370 [] ? shrink_mem_cgroup_zone+0x3ae/0x610 [] ? mem_cgroup_iter+0xfd/0x280 [] ? shrink_zone+0x63/0xb0 [] ? zone_reclaim+0x349/0x400 [] ? mempool_alloc_slab+0x15/0x20 [] ? get_page_from_freelist+0x69c/0x830 [] ? native_sched_clock+0x13/0x80 [] ? __alloc_pages_nodemask+0x113/0x8d0 [] ? blk_queue_bio+0x121/0x5d0 [] ? cache_grow+0x2cf/0x320 [] ? mempool_alloc_slab+0x15/0x20 [] ? alloc_pages_current+0xaa/0x110 [] ? __page_cache_alloc+0x87/0x90 [] ? find_or_create_page+0x4f/0xb0 [] ? __getblk+0xed/0x2a0 [] ? __breadahead+0x12/0x40 [] ? __ldiskfs_get_inode_loc+0x33e/0x3b0 [ldiskfs] [] ? ldiskfs_get_inode_loc+0x1c/0x20 [ldiskfs] [] ? ldiskfs_reserve_inode_write+0x2d/0xa0 [ldiskfs] [] ? ldiskfs_mark_inode_dirty+0x4c/0x1f0 [ldiskfs] [] ? ldiskfs_dirty_inode+0x40/0x60 [ldiskfs] [] ? osd_ldiskfs_write_record+0x2d7/0x330 [osd_ldiskfs] [] ? osd_write+0x148/0x2a0 [osd_ldiskfs] [] ? dt_record_write+0x45/0x130 [obdclass] [] ? jbd2_journal_dirty_metadata+0xff/0x150 [jbd2] [] ? llog_osd_write_blob+0x57b/0x850 [obdclass] [] ? llog_osd_write_rec+0xb5e/0x1370 [obdclass] [] ? dynlock_unlock+0x96/0x140 [ldiskfs] [] ? iam_path_release+0x42/0x70 [osd_ldiskfs] [] ? llog_write_rec+0xc8/0x290 [obdclass] [] ? llog_cat_add_rec+0xad/0x480 [obdclass] [] ? llog_add+0x91/0x1d0 [obdclass] [] ? osp_sync_add_rec+0x247/0xaa0 [osp] [] ? osp_sync_add+0x7b/0x80 [osp] [] ? osp_object_destroy+0x106/0x150 [osp] [] ? lod_object_destroy+0x1a7/0x350 [lod] [] ? mdd_finish_unlink+0x229/0x380 [mdd] [] ? mdd_unlink+0x9dc/0xe30 [mdd] [] ? mdo_unlink+0x18/0x50 [mdt] [] ? mdt_reint_unlink+0x820/0x1010 [mdt] [] ? mdt_reint_rec+0x41/0xe0 [mdt] [] ? mdt_reint_internal+0x4c3/0x780 [mdt] [] ? mdt_reint+0x44/0xe0 [mdt] [] ? mdt_handle_common+0x647/0x16d0 [mdt] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ? mds_regular_handle+0x15/0x20 [mdt] [] ? ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ? ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415118919.14332 LNet: Service thread pid 19141 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 19141, comm: mdt_rdpg03_022 Call Trace: [] start_this_handle+0x27a/0x4a0 [jbd2] [] ? autoremove_wake_function+0x0/0x40 [] jbd2_journal_start+0xd0/0x110 [jbd2] [] ? mdt_txn_start_cb+0xf9/0x340 [mdt] [] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs] [] osd_trans_start+0x1df/0x680 [osd_ldiskfs] [] lod_trans_start+0x1b9/0x250 [lod] [] mdd_trans_start+0x17/0x20 [mdd] [] mdd_attr_set+0x4a3/0x1390 [mdd] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] [] mdt_mfd_close+0x502/0x6e0 [mdt] [] mdt_close+0x67a/0xab0 [mdt] [] mdt_handle_common+0x647/0x16d0 [mdt] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] mds_readpage_handle+0x15/0x20 [mdt] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415118927.19141 LNet: Service thread pid 3712 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 3712, comm: mdt02_005 Call Trace: [] ? llog_osd_declare_write_rec+0x1b0/0x540 [obdclass] [] start_this_handle+0x27a/0x4a0 [jbd2] [] ? autoremove_wake_function+0x0/0x40 [] jbd2_journal_start+0xd0/0x110 [jbd2] [] ? mdt_txn_start_cb+0x13e/0x340 [mdt] [] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs] [] osd_trans_start+0x1df/0x680 [osd_ldiskfs] [] lod_trans_start+0x1b9/0x250 [lod] [] mdd_trans_start+0x17/0x20 [mdd] [] mdd_create+0x91a/0x1790 [mdd] [] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] [] mdt_reint_open+0x13ae/0x21d0 [mdt] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] [] ? lu_ucred+0x20/0x30 [obdclass] [] mdt_reint_rec+0x41/0xe0 [mdt] [] mdt_reint_internal+0x4c3/0x780 [mdt] [] mdt_intent_reint+0x1ed/0x520 [mdt] [] mdt_intent_policy+0x39e/0x720 [mdt] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] [] mdt_enqueue+0x46/0xe0 [mdt] [] mdt_handle_common+0x647/0x16d0 [mdt] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] mds_regular_handle+0x15/0x20 [mdt] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415118930.3712 Lustre: meerkat-MDT0000: Client 06af4715-b52b-ed1a-8266-d152291b88f8 (at 198.202.119.36@tcp) refused reconnection, still busy with 1 active RPCs Lustre: Skipped 3 previous similar messages LNet: Service thread pid 3429 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 3429, comm: mdt03_002 Call Trace: [] start_this_handle+0x27a/0x4a0 [jbd2] [] ? cache_alloc_refill+0x15b/0x240 [] ? autoremove_wake_function+0x0/0x40 [] jbd2_journal_start+0xd0/0x110 [jbd2] [] ? mdt_txn_start_cb+0xf9/0x340 [mdt] [] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs] [] osd_trans_start+0x1df/0x680 [osd_ldiskfs] [] lod_trans_start+0x1b9/0x250 [lod] [] mdd_trans_start+0x17/0x20 [mdd] [] mdd_attr_set+0x4a3/0x1390 [mdd] [] mdt_attr_set+0x268/0x560 [mdt] [] mdt_reint_setattr+0x5bd/0xcf0 [mdt] [] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] [] mdt_reint_rec+0x41/0xe0 [mdt] [] mdt_reint_internal+0x4c3/0x780 [mdt] [] mdt_reint+0x44/0xe0 [mdt] [] mdt_handle_common+0x647/0x16d0 [mdt] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] mds_regular_handle+0x15/0x20 [mdt] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415118937.3429 LNet: Service thread pid 19141 completed after 224.12s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). LNet: Service thread pid 3712 completed after 221.69s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). LNet: Skipped 2 previous similar messages Lustre: meerkat-MDT0000: Client 112aee85-df01-240f-a2da-abf3a40edd58 (at 198.202.118.93@tcp) refused reconnection, still busy with 1 active RPCs Lustre: Skipped 6 previous similar messages LNet: Service thread pid 18060 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 18060, comm: mdt01_027 Call Trace: [] ? mem_cgroup_lru_del_list+0x2b/0xb0 [] ? isolate_lru_pages.clone.0+0xd7/0x170 [] ? shrink_inactive_list+0x443/0x830 [] ? mem_cgroup_lru_del_list+0x2b/0xb0 [] ? mem_cgroup_lru_del+0x39/0x40 [] ? shrink_active_list+0x1bd/0x370 [] ? shrink_mem_cgroup_zone+0x3ae/0x610 [] ? mem_cgroup_iter+0xfd/0x280 [] ? shrink_zone+0x63/0xb0 [] ? zone_reclaim+0x349/0x400 [] ? get_page_from_freelist+0x69c/0x830 [] ? mempool_alloc_slab+0x15/0x20 [] ? zone_statistics+0x70/0xc0 [] ? __alloc_pages_nodemask+0x113/0x8d0 [] ? kmem_getpages+0x62/0x170 [] ? cache_grow+0x2cf/0x320 [] ? cache_alloc_refill+0x202/0x240 [] ? kmem_cache_alloc+0x15f/0x190 [] ? alloc_buffer_head+0x1c/0x60 [] ? alloc_page_buffers+0x3e/0xf0 [] ? __getblk+0x161/0x2a0 [] ? __breadahead+0x12/0x40 [] ? __ldiskfs_get_inode_loc+0x33e/0x3b0 [ldiskfs] [] ? ldiskfs_get_inode_loc+0x1c/0x20 [ldiskfs] [] ? ldiskfs_reserve_inode_write+0x2d/0xa0 [ldiskfs] [] ? ldiskfs_mark_inode_dirty+0x4c/0x1f0 [ldiskfs] [] ? ldiskfs_dirty_inode+0x40/0x60 [ldiskfs] [] ? osd_attr_set+0x181/0x540 [osd_ldiskfs] [] ? lod_attr_set+0x12b/0x450 [lod] [] ? mdd_attr_set_internal+0x151/0x230 [mdd] [] ? mdd_attr_set+0x107a/0x1390 [mdd] [] ? mdt_attr_set+0x268/0x560 [mdt] [] ? mdt_reint_setattr+0x5bd/0xcf0 [mdt] [] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] [] ? mdt_reint_rec+0x41/0xe0 [mdt] [] ? mdt_reint_internal+0x4c3/0x780 [mdt] [] ? mdt_reint+0x44/0xe0 [mdt] [] ? mdt_handle_common+0x647/0x16d0 [mdt] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ? mds_regular_handle+0x15/0x20 [mdt] [] ? ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ? ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415119007.18060 Lustre: meerkat-MDT0000: Client 112aee85-df01-240f-a2da-abf3a40edd58 (at 198.202.118.93@tcp) refused reconnection, still busy with 1 active RPCs Lustre: Skipped 2 previous similar messages LNet: Service thread pid 18060 completed after 284.11s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1415119126/real 0] req@ffff880316ddcc00 x1483597895269676/t0(0) o13->meerkat-OST0014-osc@172.25.32.115@tcp:7/4 lens 224/368 e 0 to 1 dl 1415119135 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3338:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Lustre: meerkat-OST0014-osc: Connection to meerkat-OST0014 (at 172.25.32.115@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 11-0: meerkat-OST0016-osc: Communicating with 172.25.32.243@tcp, operation ost_connect failed with -16. Lustre: meerkat-OST0024-osc: Connection restored to meerkat-OST0024 (at 172.25.32.115@tcp) Lustre: Skipped 15 previous similar messages Lustre: meerkat-OST0006-osc: Connection restored to meerkat-OST0006 (at 172.25.32.243@tcp) Lustre: Skipped 1 previous similar message Lustre: MGS: Client 70d5b2c7-42e0-8d9f-7c70-58b77b94ad16 (at 10.7.103.176@o2ib) reconnecting Lustre: Skipped 42 previous similar messages Lustre: 3329:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415119337/real 1415119337] req@ffff8800a91fc800 x1483597895384788/t0(0) o13->meerkat-OST001d-osc@172.25.32.244@tcp:7/4 lens 224/368 e 0 to 1 dl 1415119344 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3329:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 35 previous similar messages Lustre: meerkat-OST001d-osc: Connection to meerkat-OST001d (at 172.25.32.244@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 9 previous similar messages Lustre: meerkat-OST0038-osc: Connection to meerkat-OST0038 (at 172.25.32.118@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 8 previous similar messages Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415119341/real 1415119341] req@ffff880629fee800 x1483597895385016/t0(0) o13->meerkat-OST0003-osc@172.25.32.116@tcp:7/4 lens 224/368 e 0 to 1 dl 1415119348 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Lustre: 3322:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 25 previous similar messages Lustre: meerkat-OST0016-osc: Connection to meerkat-OST0016 (at 172.25.32.243@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 30 previous similar messages Lustre: 3412:0:(service.c:1889:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 27s req@ffff88035fa3d050 x1474651871047896/t0(0) o400->5d98b90c-829c-2af6-35c3-60ca28492e58@10.7.101.138@o2ib:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415119353/real 1415119353] req@ffff880594e08c00 x1483597895385300/t0(0) o8->meerkat-OST0032-osc@172.25.32.248@tcp:28/4 lens 400/544 e 0 to 1 dl 1415119361 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 77 previous similar messages Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415119260/real 1415119260] req@ffff88055deb3400 x1483597895380868/t0(0) o400->MGC172.25.33.53@tcp@0@lo:26/25 lens 224/224 e 1 to 1 dl 1415119389 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 23 previous similar messages LustreError: 166-1: MGC172.25.33.53@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LNet: No route to 12345-10.7.103.200@o2ib via 172.25.33.53@tcp (all routers down) LNet: No route to 12345-10.7.101.91@o2ib via 172.25.33.53@tcp (all routers down) LNet: No route to 12345-10.7.102.235@o2ib via 172.25.33.53@tcp (all routers down) LNet: Skipped 15 previous similar messages LNetError: 3305:0:(lib-move.c:1532:lnet_parse_get()) 172.25.33.53@tcp: Unable to send REPLY for GET from 12345-10.7.102.124@o2ib: -113 LNet: No route to 12345-10.7.102.144@o2ib via 172.25.33.53@tcp (all routers down) LNet: Skipped 33 previous similar messages LNet: No route to 12345-10.7.102.193@o2ib via 172.25.33.53@tcp (all routers down) LNet: Skipped 41 previous similar messages Lustre: meerkat-OST000d-osc: Connection restored to meerkat-OST000d (at 172.25.32.244@tcp) Lustre: Skipped 7 previous similar messages LNet: No route to 12345-10.7.101.131@o2ib via 172.25.33.53@tcp (all routers down) LNet: Skipped 148 previous similar messages Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1415119414/real 1415119414] req@ffff88057d79c400 x1483597895385860/t0(0) o8->meerkat-OST0038-osc@172.25.32.118@tcp:28/4 lens 400/544 e 0 to 1 dl 1415119430 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3319:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 66 previous similar messages LNet: Service thread pid 13957 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 13957, comm: ll_mgs_0031 Call Trace: [] __wait_on_freeing_inode+0x98/0xc0 [] ? wake_bit_function+0x0/0x50 [] find_inode_fast+0x58/0x80 [] ifind_fast+0x3c/0xb0 [] iget_locked+0x49/0x170 [] ldiskfs_iget+0x37/0x800 [ldiskfs] [] ? ldiskfs_find_entry+0x281/0x4a0 [ldiskfs] [] osd_iget+0x2e/0x2c0 [osd_ldiskfs] [] osd_ea_fid_get+0x176/0x2c0 [osd_ldiskfs] [] osd_index_ea_lookup+0x46c/0x850 [osd_ldiskfs] [] dt_lookup_dir+0x6f/0x130 [obdclass] [] llog_osd_open+0x485/0xc00 [obdclass] [] llog_open+0xba/0x2c0 [obdclass] [] llog_origin_handle_open+0x1f7/0x6f0 [ptlrpc] [] mgs_handle+0x686/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415119440.13957 Pid: 3413, comm: ll_mgs_0002 Call Trace: [] __wait_on_freeing_inode+0x98/0xc0 [] ? wake_bit_function+0x0/0x50 [] find_inode_fast+0x58/0x80 [] ifind_fast+0x3c/0xb0 [] iget_locked+0x49/0x170 [] ldiskfs_iget+0x37/0x800 [ldiskfs] [] ? ldiskfs_find_entry+0x281/0x4a0 [ldiskfs] [] osd_iget+0x2e/0x2c0 [osd_ldiskfs] [] osd_ea_fid_get+0x176/0x2c0 [osd_ldiskfs] [] osd_index_ea_lookup+0x46c/0x850 [osd_ldiskfs] [] dt_lookup_dir+0x6f/0x130 [obdclass] [] llog_osd_open+0x485/0xc00 [obdclass] [] llog_open+0xba/0x2c0 [obdclass] [] llog_origin_handle_open+0x1f7/0x6f0 [ptlrpc] [] mgs_handle+0x686/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415119442.3413 Pid: 13231, comm: ll_mgs_0025 Call Trace: [] __wait_on_freeing_inode+0x98/0xc0 [] ? wake_bit_function+0x0/0x50 [] find_inode_fast+0x58/0x80 [] ifind_fast+0x3c/0xb0 [] iget_locked+0x49/0x170 [] ldiskfs_iget+0x37/0x800 [ldiskfs] [] ? ldiskfs_find_entry+0x281/0x4a0 [ldiskfs] [] osd_iget+0x2e/0x2c0 [osd_ldiskfs] [] osd_ea_fid_get+0x176/0x2c0 [osd_ldiskfs] [] osd_index_ea_lookup+0x46c/0x850 [osd_ldiskfs] [] dt_lookup_dir+0x6f/0x130 [obdclass] [] llog_osd_open+0x485/0xc00 [obdclass] [] llog_open+0xba/0x2c0 [obdclass] [] llog_origin_handle_open+0x1f7/0x6f0 [ptlrpc] [] mgs_handle+0x686/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415119443.13231 Pid: 12425, comm: ll_mgs_0018 Call Trace: [] __wait_on_freeing_inode+0x98/0xc0 [] ? wake_bit_function+0x0/0x50 [] find_inode_fast+0x58/0x80 [] ifind_fast+0x3c/0xb0 [] iget_locked+0x49/0x170 [] ldiskfs_iget+0x37/0x800 [ldiskfs] [] ? ldiskfs_find_entry+0x281/0x4a0 [ldiskfs] [] osd_iget+0x2e/0x2c0 [osd_ldiskfs] [] osd_ea_fid_get+0x176/0x2c0 [osd_ldiskfs] [] osd_index_ea_lookup+0x46c/0x850 [osd_ldiskfs] [] dt_lookup_dir+0x6f/0x130 [obdclass] [] llog_osd_open+0x485/0xc00 [obdclass] [] llog_open+0xba/0x2c0 [obdclass] [] llog_origin_handle_open+0x1f7/0x6f0 [ptlrpc] [] mgs_handle+0x686/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415119447.12425 Pid: 12287, comm: ll_mgs_0015 Call Trace: [] __wait_on_freeing_inode+0x98/0xc0 [] ? wake_bit_function+0x0/0x50 [] find_inode_fast+0x58/0x80 [] ifind_fast+0x3c/0xb0 [] iget_locked+0x49/0x170 [] ldiskfs_iget+0x37/0x800 [ldiskfs] [] ? ldiskfs_find_entry+0x281/0x4a0 [ldiskfs] [] osd_iget+0x2e/0x2c0 [osd_ldiskfs] [] osd_ea_fid_get+0x176/0x2c0 [osd_ldiskfs] [] osd_index_ea_lookup+0x46c/0x850 [osd_ldiskfs] [] dt_lookup_dir+0x6f/0x130 [obdclass] [] llog_osd_open+0x485/0xc00 [obdclass] [] llog_open+0xba/0x2c0 [obdclass] [] llog_origin_handle_open+0x1f7/0x6f0 [ptlrpc] [] mgs_handle+0x686/0x11c0 [mgs] [] ? keys_fill+0x6f/0x190 [obdclass] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [] ? cfs_timer_arm+0xe/0x10 [libcfs] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [] ? __wake_up+0x53/0x70 [] ptlrpc_main+0xace/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] child_rip+0xa/0x20 [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1415119451.12287 LNet: Service thread pid 12424 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. LustreError: dumping log to /tmp/lustre-log.1415119451.12424 LNet: Service thread pid 3621 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. LustreError: dumping log to /tmp/lustre-log.1415119451.3621 LustreError: dumping log to /tmp/lustre-log.1415119451.13955 LustreError: dumping log to /tmp/lustre-log.1415119451.3625 LNet: Service thread pid 13226 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. LNet: Skipped 2 previous similar messages LustreError: dumping log to /tmp/lustre-log.1415119451.13226 LustreError: dumping log to /tmp/lustre-log.1415119452.3411 LustreError: dumping log to /tmp/lustre-log.1415119452.13227 LNet: Service thread pid 13234 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. LNet: Skipped 2 previous similar messages LustreError: dumping log to /tmp/lustre-log.1415119453.13234 LustreError: dumping log to /tmp/lustre-log.1415119453.3624 LustreError: dumping log to /tmp/lustre-log.1415119454.3622 LustreError: dumping log to /tmp/lustre-log.1415119454.3619 LustreError: dumping log to /tmp/lustre-log.1415119455.3627 LustreError: dumping log to /tmp/lustre-log.1415119455.12423 LustreError: dumping log to /tmp/lustre-log.1415119455.13230 LNet: Service thread pid 3628 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. LNet: Skipped 6 previous similar messages LustreError: dumping log to /tmp/lustre-log.1415119455.3628 LustreError: dumping log to /tmp/lustre-log.1415119455.3620 LustreError: dumping log to /tmp/lustre-log.1415119456.13956 LustreError: dumping log to /tmp/lustre-log.1415119456.13233 Lustre: meerkat-OST0037-osc: Connection to meerkat-OST0037 (at 172.25.33.114@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 23 previous similar messages LustreError: dumping log to /tmp/lustre-log.1415119457.3623 LustreError: dumping log to /tmp/lustre-log.1415119457.3618 LustreError: dumping log to /tmp/lustre-log.1415119457.13229 LustreError: dumping log to /tmp/lustre-log.1415119458.3615 LustreError: dumping log to /tmp/lustre-log.1415119458.3626 LustreError: dumping log to /tmp/lustre-log.1415119458.13228 LustreError: dumping log to /tmp/lustre-log.1415119458.13225 LustreError: dumping log to /tmp/lustre-log.1415119458.13232 Lustre: meerkat-OST0001-osc: Connection restored to meerkat-OST0001 (at 172.25.32.119@tcp) Lustre: Skipped 39 previous similar messages Lustre: 3615:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (106:109s); client may timeout. req@ffff880638ea0050 x1474564335389212/t0(0) o501->434d2f28-2924-1af6-7ae9-836565b370a4@10.7.100.55@o2ib:0/0 lens 296/240 e 1 to 0 dl 1415119364 ref 1 fl Complete:/0/0 rc 0/0 Lustre: 3615:0:(service.c:2031:ptlrpc_server_handle_request()) Skipped 1 previous similar message LNetError: 3304:0:(socklnd.c:1659:ksocknal_destroy_conn()) Completing partial receive from 12345-198.202.119.28@tcp[1], ip 198.202.119.28:1023, with error, wanted: 224, left: 224, last alive is 141 secs ago Lustre: 13229:0:(service.c:2031:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (106:110s); client may timeout. req@ffff880639225050 x1474661088425328/t0(0) o501->ed789ab0-673c-e303-d4d5-e232a571922e@10.7.101.78@o2ib:0/0 lens 296/240 e 1 to 0 dl 1415119363 ref 1 fl Complete:/0/0 rc 0/0 Lustre: 13229:0:(service.c:2031:ptlrpc_server_handle_request()) Skipped 3 previous similar messages LNet: Service thread pid 13229 completed after 215.40s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Lustre: 3615:0:(service.c:1889:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 145s req@ffff88041ddc4050 x1480884993007592/t0(0) o400->64eb259f-1e0f-8496-461a-e622379a7e09@198.202.118.231@tcp:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1 Lustre: 3615:0:(service.c:1889:ptlrpc_server_handle_req_in()) Skipped 2 previous similar messages LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3621:0:(service.c:1999:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.7.102.30@o2ib: deadline 106:109s ago req@ffff8803159a0050 x1474554026031512/t0(0) o400->7824d59f-68ce-49e7-dd06-6ccd14cb210c@10.7.102.30@o2ib:0/0 lens 224/0 e 1 to 0 dl 1415119364 ref 1 fl Interpret:/0/ffffffff rc 0/-1 LustreError: 3621:0:(service.c:1999:ptlrpc_server_handle_request()) Skipped 1 previous similar message LNetError: 3304:0:(socklnd.c:1659:ksocknal_destroy_conn()) Completing partial receive from 12345-198.202.119.80@tcp[1], ip 198.202.119.80:1023, with error, wanted: 224, left: 224, last alive is 141 secs ago LustreError: 3411:0:(pack_generic.c:593:__lustre_unpack_msg()) message length 0 too small for magic/version check LustreError: 3411:0:(sec.c:2057:sptlrpc_svc_unwrap_request()) error unpacking request from 12345-198.202.119.28@tcp x1480619549608996 LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs Lustre: mgs: This server is not able to keep up with request traffic (cpu-bound). Lustre: 13225:0:(service.c:1500:ptlrpc_at_check_timed()) earlyQ=3697 reqQ=0 recA=21, svcEst=230, delay=145024(jiff) Lustre: 13229:0:(service.c:1301:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-28s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff880574c04050 x1474601167122132/t0(0) o400->6307864d-92aa-f948-8b8f-7f70fdd26af3@10.7.102.203@o2ib:0/0 lens 224/0 e 0 to 0 dl 1415119445 ref 2 fl New:/0/ffffffff rc 0/-1 Lustre: 13229:0:(service.c:1301:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs Lustre: 13225:0:(service.c:1301:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-56s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8803e7bfb050 x1462852212492304/t0(0) o400->0d948ff1-9b46-a678-6d5e-c546783e5491@10.7.103.254@o2ib:0/0 lens 224/0 e 0 to 0 dl 1415119417 ref 2 fl New:/0/ffffffff rc 0/-1 Lustre: 13225:0:(service.c:1301:ptlrpc_at_send_early_reply()) Skipped 118 previous similar messages LustreError: 3411:0:(pack_generic.c:593:__lustre_unpack_msg()) message length 0 too small for magic/version check LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 13228:0:(service.c:1999:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.7.101.76@o2ib: deadline 100:113s ago req@ffff8803155cf850 x1474527034230980/t0(0) o250->96c32768-3fad-bb4a-798b-c3cd4870d559@10.7.101.76@o2ib:0/0 lens 400/0 e 0 to 0 dl 1415119360 ref 1 fl Interpret:/0/ffffffff rc 0/-1 LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3615:0:(sec.c:2057:sptlrpc_svc_unwrap_request()) error unpacking request from 12345-198.202.118.69@tcp x1479715673257004 LustreError: 13228:0:(service.c:1999:ptlrpc_server_handle_request()) Skipped 82 previous similar messages LustreError: 3615:0:(sec.c:2057:sptlrpc_svc_unwrap_request()) Skipped 1 previous similar message LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3411:0:(pack_generic.c:593:__lustre_unpack_msg()) Skipped 34 previous similar messages Lustre: MGS: Client 8f5c2aff-d3ac-c04a-3c83-b3a9b96f540e (at 0@lo) refused reconnection, still busy with 1 active RPCs Lustre: Skipped 2 previous similar messages LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs LustreError: 3304:0:(events.c:306:request_in_callback()) event type 2, status -5, service mgs Lustre: meerkat-OST0005-osc: Connection restored to meerkat-OST0005 (at 172.25.32.244@tcp) Lustre: Skipped 24 previous similar messages