Lustre: debug daemon will attempt to start writing to /tmp/lustre-log.barry-mds1 (65536kB max)
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330040976/real 1330040976]  req@ffff8103dda97800 x1394648644864884/t0(0) o5->lustre-OST0016-osc-MDT0000@10.37.248.64@o2ib1:7/4 lens 400/400 e 0 to 1 dl 1330040983 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Lustre: lustre-OST0016-osc-MDT0000: Connection to service lustre-OST0016 via nid 10.37.248.64@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: lustre-OST0016-osc-MDT0000: Connection restored to service lustre-OST0016 using nid 10.37.248.64@o2ib1.
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0016_UUID now active, resetting orphans
Lustre: Skipped 3 previous similar messages
LustreError: 5069:0:(osc_create.c:175:osc_interpret_create()) @@@ Unknown rc -107 from async create: failing oscc  req@ffff8103dda97800 x1394648644864884/t0(0) o5->lustre-OST0016-osc-MDT0000@10.37.248.64@o2ib1:7/4 lens 400/400 e 0 to 1 dl 1330040983 ref 1 fl Interpret:RXN/0/ffffffff rc -107/-1
LustreError: 5069:0:(lov_request.c:558:lov_update_create_set()) error creating fid 0x28 sub-object on OST idx 22/28: rc = -107
LustreError: 6022:0:(lov_obd.c:1069:lov_clear_orphans()) error in orphan recovery on OST idx 22/28: rc = -5
LustreError: 6022:0:(mds_lov.c:882:__mds_lov_synchronize()) lustre-OST0016_UUID failed at mds_lov_clear_orphans: -5
LustreError: 6022:0:(mds_lov.c:903:__mds_lov_synchronize()) lustre-OST0016_UUID sync failed -5, deactivating
Lustre: 5858:0:(lov_qos.c:472:qos_shrink_lsm()) using fewer stripes for object 40: old 28 new 27
Lustre: 5851:0:(lov_qos.c:472:qos_shrink_lsm()) using fewer stripes for object 41: old 28 new 27
Lustre: 5851:0:(lov_qos.c:472:qos_shrink_lsm()) Skipped 12 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330041208/real 1330041208]  req@ffff8103e2af1000 x1394648644870446/t0(0) o5->lustre-OST000c-osc-MDT0000@10.37.248.62@o2ib1:7/4 lens 400/400 e 0 to 1 dl 1330041215 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: lustre-OST000c-osc-MDT0000: Connection to service lustre-OST000c via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: lustre-OST000c-osc-MDT0000: Connection restored to service lustre-OST000c using nid 10.37.248.62@o2ib1.
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST000c_UUID now active, resetting orphans
Lustre: shutting down debug daemon thread...
LustreError: 5069:0:(osc_create.c:175:osc_interpret_create()) @@@ Unknown rc -107 from async create: failing oscc  req@ffff8103e2af1000 x1394648644870446/t0(0) o5->lustre-OST000c-osc-MDT0000@10.37.248.62@o2ib1:7/4 lens 400/400 e 0 to 1 dl 1330041215 ref 1 fl Interpret:RXN/0/ffffffff rc -107/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330041290/real 1330041290]  req@ffff8103df4b5800 x1394648644871239/t0(0) o400->lustre-OST0017-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330041297 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: lustre-OST0017-osc-MDT0000: Connection to service lustre-OST0017 via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: lustre-OST0017-osc-MDT0000: Connection restored to service lustre-OST0017 using nid 10.37.248.65@o2ib1.
LustreError: 6150:0:(lov_obd.c:1069:lov_clear_orphans()) error in orphan recovery on OST idx 12/28: rc = -5
LustreError: 6150:0:(mds_lov.c:882:__mds_lov_synchronize()) lustre-OST000c_UUID failed at mds_lov_clear_orphans: -5
LustreError: 6150:0:(mds_lov.c:903:__mds_lov_synchronize()) lustre-OST000c_UUID sync failed -5, deactivating
LustreError: 5069:0:(osc_create.c:175:osc_interpret_create()) @@@ Unknown rc -107 from async create: failing oscc  req@ffff81042a70e400 x1394648644870447/t0(0) o5->lustre-OST0010-osc-MDT0000@10.37.248.62@o2ib1:7/4 lens 400/400 e 0 to 1 dl 1330041297 ref 1 fl Interpret:RXN/0/ffffffff rc -107/-1
LustreError: 6202:0:(lov_obd.c:1069:lov_clear_orphans()) error in orphan recovery on OST idx 16/28: rc = -5
LustreError: 6202:0:(mds_lov.c:882:__mds_lov_synchronize()) lustre-OST0010_UUID failed at mds_lov_clear_orphans: -5
LustreError: 6202:0:(mds_lov.c:903:__mds_lov_synchronize()) lustre-OST0010_UUID sync failed -5, deactivating
Lustre: 6021:0:(lov_qos.c:472:qos_shrink_lsm()) using fewer stripes for object 49: old 26 new 25
Lustre: 6021:0:(lov_qos.c:472:qos_shrink_lsm()) Skipped 4 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330041534/real 1330041534]  req@ffff8103e1115000 x1394648644882272/t0(0) o5->lustre-OST0015-osc-MDT0000@10.37.248.63@o2ib1:7/4 lens 400/400 e 0 to 1 dl 1330041541 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-OST0015-osc-MDT0000: Connection to service lustre-OST0015 via nid 10.37.248.63@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0015-osc-MDT0000: Connection restored to service lustre-OST0015 using nid 10.37.248.63@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0015_UUID now active, resetting orphans
Lustre: Skipped 2 previous similar messages
LustreError: 5069:0:(osc_create.c:175:osc_interpret_create()) @@@ Unknown rc -107 from async create: failing oscc  req@ffff8103e1115000 x1394648644882272/t0(0) o5->lustre-OST0015-osc-MDT0000@10.37.248.63@o2ib1:7/4 lens 400/400 e 0 to 1 dl 1330041541 ref 1 fl Interpret:RXN/0/ffffffff rc -107/-1
LustreError: 6210:0:(lov_obd.c:1069:lov_clear_orphans()) error in orphan recovery on OST idx 21/28: rc = -5
LustreError: 6210:0:(mds_lov.c:882:__mds_lov_synchronize()) lustre-OST0015_UUID failed at mds_lov_clear_orphans: -5
LustreError: 6210:0:(mds_lov.c:903:__mds_lov_synchronize()) lustre-OST0015_UUID sync failed -5, deactivating
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330041991/real 1330041991]  req@ffff8103e2af1400 x1394648644903670/t0(0) o400->lustre-OST0010-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330041998 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-OST0010-osc-MDT0000: Connection to service lustre-OST0010 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0010-osc-MDT0000: Connection restored to service lustre-OST0010 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0010_UUID now active, resetting orphans
Lustre: Skipped 1 previous similar message
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330042073/real 1330042073]  req@ffff8103dc841400 x1394648644903726/t0(0) o400->lustre-OST0016-osc-MDT0000@10.37.248.64@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330042080 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: lustre-OST0016-osc-MDT0000: Connection to service lustre-OST0016 via nid 10.37.248.64@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: lustre-OST0015-osc-MDT0000: Connection restored to service lustre-OST0015 using nid 10.37.248.63@o2ib1.
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330042380/real 1330042380]  req@ffff8103dd4c1400 x1394648644904332/t0(0) o400->lustre-OST0011-osc-MDT0000@10.37.248.63@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330042387 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-OST0011-osc-MDT0000: Connection to service lustre-OST0011 via nid 10.37.248.63@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0011-osc-MDT0000: Connection restored to service lustre-OST0011 using nid 10.37.248.63@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0017-osc-MDT0000: Connection to service lustre-OST0017 via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0017-osc-MDT0000: Connection restored to service lustre-OST0017 using nid 10.37.248.65@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0013-osc-MDT0000: Connection to service lustre-OST0013 via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST000c-osc-MDT0000: Connection restored to service lustre-OST000c using nid 10.37.248.62@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST000c_UUID now active, resetting orphans
Lustre: Skipped 6 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330042701/real 1330042701]  req@ffff810425c4f400 x1394648644904487/t0(0) o400->lustre-OST000f-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330042708 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Lustre: lustre-OST000f-osc-MDT0000: Connection to service lustre-OST000f via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST000f-osc-MDT0000: Connection restored to service lustre-OST000f using nid 10.37.248.65@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0001-osc-MDT0000: Connection to service lustre-OST0001 via nid 10.37.248.63@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0001-osc-MDT0000: Connection restored to service lustre-OST0001 using nid 10.37.248.63@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0001_UUID now active, resetting orphans
Lustre: Skipped 3 previous similar messages
LustreError: 5069:0:(osc_create.c:175:osc_interpret_create()) @@@ Unknown rc -107 from async create: failing oscc  req@ffff81041cd72400 x1394648644904693/t0(0) o5->lustre-OST0001-osc-MDT0000@10.37.248.63@o2ib1:7/4 lens 400/400 e 0 to 1 dl 1330043173 ref 1 fl Interpret:RXN/0/ffffffff rc -107/-1
LustreError: 5069:0:(osc_create.c:175:osc_interpret_create()) Skipped 1 previous similar message
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330043248/real 1330043248]  req@ffff8103dc841000 x1394648644904726/t0(0) o400->lustre-OST0012-osc-MDT0000@10.37.248.64@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330043255 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
LustreError: 6367:0:(lov_obd.c:1069:lov_clear_orphans()) error in orphan recovery on OST idx 1/28: rc = -5
LustreError: 6367:0:(lov_obd.c:1069:lov_clear_orphans()) Skipped 1 previous similar message
LustreError: 6367:0:(mds_lov.c:882:__mds_lov_synchronize()) lustre-OST0001_UUID failed at mds_lov_clear_orphans: -5
LustreError: 6367:0:(mds_lov.c:882:__mds_lov_synchronize()) Skipped 1 previous similar message
LustreError: 6367:0:(mds_lov.c:903:__mds_lov_synchronize()) lustre-OST0001_UUID sync failed -5, deactivating
LustreError: 6367:0:(mds_lov.c:903:__mds_lov_synchronize()) Skipped 1 previous similar message
LustreError: 6368:0:(lov_obd.c:1069:lov_clear_orphans()) error in orphan recovery on OST idx 1/28: rc = -5
LustreError: 6368:0:(mds_lov.c:882:__mds_lov_synchronize()) lustre-OST0001_UUID failed at mds_lov_clear_orphans: -5
LustreError: 6368:0:(mds_lov.c:903:__mds_lov_synchronize()) lustre-OST0001_UUID sync failed -5, deactivating
Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 4 previous similar messages
Lustre: lustre-OST0000-osc-MDT0000: Connection restored to service lustre-OST0000 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 4 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330044019/real 1330044019]  req@ffff8103e2af5c00 x1394648644905039/t0(0) o400->lustre-OST000f-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330044026 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST000f_UUID now active, resetting orphans
Lustre: Skipped 6 previous similar messages
Lustre: lustre-OST0013-osc-MDT0000: Connection to service lustre-OST0013 via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 3 previous similar messages
Lustre: lustre-OST0013-osc-MDT0000: Connection restored to service lustre-OST0013 using nid 10.37.248.65@o2ib1.
Lustre: Skipped 3 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330044715/real 1330044715]  req@ffff8103e29c7c00 x1394648644905347/t0(0) o400->lustre-OST0018-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330044722 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Lustre: lustre-OST0018-osc-MDT0000: Connection to service lustre-OST0018 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 3 previous similar messages
Lustre: lustre-OST0018-osc-MDT0000: Connection restored to service lustre-OST0018 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 3 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0018_UUID now active, resetting orphans
Lustre: Skipped 5 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330045439/real 1330045439]  req@ffff8104247b7000 x1394648644905695/t0(0) o400->lustre-OST000f-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330045446 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
Lustre: lustre-OST000f-osc-MDT0000: Connection to service lustre-OST000f via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 13 previous similar messages
Lustre: lustre-OST000f-osc-MDT0000: Connection restored to service lustre-OST000f using nid 10.37.248.65@o2ib1.
Lustre: Skipped 13 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0008_UUID now active, resetting orphans
Lustre: Skipped 13 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330046299/real 1330046299]  req@ffff810426d88000 x1394648644906090/t0(0) o400->lustre-OST0019-osc-MDT0000@10.37.248.63@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330046306 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
Lustre: lustre-OST0019-osc-MDT0000: Connection to service lustre-OST0019 via nid 10.37.248.63@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 9 previous similar messages
Lustre: lustre-OST0019-osc-MDT0000: Connection restored to service lustre-OST0019 using nid 10.37.248.63@o2ib1.
Lustre: Skipped 9 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0019_UUID now active, resetting orphans
Lustre: Skipped 9 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330047738/real 1330047738]  req@ffff8103e2553000 x1394648644906659/t0(0) o400->lustre-OST0013-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330047745 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Lustre: lustre-OST0013-osc-MDT0000: Connection to service lustre-OST0013 via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 3 previous similar messages
Lustre: lustre-OST0013-osc-MDT0000: Connection restored to service lustre-OST0013 using nid 10.37.248.65@o2ib1.
Lustre: Skipped 3 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0013_UUID now active, resetting orphans
Lustre: Skipped 3 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330047970/real 1330047970]  req@ffff8103e2589c00 x1394648644906764/t0(0) o400->lustre-OST0019-osc-MDT0000@10.37.248.63@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330047977 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-OST0019-osc-MDT0000: Connection to service lustre-OST0019 via nid 10.37.248.63@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0018-osc-MDT0000: Connection restored to service lustre-OST0018 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0018_UUID now active, resetting orphans
Lustre: Skipped 1 previous similar message
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330048134/real 1330048134]  req@ffff8103e2553800 x1394648644906829/t0(0) o400->lustre-OST0004-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330048141 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Lustre: lustre-OST0004-osc-MDT0000: Connection to service lustre-OST0004 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 3 previous similar messages
Lustre: lustre-OST0004-osc-MDT0000: Connection restored to service lustre-OST0004 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 3 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0004_UUID now active, resetting orphans
Lustre: Skipped 3 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330049048/real 1330049048]  req@ffff8103d6fca000 x1394648644907204/t0(0) o400->lustre-OST000d-osc-MDT0000@10.37.248.63@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330049055 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Lustre: lustre-OST000d-osc-MDT0000: Connection to service lustre-OST000d via nid 10.37.248.63@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 2 previous similar messages
Lustre: lustre-OST000d-osc-MDT0000: Connection restored to service lustre-OST000d using nid 10.37.248.63@o2ib1.
Lustre: Skipped 2 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST000d_UUID now active, resetting orphans
Lustre: Skipped 2 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330049430/real 1330049430]  req@ffff81041cd72400 x1394648644907371/t0(0) o400->lustre-OST0017-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330049437 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-OST0017-osc-MDT0000: Connection to service lustre-OST0017 via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0011-osc-MDT0000: Connection restored to service lustre-OST0011 using nid 10.37.248.63@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0011_UUID now active, resetting orphans
Lustre: Skipped 1 previous similar message
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330049737/real 1330049737]  req@ffff8103de7c8000 x1394648644907495/t0(0) o400->lustre-OST0013-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330049744 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-OST0013-osc-MDT0000: Connection to service lustre-OST0013 via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0013-osc-MDT0000: Connection restored to service lustre-OST0013 using nid 10.37.248.65@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0013_UUID now active, resetting orphans
Lustre: Skipped 1 previous similar message
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330049901/real 1330049901]  req@ffff8103e2790400 x1394648644907583/t0(0) o400->lustre-OST0016-osc-MDT0000@10.37.248.64@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330049908 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Lustre: lustre-OST0016-osc-MDT0000: Connection to service lustre-OST0016 via nid 10.37.248.64@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 3 previous similar messages
Lustre: lustre-OST0016-osc-MDT0000: Connection restored to service lustre-OST0016 using nid 10.37.248.64@o2ib1.
Lustre: Skipped 3 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0016_UUID now active, resetting orphans
Lustre: Skipped 3 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330050283/real 1330050283]  req@ffff8103e32fc000 x1394648644907743/t0(0) o400->lustre-OST0018-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330050290 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-OST0018-osc-MDT0000: Connection to service lustre-OST0018 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0018-osc-MDT0000: Connection restored to service lustre-OST0018 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0018_UUID now active, resetting orphans
Lustre: Skipped 1 previous similar message
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330050890/real 1330050890]  req@ffff8103e32fc800 x1394648644907964/t0(0) o400->lustre-OST0001-osc-MDT0000@10.37.248.63@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330050897 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-OST0001-osc-MDT0000: Connection to service lustre-OST0001 via nid 10.37.248.63@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0001-osc-MDT0000: Connection restored to service lustre-OST0001 using nid 10.37.248.63@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0001_UUID now active, resetting orphans
Lustre: Skipped 1 previous similar message
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330051539/real 1330051539]  req@ffff8103e292c800 x1394648644908297/t0(0) o400->lustre-OST0017-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330051546 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
Lustre: lustre-OST0017-osc-MDT0000: Connection to service lustre-OST0017 via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 11 previous similar messages
Lustre: lustre-OST0007-osc-MDT0000: Connection restored to service lustre-OST0007 using nid 10.37.248.65@o2ib1.
Lustre: Skipped 11 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0017_UUID now active, resetting orphans
Lustre: Skipped 11 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330052181/real 1330052181]  req@ffff8103e2af5000 x1394648644908606/t0(0) o400->lustre-OST0011-osc-MDT0000@10.37.248.63@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330052188 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Lustre: lustre-OST0011-osc-MDT0000: Connection to service lustre-OST0011 via nid 10.37.248.63@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 11 previous similar messages
Lustre: lustre-OST0011-osc-MDT0000: Connection restored to service lustre-OST0011 using nid 10.37.248.63@o2ib1.
Lustre: Skipped 11 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0011_UUID now active, resetting orphans
Lustre: Skipped 11 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330052809/real 1330052809]  req@ffff8103d6fca400 x1394648644908873/t0(0) o400->lustre-OST0002-osc-MDT0000@10.37.248.64@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330052816 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Lustre: lustre-OST0002-osc-MDT0000: Connection to service lustre-OST0002 via nid 10.37.248.64@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 6 previous similar messages
Lustre: lustre-OST0002-osc-MDT0000: Connection restored to service lustre-OST0002 using nid 10.37.248.64@o2ib1.
Lustre: Skipped 6 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0002_UUID now active, resetting orphans
Lustre: Skipped 6 previous similar messages
LustreError: 5069:0:(osc_create.c:175:osc_interpret_create()) @@@ Unknown rc -107 from async create: failing oscc  req@ffff8104127ba000 x1394648644908906/t0(0) o5->lustre-OST0002-osc-MDT0000@10.37.248.64@o2ib1:7/4 lens 400/400 e 0 to 1 dl 1330052823 ref 1 fl Interpret:RXN/0/ffffffff rc -107/-1
LustreError: 7000:0:(lov_obd.c:1069:lov_clear_orphans()) error in orphan recovery on OST idx 2/28: rc = -5
LustreError: 7000:0:(mds_lov.c:882:__mds_lov_synchronize()) lustre-OST0002_UUID failed at mds_lov_clear_orphans: -5
LustreError: 7000:0:(mds_lov.c:903:__mds_lov_synchronize()) lustre-OST0002_UUID sync failed -5, deactivating
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330053798/real 1330053798]  req@ffff8103dcf26000 x1394648644909289/t0(0) o400->lustre-OST0015-osc-MDT0000@10.37.248.63@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330053805 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Lustre: lustre-OST0015-osc-MDT0000: Connection to service lustre-OST0015 via nid 10.37.248.63@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 2 previous similar messages
Lustre: lustre-OST0015-osc-MDT0000: Connection restored to service lustre-OST0015 using nid 10.37.248.63@o2ib1.
Lustre: Skipped 2 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0015_UUID now active, resetting orphans
Lustre: Skipped 2 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330054433/real 1330054433]  req@ffff810422d94800 x1394648644909580/t0(0) o400->lustre-OST000f-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330054440 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
Lustre: lustre-OST000f-osc-MDT0000: Connection to service lustre-OST000f via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 9 previous similar messages
Lustre: lustre-OST000f-osc-MDT0000: Connection restored to service lustre-OST000f using nid 10.37.248.65@o2ib1.
Lustre: Skipped 9 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST000f_UUID now active, resetting orphans
Lustre: Skipped 9 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330055047/real 1330055047]  req@ffff8103e2af1400 x1394648644909843/t0(0) o400->lustre-OST0015-osc-MDT0000@10.37.248.63@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330055054 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Lustre: lustre-OST0015-osc-MDT0000: Connection to service lustre-OST0015 via nid 10.37.248.63@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 3 previous similar messages
Lustre: lustre-OST0015-osc-MDT0000: Connection restored to service lustre-OST0015 using nid 10.37.248.63@o2ib1.
Lustre: Skipped 3 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0015_UUID now active, resetting orphans
Lustre: Skipped 3 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330055750/real 1330055750]  req@ffff8103dfe7d400 x1394648644910150/t0(0) o400->lustre-OST0017-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330055757 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Lustre: lustre-OST0017-osc-MDT0000: Connection to service lustre-OST0017 via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 6 previous similar messages
Lustre: lustre-OST0017-osc-MDT0000: Connection restored to service lustre-OST0017 using nid 10.37.248.65@o2ib1.
Lustre: Skipped 6 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0017_UUID now active, resetting orphans
Lustre: Skipped 6 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330056385/real 1330056385]  req@ffff8103e2519c00 x1394648644910443/t0(0) o400->lustre-OST0013-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330056392 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
Lustre: lustre-OST0013-osc-MDT0000: Connection to service lustre-OST0013 via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 9 previous similar messages
Lustre: lustre-OST0013-osc-MDT0000: Connection restored to service lustre-OST0013 using nid 10.37.248.65@o2ib1.
Lustre: Skipped 9 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0013_UUID now active, resetting orphans
Lustre: Skipped 9 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330057006/real 1330057006]  req@ffff810422d94800 x1394648644910715/t0(0) o400->lustre-OST0014-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330057013 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Lustre: lustre-OST0014-osc-MDT0000: Connection to service lustre-OST0014 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 5 previous similar messages
Lustre: lustre-OST0014-osc-MDT0000: Connection restored to service lustre-OST0014 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 5 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0014_UUID now active, resetting orphans
Lustre: Skipped 5 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330057648/real 1330057648]  req@ffff8103e2af5400 x1394648644911011/t0(0) o400->lustre-OST0004-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330057655 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Lustre: lustre-OST0004-osc-MDT0000: Connection to service lustre-OST0004 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 11 previous similar messages
Lustre: lustre-OST0004-osc-MDT0000: Connection restored to service lustre-OST0004 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 11 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0004_UUID now active, resetting orphans
Lustre: Skipped 11 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330058351/real 1330058351]  req@ffff8103ddf07800 x1394648644911340/t0(0) o400->lustre-OST001a-osc-MDT0000@10.37.248.64@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330058358 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Lustre: lustre-OST001a-osc-MDT0000: Connection to service lustre-OST001a via nid 10.37.248.64@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 6 previous similar messages
Lustre: lustre-OST0019-osc-MDT0000: Connection restored to service lustre-OST0019 using nid 10.37.248.63@o2ib1.
Lustre: Skipped 6 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0019_UUID now active, resetting orphans
Lustre: Skipped 6 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330059122/real 1330059122]  req@ffff8103e3aa3c00 x1394648644911655/t0(0) o400->lustre-OST000f-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330059129 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Lustre: lustre-OST000f-osc-MDT0000: Connection to service lustre-OST000f via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 5 previous similar messages
Lustre: lustre-OST000f-osc-MDT0000: Connection restored to service lustre-OST000f using nid 10.37.248.65@o2ib1.
Lustre: Skipped 5 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST000f_UUID now active, resetting orphans
Lustre: Skipped 5 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330059750/real 1330059750]  req@ffff8103e2af1800 x1394648644911942/t0(0) o400->lustre-OST0013-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330059757 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Lustre: lustre-OST0013-osc-MDT0000: Connection to service lustre-OST0013 via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 7 previous similar messages
Lustre: lustre-OST000c-osc-MDT0000: Connection restored to service lustre-OST000c using nid 10.37.248.62@o2ib1.
Lustre: Skipped 7 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0013_UUID now active, resetting orphans
Lustre: Skipped 7 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330060446/real 1330060446]  req@ffff810426d88000 x1394648644912245/t0(0) o400->lustre-OST0018-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330060453 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Lustre: lustre-OST0018-osc-MDT0000: Connection to service lustre-OST0018 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 5 previous similar messages
Lustre: lustre-OST0017-osc-MDT0000: Connection restored to service lustre-OST0017 using nid 10.37.248.65@o2ib1.
Lustre: Skipped 5 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0017_UUID now active, resetting orphans
Lustre: Skipped 5 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330061224/real 1330061224]  req@ffff8103e2790400 x1394648644912580/t0(0) o400->lustre-OST0013-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330061231 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Lustre: lustre-OST0013-osc-MDT0000: Connection to service lustre-OST0013 via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 7 previous similar messages
Lustre: lustre-OST0013-osc-MDT0000: Connection restored to service lustre-OST0013 using nid 10.37.248.65@o2ib1.
Lustre: Skipped 7 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0013_UUID now active, resetting orphans
Lustre: Skipped 7 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330061831/real 1330061831]  req@ffff8103e2863000 x1394648644912807/t0(0) o400->lustre-OST0002-osc-MDT0000@10.37.248.64@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330061838 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-OST0002-osc-MDT0000: Connection to service lustre-OST0002 via nid 10.37.248.64@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0002-osc-MDT0000: Connection restored to service lustre-OST0002 using nid 10.37.248.64@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0002_UUID now active, resetting orphans
Lustre: Skipped 1 previous similar message
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330062752/real 1330062752]  req@ffff8103e2af7800 x1394648644913202/t0(0) o400->lustre-OST000f-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330062759 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
Lustre: lustre-OST000f-osc-MDT0000: Connection to service lustre-OST000f via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 4 previous similar messages
Lustre: lustre-OST000f-osc-MDT0000: Connection restored to service lustre-OST000f using nid 10.37.248.65@o2ib1.
Lustre: Skipped 4 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST000f_UUID now active, resetting orphans
Lustre: Skipped 4 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330063483/real 1330063483]  req@ffff8103de85d800 x1394648644913560/t0(0) o400->lustre-OST0010-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330063490 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
Lustre: lustre-OST0010-osc-MDT0000: Connection to service lustre-OST0010 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 13 previous similar messages
Lustre: lustre-OST0010-osc-MDT0000: Connection restored to service lustre-OST0010 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 13 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0010_UUID now active, resetting orphans
Lustre: Skipped 13 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330065325/real 1330065325]  req@ffff8103e2af5000 x1394648644914322/t0(0) o400->lustre-OST0014-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330065332 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
Lustre: lustre-OST0014-osc-MDT0000: Connection to service lustre-OST0014 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 8 previous similar messages
Lustre: lustre-OST0014-osc-MDT0000: Connection restored to service lustre-OST0014 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 8 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0014_UUID now active, resetting orphans
Lustre: Skipped 8 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330065707/real 1330065707]  req@ffff8104127ba000 x1394648644914476/t0(0) o400->lustre-OST0011-osc-MDT0000@10.37.248.63@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330065714 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-OST0011-osc-MDT0000: Connection to service lustre-OST0011 via nid 10.37.248.63@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0011-osc-MDT0000: Connection restored to service lustre-OST0011 using nid 10.37.248.63@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0011_UUID now active, resetting orphans
Lustre: Skipped 1 previous similar message
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330065864/real 1330065864]  req@ffff8103e24b3800 x1394648644914550/t0(0) o400->lustre-OST0015-osc-MDT0000@10.37.248.63@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330065871 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-OST0015-osc-MDT0000: Connection to service lustre-OST0015 via nid 10.37.248.63@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0015-osc-MDT0000: Connection restored to service lustre-OST0015 using nid 10.37.248.63@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0015_UUID now active, resetting orphans
Lustre: Skipped 1 previous similar message
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330066185/real 1330066185]  req@ffff8103e0d41400 x1394648644914707/t0(0) o400->lustre-OST0017-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330066192 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Lustre: lustre-OST0017-osc-MDT0000: Connection to service lustre-OST0017 via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 5 previous similar messages
Lustre: lustre-OST0016-osc-MDT0000: Connection restored to service lustre-OST0016 using nid 10.37.248.64@o2ib1.
Lustre: Skipped 5 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0016_UUID now active, resetting orphans
Lustre: Skipped 5 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330067106/real 1330067106]  req@ffff8103e2aa5c00 x1394648644915085/t0(0) o400->lustre-OST000e-osc-MDT0000@10.37.248.64@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330067113 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Lustre: lustre-OST000e-osc-MDT0000: Connection to service lustre-OST000e via nid 10.37.248.64@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 5 previous similar messages
Lustre: lustre-OST000e-osc-MDT0000: Connection restored to service lustre-OST000e using nid 10.37.248.64@o2ib1.
Lustre: Skipped 5 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST000e_UUID now active, resetting orphans
Lustre: Skipped 5 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330067816/real 1330067816]  req@ffff8103e2af1800 x1394648644915413/t0(0) o400->lustre-OST0011-osc-MDT0000@10.37.248.63@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330067823 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
Lustre: lustre-OST0011-osc-MDT0000: Connection to service lustre-OST0011 via nid 10.37.248.63@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 9 previous similar messages
Lustre: lustre-OST0011-osc-MDT0000: Connection restored to service lustre-OST0011 using nid 10.37.248.63@o2ib1.
Lustre: Skipped 9 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0011_UUID now active, resetting orphans
Lustre: Skipped 9 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330068540/real 1330068540]  req@ffff8103e2e94000 x1394648644915779/t0(0) o400->lustre-OST001a-osc-MDT0000@10.37.248.64@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330068547 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
Lustre: lustre-OST001a-osc-MDT0000: Connection to service lustre-OST001a via nid 10.37.248.64@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 13 previous similar messages
Lustre: lustre-OST0019-osc-MDT0000: Connection restored to service lustre-OST0019 using nid 10.37.248.63@o2ib1.
Lustre: Skipped 13 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0019_UUID now active, resetting orphans
Lustre: Skipped 13 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330069168/real 1330069168]  req@ffff81042952f800 x1394648644916056/t0(0) o400->lustre-OST0014-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330069175 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Lustre: lustre-OST0014-osc-MDT0000: Connection to service lustre-OST0014 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 7 previous similar messages
Lustre: lustre-OST000c-osc-MDT0000: Connection restored to service lustre-OST000c using nid 10.37.248.62@o2ib1.
Lustre: Skipped 7 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST000c_UUID now active, resetting orphans
Lustre: Skipped 7 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330069878/real 1330069878]  req@ffff81041cd72400 x1394648644916389/t0(0) o400->lustre-OST0018-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330069885 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
Lustre: lustre-OST0018-osc-MDT0000: Connection to service lustre-OST0018 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 9 previous similar messages
Lustre: lustre-OST0017-osc-MDT0000: Connection restored to service lustre-OST0017 using nid 10.37.248.65@o2ib1.
Lustre: Skipped 9 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0017_UUID now active, resetting orphans
Lustre: Skipped 9 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330070595/real 1330070595]  req@ffff8103e16ccc00 x1394648644916718/t0(0) o400->lustre-OST000e-osc-MDT0000@10.37.248.64@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330070602 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Lustre: lustre-OST000e-osc-MDT0000: Connection to service lustre-OST000e via nid 10.37.248.64@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 11 previous similar messages
Lustre: lustre-OST000e-osc-MDT0000: Connection restored to service lustre-OST000e using nid 10.37.248.64@o2ib1.
Lustre: Skipped 11 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST000e_UUID now active, resetting orphans
Lustre: Skipped 11 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330071298/real 1330071298]  req@ffff8103e2493c00 x1394648644917033/t0(0) o400->lustre-OST0016-osc-MDT0000@10.37.248.64@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330071305 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Lustre: lustre-OST0016-osc-MDT0000: Connection to service lustre-OST0016 via nid 10.37.248.64@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 6 previous similar messages
Lustre: lustre-OST0016-osc-MDT0000: Connection restored to service lustre-OST0016 using nid 10.37.248.64@o2ib1.
Lustre: Skipped 6 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0016_UUID now active, resetting orphans
Lustre: Skipped 6 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330072308/real 1330072308]  req@ffff8103e292c800 x1394648644917476/t0(0) o400->lustre-OST0018-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330072315 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
Lustre: lustre-OST0018-osc-MDT0000: Connection to service lustre-OST0018 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 9 previous similar messages
Lustre: lustre-OST0018-osc-MDT0000: Connection restored to service lustre-OST0018 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 9 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0018_UUID now active, resetting orphans
Lustre: Skipped 9 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330073072/real 1330073072]  req@ffff8103e3583400 x1394648644917787/t0(0) o400->lustre-OST0015-osc-MDT0000@10.37.248.63@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330073079 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Lustre: lustre-OST0015-osc-MDT0000: Connection to service lustre-OST0015 via nid 10.37.248.63@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 3 previous similar messages
Lustre: lustre-OST0015-osc-MDT0000: Connection restored to service lustre-OST0015 using nid 10.37.248.63@o2ib1.
Lustre: Skipped 3 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0015_UUID now active, resetting orphans
Lustre: Skipped 3 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330073714/real 1330073714]  req@ffff8103e2553000 x1394648644918091/t0(0) o400->lustre-OST000e-osc-MDT0000@10.37.248.64@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330073721 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Lustre: lustre-OST000e-osc-MDT0000: Connection to service lustre-OST000e via nid 10.37.248.64@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 11 previous similar messages
Lustre: lustre-OST000e-osc-MDT0000: Connection restored to service lustre-OST000e using nid 10.37.248.64@o2ib1.
Lustre: Skipped 11 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST000e_UUID now active, resetting orphans
Lustre: Skipped 11 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330074471/real 1330074471]  req@ffff8103e2af1800 x1394648644918397/t0(0) o400->lustre-OST0010-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330074478 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-OST0010-osc-MDT0000: Connection to service lustre-OST0010 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0010-osc-MDT0000: Connection restored to service lustre-OST0010 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0010_UUID now active, resetting orphans
Lustre: Skipped 1 previous similar message
LustreError: 5069:0:(osc_create.c:175:osc_interpret_create()) @@@ Unknown rc -107 from async create: failing oscc  req@ffff8103e292c000 x1394648644918605/t0(0) o5->lustre-OST0016-osc-MDT0000@10.37.248.64@o2ib1:7/4 lens 400/400 e 0 to 1 dl 1330074881 ref 1 fl Interpret:RXN/0/ffffffff rc -107/-1
LustreError: 9384:0:(lov_obd.c:1069:lov_clear_orphans()) error in orphan recovery on OST idx 22/28: rc = -5
LustreError: 9384:0:(mds_lov.c:882:__mds_lov_synchronize()) lustre-OST0016_UUID failed at mds_lov_clear_orphans: -5
LustreError: 9384:0:(mds_lov.c:903:__mds_lov_synchronize()) lustre-OST0016_UUID sync failed -5, deactivating
LustreError: 9385:0:(lov_obd.c:1069:lov_clear_orphans()) error in orphan recovery on OST idx 22/28: rc = -5
LustreError: 9385:0:(mds_lov.c:882:__mds_lov_synchronize()) lustre-OST0016_UUID failed at mds_lov_clear_orphans: -5
LustreError: 9385:0:(mds_lov.c:903:__mds_lov_synchronize()) lustre-OST0016_UUID sync failed -5, deactivating
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330075106/real 1330075106]  req@ffff8103e3aa3800 x1394648644918696/t0(0) o400->lustre-OST0019-osc-MDT0000@10.37.248.63@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330075113 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Lustre: lustre-OST0019-osc-MDT0000: Connection to service lustre-OST0019 via nid 10.37.248.63@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 8 previous similar messages
Lustre: lustre-OST0019-osc-MDT0000: Connection restored to service lustre-OST0019 using nid 10.37.248.63@o2ib1.
Lustre: Skipped 8 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0019_UUID now active, resetting orphans
Lustre: Skipped 8 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330075898/real 1330075898]  req@ffff8103e3583400 x1394648644919069/t0(0) o400->lustre-OST0018-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330075905 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Lustre: lustre-OST0018-osc-MDT0000: Connection to service lustre-OST0018 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 11 previous similar messages
Lustre: lustre-OST0018-osc-MDT0000: Connection restored to service lustre-OST0018 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 11 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0018_UUID now active, resetting orphans
Lustre: Skipped 11 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330076969/real 1330076969]  req@ffff8103e2af5000 x1394648644919491/t0(0) o400->lustre-OST0010-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330076976 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Lustre: lustre-OST0010-osc-MDT0000: Connection to service lustre-OST0010 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 3 previous similar messages
Lustre: lustre-OST0010-osc-MDT0000: Connection restored to service lustre-OST0010 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 3 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0010_UUID now active, resetting orphans
Lustre: Skipped 3 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330077597/real 1330077597]  req@ffff8103e292c000 x1394648644919771/t0(0) o400->lustre-OST0013-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330077604 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Lustre: lustre-OST0013-osc-MDT0000: Connection to service lustre-OST0013 via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 6 previous similar messages
Lustre: lustre-OST0013-osc-MDT0000: Connection restored to service lustre-OST0013 using nid 10.37.248.65@o2ib1.
Lustre: Skipped 6 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0013_UUID now active, resetting orphans
Lustre: Skipped 6 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330078839/real 1330078839]  req@ffff8103e2af5000 x1394648644920306/t0(0) o400->lustre-OST000e-osc-MDT0000@10.37.248.64@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330078846 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Lustre: lustre-OST000e-osc-MDT0000: Connection to service lustre-OST000e via nid 10.37.248.64@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 11 previous similar messages
Lustre: lustre-OST000e-osc-MDT0000: Connection restored to service lustre-OST000e using nid 10.37.248.64@o2ib1.
Lustre: Skipped 11 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST000e_UUID now active, resetting orphans
Lustre: Skipped 11 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330079971/real 1330079971]  req@ffff8103e2589c00 x1394648644920752/t0(0) o400->lustre-OST000f-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330079978 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-OST000f-osc-MDT0000: Connection to service lustre-OST000f via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST000f-osc-MDT0000: Connection restored to service lustre-OST000f using nid 10.37.248.65@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST000f_UUID now active, resetting orphans
Lustre: Skipped 1 previous similar message
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330080053/real 1330080053]  req@ffff8103e32fc000 x1394648644920798/t0(0) o400->lustre-OST0014-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330080060 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-OST0014-osc-MDT0000: Connection to service lustre-OST0014 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST000c-osc-MDT0000: Connection restored to service lustre-OST000c using nid 10.37.248.62@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST000c_UUID now active, resetting orphans
Lustre: Skipped 1 previous similar message
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330080210/real 1330080210]  req@ffff810414e74800 x1394648644920874/t0(0) o400->lustre-OST0018-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330080217 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-OST0018-osc-MDT0000: Connection to service lustre-OST0018 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0018-osc-MDT0000: Connection restored to service lustre-OST0018 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0018_UUID now active, resetting orphans
Lustre: Skipped 1 previous similar message
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330080292/real 1330080292]  req@ffff8103e29c7c00 x1394648644920912/t0(0) o400->lustre-OST0014-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330080299 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Lustre: lustre-OST0014-osc-MDT0000: Connection to service lustre-OST0014 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST000d-osc-MDT0000: Connection restored to service lustre-OST000d using nid 10.37.248.63@o2ib1.
Lustre: Skipped 1 previous similar message
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST000d_UUID now active, resetting orphans
Lustre: Skipped 1 previous similar message
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330080456/real 1330080456]  req@ffff8103e2553800 x1394648644920994/t0(0) o400->lustre-OST0017-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330080463 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Lustre: lustre-OST0017-osc-MDT0000: Connection to service lustre-OST0017 via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 2 previous similar messages
Lustre: lustre-OST0017-osc-MDT0000: Connection restored to service lustre-OST0017 using nid 10.37.248.65@o2ib1.
Lustre: Skipped 2 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0017_UUID now active, resetting orphans
Lustre: Skipped 2 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330080620/real 1330080620]  req@ffff8103e2519c00 x1394648644921079/t0(0) o400->lustre-OST0018-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330080627 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Lustre: lustre-OST0018-osc-MDT0000: Connection to service lustre-OST0018 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 3 previous similar messages
Lustre: lustre-OST0018-osc-MDT0000: Connection restored to service lustre-OST0018 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 3 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0018_UUID now active, resetting orphans
Lustre: Skipped 3 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330080934/real 1330080934]  req@ffff81040a3ff400 x1394648644921215/t0(0) o400->lustre-OST0017-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330080941 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Lustre: lustre-OST0017-osc-MDT0000: Connection to service lustre-OST0017 via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 2 previous similar messages
Lustre: lustre-OST000a-osc-MDT0000: Connection restored to service lustre-OST000a using nid 10.37.248.64@o2ib1.
Lustre: Skipped 2 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST000a_UUID now active, resetting orphans
Lustre: Skipped 2 previous similar messages
LustreError: 166-1: MGC10.37.248.56@o2ib1: Connection to service MGS via nid 10.37.248.56@o2ib1 was lost; in progress operations using this service will fail.
Lustre: MGC10.37.248.56@o2ib1: Reactivating import
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330081869/real 1330081869]  req@ffff810414e74000 x1394648644921623/t0(0) o400->lustre-OST001b-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330081876 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Lustre: lustre-OST001b-osc-MDT0000: Connection to service lustre-OST001b via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 6 previous similar messages
Lustre: lustre-OST0014-osc-MDT0000: Connection restored to service lustre-OST0014 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 7 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0014_UUID now active, resetting orphans
Lustre: Skipped 6 previous similar messages
LustreError: 166-1: MGC10.37.248.56@o2ib1: Connection to service MGS via nid 10.37.248.56@o2ib1 was lost; in progress operations using this service will fail.
Lustre: MGC10.37.248.56@o2ib1: Reactivating import
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330082525/real 1330082525]  req@ffff8103e2e94800 x1394648644921921/t0(0) o400->lustre-OST0000-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330082532 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
Lustre: lustre-OST0000-osc-MDT0000: Connection to service lustre-OST0000 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 11 previous similar messages
Lustre: lustre-OST0000-osc-MDT0000: Connection restored to service lustre-OST0000 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 12 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0000_UUID now active, resetting orphans
Lustre: Skipped 11 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330083167/real 1330083167]  req@ffff8103e2aa5400 x1394648644922235/t0(0) o400->lustre-OST001b-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330083174 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Lustre: lustre-OST001b-osc-MDT0000: Connection to service lustre-OST001b via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 7 previous similar messages
Lustre: lustre-OST0014-osc-MDT0000: Connection restored to service lustre-OST0014 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 7 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0014_UUID now active, resetting orphans
Lustre: Skipped 7 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330083856/real 1330083856]  req@ffff8103defd0000 x1394648644922520/t0(0) o400->lustre-OST001b-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330083863 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Lustre: lustre-OST001b-osc-MDT0000: Connection to service lustre-OST001b via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 3 previous similar messages
Lustre: lustre-OST001b-osc-MDT0000: Connection restored to service lustre-OST001b using nid 10.37.248.65@o2ib1.
Lustre: Skipped 3 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST001b_UUID now active, resetting orphans
Lustre: Skipped 3 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330084484/real 1330084484]  req@ffff8103e2589800 x1394648644922801/t0(0) o400->lustre-OST0018-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330084491 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Lustre: lustre-OST0018-osc-MDT0000: Connection to service lustre-OST0018 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 7 previous similar messages
Lustre: lustre-OST0018-osc-MDT0000: Connection restored to service lustre-OST0018 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 7 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0018_UUID now active, resetting orphans
Lustre: Skipped 7 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330085119/real 1330085119]  req@ffff8103defd0400 x1394648644923098/t0(0) o400->lustre-OST0017-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330085126 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
Lustre: lustre-OST0017-osc-MDT0000: Connection to service lustre-OST0017 via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 9 previous similar messages
Lustre: lustre-OST0017-osc-MDT0000: Connection restored to service lustre-OST0017 using nid 10.37.248.65@o2ib1.
Lustre: Skipped 9 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0017_UUID now active, resetting orphans
Lustre: Skipped 9 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330085815/real 1330085815]  req@ffff8104127ba000 x1394648644923395/t0(0) o400->lustre-OST0018-osc-MDT0000@10.37.248.62@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330085822 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
Lustre: lustre-OST0018-osc-MDT0000: Connection to service lustre-OST0018 via nid 10.37.248.62@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 4 previous similar messages
Lustre: lustre-OST0018-osc-MDT0000: Connection restored to service lustre-OST0018 using nid 10.37.248.62@o2ib1.
Lustre: Skipped 4 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0018_UUID now active, resetting orphans
Lustre: Skipped 4 previous similar messages
LustreError: 166-1: MGC10.37.248.56@o2ib1: Connection to service MGS via nid 10.37.248.56@o2ib1 was lost; in progress operations using this service will fail.
Lustre: MGC10.37.248.56@o2ib1: Reactivating import
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330086675/real 1330086675]  req@ffff810423459c00 x1394648644923775/t0(0) o400->lustre-OST001b-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330086682 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Lustre: lustre-OST001b-osc-MDT0000: Connection to service lustre-OST001b via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 6 previous similar messages
Lustre: lustre-OST001b-osc-MDT0000: Connection restored to service lustre-OST001b using nid 10.37.248.65@o2ib1.
Lustre: Skipped 7 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST001b_UUID now active, resetting orphans
Lustre: Skipped 6 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330087378/real 1330087378]  req@ffff8103e0d41800 x1394648644924077/t0(0) o400->lustre-OST001b-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330087385 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Lustre: lustre-OST001b-osc-MDT0000: Connection to service lustre-OST001b via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 5 previous similar messages
Lustre: lustre-OST001b-osc-MDT0000: Connection restored to service lustre-OST001b using nid 10.37.248.65@o2ib1.
Lustre: Skipped 5 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST001b_UUID now active, resetting orphans
Lustre: Skipped 5 previous similar messages
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1330087992/real 1330087992]  req@ffff8103e2553000 x1394648644924320/t0(0) o400->lustre-OST0013-osc-MDT0000@10.37.248.65@o2ib1:28/4 lens 192/192 e 0 to 1 dl 1330087999 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
Lustre: 5069:0:(client.c:1789:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Lustre: lustre-OST0013-osc-MDT0000: Connection to service lustre-OST0013 via nid 10.37.248.65@o2ib1 was lost; in progress operations using this service will wait for recovery to complete.
Lustre: Skipped 2 previous similar messages
Lustre: lustre-OST0013-osc-MDT0000: Connection restored to service lustre-OST0013 using nid 10.37.248.65@o2ib1.
Lustre: Skipped 2 previous similar messages
Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0013_UUID now active, resetting orphans
Lustre: Skipped 2 previous similar messages