Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: oak-OST0051: Recovery already passed deadline 1:52. If you do not want to wait more, please abort the recovery by force.
Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: oak-OST0035: Recovery already passed deadline 1:52. If you do not want to wait more, please abort the recovery by force.
Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: oak-OST0043: Recovery already passed deadline 1:54. If you do not want to wait more, please abort the recovery by force.
Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: oak-OST003b: Recovery already passed deadline 1:56. If you do not want to wait more, please abort the recovery by force.
Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: oak-OST0035: Connection restored to 566c789c-f3a6-ac0c-3466-186fc4c4470b (at 10.12.4.20@o2ib)
Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: oak-OST003b: Connection restored to 566c789c-f3a6-ac0c-3466-186fc4c4470b (at 10.12.4.20@o2ib)
Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: Skipped 24558 previous similar messages
Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: Skipped 24558 previous similar messages
Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: oak-OST003b: Recovery already passed deadline 1:55. If you do not want to wait more, please abort the recovery by force.
Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: oak-OST004b: Recovery already passed deadline 1:57. If you do not want to wait more, please abort the recovery by force.
Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: oak-OST0051: Recovery already passed deadline 1:51. If you do not want to wait more, please abort the recovery by force.
Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: oak-OST0047: Recovery already passed deadline 1:55. If you do not want to wait more, please abort the recovery by force.
Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: Skipped 25 previous similar messages
Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: Skipped 25 previous similar messages
Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: Skipped 25 previous similar messages
Feb 21 11:26:47 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 21 11:26:49 oak-io2-s1 kernel: Lustre: oak-OST004b: Recovery already passed deadline 1:56. If you do not want to wait more, please abort the recovery by force.
Feb 21 11:26:49 oak-io2-s1 kernel: Lustre: Skipped 75 previous similar messages
Feb 21 11:26:51 oak-io2-s1 kernel: Lustre: oak-OST0053: Recovery already passed deadline 1:56. If you do not want to wait more, please abort the recovery by force.
Feb 21 11:26:51 oak-io2-s1 kernel: Lustre: oak-OST0033: Recovery already passed deadline 1:54. If you do not want to wait more, please abort the recovery by force.
Feb 21 11:26:51 oak-io2-s1 kernel: Lustre: oak-OST004b: Recovery already passed deadline 1:54. If you do not want to wait more, please abort the recovery by force.
Feb 21 11:26:51 oak-io2-s1 kernel: Lustre: oak-OST0045: Recovery already passed deadline 1:56. If you do not want to wait more, please abort the recovery by force.
Feb 21 11:26:51 oak-io2-s1 kernel: Lustre: oak-OST004d: Recovery already passed deadline 1:54. If you do not want to wait more, please abort the recovery by force.
Feb 21 11:26:51 oak-io2-s1 kernel: Lustre: oak-OST0037: Recovery already passed deadline 1:54. If you do not want to wait more, please abort the recovery by force.
Feb 21 11:26:51 oak-io2-s1 kernel: Lustre: Skipped 295 previous similar messages
Feb 21 11:26:51 oak-io2-s1 kernel: Lustre: Skipped 295 previous similar messages
Feb 21 11:26:51 oak-io2-s1 kernel: Lustre: Skipped 295 previous similar messages
Feb 21 11:26:51 oak-io2-s1 kernel: Lustre: Skipped 295 previous similar messages
Feb 21 11:26:51 oak-io2-s1 kernel: Lustre: Skipped 295 previous similar messages
Feb 21 11:26:51 oak-io2-s1 kernel: Lustre: Skipped 21 previous similar messages
Feb 21 11:26:59 oak-io2-s1 kernel: Lustre: oak-OST0039: Recovery already passed deadline 1:50. If you do not want to wait more, please abort the recovery by force.
Feb 21 11:26:59 oak-io2-s1 kernel: Lustre: oak-OST004f: Recovery already passed deadline 1:50. If you do not want to wait more, please abort the recovery by force.
Feb 21 11:26:59 oak-io2-s1 kernel: Lustre: oak-OST003d: Recovery already passed deadline 1:50. If you do not want to wait more, please abort the recovery by force.
Feb 21 11:26:59 oak-io2-s1 kernel: Lustre: Skipped 546 previous similar messages
Feb 21 11:26:59 oak-io2-s1 kernel: Lustre: Skipped 546 previous similar messages
Feb 21 11:28:32 oak-io2-s1 kernel: Lustre: oak-OST0031: Denying connection for new client 47585c16-2c65-1a36-2519-62a0145c0305(at 10.8.9.8@o2ib6), waiting for 1238 known clients (1220 recovered, 4 in progress, and 0 evicted) to recover in 0:09
Feb 21 11:28:32 oak-io2-s1 kernel: Lustre: Skipped 11759 previous similar messages
Feb 21 11:28:39 oak-io2-s1 kernel: Lustre: oak-OST0035: recovery is timed out, evict stale exports
Feb 21 11:28:39 oak-io2-s1 kernel: Lustre: oak-OST0051: disconnecting 14 stale clients
Feb 21 11:28:39 oak-io2-s1 kernel: Lustre: 210671:0:(ldlm_lib.c:1773:extend_recovery_timer()) oak-OST0051: extended recovery timer reaching hard limit: 900, extend: 1
Feb 21 11:28:39 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
Feb 21 11:28:39 oak-io2-s1 kernel: Lustre: oak-OST0051: deleting orphan objects from 0x0:156881 to 0x0:156897
Feb 21 11:28:39 oak-io2-s1 kernel: Lustre: oak-OST0051: Recovery over after 14:37, of 1238 clients 1224 recovered and 14 were evicted.
Feb 21 11:28:39 oak-io2-s1 kernel: LustreError: 184234:0:(ofd_io.c:616:ofd_preprw_write()) oak-OST0035: BRW to missing obj 0x0:4346882
Feb 21 11:28:39 oak-io2-s1 kernel: LustreError: 173730:0:(lustre_dlm.h:1372:ldlm_res_lvbo_update()) delayed lvb init failed (rc -2)
Feb 21 11:28:41 oak-io2-s1 kernel: Lustre: oak-OST0043: recovery is timed out, evict stale exports
Feb 21 11:28:41 oak-io2-s1 kernel: Lustre: oak-OST003f: disconnecting 12 stale clients
Feb 21 11:28:41 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
Feb 21 11:28:41 oak-io2-s1 kernel: Lustre: 210774:0:(ldlm_lib.c:1773:extend_recovery_timer()) oak-OST003f: extended recovery timer reaching hard limit: 900, extend: 1
Feb 21 11:28:41 oak-io2-s1 kernel: Lustre: 210774:0:(ldlm_lib.c:1773:extend_recovery_timer()) Skipped 16 previous similar messages
Feb 21 11:28:41 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 21 11:28:41 oak-io2-s1 kernel: Lustre: oak-OST003f: Recovery over after 14:37, of 1257 clients 1245 recovered and 12 were evicted.
Feb 21 11:28:41 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
Feb 21 11:28:41 oak-io2-s1 kernel: Lustre: oak-OST003f: deleting orphan objects from 0x0:4411374 to 0x0:4411396
Feb 21 11:28:43 oak-io2-s1 kernel: Lustre: oak-OST003b: recovery is timed out, evict stale exports
Feb 21 11:28:43 oak-io2-s1 kernel: Lustre: oak-OST0047: disconnecting 14 stale clients
Feb 21 11:28:43 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 21 11:28:43 oak-io2-s1 kernel: Lustre: 210910:0:(ldlm_lib.c:1773:extend_recovery_timer()) oak-OST0047: extended recovery timer reaching hard limit: 900, extend: 1
Feb 21 11:28:43 oak-io2-s1 kernel: Lustre: 210910:0:(ldlm_lib.c:1773:extend_recovery_timer()) Skipped 32 previous similar messages
Feb 21 11:28:43 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
Feb 21 11:28:43 oak-io2-s1 kernel: Lustre: oak-OST0047: Recovery over after 14:36, of 1238 clients 1224 recovered and 14 were evicted.
Feb 21 11:28:43 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 21 11:28:43 oak-io2-s1 kernel: LustreError: 175762:0:(ofd_io.c:616:ofd_preprw_write()) oak-OST0047: BRW to missing obj 0x0:4389059
Feb 21 11:28:43 oak-io2-s1 kernel: LustreError: 173730:0:(lustre_dlm.h:1372:ldlm_res_lvbo_update()) delayed lvb init failed (rc -2)
Feb 21 11:28:43 oak-io2-s1 kernel: format at lustre_dlm.h:1097:ldlm_lvbo_fill doesn't end in newline
Feb 21 11:28:45 oak-io2-s1 kernel: Lustre: oak-OST0033: recovery is timed out, evict stale exports
Feb 21 11:28:45 oak-io2-s1 kernel: Lustre: oak-OST0049: recovery is timed out, evict stale exports
Feb 21 11:28:45 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
Feb 21 11:28:45 oak-io2-s1 kernel: Lustre: oak-OST0037: disconnecting 14 stale clients
Feb 21 11:28:45 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
Feb 21 11:28:45 oak-io2-s1 kernel: Lustre: 211040:0:(ldlm_lib.c:1773:extend_recovery_timer()) oak-OST0049: extended recovery timer reaching hard limit: 900, extend: 1
Feb 21 11:28:45 oak-io2-s1 kernel: Lustre: 211040:0:(ldlm_lib.c:1773:extend_recovery_timer()) Skipped 24 previous similar messages
Feb 21 11:28:45 oak-io2-s1 kernel: Lustre: oak-OST0049: deleting orphan objects from 0x0:3442723 to 0x0:3442849
Feb 21 11:28:45 oak-io2-s1 kernel: Lustre: oak-OST0041: deleting orphan objects from 0x0:4427080 to 0x0:4427105
Feb 21 11:28:45 oak-io2-s1 kernel: Lustre: oak-OST0037: Recovery over after 14:36, of 1238 clients 1224 recovered and 14 were evicted.
Feb 21 11:28:45 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
Feb 21 11:28:45 oak-io2-s1 kernel: Lustre: oak-OST004d: deleting orphan objects from 0x0:3449716 to 0x0:3449761
Feb 21 11:28:45 oak-io2-s1 kernel: LustreError: 251672:0:(ofd_io.c:616:ofd_preprw_write()) oak-OST0037: BRW to missing obj 0x0:4426498
Feb 21 11:28:45 oak-io2-s1 kernel: LustreError: 251672:0:(ofd_io.c:616:ofd_preprw_write()) Skipped 6 previous similar messages
Feb 21 11:28:45 oak-io2-s1 kernel: LustreError: 131944:0:(lustre_dlm.h:1372:ldlm_res_lvbo_update()) delayed lvb init failed (rc -2)
Feb 21 11:28:45 oak-io2-s1 kernel: LustreError: 131944:0:(lustre_dlm.h:1372:ldlm_res_lvbo_update()) Skipped 6 previous similar messages
Feb 21 11:28:45 oak-io2-s1 kernel: Lustre: oak-OST0033: deleting orphan objects from 0x0:4409626 to 0x0:4409665
Feb 21 11:28:45 oak-io2-s1 kernel: Lustre: oak-OST004b: deleting orphan objects from 0x0:3443976 to 0x0:3444011
Feb 21 11:28:47 oak-io2-s1 kernel: Lustre: oak-OST0045: deleting orphan objects from 0x0:4406757 to 0x0:4406785
Feb 21 11:28:47 oak-io2-s1 kernel: LustreError: 63779:0:(ofd_io.c:616:ofd_preprw_write()) oak-OST0053: BRW to missing obj 0x0:156930
Feb 21 11:28:47 oak-io2-s1 kernel: LustreError: 258392:0:(lustre_dlm.h:1372:ldlm_res_lvbo_update()) delayed lvb init failed (rc -2)
Feb 21 11:28:49 oak-io2-s1 kernel: Lustre: oak-OST004f: disconnecting 14 stale clients
Feb 21 11:28:49 oak-io2-s1 kernel: Lustre: Skipped 8 previous similar messages
Feb 21 11:28:49 oak-io2-s1 kernel: Lustre: 211183:0:(ldlm_lib.c:1773:extend_recovery_timer()) oak-OST004f: extended recovery timer reaching hard limit: 900, extend: 1
Feb 21 11:28:49 oak-io2-s1 kernel: Lustre: 211183:0:(ldlm_lib.c:1773:extend_recovery_timer()) Skipped 60 previous similar messages
Feb 21 11:28:49 oak-io2-s1 kernel: Lustre: oak-OST004f: Recovery over after 14:38, of 1238 clients 1224 recovered and 14 were evicted.
Feb 21 11:28:49 oak-io2-s1 kernel: Lustre: Skipped 7 previous similar messages
Feb 21 11:28:49 oak-io2-s1 kernel: Lustre: oak-OST0039: deleting orphan objects from 0x0:4333910 to 0x0:4333953
Feb 21 11:42:51 oak-io2-s1 kernel: md: md24: data-check done.
Feb 21 11:43:40 oak-io2-s1 kernel: md: data-check of RAID array md18
Feb 21 11:44:07 oak-io2-s1 kernel: md: md12: data-check done.
Feb 21 11:44:44 oak-io2-s1 kernel: md: md10: data-check done.
Feb 21 11:44:47 oak-io2-s1 kernel: md: data-check of RAID array md8
Feb 21 11:44:53 oak-io2-s1 kernel: md: data-check of RAID array md0
Feb 21 11:45:22 oak-io2-s1 kernel: md: md6: data-check done.
Feb 21 11:45:44 oak-io2-s1 kernel: md: md28: data-check done.
Feb 21 11:45:59 oak-io2-s1 kernel: md: data-check of RAID array md2
Feb 21 11:46:01 oak-io2-s1 kernel: md: md26: data-check done.
Feb 21 11:46:05 oak-io2-s1 kernel: md: data-check of RAID array md30
Feb 21 11:46:12 oak-io2-s1 kernel: md: md16: data-check done.
Feb 21 11:46:59 oak-io2-s1 kernel: md: md22: data-check done.
Feb 21 11:47:02 oak-io2-s1 kernel: md: md20: data-check done.
Feb 21 11:47:55 oak-io2-s1 kernel: md: md14: data-check done.
Feb 21 11:48:24 oak-io2-s1 kernel: md: md32: data-check done.
Feb 21 11:48:26 oak-io2-s1 kernel: md: md34: data-check done.
Feb 21 11:50:57 oak-io2-s1 kernel: md: md4: data-check done.
Feb 21 12:04:37 oak-io2-s1 kernel: LustreError: 11-0: oak-MDT0000-lwp-OST0032: operation obd_ping to node 10.0.2.52@o2ib5 failed: rc = -107
Feb 21 12:04:37 oak-io2-s1 kernel: LustreError: 11-0: oak-MDT0000-lwp-OST0038: operation obd_ping to node 10.0.2.52@o2ib5 failed: rc = -107
Feb 21 12:04:37 oak-io2-s1 kernel: LustreError: Skipped 2 previous similar messages
Feb 21 12:04:37 oak-io2-s1 kernel: Lustre: oak-MDT0000-lwp-OST0046: Connection to oak-MDT0000 (at 10.0.2.52@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
Feb 21 12:04:37 oak-io2-s1 kernel: Lustre: Skipped 17 previous similar messages
Feb 21 12:04:37 oak-io2-s1 kernel: LustreError: Skipped 31 previous similar messages
Feb 21 12:05:02 oak-io2-s1 kernel: LustreError: 11-0: oak-MDT0000-lwp-OST003a: operation obd_ping to node 10.0.2.52@o2ib5 failed: rc = -107
Feb 21 12:05:02 oak-io2-s1 kernel: LustreError: Skipped 1 previous similar message
Feb 21 12:05:02 oak-io2-s1 kernel: Lustre: oak-MDT0000-lwp-OST003a: Connection to oak-MDT0000 (at 10.0.2.52@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
Feb 21 12:05:02 oak-io2-s1 kernel: Lustre: Skipped 32 previous similar messages
Feb 21 12:05:08 oak-io2-s1 kernel: Lustre: 129563:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519243502/real 1519243502]  req@ffff8829baab7200 x1592481950950448/t0(0) o38->oak-MDT0000-lwp-OST0049@10.0.2.51@o2ib5:12/10 lens 520/544 e 0 to 1 dl 1519243508 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Feb 21 12:05:08 oak-io2-s1 kernel: Lustre: 129563:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 53 previous similar messages
Feb 21 12:05:33 oak-io2-s1 kernel: Lustre: 129563:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519243527/real 1519243527]  req@ffff8829baab2700 x1592481950950704/t0(0) o38->oak-MDT0000-lwp-OST003a@10.0.2.51@o2ib5:12/10 lens 520/544 e 0 to 1 dl 1519243533 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Feb 21 12:05:33 oak-io2-s1 kernel: Lustre: 129563:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 34 previous similar messages
Feb 21 12:06:03 oak-io2-s1 kernel: Lustre: 129563:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519243552/real 1519243552]  req@ffff8829baab4800 x1592481950951680/t0(0) o38->oak-MDT0000-lwp-OST0049@10.0.2.51@o2ib5:12/10 lens 520/544 e 0 to 1 dl 1519243563 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Feb 21 12:06:58 oak-io2-s1 kernel: Lustre: 129563:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519243602/real 1519243602]  req@ffff8829baab3600 x1592481950953056/t0(0) o38->oak-MDT0000-lwp-OST0039@10.0.2.51@o2ib5:12/10 lens 520/544 e 0 to 1 dl 1519243618 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Feb 21 12:06:58 oak-io2-s1 kernel: Lustre: 129563:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 35 previous similar messages
Feb 21 12:07:05 oak-io2-s1 kernel: Lustre: oak-OST0035: haven't heard from client 7de7feed-7511-95a2-ea1d-cf787ba38328 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d45bfc000, cur 1519243625 expire 1519243475 last 1519243398
Feb 21 12:07:05 oak-io2-s1 kernel: Lustre: Skipped 6 previous similar messages
Feb 21 12:07:07 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to oak-MDT0000-mdtlov_UUID (at 10.0.2.52@o2ib5)
Feb 21 12:07:07 oak-io2-s1 kernel: Lustre: Skipped 1701 previous similar messages
Feb 21 12:07:32 oak-io2-s1 kernel: LustreError: 167-0: oak-MDT0000-lwp-OST0032: This client was evicted by oak-MDT0000; in progress operations using this service will fail.
Feb 21 12:07:32 oak-io2-s1 kernel: LustreError: Skipped 17 previous similar messages
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0046: deleting orphan objects from 0x0:4447028 to 0x0:4447073
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0049: deleting orphan objects from 0x0:3442854 to 0x0:3442881
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0039: deleting orphan objects from 0x0:4333957 to 0x0:4333985
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0048: deleting orphan objects from 0x0:3426362 to 0x0:3426401
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST004b: deleting orphan objects from 0x0:3444015 to 0x0:3444043
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0047: deleting orphan objects from 0x0:4389160 to 0x0:4389186
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0032: deleting orphan objects from 0x0:4410704 to 0x0:4410721
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0051: deleting orphan objects from 0x0:156900 to 0x0:156929
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST003c: deleting orphan objects from 0x0:4422920 to 0x0:4422945
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0037: deleting orphan objects from 0x0:4426499 to 0x0:4426530
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0034: deleting orphan objects from 0x0:4276398 to 0x0:4276417
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST004c: deleting orphan objects from 0x0:3440273 to 0x0:3440289
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0033: deleting orphan objects from 0x0:4409668 to 0x0:4409697
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0036: deleting orphan objects from 0x0:4236441 to 0x0:4236481
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0031: deleting orphan objects from 0x0:4387253 to 0x0:4387265
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0040: deleting orphan objects from 0x0:4430065 to 0x0:4430081
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0042: deleting orphan objects from 0x0:4319182 to 0x0:4319201
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST003a: deleting orphan objects from 0x0:4396720 to 0x0:4396737
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0045: deleting orphan objects from 0x0:4406791 to 0x0:4406817
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0030: deleting orphan objects from 0x0:4458008 to 0x0:4458049
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST004e: deleting orphan objects from 0x0:156784 to 0x0:156801
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0041: deleting orphan objects from 0x0:4427111 to 0x0:4427137
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0044: deleting orphan objects from 0x0:4391655 to 0x0:4391681
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0043: deleting orphan objects from 0x0:4389258 to 0x0:4389345
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0050: deleting orphan objects from 0x0:156058 to 0x0:156097
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST003e: deleting orphan objects from 0x0:4275091 to 0x0:4275137
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST003f: deleting orphan objects from 0x0:4411400 to 0x0:4411428
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST004d: deleting orphan objects from 0x0:3449764 to 0x0:3449793
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST004a: deleting orphan objects from 0x0:3422486 to 0x0:3422529
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0052: deleting orphan objects from 0x0:156798 to 0x0:156833
Feb 21 12:21:46 oak-io2-s1 kernel: Lustre: oak-OST0038: deleting orphan objects from 0x0:4374546 to 0x0:4374561
Feb 21 12:44:28 oak-io2-s1 kernel: Lustre: oak-OST0045: haven't heard from client 17dab95c-3765-c601-9b27-f4222ca95dab (at 10.9.112.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882d06906800, cur 1519245868 expire 1519245718 last 1519245641
Feb 21 12:44:28 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 21 13:18:04 oak-io2-s1 kernel: Lustre: oak-OST0045: haven't heard from client 5552bcd3-4f27-e74c-8ee1-179309abb500 (at 10.9.113.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804a811f400, cur 1519247884 expire 1519247734 last 1519247657
Feb 21 13:18:04 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 21 13:18:07 oak-io2-s1 kernel: Lustre: oak-OST004a: haven't heard from client 5552bcd3-4f27-e74c-8ee1-179309abb500 (at 10.9.113.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d2788b400, cur 1519247887 expire 1519247737 last 1519247660
Feb 21 13:18:13 oak-io2-s1 kernel: Lustre: oak-OST0041: haven't heard from client 5552bcd3-4f27-e74c-8ee1-179309abb500 (at 10.9.113.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b37f6dc00, cur 1519247893 expire 1519247743 last 1519247666
Feb 21 13:18:16 oak-io2-s1 kernel: Lustre: oak-OST0031: haven't heard from client 5552bcd3-4f27-e74c-8ee1-179309abb500 (at 10.9.113.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883c42fb3c00, cur 1519247896 expire 1519247746 last 1519247669
Feb 21 13:18:16 oak-io2-s1 kernel: Lustre: Skipped 29 previous similar messages
Feb 21 13:27:37 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 5552bcd3-4f27-e74c-8ee1-179309abb500 (at 10.9.113.6@o2ib4)
Feb 21 13:27:37 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to 5552bcd3-4f27-e74c-8ee1-179309abb500 (at 10.9.113.6@o2ib4)
Feb 21 13:27:37 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 5552bcd3-4f27-e74c-8ee1-179309abb500 (at 10.9.113.6@o2ib4)
Feb 21 13:27:37 oak-io2-s1 kernel: Lustre: oak-OST0038: Connection restored to 5552bcd3-4f27-e74c-8ee1-179309abb500 (at 10.9.113.6@o2ib4)
Feb 21 13:27:37 oak-io2-s1 kernel: Lustre: Skipped 67 previous similar messages
Feb 21 13:27:37 oak-io2-s1 kernel: Lustre: Skipped 67 previous similar messages
Feb 21 13:27:37 oak-io2-s1 kernel: Lustre: Skipped 67 previous similar messages
Feb 21 13:27:37 oak-io2-s1 kernel: Lustre: Skipped 10 previous similar messages
Feb 21 13:28:02 oak-io2-s1 kernel: Lustre: oak-OST0031: Connection restored to 5552bcd3-4f27-e74c-8ee1-179309abb500 (at 10.9.113.6@o2ib4)
Feb 21 13:28:02 oak-io2-s1 kernel: Lustre: Skipped 19 previous similar messages
Feb 21 13:58:17 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to  (at 10.8.15.2@o2ib6)
Feb 21 13:58:17 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to  (at 10.8.15.2@o2ib6)
Feb 21 13:58:17 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 21 13:58:43 oak-io2-s1 kernel: Lustre: oak-OST0031: Connection restored to  (at 10.8.15.2@o2ib6)
Feb 21 13:58:43 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 21 14:00:26 oak-io2-s1 kernel: Lustre: oak-OST0041: haven't heard from client 1eb55616-e815-1eed-5ced-241990d7e40c (at 10.9.112.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b37f68400, cur 1519250426 expire 1519250276 last 1519250199
Feb 21 14:00:26 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
Feb 21 14:09:50 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to 1eb55616-e815-1eed-5ced-241990d7e40c (at 10.9.112.8@o2ib4)
Feb 21 14:09:50 oak-io2-s1 kernel: Lustre: oak-OST0038: Connection restored to 1eb55616-e815-1eed-5ced-241990d7e40c (at 10.9.112.8@o2ib4)
Feb 21 14:09:50 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 1eb55616-e815-1eed-5ced-241990d7e40c (at 10.9.112.8@o2ib4)
Feb 21 14:09:50 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 21 14:09:50 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 21 14:09:50 oak-io2-s1 kernel: Lustre: Skipped 13 previous similar messages
Feb 21 14:10:15 oak-io2-s1 kernel: Lustre: oak-OST0031: Connection restored to 1eb55616-e815-1eed-5ced-241990d7e40c (at 10.9.112.8@o2ib4)
Feb 21 14:10:15 oak-io2-s1 kernel: Lustre: Skipped 15 previous similar messages
Feb 21 15:44:20 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 2e9ea172-e8f6-6fd6-a6b7-22e2a9ad7a6b (at 10.9.101.60@o2ib4)
Feb 21 15:44:20 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
Feb 21 15:44:45 oak-io2-s1 kernel: Lustre: oak-OST0031: Connection restored to 2e9ea172-e8f6-6fd6-a6b7-22e2a9ad7a6b (at 10.9.101.60@o2ib4)
Feb 21 15:44:45 oak-io2-s1 kernel: Lustre: Skipped 20 previous similar messages
Feb 21 15:51:14 oak-io2-s1 kernel: Lustre: oak-OST004d: haven't heard from client d16097e6-b240-aa27-27eb-24d11bff4c92 (at 10.9.113.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88156d4cec00, cur 1519257074 expire 1519256924 last 1519256847
Feb 21 15:51:14 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 21 15:51:29 oak-io2-s1 kernel: Lustre: oak-OST004c: haven't heard from client d16097e6-b240-aa27-27eb-24d11bff4c92 (at 10.9.113.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883ded6c9000, cur 1519257089 expire 1519256939 last 1519256862
Feb 21 15:51:33 oak-io2-s1 kernel: Lustre: oak-OST003d: haven't heard from client d16097e6-b240-aa27-27eb-24d11bff4c92 (at 10.9.113.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88155c517400, cur 1519257093 expire 1519256943 last 1519256866
Feb 21 15:51:33 oak-io2-s1 kernel: Lustre: Skipped 30 previous similar messages
Feb 21 15:51:41 oak-io2-s1 kernel: Lustre: oak-OST0034: haven't heard from client d16097e6-b240-aa27-27eb-24d11bff4c92 (at 10.9.113.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883cda8d5800, cur 1519257101 expire 1519256951 last 1519256874
Feb 21 16:00:19 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to d16097e6-b240-aa27-27eb-24d11bff4c92 (at 10.9.113.7@o2ib4)
Feb 21 16:00:19 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to d16097e6-b240-aa27-27eb-24d11bff4c92 (at 10.9.113.7@o2ib4)
Feb 21 16:00:19 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to d16097e6-b240-aa27-27eb-24d11bff4c92 (at 10.9.113.7@o2ib4)
Feb 21 16:00:19 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to d16097e6-b240-aa27-27eb-24d11bff4c92 (at 10.9.113.7@o2ib4)
Feb 21 16:00:19 oak-io2-s1 kernel: Lustre: Skipped 10 previous similar messages
Feb 21 16:00:19 oak-io2-s1 kernel: Lustre: Skipped 10 previous similar messages
Feb 21 16:00:19 oak-io2-s1 kernel: Lustre: Skipped 10 previous similar messages
Feb 21 16:00:19 oak-io2-s1 kernel: Lustre: Skipped 12 previous similar messages
Feb 21 16:00:44 oak-io2-s1 kernel: Lustre: oak-OST0031: Connection restored to d16097e6-b240-aa27-27eb-24d11bff4c92 (at 10.9.113.7@o2ib4)
Feb 21 16:00:44 oak-io2-s1 kernel: Lustre: Skipped 17 previous similar messages
Feb 21 16:25:51 oak-io2-s1 kernel: Lustre: oak-OST003f: haven't heard from client 674e65bd-d9a0-971c-e78d-61c4a8a708bd (at 10.210.47.106@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882a2bda4400, cur 1519259151 expire 1519259001 last 1519258924
Feb 21 16:25:51 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 21 16:39:05 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to  (at 10.210.47.106@o2ib3)
Feb 21 16:39:05 oak-io2-s1 kernel: Lustre: Skipped 14 previous similar messages
Feb 21 16:39:31 oak-io2-s1 kernel: Lustre: oak-OST0045: Connection restored to  (at 10.210.47.106@o2ib3)
Feb 21 16:39:31 oak-io2-s1 kernel: Lustre: oak-OST0031: Connection restored to  (at 10.210.47.106@o2ib3)
Feb 21 16:39:31 oak-io2-s1 kernel: Lustre: oak-OST0037: Connection restored to  (at 10.210.47.106@o2ib3)
Feb 21 16:39:31 oak-io2-s1 kernel: Lustre: Skipped 4 previous similar messages
Feb 21 16:39:31 oak-io2-s1 kernel: Lustre: Skipped 4 previous similar messages
Feb 21 16:39:31 oak-io2-s1 kernel: Lustre: Skipped 8 previous similar messages
Feb 21 16:40:15 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to db6d3d40-4b8c-e093-15a4-aa74120c714b (at 10.210.46.49@o2ib3)
Feb 21 16:40:15 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
Feb 21 16:40:26 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 7926e9f4-ad37-af82-48cd-0434817ff711 (at 10.210.44.47@o2ib3)
Feb 21 16:40:26 oak-io2-s1 kernel: Lustre: Skipped 15 previous similar messages
Feb 21 16:40:40 oak-io2-s1 kernel: Lustre: oak-OST003d: Connection restored to db6d3d40-4b8c-e093-15a4-aa74120c714b (at 10.210.46.49@o2ib3)
Feb 21 16:40:40 oak-io2-s1 kernel: Lustre: oak-OST0039: Connection restored to db6d3d40-4b8c-e093-15a4-aa74120c714b (at 10.210.46.49@o2ib3)
Feb 21 16:40:40 oak-io2-s1 kernel: Lustre: oak-OST0031: Connection restored to db6d3d40-4b8c-e093-15a4-aa74120c714b (at 10.210.46.49@o2ib3)
Feb 21 16:40:40 oak-io2-s1 kernel: Lustre: oak-OST0037: Connection restored to db6d3d40-4b8c-e093-15a4-aa74120c714b (at 10.210.46.49@o2ib3)
Feb 21 16:40:40 oak-io2-s1 kernel: Lustre: oak-OST0035: Connection restored to db6d3d40-4b8c-e093-15a4-aa74120c714b (at 10.210.46.49@o2ib3)
Feb 21 16:40:40 oak-io2-s1 kernel: Lustre: oak-OST0033: Connection restored to db6d3d40-4b8c-e093-15a4-aa74120c714b (at 10.210.46.49@o2ib3)
Feb 21 16:40:40 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 21 16:40:40 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 21 16:40:40 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 21 16:40:40 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 21 16:40:40 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 21 16:40:40 oak-io2-s1 kernel: Lustre: Skipped 9 previous similar messages
Feb 21 18:48:20 oak-io2-s1 kernel: Lustre: oak-OST0049: Client 7461b1df-58fb-1087-c619-45f28694768c (at 10.9.104.12@o2ib4) reconnecting
Feb 21 18:48:20 oak-io2-s1 kernel: Lustre: oak-OST0031: Client 7461b1df-58fb-1087-c619-45f28694768c (at 10.9.104.12@o2ib4) reconnecting
Feb 21 18:48:20 oak-io2-s1 kernel: Lustre: oak-OST003f: Client 7461b1df-58fb-1087-c619-45f28694768c (at 10.9.104.12@o2ib4) reconnecting
Feb 21 18:48:20 oak-io2-s1 kernel: Lustre: oak-OST0045: Client 7461b1df-58fb-1087-c619-45f28694768c (at 10.9.104.12@o2ib4) reconnecting
Feb 21 18:48:20 oak-io2-s1 kernel: Lustre: Skipped 71 previous similar messages
Feb 21 18:48:20 oak-io2-s1 kernel: Lustre: Skipped 71 previous similar messages
Feb 21 18:48:20 oak-io2-s1 kernel: Lustre: Skipped 71 previous similar messages
Feb 21 18:48:20 oak-io2-s1 kernel: Lustre: oak-OST0045: Connection restored to 7461b1df-58fb-1087-c619-45f28694768c (at 10.9.104.12@o2ib4)
Feb 21 18:48:20 oak-io2-s1 kernel: Lustre: oak-OST003f: Connection restored to 7461b1df-58fb-1087-c619-45f28694768c (at 10.9.104.12@o2ib4)
Feb 21 18:48:20 oak-io2-s1 kernel: Lustre: Skipped 15 previous similar messages
Feb 21 18:48:20 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 21 18:48:20 oak-io2-s1 kernel: Lustre: Skipped 8 previous similar messages
Feb 21 18:49:10 oak-io2-s1 kernel: Lustre: oak-OST0030: Client 1a82bd75-72e1-8040-9fd0-f227412ed7d7 (at 10.210.45.38@o2ib3) reconnecting
Feb 21 18:49:10 oak-io2-s1 kernel: Lustre: oak-OST0031: Connection restored to 1a82bd75-72e1-8040-9fd0-f227412ed7d7 (at 10.210.45.38@o2ib3)
Feb 21 18:49:10 oak-io2-s1 kernel: Lustre: Skipped 13005 previous similar messages
Feb 21 18:49:10 oak-io2-s1 kernel: Lustre: Skipped 13005 previous similar messages
Feb 21 18:49:59 oak-io2-s1 kernel: Lustre: oak-OST0052: Connection restored to dbad0532-ffa9-7d4e-e8f2-8bbf137ac40a (at 10.9.103.16@o2ib4)
Feb 21 18:49:59 oak-io2-s1 kernel: Lustre: oak-OST0051: Connection restored to dbad0532-ffa9-7d4e-e8f2-8bbf137ac40a (at 10.9.103.16@o2ib4)
Feb 21 18:49:59 oak-io2-s1 kernel: Lustre: Skipped 24 previous similar messages
Feb 21 18:49:59 oak-io2-s1 kernel: Lustre: Skipped 19 previous similar messages
Feb 21 18:50:02 oak-io2-s1 kernel: LustreError: 62502:0:(ldlm_lib.c:3236:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8817f95f6850 x1591095117516672/t0(0) o4->89ace3c6-2eb8-89f3-a103-a2495018f228@10.9.101.55@o2ib4:182/0 lens 608/448 e 0 to 0 dl 1519267807 ref 1 fl Interpret:/0/0 rc 0/0
Feb 21 18:50:02 oak-io2-s1 kernel: Lustre: oak-OST003c: Bulk IO write error with 89ace3c6-2eb8-89f3-a103-a2495018f228 (at 10.9.101.55@o2ib4), client will retry: rc = -110
Feb 21 18:50:02 oak-io2-s1 kernel: Lustre: Skipped 24 previous similar messages
Feb 21 18:50:15 oak-io2-s1 kernel: LustreError: 62502:0:(ldlm_lib.c:3236:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff88039624cc50 x1591878701963744/t0(0) o4->70ea5860-ea02-c80c-735a-6a5e8618d73c@10.8.18.14@o2ib6:195/0 lens 608/448 e 0 to 0 dl 1519267820 ref 1 fl Interpret:/0/0 rc 0/0
Feb 21 18:50:15 oak-io2-s1 kernel: Lustre: oak-OST003b: Bulk IO write error with 70ea5860-ea02-c80c-735a-6a5e8618d73c (at 10.8.18.14@o2ib6), client will retry: rc = -110
Feb 21 18:50:20 oak-io2-s1 kernel: Lustre: oak-OST0031: Client 18aab27a-dacb-a4b4-76d0-192734850c7d (at 10.8.18.24@o2ib6) reconnecting
Feb 21 18:50:20 oak-io2-s1 kernel: Lustre: Skipped 8774 previous similar messages
Feb 21 18:51:04 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 51ac6e63-c3f7-ca59-16ef-4eb91d23dbd7 (at 10.9.104.51@o2ib4)
Feb 21 18:51:04 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to 51ac6e63-c3f7-ca59-16ef-4eb91d23dbd7 (at 10.9.104.51@o2ib4)
Feb 21 18:51:04 oak-io2-s1 kernel: Lustre: Skipped 16852 previous similar messages
Feb 21 18:51:04 oak-io2-s1 kernel: Lustre: Skipped 15 previous similar messages
Feb 21 18:51:38 oak-io2-s1 kernel: LustreError: 135003:0:(ldlm_lockd.c:2365:ldlm_cancel_handler()) ldlm_cancel from 10.9.104.28@o2ib4 arrived at 1519267898 with bad export cookie 0
Feb 21 18:51:39 oak-io2-s1 kernel: LustreError: 301774:0:(ldlm_lib.c:3236:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff881fde03e450 x1591095117516656/t0(0) o4->89ace3c6-2eb8-89f3-a103-a2495018f228@10.9.101.55@o2ib4:282/0 lens 608/448 e 4 to 0 dl 1519267907 ref 1 fl Interpret:/0/0 rc 0/0
Feb 21 18:51:39 oak-io2-s1 kernel: Lustre: oak-OST0051: Bulk IO write error with 89ace3c6-2eb8-89f3-a103-a2495018f228 (at 10.9.101.55@o2ib4), client will retry: rc = -110
Feb 21 18:52:12 oak-io2-s1 kernel: Lustre: oak-OST0030: haven't heard from client ea411769-8915-6823-c572-b4d43a6ebe81 (at 10.210.47.41@o2ib3) in 226 seconds. I think it's dead, and I am evicting it. exp ffff881ff1b0f400, cur 1519267932 expire 1519267782 last 1519267706
Feb 21 18:52:12 oak-io2-s1 kernel: Lustre: Skipped 107 previous similar messages
Feb 21 18:52:13 oak-io2-s1 kernel: Lustre: oak-OST003c: haven't heard from client 9f775363-f028-046a-4f86-6a31f8156b5f (at 10.9.105.58@o2ib4) in 197 seconds. I think it's dead, and I am evicting it. exp ffff883ca7ede400, cur 1519267933 expire 1519267783 last 1519267736
Feb 21 18:52:13 oak-io2-s1 kernel: Lustre: Skipped 7 previous similar messages
Feb 21 18:52:28 oak-io2-s1 kernel: Lustre: oak-OST0030: Client 94bd7708-fcb9-3baa-2082-e172ea10b753 (at 10.210.46.73@o2ib3) reconnecting
Feb 21 18:52:28 oak-io2-s1 kernel: Lustre: Skipped 63957 previous similar messages
Feb 21 19:56:53 oak-io2-s1 kernel: Lustre: oak-OST0031: Client 09979287-78e8-d016-8f34-973ac95dc589 (at 10.9.104.19@o2ib4) reconnecting
Feb 21 19:56:53 oak-io2-s1 kernel: Lustre: oak-OST0038: Connection restored to 09979287-78e8-d016-8f34-973ac95dc589 (at 10.9.104.19@o2ib4)
Feb 21 19:56:53 oak-io2-s1 kernel: Lustre: Skipped 65314 previous similar messages
Feb 21 19:56:53 oak-io2-s1 kernel: Lustre: Skipped 9415 previous similar messages
Feb 21 19:57:28 oak-io2-s1 kernel: Lustre: oak-OST0039: Client fab6a15a-3f78-a8e4-40d4-43e3c90d9b87 (at 10.9.102.65@o2ib4) reconnecting
Feb 21 19:57:28 oak-io2-s1 kernel: Lustre: Skipped 92 previous similar messages
Feb 21 19:57:28 oak-io2-s1 kernel: Lustre: oak-OST0039: Connection restored to fab6a15a-3f78-a8e4-40d4-43e3c90d9b87 (at 10.9.102.65@o2ib4)
Feb 21 19:57:28 oak-io2-s1 kernel: Lustre: Skipped 99 previous similar messages
Feb 21 19:57:45 oak-io2-s1 kernel: LustreError: 185052:0:(ldlm_lib.c:3236:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff883549921050 x1591055802559504/t0(0) o4->7f22b6e5-5e60-0633-6d43-41c391054a8e@10.9.102.26@o2ib4:470/0 lens 608/448 e 0 to 0 dl 1519271870 ref 1 fl Interpret:/0/0 rc 0/0
Feb 21 19:57:45 oak-io2-s1 kernel: LustreError: 185052:0:(ldlm_lib.c:3236:target_bulk_io()) Skipped 1 previous similar message
Feb 21 19:57:45 oak-io2-s1 kernel: Lustre: oak-OST003e: Bulk IO write error with 7f22b6e5-5e60-0633-6d43-41c391054a8e (at 10.9.102.26@o2ib4), client will retry: rc = -110
Feb 21 19:57:45 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
Feb 21 19:58:32 oak-io2-s1 kernel: Lustre: oak-OST0032: Client 1023aa9d-1c81-17bf-ad08-9f615d992694 (at 10.210.47.116@o2ib3) reconnecting
Feb 21 19:58:32 oak-io2-s1 kernel: Lustre: oak-OST0035: Connection restored to 1023aa9d-1c81-17bf-ad08-9f615d992694 (at 10.210.47.116@o2ib3)
Feb 21 19:58:32 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to 1023aa9d-1c81-17bf-ad08-9f615d992694 (at 10.210.47.116@o2ib3)
Feb 21 19:58:32 oak-io2-s1 kernel: Lustre: Skipped 53799 previous similar messages
Feb 21 19:58:32 oak-io2-s1 kernel: Lustre: Skipped 53799 previous similar messages
Feb 21 19:58:32 oak-io2-s1 kernel: Lustre: Skipped 53826 previous similar messages
Feb 21 22:31:03 oak-io2-s1 kernel: Lustre: oak-OST0044: haven't heard from client 51e89c38-fbba-0757-e10c-4872b8f7d281 (at 10.8.28.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883caae7f400, cur 1519281063 expire 1519280913 last 1519280836
Feb 21 22:31:03 oak-io2-s1 kernel: Lustre: Skipped 36 previous similar messages
Feb 22 06:10:01 oak-io2-s1 kernel: Lustre: oak-OST004c: haven't heard from client 35ff4bab-ea66-ec8e-ca5d-3c6ff7075194 (at 10.9.112.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880edb099400, cur 1519308601 expire 1519308451 last 1519308374
Feb 22 06:10:01 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 22 07:28:01 oak-io2-s1 kernel: Lustre: oak-OST004b: haven't heard from client ab3fefa7-8cee-4359-a92b-5ff5ee75eb32 (at 10.210.46.127@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fde2a3c00, cur 1519313281 expire 1519313131 last 1519313054
Feb 22 07:28:01 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 22 07:40:02 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to ab3fefa7-8cee-4359-a92b-5ff5ee75eb32 (at 10.210.46.127@o2ib3)
Feb 22 07:40:02 oak-io2-s1 kernel: Lustre: Skipped 1818 previous similar messages
Feb 22 07:40:35 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to aea09ef5-c649-fac0-d976-164232fcb59b (at 10.210.47.108@o2ib3)
Feb 22 07:40:35 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to aea09ef5-c649-fac0-d976-164232fcb59b (at 10.210.47.108@o2ib3)
Feb 22 07:40:35 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to aea09ef5-c649-fac0-d976-164232fcb59b (at 10.210.47.108@o2ib3)
Feb 22 07:40:35 oak-io2-s1 kernel: Lustre: oak-OST003a: Connection restored to aea09ef5-c649-fac0-d976-164232fcb59b (at 10.210.47.108@o2ib3)
Feb 22 07:40:35 oak-io2-s1 kernel: Lustre: oak-OST0038: Connection restored to aea09ef5-c649-fac0-d976-164232fcb59b (at 10.210.47.108@o2ib3)
Feb 22 07:40:35 oak-io2-s1 kernel: Lustre: Skipped 38 previous similar messages
Feb 22 07:40:35 oak-io2-s1 kernel: Lustre: Skipped 38 previous similar messages
Feb 22 07:40:35 oak-io2-s1 kernel: Lustre: Skipped 38 previous similar messages
Feb 22 07:40:35 oak-io2-s1 kernel: Lustre: Skipped 38 previous similar messages
Feb 22 07:40:35 oak-io2-s1 kernel: Lustre: Skipped 11 previous similar messages
Feb 22 08:21:41 oak-io2-s1 kernel: Lustre: oak-OST0049: haven't heard from client ef54ddaa-b488-a9e3-469f-c7a53917244f (at 10.210.47.105@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8839c813d000, cur 1519316501 expire 1519316351 last 1519316274
Feb 22 08:21:41 oak-io2-s1 kernel: Lustre: Skipped 215 previous similar messages
Feb 22 08:43:01 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 2e9ea172-e8f6-6fd6-a6b7-22e2a9ad7a6b (at 10.9.101.60@o2ib4)
Feb 22 08:43:01 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 2e9ea172-e8f6-6fd6-a6b7-22e2a9ad7a6b (at 10.9.101.60@o2ib4)
Feb 22 08:43:01 oak-io2-s1 kernel: Lustre: Skipped 167 previous similar messages
Feb 22 08:43:01 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 22 08:50:09 oak-io2-s1 kernel: Lustre: oak-OST0032: haven't heard from client 05d353d5-c0ad-f363-77dd-e7ed5dacee45 (at 10.9.101.60@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8813f8fd1800, cur 1519318209 expire 1519318059 last 1519317982
Feb 22 08:50:09 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 22 09:01:50 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 2e9ea172-e8f6-6fd6-a6b7-22e2a9ad7a6b (at 10.9.101.60@o2ib4)
Feb 22 09:01:50 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 2e9ea172-e8f6-6fd6-a6b7-22e2a9ad7a6b (at 10.9.101.60@o2ib4)
Feb 22 09:01:50 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 22 09:01:50 oak-io2-s1 kernel: Lustre: Skipped 10 previous similar messages
Feb 22 09:02:16 oak-io2-s1 kernel: Lustre: oak-OST0031: Connection restored to 2e9ea172-e8f6-6fd6-a6b7-22e2a9ad7a6b (at 10.9.101.60@o2ib4)
Feb 22 09:02:16 oak-io2-s1 kernel: Lustre: Skipped 21 previous similar messages
Feb 22 09:39:58 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to e8d03940-2841-ddc6-88a1-756abe60edf4 (at 10.9.0.1@o2ib4)
Feb 22 09:39:58 oak-io2-s1 kernel: Lustre: Skipped 4 previous similar messages
Feb 22 09:40:23 oak-io2-s1 kernel: Lustre: oak-OST0031: Connection restored to e8d03940-2841-ddc6-88a1-756abe60edf4 (at 10.9.0.1@o2ib4)
Feb 22 09:40:23 oak-io2-s1 kernel: Lustre: Skipped 27 previous similar messages
Feb 22 09:45:13 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to  (at 10.8.1.27@o2ib6)
Feb 22 09:45:13 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to  (at 10.8.1.27@o2ib6)
Feb 22 09:45:13 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
Feb 22 09:45:13 oak-io2-s1 kernel: Lustre: Skipped 15 previous similar messages
Feb 22 09:45:38 oak-io2-s1 kernel: Lustre: oak-OST003b: Connection restored to  (at 10.8.1.27@o2ib6)
Feb 22 09:45:38 oak-io2-s1 kernel: Lustre: oak-OST0037: Connection restored to  (at 10.8.1.27@o2ib6)
Feb 22 09:45:38 oak-io2-s1 kernel: Lustre: Skipped 4 previous similar messages
Feb 22 09:45:38 oak-io2-s1 kernel: Lustre: Skipped 10 previous similar messages
Feb 22 09:51:27 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to aa0141b0-d017-626c-c09d-b75d6daae2e4 (at 10.8.1.28@o2ib6)
Feb 22 09:51:27 oak-io2-s1 kernel: Lustre: Skipped 15 previous similar messages
Feb 22 10:09:46 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 2e9ea172-e8f6-6fd6-a6b7-22e2a9ad7a6b (at 10.9.101.60@o2ib4)
Feb 22 10:09:46 oak-io2-s1 kernel: Lustre: Skipped 23 previous similar messages
Feb 22 10:10:11 oak-io2-s1 kernel: Lustre: oak-OST0035: Connection restored to 2e9ea172-e8f6-6fd6-a6b7-22e2a9ad7a6b (at 10.9.101.60@o2ib4)
Feb 22 10:10:11 oak-io2-s1 kernel: Lustre: oak-OST0039: Connection restored to 2e9ea172-e8f6-6fd6-a6b7-22e2a9ad7a6b (at 10.9.101.60@o2ib4)
Feb 22 10:10:11 oak-io2-s1 kernel: Lustre: oak-OST0031: Connection restored to 2e9ea172-e8f6-6fd6-a6b7-22e2a9ad7a6b (at 10.9.101.60@o2ib4)
Feb 22 10:10:11 oak-io2-s1 kernel: Lustre: Skipped 6 previous similar messages
Feb 22 10:10:11 oak-io2-s1 kernel: Lustre: Skipped 6 previous similar messages
Feb 22 10:10:11 oak-io2-s1 kernel: Lustre: Skipped 15 previous similar messages
Feb 22 10:12:36 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to a6968ebc-f439-75ac-3be1-ab11e5061596 (at 10.8.26.33@o2ib6)
Feb 22 10:12:36 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to a6968ebc-f439-75ac-3be1-ab11e5061596 (at 10.8.26.33@o2ib6)
Feb 22 10:12:36 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to a6968ebc-f439-75ac-3be1-ab11e5061596 (at 10.8.26.33@o2ib6)
Feb 22 10:12:36 oak-io2-s1 kernel: Lustre: Skipped 14 previous similar messages
Feb 22 10:17:01 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 59b13195-34b1-8dc0-1073-e82bd675167c (at 10.9.105.51@o2ib4)
Feb 22 10:17:01 oak-io2-s1 kernel: Lustre: Skipped 3648 previous similar messages
Feb 22 10:30:25 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c2359b6b-1fee-229e-dce9-c36c4371d32f (at 10.9.102.39@o2ib4)
Feb 22 10:30:25 oak-io2-s1 kernel: Lustre: Skipped 279 previous similar messages
Feb 22 10:45:25 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 50b56427-cfcb-b700-037c-334f11d1cb00 (at 10.9.101.48@o2ib4)
Feb 22 10:45:25 oak-io2-s1 kernel: Lustre: Skipped 3297 previous similar messages
Feb 22 11:04:21 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 64597622-66c8-e766-d445-7261952d7072 (at 10.9.101.18@o2ib4)
Feb 22 11:04:21 oak-io2-s1 kernel: Lustre: Skipped 34 previous similar messages
Feb 22 11:14:43 oak-io2-s1 kernel: Lustre: oak-OST004f: Client f1d1d5b3-77a3-d711-f284-7184f5444cdb (at 10.210.44.249@o2ib3) reconnecting
Feb 22 11:14:43 oak-io2-s1 kernel: Lustre: oak-OST0035: Client f1d1d5b3-77a3-d711-f284-7184f5444cdb (at 10.210.44.249@o2ib3) reconnecting
Feb 22 11:14:43 oak-io2-s1 kernel: Lustre: oak-OST0050: Client f1d1d5b3-77a3-d711-f284-7184f5444cdb (at 10.210.44.249@o2ib3) reconnecting
Feb 22 11:14:43 oak-io2-s1 kernel: Lustre: Skipped 1777 previous similar messages
Feb 22 11:14:43 oak-io2-s1 kernel: Lustre: Skipped 1777 previous similar messages
Feb 22 11:14:43 oak-io2-s1 kernel: Lustre: oak-OST0035: Connection restored to  (at 10.210.44.249@o2ib3)
Feb 22 11:14:43 oak-io2-s1 kernel: Lustre: Skipped 168 previous similar messages
Feb 22 11:14:43 oak-io2-s1 kernel: Lustre: Skipped 31 previous similar messages
Feb 22 11:46:17 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to b6338d77-1fa9-be10-93d4-b9041c881f67 (at 10.9.113.11@o2ib4)
Feb 22 11:46:17 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to b6338d77-1fa9-be10-93d4-b9041c881f67 (at 10.9.113.11@o2ib4)
Feb 22 11:46:17 oak-io2-s1 kernel: Lustre: Skipped 33 previous similar messages
Feb 22 11:46:17 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 22 11:47:49 oak-io2-s1 kernel: Lustre: oak-OST0033: Connection restored to  (at 10.9.101.50@o2ib4)
Feb 22 11:47:49 oak-io2-s1 kernel: Lustre: Skipped 84 previous similar messages
Feb 22 11:52:16 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to b0a7599d-e6fa-20c7-5218-7059666ae4c4 (at 10.9.114.7@o2ib4)
Feb 22 11:52:16 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to b0a7599d-e6fa-20c7-5218-7059666ae4c4 (at 10.9.114.7@o2ib4)
Feb 22 11:52:16 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to b0a7599d-e6fa-20c7-5218-7059666ae4c4 (at 10.9.114.7@o2ib4)
Feb 22 11:52:16 oak-io2-s1 kernel: Lustre: Skipped 14 previous similar messages
Feb 22 12:01:20 oak-io2-s1 kernel: Lustre: oak-OST0038: Connection restored to 3e650e62-845d-6d60-26a3-3d4109f50747 (at 10.9.112.13@o2ib4)
Feb 22 12:01:20 oak-io2-s1 kernel: Lustre: oak-OST003a: Connection restored to 3e650e62-845d-6d60-26a3-3d4109f50747 (at 10.9.112.13@o2ib4)
Feb 22 12:01:20 oak-io2-s1 kernel: Lustre: Skipped 813 previous similar messages
Feb 22 12:01:20 oak-io2-s1 kernel: Lustre: Skipped 11 previous similar messages
Feb 22 12:11:28 oak-io2-s1 kernel: Lustre: oak-OST0031: Connection restored to 51e89c38-fbba-0757-e10c-4872b8f7d281 (at 10.8.28.12@o2ib6)
Feb 22 12:11:28 oak-io2-s1 kernel: Lustre: oak-OST0033: Connection restored to 51e89c38-fbba-0757-e10c-4872b8f7d281 (at 10.8.28.12@o2ib6)
Feb 22 12:11:28 oak-io2-s1 kernel: Lustre: Skipped 36 previous similar messages
Feb 22 12:11:28 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 22 13:11:13 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to 00d4168e-b282-fe05-71fa-6d4c2929a322 (at 10.8.28.2@o2ib6)
Feb 22 13:11:13 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 00d4168e-b282-fe05-71fa-6d4c2929a322 (at 10.8.28.2@o2ib6)
Feb 22 13:11:13 oak-io2-s1 kernel: Lustre: oak-OST0038: Connection restored to 00d4168e-b282-fe05-71fa-6d4c2929a322 (at 10.8.28.2@o2ib6)
Feb 22 13:11:13 oak-io2-s1 kernel: Lustre: Skipped 5 previous similar messages
Feb 22 13:11:13 oak-io2-s1 kernel: Lustre: Skipped 5 previous similar messages
Feb 22 13:11:13 oak-io2-s1 kernel: Lustre: Skipped 10 previous similar messages
Feb 22 13:12:35 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 89ace3c6-2eb8-89f3-a103-a2495018f228 (at 10.9.101.55@o2ib4)
Feb 22 13:12:35 oak-io2-s1 kernel: Lustre: Skipped 356 previous similar messages
Feb 22 13:18:17 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 6cfc17b8-b5a8-7bc9-3ba4-f9d50bb01c21 (at 10.9.112.17@o2ib4)
Feb 22 13:18:17 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 6cfc17b8-b5a8-7bc9-3ba4-f9d50bb01c21 (at 10.9.112.17@o2ib4)
Feb 22 13:18:17 oak-io2-s1 kernel: Lustre: Skipped 97 previous similar messages
Feb 22 13:18:17 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 22 14:25:36 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 22 14:25:36 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 22 14:25:36 oak-io2-s1 kernel: Lustre: Skipped 18 previous similar messages
Feb 22 14:25:36 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 22 14:27:12 oak-io2-s1 kernel: Lustre: oak-OST0039: Connection restored to baccbd1e-f756-b740-0a2c-485ab7fddb76 (at 10.9.101.53@o2ib4)
Feb 22 14:27:12 oak-io2-s1 kernel: Lustre: Skipped 172 previous similar messages
Feb 22 14:29:48 oak-io2-s1 kernel: Lustre: oak-OST0037: Connection restored to c7d5dc36-f107-849d-e976-61d2b500e8cd (at 10.9.101.52@o2ib4)
Feb 22 14:29:48 oak-io2-s1 kernel: Lustre: oak-OST0033: Connection restored to c7d5dc36-f107-849d-e976-61d2b500e8cd (at 10.9.101.52@o2ib4)
Feb 22 14:29:48 oak-io2-s1 kernel: Lustre: oak-OST003f: Connection restored to c7d5dc36-f107-849d-e976-61d2b500e8cd (at 10.9.101.52@o2ib4)
Feb 22 14:29:48 oak-io2-s1 kernel: Lustre: Skipped 96 previous similar messages
Feb 22 14:29:48 oak-io2-s1 kernel: Lustre: Skipped 96 previous similar messages
Feb 22 14:29:48 oak-io2-s1 kernel: Lustre: Skipped 10 previous similar messages
Feb 22 14:51:29 oak-io2-s1 kernel: Lustre: oak-OST0042: haven't heard from client 6a9f55f7-322f-0aa5-b322-370a427863d5 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8811b783fc00, cur 1519339889 expire 1519339739 last 1519339662
Feb 22 14:51:29 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 22 15:10:22 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 70d66984-cbf7-71f7-cfbf-7f3f48d8a9ef (at 10.9.114.8@o2ib4)
Feb 22 15:10:22 oak-io2-s1 kernel: Lustre: Skipped 17 previous similar messages
Feb 22 15:51:02 oak-io2-s1 kernel: Lustre: oak-OST0049: haven't heard from client 0a268942-d0a2-02b0-c5d8-38e968656e83 (at 10.210.47.108@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882bccee8000, cur 1519343462 expire 1519343312 last 1519343235
Feb 22 15:51:02 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 22 17:03:39 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0040: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8dda:0x0] object 0x0:4434610 extent [4194304-8388607], client returned csum 54fde0b7 (type 4), server csum 1f41acdd (type 4)
Feb 22 17:03:39 oak-io2-s1 kernel: LustreError: Skipped 6 previous similar messages
Feb 22 17:03:59 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0040: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8dda:0x0] object 0x0:4434610 extent [4194304-8388607], client returned csum 54fde0b7 (type 4), server csum 1f41acdd (type 4)
Feb 22 17:03:59 oak-io2-s1 kernel: LustreError: Skipped 4 previous similar messages
Feb 22 17:04:33 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0040: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8dda:0x0] object 0x0:4434610 extent [4194304-8388607], client returned csum 54fde0b7 (type 4), server csum 1f41acdd (type 4)
Feb 22 17:04:33 oak-io2-s1 kernel: LustreError: Skipped 3 previous similar messages
Feb 22 17:05:38 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0040: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8dda:0x0] object 0x0:4434610 extent [4194304-7237631], client returned csum 9ac3682f (type 4), server csum 664ab593 (type 4)
Feb 22 17:05:38 oak-io2-s1 kernel: LustreError: Skipped 12 previous similar messages
Feb 22 17:07:50 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0040: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8dda:0x0] object 0x0:4434610 extent [4194304-8388607], client returned csum 54fde0b7 (type 4), server csum 1f41acdd (type 4)
Feb 22 17:07:50 oak-io2-s1 kernel: LustreError: Skipped 22 previous similar messages
Feb 22 17:11:17 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to f2fc2008-bb5f-07e6-6fbb-ac07090ab2fb (at 10.8.18.35@o2ib6)
Feb 22 17:11:17 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to f2fc2008-bb5f-07e6-6fbb-ac07090ab2fb (at 10.8.18.35@o2ib6)
Feb 22 17:11:17 oak-io2-s1 kernel: Lustre: Skipped 17 previous similar messages
Feb 22 17:11:17 oak-io2-s1 kernel: Lustre: Skipped 15 previous similar messages
Feb 22 17:11:26 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to dae73673-b89d-dcde-ce37-464280cdd141 (at 10.8.2.16@o2ib6)
Feb 22 17:11:26 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to dae73673-b89d-dcde-ce37-464280cdd141 (at 10.8.2.16@o2ib6)
Feb 22 17:11:26 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to dae73673-b89d-dcde-ce37-464280cdd141 (at 10.8.2.16@o2ib6)
Feb 22 17:11:26 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to dae73673-b89d-dcde-ce37-464280cdd141 (at 10.8.2.16@o2ib6)
Feb 22 17:11:26 oak-io2-s1 kernel: Lustre: Skipped 374 previous similar messages
Feb 22 17:11:26 oak-io2-s1 kernel: Lustre: Skipped 374 previous similar messages
Feb 22 17:11:26 oak-io2-s1 kernel: Lustre: Skipped 374 previous similar messages
Feb 22 17:11:26 oak-io2-s1 kernel: Lustre: Skipped 14 previous similar messages
Feb 22 17:11:45 oak-io2-s1 kernel: Lustre: oak-OST003b: Connection restored to 8c6c7eba-cfd3-a8a4-7d4e-c115b33d2d47 (at 10.9.105.18@o2ib4)
Feb 22 17:11:45 oak-io2-s1 kernel: Lustre: oak-OST0033: Connection restored to 8c6c7eba-cfd3-a8a4-7d4e-c115b33d2d47 (at 10.9.105.18@o2ib4)
Feb 22 17:11:45 oak-io2-s1 kernel: Lustre: oak-OST0037: Connection restored to 8c6c7eba-cfd3-a8a4-7d4e-c115b33d2d47 (at 10.9.105.18@o2ib4)
Feb 22 17:11:45 oak-io2-s1 kernel: Lustre: Skipped 408 previous similar messages
Feb 22 17:11:45 oak-io2-s1 kernel: Lustre: Skipped 408 previous similar messages
Feb 22 17:11:45 oak-io2-s1 kernel: Lustre: Skipped 15 previous similar messages
Feb 22 17:24:20 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 31f88457-4737-4098-430b-1de18cc9c3b2 (at 10.9.101.63@o2ib4)
Feb 22 17:24:20 oak-io2-s1 kernel: Lustre: Skipped 543 previous similar messages
Feb 22 17:27:29 oak-io2-s1 kernel: Lustre: oak-OST0051: haven't heard from client 1eac496c-e83a-4227-c7cc-532c9a767740 (at 10.9.114.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881200880c00, cur 1519349249 expire 1519349099 last 1519349022
Feb 22 17:27:29 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 22 17:36:05 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 70d66984-cbf7-71f7-cfbf-7f3f48d8a9ef (at 10.9.114.8@o2ib4)
Feb 22 17:36:05 oak-io2-s1 kernel: Lustre: Skipped 74 previous similar messages
Feb 22 23:21:36 oak-io2-s1 kernel: Lustre: oak-OST0036: haven't heard from client 2a67fa2c-d9c4-c535-3b45-fb4ccd799362 (at 10.9.112.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883cac657400, cur 1519370496 expire 1519370346 last 1519370269
Feb 22 23:21:36 oak-io2-s1 kernel: Lustre: Skipped 71 previous similar messages
Feb 23 00:49:47 oak-io2-s1 kernel: Lustre: DEBUG MARKER: Fri Feb 23 00:49:46 2018
Feb 23 02:38:01 oak-io2-s1 kernel: Lustre: oak-OST003b: haven't heard from client 3329e82e-efa7-879b-9e0f-29c2077791d5 (at 10.9.112.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88287201cc00, cur 1519382281 expire 1519382131 last 1519382054
Feb 23 02:38:01 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 09:33:42 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to 7a15e73a-a0ee-6bca-ab92-aa573877def2 (at 10.9.112.5@o2ib4)
Feb 23 09:33:42 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 7a15e73a-a0ee-6bca-ab92-aa573877def2 (at 10.9.112.5@o2ib4)
Feb 23 09:33:42 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to 7a15e73a-a0ee-6bca-ab92-aa573877def2 (at 10.9.112.5@o2ib4)
Feb 23 09:33:42 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 7a15e73a-a0ee-6bca-ab92-aa573877def2 (at 10.9.112.5@o2ib4)
Feb 23 09:33:42 oak-io2-s1 kernel: Lustre: Skipped 19 previous similar messages
Feb 23 09:33:42 oak-io2-s1 kernel: Lustre: Skipped 19 previous similar messages
Feb 23 09:33:42 oak-io2-s1 kernel: Lustre: Skipped 19 previous similar messages
Feb 23 09:33:42 oak-io2-s1 kernel: Lustre: Skipped 14 previous similar messages
Feb 23 09:33:52 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to a3309323-63b0-d61f-19a3-ab31f183f10e (at 10.9.104.54@o2ib4)
Feb 23 09:33:52 oak-io2-s1 kernel: Lustre: Skipped 324 previous similar messages
Feb 23 09:34:11 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 51ac6e63-c3f7-ca59-16ef-4eb91d23dbd7 (at 10.9.104.51@o2ib4)
Feb 23 09:34:11 oak-io2-s1 kernel: Lustre: Skipped 378 previous similar messages
Feb 23 09:34:50 oak-io2-s1 kernel: Lustre: oak-OST0037: Connection restored to 590cd3e3-fa19-0835-a53e-1ffc2c2d0012 (at 10.8.28.9@o2ib6)
Feb 23 09:34:50 oak-io2-s1 kernel: Lustre: oak-OST0031: Connection restored to 590cd3e3-fa19-0835-a53e-1ffc2c2d0012 (at 10.8.28.9@o2ib6)
Feb 23 09:34:50 oak-io2-s1 kernel: Lustre: Skipped 303 previous similar messages
Feb 23 09:34:50 oak-io2-s1 kernel: Lustre: Skipped 13 previous similar messages
Feb 23 09:38:03 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 6753e7eb-522c-7be3-4e53-e46f453e0ada (at 10.9.113.3@o2ib4)
Feb 23 09:38:03 oak-io2-s1 kernel: Lustre: Skipped 250 previous similar messages
Feb 23 10:03:36 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 6cfc17b8-b5a8-7bc9-3ba4-f9d50bb01c21 (at 10.9.112.17@o2ib4)
Feb 23 10:03:36 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to 6cfc17b8-b5a8-7bc9-3ba4-f9d50bb01c21 (at 10.9.112.17@o2ib4)
Feb 23 10:03:36 oak-io2-s1 kernel: Lustre: oak-OST0038: Connection restored to 6cfc17b8-b5a8-7bc9-3ba4-f9d50bb01c21 (at 10.9.112.17@o2ib4)
Feb 23 10:03:36 oak-io2-s1 kernel: Lustre: Skipped 49 previous similar messages
Feb 23 10:03:36 oak-io2-s1 kernel: Lustre: Skipped 49 previous similar messages
Feb 23 10:03:36 oak-io2-s1 kernel: Lustre: Skipped 12 previous similar messages
Feb 23 10:03:56 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 10:03:56 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 10:03:56 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 23 10:03:56 oak-io2-s1 kernel: Lustre: Skipped 13 previous similar messages
Feb 23 10:04:21 oak-io2-s1 kernel: Lustre: oak-OST0031: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 10:04:21 oak-io2-s1 kernel: Lustre: Skipped 50 previous similar messages
Feb 23 10:10:19 oak-io2-s1 kernel: Lustre: oak-OST0036: haven't heard from client 5441a0ea-9d87-2364-d972-e431616c1d62 (at 10.9.112.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814ec3aa800, cur 1519409419 expire 1519409269 last 1519409192
Feb 23 10:10:19 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 10:15:36 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 6cfc17b8-b5a8-7bc9-3ba4-f9d50bb01c21 (at 10.9.112.17@o2ib4)
Feb 23 10:15:36 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to 6cfc17b8-b5a8-7bc9-3ba4-f9d50bb01c21 (at 10.9.112.17@o2ib4)
Feb 23 10:15:36 oak-io2-s1 kernel: Lustre: Skipped 21 previous similar messages
Feb 23 10:15:36 oak-io2-s1 kernel: Lustre: Skipped 13 previous similar messages
Feb 23 10:15:57 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 10:15:57 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 10:15:57 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 10:15:57 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
Feb 23 10:15:57 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
Feb 23 10:15:57 oak-io2-s1 kernel: Lustre: Skipped 14 previous similar messages
Feb 23 10:16:22 oak-io2-s1 kernel: Lustre: oak-OST0039: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 10:16:22 oak-io2-s1 kernel: Lustre: oak-OST0031: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 10:16:22 oak-io2-s1 kernel: Lustre: oak-OST0035: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 10:16:22 oak-io2-s1 kernel: Lustre: Skipped 34 previous similar messages
Feb 23 10:16:22 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 10:16:22 oak-io2-s1 kernel: Lustre: Skipped 13 previous similar messages
Feb 23 10:22:15 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to 57635f11-dca4-393d-e47e-90cf0165d384 (at 10.9.112.14@o2ib4)
Feb 23 10:22:15 oak-io2-s1 kernel: Lustre: oak-OST003a: Connection restored to 57635f11-dca4-393d-e47e-90cf0165d384 (at 10.9.112.14@o2ib4)
Feb 23 10:22:15 oak-io2-s1 kernel: Lustre: Skipped 18 previous similar messages
Feb 23 10:22:15 oak-io2-s1 kernel: Lustre: Skipped 12 previous similar messages
Feb 23 10:22:18 oak-io2-s1 kernel: Lustre: oak-OST004f: haven't heard from client a7974e92-7914-f271-fc98-69afe5d66cad (at 10.9.112.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883cdbdcf800, cur 1519410138 expire 1519409988 last 1519409911
Feb 23 10:22:18 oak-io2-s1 kernel: Lustre: Skipped 107 previous similar messages
Feb 23 10:27:32 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 6cfc17b8-b5a8-7bc9-3ba4-f9d50bb01c21 (at 10.9.112.17@o2ib4)
Feb 23 10:27:32 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to 6cfc17b8-b5a8-7bc9-3ba4-f9d50bb01c21 (at 10.9.112.17@o2ib4)
Feb 23 10:27:32 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 6cfc17b8-b5a8-7bc9-3ba4-f9d50bb01c21 (at 10.9.112.17@o2ib4)
Feb 23 10:27:32 oak-io2-s1 kernel: Lustre: Skipped 14 previous similar messages
Feb 23 10:27:32 oak-io2-s1 kernel: Lustre: Skipped 14 previous similar messages
Feb 23 10:27:32 oak-io2-s1 kernel: Lustre: Skipped 13 previous similar messages
Feb 23 10:28:57 oak-io2-s1 kernel: Lustre: oak-OST0041: haven't heard from client 133a7287-8266-4fa9-7928-bdda96a5c75f (at 10.9.112.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883c89313000, cur 1519410537 expire 1519410387 last 1519410310
Feb 23 10:28:57 oak-io2-s1 kernel: Lustre: Skipped 107 previous similar messages
Feb 23 10:33:49 oak-io2-s1 kernel: Lustre: oak-OST0039: haven't heard from client b0216e63-f0c0-71c1-b5ce-a54eae556ef1 (at 10.9.112.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ff1953400, cur 1519410829 expire 1519410679 last 1519410602
Feb 23 10:33:49 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 10:39:55 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 10:39:55 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 10:39:55 oak-io2-s1 kernel: Lustre: Skipped 119 previous similar messages
Feb 23 10:39:55 oak-io2-s1 kernel: Lustre: Skipped 13 previous similar messages
Feb 23 10:40:53 oak-io2-s1 kernel: Lustre: oak-OST0039: haven't heard from client bcd470bd-5253-e0f7-85b7-4ad3308a65f2 (at 10.9.112.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883ca7843000, cur 1519411253 expire 1519411103 last 1519411026
Feb 23 10:40:53 oak-io2-s1 kernel: Lustre: Skipped 107 previous similar messages
Feb 23 10:46:37 oak-io2-s1 kernel: Lustre: oak-OST0038: haven't heard from client 9097c84d-615a-9440-6dc7-ae9e191b17c4 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88130f2e9000, cur 1519411597 expire 1519411447 last 1519411370
Feb 23 10:46:37 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 10:51:55 oak-io2-s1 kernel: Lustre: oak-OST003a: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 10:51:55 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 10:51:55 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 10:51:55 oak-io2-s1 kernel: Lustre: Skipped 89 previous similar messages
Feb 23 10:51:55 oak-io2-s1 kernel: Lustre: Skipped 90 previous similar messages
Feb 23 10:51:55 oak-io2-s1 kernel: Lustre: Skipped 11 previous similar messages
Feb 23 10:52:14 oak-io2-s1 kernel: Lustre: oak-OST0033: haven't heard from client 8698f9f5-c30e-2993-4c7b-90071d4953d1 (at 10.9.112.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883d1997dc00, cur 1519411934 expire 1519411784 last 1519411707
Feb 23 10:52:14 oak-io2-s1 kernel: Lustre: Skipped 71 previous similar messages
Feb 23 10:58:37 oak-io2-s1 kernel: Lustre: oak-OST003b: haven't heard from client f38dd9b5-0d59-2122-09f9-20c634354cef (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b78a18000, cur 1519412317 expire 1519412167 last 1519412090
Feb 23 10:58:37 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 11:03:51 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 11:03:51 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 11:03:51 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 11:03:51 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 11:03:51 oak-io2-s1 kernel: Lustre: Skipped 17 previous similar messages
Feb 23 11:03:51 oak-io2-s1 kernel: Lustre: Skipped 17 previous similar messages
Feb 23 11:03:51 oak-io2-s1 kernel: Lustre: Skipped 17 previous similar messages
Feb 23 11:03:51 oak-io2-s1 kernel: Lustre: Skipped 14 previous similar messages
Feb 23 11:10:34 oak-io2-s1 kernel: Lustre: oak-OST0031: haven't heard from client 78bd8f8c-2820-9c7b-52b2-caf4b2ebee7d (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805c027a400, cur 1519413034 expire 1519412884 last 1519412807
Feb 23 11:10:34 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 11:13:31 oak-io2-s1 kernel: Lustre: DEBUG MARKER: Fri Feb 23 11:13:31 2018
Feb 23 11:15:54 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 11:15:54 oak-io2-s1 kernel: Lustre: Skipped 62 previous similar messages
Feb 23 11:16:08 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0045: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200008c53:0xabf5:0x0] object 0x0:3018242 extent [46137344-50331647], client returned csum b2e206b3 (type 4), server csum 8a0c8ff7 (type 4)
Feb 23 11:16:08 oak-io2-s1 kernel: LustreError: Skipped 14 previous similar messages
Feb 23 11:16:37 oak-io2-s1 kernel: Lustre: oak-OST003d: haven't heard from client aa81197f-aba0-64b1-d844-c09875fc29be (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883cac598400, cur 1519413397 expire 1519413247 last 1519413170
Feb 23 11:16:37 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 11:16:43 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0045: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200008c53:0xabf5:0x0] object 0x0:3018242 extent [46137344-50331647], client returned csum b2e206b3 (type 4), server csum 8a0c8ff7 (type 4)
Feb 23 11:16:43 oak-io2-s1 kernel: LustreError: Skipped 6 previous similar messages
Feb 23 11:17:48 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0045: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200008c53:0xabf5:0x0] object 0x0:3018242 extent [46137344-50331647], client returned csum b2e206b3 (type 4), server csum 8a0c8ff7 (type 4)
Feb 23 11:17:48 oak-io2-s1 kernel: LustreError: Skipped 10 previous similar messages
Feb 23 11:22:36 oak-io2-s1 kernel: Lustre: oak-OST0030: haven't heard from client 3ee6ad51-0711-75b0-7bc9-b392a6b44a05 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881cffa60400, cur 1519413756 expire 1519413606 last 1519413529
Feb 23 11:22:36 oak-io2-s1 kernel: Lustre: Skipped 71 previous similar messages
Feb 23 11:27:49 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 11:27:49 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 11:27:49 oak-io2-s1 kernel: Lustre: Skipped 122 previous similar messages
Feb 23 11:27:49 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 23 11:34:31 oak-io2-s1 kernel: Lustre: oak-OST0034: haven't heard from client 42f302db-a226-61a7-d35e-4446d1e1ac21 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883cbe5d5800, cur 1519414471 expire 1519414321 last 1519414244
Feb 23 11:34:31 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 11:39:50 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 11:39:50 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 11:39:50 oak-io2-s1 kernel: Lustre: Skipped 52 previous similar messages
Feb 23 11:39:50 oak-io2-s1 kernel: Lustre: Skipped 15 previous similar messages
Feb 23 11:40:34 oak-io2-s1 kernel: Lustre: oak-OST0030: haven't heard from client b573c12b-5a94-ff00-20c6-26d5fff91b01 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883c91afe800, cur 1519414834 expire 1519414684 last 1519414607
Feb 23 11:40:34 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 11:46:03 oak-io2-s1 kernel: Lustre: oak-OST0036: haven't heard from client 39e0cc97-3498-7d4d-9f0f-2cb2418b7344 (at 10.9.101.60@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88349c5d7800, cur 1519415163 expire 1519415013 last 1519414936
Feb 23 11:46:03 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 11:58:14 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 0fb09ad7-848f-0329-d2d6-a5d10baed2d0 (at 10.9.113.8@o2ib4)
Feb 23 11:58:14 oak-io2-s1 kernel: Lustre: Skipped 195 previous similar messages
Feb 23 12:02:01 oak-io2-s1 kernel: Lustre: oak-OST004f: haven't heard from client 7caaf3bb-2673-7c3e-80a2-f6a25b84558e (at 10.8.9.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880751a6b000, cur 1519416121 expire 1519415971 last 1519415894
Feb 23 12:02:01 oak-io2-s1 kernel: Lustre: Skipped 71 previous similar messages
Feb 23 12:21:07 oak-io2-s1 kernel: Lustre: oak-OST003c: haven't heard from client ae8efa8b-fbec-cb2a-23a5-9c09f69719cd (at 10.12.4.68@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881cfe4c9400, cur 1519417267 expire 1519417117 last 1519417040
Feb 23 12:21:07 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 12:21:33 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to f4b38ec1-850e-7e45-feb8-6ea6e1013f81 (at 10.12.4.68@o2ib)
Feb 23 12:21:33 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to f4b38ec1-850e-7e45-feb8-6ea6e1013f81 (at 10.12.4.68@o2ib)
Feb 23 12:21:33 oak-io2-s1 kernel: Lustre: Skipped 150 previous similar messages
Feb 23 12:21:33 oak-io2-s1 kernel: Lustre: Skipped 13 previous similar messages
Feb 23 12:22:48 oak-io2-s1 kernel: Lustre: oak-OST0031: Connection restored to f4b38ec1-850e-7e45-feb8-6ea6e1013f81 (at 10.12.4.68@o2ib)
Feb 23 12:22:48 oak-io2-s1 kernel: Lustre: Skipped 17 previous similar messages
Feb 23 12:23:18 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0051: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8dd8:0x0] object 0x0:162805 extent [1539309568-1543503871], client returned csum 7d3a3f7d (type 4), server csum 111ca6a0 (type 4)
Feb 23 12:23:18 oak-io2-s1 kernel: LustreError: Skipped 1 previous similar message
Feb 23 12:23:38 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0051: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8dd8:0x0] object 0x0:162805 extent [1539309568-1543503871], client returned csum 7d3a3f7d (type 4), server csum 111ca6a0 (type 4)
Feb 23 12:23:38 oak-io2-s1 kernel: LustreError: Skipped 4 previous similar messages
Feb 23 12:26:08 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0051: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8dd8:0x0] object 0x0:162805 extent [1539309568-1543503871], client returned csum 7d3a3f7d (type 4), server csum 111ca6a0 (type 4)
Feb 23 12:26:08 oak-io2-s1 kernel: LustreError: Skipped 4 previous similar messages
Feb 23 12:27:15 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0051: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8dd8:0x0] object 0x0:162805 extent [1539309568-1543503871], client returned csum 7d3a3f7d (type 4), server csum 111ca6a0 (type 4)
Feb 23 12:27:15 oak-io2-s1 kernel: LustreError: Skipped 12 previous similar messages
Feb 23 12:29:05 oak-io2-s1 kernel: Lustre: oak-OST003b: haven't heard from client f39f0e19-5eaa-43e3-c3ea-2e223296d1a4 (at 10.12.4.68@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881a8522b800, cur 1519417745 expire 1519417595 last 1519417518
Feb 23 12:29:05 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 12:29:39 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to f4b38ec1-850e-7e45-feb8-6ea6e1013f81 (at 10.12.4.68@o2ib)
Feb 23 12:29:39 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to f4b38ec1-850e-7e45-feb8-6ea6e1013f81 (at 10.12.4.68@o2ib)
Feb 23 12:29:39 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 23 12:30:54 oak-io2-s1 kernel: Lustre: oak-OST0033: Connection restored to f4b38ec1-850e-7e45-feb8-6ea6e1013f81 (at 10.12.4.68@o2ib)
Feb 23 12:30:54 oak-io2-s1 kernel: Lustre: oak-OST0031: Connection restored to f4b38ec1-850e-7e45-feb8-6ea6e1013f81 (at 10.12.4.68@o2ib)
Feb 23 12:30:54 oak-io2-s1 kernel: Lustre: Skipped 15 previous similar messages
Feb 23 12:35:54 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to f4b38ec1-850e-7e45-feb8-6ea6e1013f81 (at 10.12.4.68@o2ib)
Feb 23 12:35:54 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to f4b38ec1-850e-7e45-feb8-6ea6e1013f81 (at 10.12.4.68@o2ib)
Feb 23 12:35:54 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
Feb 23 12:35:54 oak-io2-s1 kernel: Lustre: Skipped 15 previous similar messages
Feb 23 12:35:56 oak-io2-s1 kernel: Lustre: oak-OST0036: haven't heard from client 3aa01694-09dd-7d04-e8f1-1e3426332284 (at 10.12.4.68@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b3949e400, cur 1519418156 expire 1519418006 last 1519417929
Feb 23 12:35:56 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 13:17:02 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0031: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8deb:0x0] object 0x0:4391627 extent [3124756480-3128950783], client returned csum e6cf8ef (type 4), server csum 63d56461 (type 4)
Feb 23 13:17:02 oak-io2-s1 kernel: LustreError: Skipped 6 previous similar messages
Feb 23 13:17:22 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0031: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8deb:0x0] object 0x0:4391627 extent [3124756480-3128950783], client returned csum e6cf8ef (type 4), server csum 63d56461 (type 4)
Feb 23 13:17:22 oak-io2-s1 kernel: LustreError: Skipped 4 previous similar messages
Feb 23 13:17:56 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0031: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8deb:0x0] object 0x0:4391627 extent [3124756480-3128950783], client returned csum e6cf8ef (type 4), server csum 63d56461 (type 4)
Feb 23 13:17:56 oak-io2-s1 kernel: LustreError: Skipped 3 previous similar messages
Feb 23 13:20:16 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0031: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8deb:0x0] object 0x0:4391627 extent [3124756480-3128950783], client returned csum e6cf8ef (type 4), server csum 63d56461 (type 4)
Feb 23 13:22:31 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0031: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8deb:0x0] object 0x0:4391627 extent [3124756480-3128950783], client returned csum e6cf8ef (type 4), server csum 63d56461 (type 4)
Feb 23 13:22:31 oak-io2-s1 kernel: LustreError: Skipped 17 previous similar messages
Feb 23 13:36:23 oak-io2-s1 kernel: Lustre: oak-OST0050: haven't heard from client a3d3e82a-aa6f-03a1-1e54-d2436b9b790a (at 10.210.45.46@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883ca6ea2c00, cur 1519421783 expire 1519421633 last 1519421556
Feb 23 13:36:23 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 14:19:56 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 06585daa-72d5-9293-64a7-1d0994d10593 (at 10.8.27.24@o2ib6)
Feb 23 14:19:56 oak-io2-s1 kernel: Lustre: Skipped 34 previous similar messages
Feb 23 14:34:12 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0040: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8dda:0x0] object 0x0:4434610 extent [4676648960-4680843263], client returned csum 657ce454 (type 4), server csum 87204dc7 (type 4)
Feb 23 14:34:12 oak-io2-s1 kernel: LustreError: Skipped 1 previous similar message
Feb 23 14:34:47 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0040: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8dda:0x0] object 0x0:4434610 extent [4676648960-4680843263], client returned csum 657ce454 (type 4), server csum 87204dc7 (type 4)
Feb 23 14:34:47 oak-io2-s1 kernel: LustreError: Skipped 6 previous similar messages
Feb 23 14:39:16 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0040: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8dda:0x0] object 0x0:4434610 extent [4676648960-4680843263], client returned csum 657ce454 (type 4), server csum 87204dc7 (type 4)
Feb 23 14:39:16 oak-io2-s1 kernel: LustreError: Skipped 2 previous similar messages
Feb 23 14:41:29 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0040: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8dda:0x0] object 0x0:4434610 extent [4676648960-4680843263], client returned csum 657ce454 (type 4), server csum 87204dc7 (type 4)
Feb 23 14:41:29 oak-io2-s1 kernel: LustreError: Skipped 18 previous similar messages
Feb 23 14:45:52 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0040: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8dda:0x0] object 0x0:4434610 extent [4678746112-4680843263], client returned csum dca056d7 (type 4), server csum 3efcff44 (type 4)
Feb 23 14:45:52 oak-io2-s1 kernel: LustreError: Skipped 46 previous similar messages
Feb 23 15:25:44 oak-io2-s1 kernel: Lustre: oak-OST0053: haven't heard from client a6c83b55-0c57-d887-4d6c-5b5686fbfa7e (at 10.210.44.17@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c4d185000, cur 1519428344 expire 1519428194 last 1519428117
Feb 23 15:25:44 oak-io2-s1 kernel: Lustre: Skipped 71 previous similar messages
Feb 23 15:29:23 oak-io2-s1 kernel: Lustre: oak-OST0040: haven't heard from client f98872db-7abe-2a9a-2f3c-36218aaa6cf2 (at 10.210.44.18@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883cd7b24800, cur 1519428563 expire 1519428413 last 1519428336
Feb 23 15:29:23 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 15:39:27 oak-io2-s1 kernel: Lustre: oak-OST0034: haven't heard from client 551963be-92d7-db3e-b398-33a21bf055e3 (at 10.210.44.37@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883cd9dddc00, cur 1519429167 expire 1519429017 last 1519428940
Feb 23 15:39:27 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 15:40:43 oak-io2-s1 kernel: Lustre: oak-OST0049: haven't heard from client 99f21505-d405-7b58-c0ca-66eb991e0c05 (at 10.210.44.39@o2ib3) in 198 seconds. I think it's dead, and I am evicting it. exp ffff8818aaeea400, cur 1519429243 expire 1519429093 last 1519429045
Feb 23 15:40:43 oak-io2-s1 kernel: Lustre: Skipped 71 previous similar messages
Feb 23 15:41:59 oak-io2-s1 kernel: Lustre: oak-OST003c: haven't heard from client ed6a567d-bffe-57f8-2594-d2ad03a7bd32 (at 10.210.45.12@o2ib3) in 220 seconds. I think it's dead, and I am evicting it. exp ffff883ca7f3e400, cur 1519429319 expire 1519429169 last 1519429099
Feb 23 15:41:59 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 16:03:55 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0051: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8dd8:0x0] object 0x0:162805 extent [5897191424-5899550719], client returned csum 882d5a82 (type 4), server csum 4ad47bbc (type 4)
Feb 23 16:03:55 oak-io2-s1 kernel: LustreError: Skipped 53 previous similar messages
Feb 23 16:05:04 oak-io2-s1 kernel: LustreError: 132-0: oak-OST0051: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.0.2.230@o2ib5 inode [0x200010201:0x8dd8:0x0] object 0x0:162805 extent [5897191424-5901385727], client returned csum dbc0eb5f (type 4), server csum 376d86ab (type 4)
Feb 23 16:05:04 oak-io2-s1 kernel: LustreError: Skipped 14 previous similar messages
Feb 23 16:07:55 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 77bd6f8f-dfb2-10d9-8d00-57a663f314f1 (at 10.210.45.13@o2ib3)
Feb 23 16:07:55 oak-io2-s1 kernel: Lustre: Skipped 67 previous similar messages
Feb 23 16:11:49 oak-io2-s1 kernel: Lustre: oak-OST003a: haven't heard from client 77bd6f8f-dfb2-10d9-8d00-57a663f314f1 (at 10.210.45.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883d272be800, cur 1519431109 expire 1519430959 last 1519430882
Feb 23 16:11:49 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 16:34:46 oak-io2-s1 kernel: Lustre: 141924:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519432479/real 1519432479]  req@ffff88005c226f00 x1592481984075968/t0(0) o106->oak-OST0039@10.8.2.20@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1519432486 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Feb 23 16:34:46 oak-io2-s1 kernel: Lustre: 17272:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519432479/real 1519432479]  req@ffff880053ce7800 x1592481984075984/t0(0) o106->oak-OST0046@10.8.2.20@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1519432486 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Feb 23 16:34:46 oak-io2-s1 kernel: Lustre: 17272:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 36 previous similar messages
Feb 23 16:58:43 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to f30803fa-607a-2a67-3576-43087423cbf2 (at 10.8.8.34@o2ib6)
Feb 23 16:58:43 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to f30803fa-607a-2a67-3576-43087423cbf2 (at 10.8.8.34@o2ib6)
Feb 23 16:58:43 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 23 16:58:43 oak-io2-s1 kernel: Lustre: Skipped 15 previous similar messages
Feb 23 16:59:08 oak-io2-s1 kernel: Lustre: oak-OST0031: Connection restored to f30803fa-607a-2a67-3576-43087423cbf2 (at 10.8.8.34@o2ib6)
Feb 23 16:59:08 oak-io2-s1 kernel: Lustre: Skipped 100 previous similar messages
Feb 23 17:04:01 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 17:04:01 oak-io2-s1 kernel: Lustre: Skipped 102 previous similar messages
Feb 23 17:04:26 oak-io2-s1 kernel: Lustre: oak-OST004b: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 17:04:26 oak-io2-s1 kernel: Lustre: oak-OST0053: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 17:04:26 oak-io2-s1 kernel: Lustre: oak-OST0043: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 17:04:26 oak-io2-s1 kernel: Lustre: oak-OST0047: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 17:04:26 oak-io2-s1 kernel: Lustre: Skipped 8 previous similar messages
Feb 23 17:04:26 oak-io2-s1 kernel: Lustre: Skipped 8 previous similar messages
Feb 23 17:04:26 oak-io2-s1 kernel: Lustre: Skipped 8 previous similar messages
Feb 23 17:04:26 oak-io2-s1 kernel: Lustre: Skipped 5 previous similar messages
Feb 23 17:10:11 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 17:10:11 oak-io2-s1 kernel: Lustre: Skipped 13 previous similar messages
Feb 23 17:10:43 oak-io2-s1 kernel: Lustre: oak-OST0030: haven't heard from client 121dcbd3-8e5d-9112-5d3b-a9c0f4eb8bb9 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882eebc35c00, cur 1519434643 expire 1519434493 last 1519434416
Feb 23 17:10:43 oak-io2-s1 kernel: Lustre: Skipped 17 previous similar messages
Feb 23 17:10:46 oak-io2-s1 kernel: Lustre: oak-OST0043: haven't heard from client 121dcbd3-8e5d-9112-5d3b-a9c0f4eb8bb9 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88083ad28000, cur 1519434646 expire 1519434496 last 1519434419
Feb 23 17:10:46 oak-io2-s1 kernel: Lustre: Skipped 32 previous similar messages
Feb 23 17:10:52 oak-io2-s1 kernel: Lustre: oak-OST0039: haven't heard from client 121dcbd3-8e5d-9112-5d3b-a9c0f4eb8bb9 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808bd4e9400, cur 1519434652 expire 1519434502 last 1519434425
Feb 23 17:16:52 oak-io2-s1 kernel: Lustre: oak-OST003d: haven't heard from client 6bafd216-b326-77e1-e921-89a03111fc84 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8823dd911400, cur 1519435012 expire 1519434862 last 1519434785
Feb 23 17:16:52 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
Feb 23 17:19:02 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to e79fe59f-4a6e-3dd9-6e7e-c0551da1735f (at 10.210.46.123@o2ib3)
Feb 23 17:19:02 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to e79fe59f-4a6e-3dd9-6e7e-c0551da1735f (at 10.210.46.123@o2ib3)
Feb 23 17:19:02 oak-io2-s1 kernel: Lustre: Skipped 22 previous similar messages
Feb 23 17:19:02 oak-io2-s1 kernel: Lustre: Skipped 13 previous similar messages
Feb 23 17:28:56 oak-io2-s1 kernel: Lustre: oak-OST0040: haven't heard from client 7eb29d38-a42b-5ed7-c22f-5aa05669584c (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88067fc92800, cur 1519435736 expire 1519435586 last 1519435509
Feb 23 17:28:56 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 17:34:18 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 17:34:18 oak-io2-s1 kernel: Lustre: oak-OST0038: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 17:34:18 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 17:34:18 oak-io2-s1 kernel: Lustre: Skipped 53 previous similar messages
Feb 23 17:34:18 oak-io2-s1 kernel: Lustre: Skipped 53 previous similar messages
Feb 23 17:34:18 oak-io2-s1 kernel: Lustre: Skipped 12 previous similar messages
Feb 23 17:37:34 oak-io2-s1 kernel: Lustre: 249468:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519436247/real 1519436247]  req@ffff8809e272ec00 x1592481985110224/t0(0) o106->oak-OST0049@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519436254 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Feb 23 17:37:34 oak-io2-s1 kernel: Lustre: 249474:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519436247/real 1519436247]  req@ffff880028afbf00 x1592481985110208/t0(0) o106->oak-OST0042@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519436254 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Feb 23 17:37:41 oak-io2-s1 kernel: Lustre: 249474:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519436254/real 1519436254]  req@ffff880028afbf00 x1592481985110208/t0(0) o106->oak-OST0042@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519436261 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 17:37:48 oak-io2-s1 kernel: Lustre: 249474:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519436261/real 1519436261]  req@ffff880028afbf00 x1592481985110208/t0(0) o106->oak-OST0042@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519436268 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 17:37:48 oak-io2-s1 kernel: Lustre: 249474:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Feb 23 17:37:55 oak-io2-s1 kernel: Lustre: 249468:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519436268/real 1519436268]  req@ffff8809e272ec00 x1592481985110224/t0(0) o106->oak-OST0049@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519436275 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 17:37:55 oak-io2-s1 kernel: Lustre: 249468:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Feb 23 17:38:09 oak-io2-s1 kernel: Lustre: 249474:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519436282/real 1519436282]  req@ffff880028afbf00 x1592481985110208/t0(0) o106->oak-OST0042@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519436289 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 17:38:09 oak-io2-s1 kernel: Lustre: 249474:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Feb 23 17:38:30 oak-io2-s1 kernel: Lustre: 249468:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519436303/real 1519436303]  req@ffff8809e272ec00 x1592481985110224/t0(0) o106->oak-OST0049@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519436310 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 17:38:30 oak-io2-s1 kernel: Lustre: 249474:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519436303/real 1519436303]  req@ffff880028afbf00 x1592481985110208/t0(0) o106->oak-OST0042@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519436310 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 17:38:30 oak-io2-s1 kernel: Lustre: 249474:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
Feb 23 17:39:12 oak-io2-s1 kernel: Lustre: 249474:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519436345/real 1519436345]  req@ffff880028afbf00 x1592481985110208/t0(0) o106->oak-OST0042@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519436352 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 17:39:12 oak-io2-s1 kernel: Lustre: 249474:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
Feb 23 17:40:21 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 17:40:21 oak-io2-s1 kernel: Lustre: Skipped 30 previous similar messages
Feb 23 17:40:22 oak-io2-s1 kernel: LustreError: 249474:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.112.15@o2ib4) returned error from glimpse AST (req@ffff880028afbf00 x1592481985110208 status -107 rc -107), evict it ns: filter-oak-OST0042_UUID lock: ffff8800669ca000/0x806f959362256fd7 lrc: 4/0,0 mode: PW/PW res: [0x421917:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x40000000000000 nid: 10.9.112.15@o2ib4 remote: 0x574cd27843e52b31 expref: 5 pid: 249335 timeout: 0 lvb_type: 0
Feb 23 17:40:22 oak-io2-s1 kernel: LustreError: 138-a: oak-OST0049: A client on nid 10.9.112.15@o2ib4 was evicted due to a lock glimpse callback time out: rc -107
Feb 23 17:40:22 oak-io2-s1 kernel: LustreError: 249474:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) Skipped 1 previous similar message
Feb 23 17:41:00 oak-io2-s1 kernel: Lustre: oak-OST0033: haven't heard from client 383da4a6-81fa-285e-647b-4cc6d0a5aef5 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cc63cd000, cur 1519436460 expire 1519436310 last 1519436233
Feb 23 17:41:00 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 17:47:03 oak-io2-s1 kernel: Lustre: oak-OST0032: haven't heard from client f243424f-9c95-754e-10c7-7902bc37011f (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88159cd72800, cur 1519436823 expire 1519436673 last 1519436596
Feb 23 17:47:03 oak-io2-s1 kernel: Lustre: Skipped 31 previous similar messages
Feb 23 17:58:25 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 17:58:25 oak-io2-s1 kernel: Lustre: Skipped 32 previous similar messages
Feb 23 17:58:51 oak-io2-s1 kernel: Lustre: oak-OST0033: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 17:58:51 oak-io2-s1 kernel: Lustre: Skipped 9 previous similar messages
Feb 23 18:04:27 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 18:04:27 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 18:04:27 oak-io2-s1 kernel: Lustre: Skipped 14 previous similar messages
Feb 23 18:05:05 oak-io2-s1 kernel: Lustre: oak-OST003c: haven't heard from client b82d75f2-2bf9-91ed-5de2-71f8d922db6c (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883d3a970400, cur 1519437905 expire 1519437755 last 1519437678
Feb 23 18:05:05 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 18:05:08 oak-io2-s1 kernel: Lustre: oak-OST003d: haven't heard from client b82d75f2-2bf9-91ed-5de2-71f8d922db6c (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b35160000, cur 1519437908 expire 1519437758 last 1519437681
Feb 23 18:05:08 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
Feb 23 18:05:14 oak-io2-s1 kernel: Lustre: oak-OST0030: haven't heard from client b82d75f2-2bf9-91ed-5de2-71f8d922db6c (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883d2e372c00, cur 1519437914 expire 1519437764 last 1519437687
Feb 23 18:05:14 oak-io2-s1 kernel: Lustre: Skipped 30 previous similar messages
Feb 23 18:05:55 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to b90740e3-d61c-ff37-5776-dec50cda45bb (at 10.9.102.42@o2ib4)
Feb 23 18:05:55 oak-io2-s1 kernel: Lustre: Skipped 23 previous similar messages
Feb 23 18:08:12 oak-io2-s1 kernel: Lustre: oak-OST003b: haven't heard from client e95d51b7-d047-0aab-47c7-7b9fa69b0296 (at 10.9.113.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882df0493c00, cur 1519438092 expire 1519437942 last 1519437865
Feb 23 18:08:12 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 23 18:09:23 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to 06585daa-72d5-9293-64a7-1d0994d10593 (at 10.8.27.24@o2ib6)
Feb 23 18:09:23 oak-io2-s1 kernel: Lustre: Skipped 325 previous similar messages
Feb 23 18:11:09 oak-io2-s1 kernel: Lustre: oak-OST004e: haven't heard from client 569de15f-8859-521c-aecf-e9e2ad56118e (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883ccbe03800, cur 1519438269 expire 1519438119 last 1519438042
Feb 23 18:11:09 oak-io2-s1 kernel: Lustre: Skipped 107 previous similar messages
Feb 23 18:16:28 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 18:16:28 oak-io2-s1 kernel: Lustre: Skipped 67 previous similar messages
Feb 23 18:19:38 oak-io2-s1 kernel: Lustre: 390779:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519438771/real 1519438771]  req@ffff880b91e54b00 x1592481986161856/t0(0) o106->oak-OST0033@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519438778 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Feb 23 18:19:38 oak-io2-s1 kernel: Lustre: 390779:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
Feb 23 18:19:59 oak-io2-s1 kernel: Lustre: 390779:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519438792/real 1519438792]  req@ffff880b91e54b00 x1592481986161856/t0(0) o106->oak-OST0033@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519438799 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 18:19:59 oak-io2-s1 kernel: Lustre: 390779:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
Feb 23 18:20:37 oak-io2-s1 kernel: Lustre: 390793:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519438826/real 1519438826]  req@ffff88005721e300 x1592481986161872/t0(0) o106->oak-OST003a@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519438837 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 18:20:37 oak-io2-s1 kernel: Lustre: 390793:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Feb 23 18:21:54 oak-io2-s1 kernel: Lustre: 390793:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519438903/real 1519438903]  req@ffff88005721e300 x1592481986161872/t0(0) o106->oak-OST003a@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519438914 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 18:21:54 oak-io2-s1 kernel: Lustre: 390793:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
Feb 23 18:22:26 oak-io2-s1 kernel: LustreError: 390779:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.112.15@o2ib4) returned error from glimpse AST (req@ffff880b91e54b00 x1592481986161856 status -107 rc -107), evict it ns: filter-oak-OST0033_UUID lock: ffff88004973f000/0x806f9593622e3c4f lrc: 4/0,0 mode: PW/PW res: [0x437cf4:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x40000000000000 nid: 10.9.112.15@o2ib4 remote: 0x3895f6c5428b1997 expref: 5 pid: 362684 timeout: 0 lvb_type: 0
Feb 23 18:22:26 oak-io2-s1 kernel: LustreError: 390779:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) Skipped 2 previous similar messages
Feb 23 18:22:26 oak-io2-s1 kernel: LustreError: 138-a: oak-OST0033: A client on nid 10.9.112.15@o2ib4 was evicted due to a lock glimpse callback time out: rc -107
Feb 23 18:22:26 oak-io2-s1 kernel: LustreError: Skipped 3 previous similar messages
Feb 23 18:22:27 oak-io2-s1 kernel: LustreError: 390793:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.112.15@o2ib4) returned error from glimpse AST (req@ffff88005721e300 x1592481986161872 status -107 rc -107), evict it ns: filter-oak-OST003a_UUID lock: ffff8808af16c800/0x806f9593622e452b lrc: 4/0,0 mode: PW/PW res: [0x434ae0:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x40000000000000 nid: 10.9.112.15@o2ib4 remote: 0x3895f6c5428b19eb expref: 5 pid: 132963 timeout: 0 lvb_type: 0
Feb 23 18:22:27 oak-io2-s1 kernel: LustreError: 138-a: oak-OST003a: A client on nid 10.9.112.15@o2ib4 was evicted due to a lock glimpse callback time out: rc -107
Feb 23 18:23:04 oak-io2-s1 kernel: Lustre: oak-OST0033: haven't heard from client 37f505d3-7f6e-32cf-6b6b-ca723026d0a8 (at 10.8.27.24@o2ib6) in 196 seconds. I think it's dead, and I am evicting it. exp ffff881ff876e800, cur 1519438984 expire 1519438834 last 1519438788
Feb 23 18:23:04 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 18:23:09 oak-io2-s1 kernel: Lustre: oak-OST0050: haven't heard from client 718597f3-70cf-fd4e-1061-fb6ca2fd6515 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d03b35000, cur 1519438989 expire 1519438839 last 1519438762
Feb 23 18:23:09 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
Feb 23 18:23:13 oak-io2-s1 kernel: Lustre: oak-OST0040: haven't heard from client 37f505d3-7f6e-32cf-6b6b-ca723026d0a8 (at 10.8.27.24@o2ib6) in 205 seconds. I think it's dead, and I am evicting it. exp ffff881ecaa8f000, cur 1519438993 expire 1519438843 last 1519438788
Feb 23 18:23:13 oak-io2-s1 kernel: Lustre: Skipped 92 previous similar messages
Feb 23 18:28:30 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 18:28:30 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 18:28:30 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 18:28:30 oak-io2-s1 kernel: Lustre: Skipped 86 previous similar messages
Feb 23 18:28:30 oak-io2-s1 kernel: Lustre: Skipped 86 previous similar messages
Feb 23 18:28:30 oak-io2-s1 kernel: Lustre: Skipped 15 previous similar messages
Feb 23 18:29:08 oak-io2-s1 kernel: Lustre: oak-OST0041: haven't heard from client a479ea1b-a2f9-9649-f834-362d1d93f7e9 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880248a9c400, cur 1519439348 expire 1519439198 last 1519439121
Feb 23 18:29:08 oak-io2-s1 kernel: Lustre: Skipped 5 previous similar messages
Feb 23 18:35:08 oak-io2-s1 kernel: Lustre: oak-OST0047: haven't heard from client 5c3b5bec-b026-9492-faa5-22fc5fad13fb (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e096ef400, cur 1519439708 expire 1519439558 last 1519439481
Feb 23 18:35:08 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 18:40:30 oak-io2-s1 kernel: Lustre: oak-OST003c: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 18:40:30 oak-io2-s1 kernel: Lustre: oak-OST0038: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 18:40:30 oak-io2-s1 kernel: Lustre: oak-OST003e: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 18:40:30 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 18:40:30 oak-io2-s1 kernel: Lustre: Skipped 50 previous similar messages
Feb 23 18:40:30 oak-io2-s1 kernel: Lustre: Skipped 50 previous similar messages
Feb 23 18:40:30 oak-io2-s1 kernel: Lustre: Skipped 50 previous similar messages
Feb 23 18:40:30 oak-io2-s1 kernel: Lustre: Skipped 8 previous similar messages
Feb 23 18:41:13 oak-io2-s1 kernel: Lustre: oak-OST003f: haven't heard from client 4564b1e6-cbab-5450-0ad6-4f7e8b138df2 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a029afc00, cur 1519440073 expire 1519439923 last 1519439846
Feb 23 18:41:13 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 18:47:12 oak-io2-s1 kernel: Lustre: oak-OST004b: haven't heard from client 67357652-0bb5-c580-e574-58a776fb9161 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8827d8aa0400, cur 1519440432 expire 1519440282 last 1519440205
Feb 23 18:47:12 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 18:52:37 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 18:52:37 oak-io2-s1 kernel: Lustre: Skipped 98 previous similar messages
Feb 23 18:59:16 oak-io2-s1 kernel: Lustre: oak-OST0051: haven't heard from client 1d8dfad6-be37-8a9e-fe0f-43045062e957 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88141e797400, cur 1519441156 expire 1519441006 last 1519440929
Feb 23 18:59:16 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 19:01:50 oak-io2-s1 kernel: Lustre: 17265:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519441303/real 1519441303]  req@ffff8819da8d0900 x1592481987629072/t0(0) o106->oak-OST0052@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519441310 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Feb 23 19:01:50 oak-io2-s1 kernel: Lustre: 17265:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Feb 23 19:02:11 oak-io2-s1 kernel: Lustre: 17265:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519441324/real 1519441324]  req@ffff8819da8d0900 x1592481987629072/t0(0) o106->oak-OST0052@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519441331 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 19:02:11 oak-io2-s1 kernel: Lustre: 17265:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Feb 23 19:02:53 oak-io2-s1 kernel: Lustre: 17265:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519441366/real 1519441366]  req@ffff8819da8d0900 x1592481987629072/t0(0) o106->oak-OST0052@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519441373 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 19:02:53 oak-io2-s1 kernel: Lustre: 17265:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Feb 23 19:04:10 oak-io2-s1 kernel: Lustre: 17265:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519441443/real 1519441443]  req@ffff8819da8d0900 x1592481987629072/t0(0) o106->oak-OST0052@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519441450 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 19:04:10 oak-io2-s1 kernel: Lustre: 17265:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
Feb 23 19:05:04 oak-io2-s1 kernel: LNet: Service thread pid 17265 was inactive for 200.61s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 23 19:05:04 oak-io2-s1 kernel: Pid: 17265, comm: ll_ost00_032
Feb 23 19:05:04 oak-io2-s1 kernel: #012Call Trace:
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffff816a6f34>] schedule_timeout+0x174/0x2c0
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffff81098b20>] ? process_timeout+0x0/0x10
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffffc0c46e40>] ptlrpc_set_wait+0x4c0/0x910 [ptlrpc]
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffffc0c04df3>] ldlm_run_ast_work+0xd3/0x3a0 [ptlrpc]
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffffc0c258bb>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc]
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffffc12316c4>] ofd_intent_policy+0x444/0xa30 [ofd]
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffffc0c04277>] ldlm_lock_enqueue+0x387/0x970 [ptlrpc]
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffffc0c2d903>] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc]
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffffc0c55ae0>] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc]
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffffc0cb2ea2>] tgt_enqueue+0x62/0x210 [ptlrpc]
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 19:05:04 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 19:05:04 oak-io2-s1 kernel: 
Feb 23 19:05:04 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519441504.17265
Feb 23 19:05:06 oak-io2-s1 kernel: Lustre: oak-OST0033: haven't heard from client 4b6463cc-fa45-ef77-7f7b-f455bd7e7e80 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d2cb22000, cur 1519441506 expire 1519441356 last 1519441279
Feb 23 19:05:06 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 19:05:17 oak-io2-s1 kernel: LNet: Service thread pid 17265 completed after 213.45s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Feb 23 19:05:23 oak-io2-s1 kernel: Lustre: oak-OST0051: haven't heard from client 4b6463cc-fa45-ef77-7f7b-f455bd7e7e80 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c062bcc00, cur 1519441523 expire 1519441373 last 1519441296
Feb 23 19:05:23 oak-io2-s1 kernel: Lustre: Skipped 34 previous similar messages
Feb 23 19:10:41 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 19:10:41 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 19:10:41 oak-io2-s1 kernel: Lustre: Skipped 53 previous similar messages
Feb 23 19:10:41 oak-io2-s1 kernel: Lustre: Skipped 15 previous similar messages
Feb 23 19:17:23 oak-io2-s1 kernel: Lustre: oak-OST0035: haven't heard from client dc520497-0ed3-7fcc-0ff7-ef6979cd6d72 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e033ab400, cur 1519442243 expire 1519442093 last 1519442016
Feb 23 19:23:15 oak-io2-s1 kernel: Lustre: oak-OST0038: haven't heard from client 57fe3b57-b29a-46d5-c4e3-434d3c2841a8 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883edce3bc00, cur 1519442595 expire 1519442445 last 1519442368
Feb 23 19:23:15 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 19:23:26 oak-io2-s1 kernel: Lustre: oak-OST0046: haven't heard from client 57fe3b57-b29a-46d5-c4e3-434d3c2841a8 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88153b1e7000, cur 1519442606 expire 1519442456 last 1519442379
Feb 23 19:23:26 oak-io2-s1 kernel: Lustre: Skipped 32 previous similar messages
Feb 23 19:34:47 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 19:34:47 oak-io2-s1 kernel: Lustre: Skipped 66 previous similar messages
Feb 23 19:41:29 oak-io2-s1 kernel: Lustre: oak-OST003a: haven't heard from client 88674c29-77b1-6988-7736-4d705a1911ef (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bfacdb800, cur 1519443689 expire 1519443539 last 1519443462
Feb 23 19:41:29 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 23 19:58:55 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 19:58:55 oak-io2-s1 kernel: Lustre: Skipped 32 previous similar messages
Feb 23 19:59:20 oak-io2-s1 kernel: Lustre: oak-OST0035: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 19:59:20 oak-io2-s1 kernel: Lustre: oak-OST0043: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 19:59:20 oak-io2-s1 kernel: Lustre: Skipped 7 previous similar messages
Feb 23 19:59:20 oak-io2-s1 kernel: Lustre: Skipped 7 previous similar messages
Feb 23 20:04:54 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 20:04:54 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 20:04:54 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 23 20:05:37 oak-io2-s1 kernel: Lustre: oak-OST0034: haven't heard from client 5352dd19-0cc0-609f-cada-baa09ad688bb (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e7a36f800, cur 1519445137 expire 1519444987 last 1519444910
Feb 23 20:05:37 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 20:08:09 oak-io2-s1 kernel: Lustre: 132016:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519445282/real 1519445282]  req@ffff88005f1c1800 x1592481989308912/t0(0) o106->oak-OST0042@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519445289 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Feb 23 20:08:09 oak-io2-s1 kernel: Lustre: 132016:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
Feb 23 20:08:30 oak-io2-s1 kernel: Lustre: 132016:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519445303/real 1519445303]  req@ffff88005f1c1800 x1592481989308912/t0(0) o106->oak-OST0042@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519445310 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 20:08:30 oak-io2-s1 kernel: Lustre: 132016:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Feb 23 20:09:12 oak-io2-s1 kernel: Lustre: 132016:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519445345/real 1519445345]  req@ffff88005f1c1800 x1592481989308912/t0(0) o106->oak-OST0042@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519445352 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 20:09:12 oak-io2-s1 kernel: Lustre: 132016:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Feb 23 20:10:29 oak-io2-s1 kernel: Lustre: 132016:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519445422/real 1519445422]  req@ffff88005f1c1800 x1592481989308912/t0(0) o106->oak-OST0042@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519445429 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 20:10:29 oak-io2-s1 kernel: Lustre: 132016:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
Feb 23 20:11:22 oak-io2-s1 kernel: LNet: Service thread pid 132016 was inactive for 200.29s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 23 20:11:22 oak-io2-s1 kernel: Pid: 132016, comm: ll_ost00_005
Feb 23 20:11:22 oak-io2-s1 kernel: #012Call Trace:
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffff816a6f34>] schedule_timeout+0x174/0x2c0
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffff81098b20>] ? process_timeout+0x0/0x10
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffffc0c46e40>] ptlrpc_set_wait+0x4c0/0x910 [ptlrpc]
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffffc0c04df3>] ldlm_run_ast_work+0xd3/0x3a0 [ptlrpc]
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffffc0c258bb>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc]
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffffc12316c4>] ofd_intent_policy+0x444/0xa30 [ofd]
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffffc0c04277>] ldlm_lock_enqueue+0x387/0x970 [ptlrpc]
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffffc0c2d903>] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc]
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffffc0c55ae0>] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc]
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffffc0cb2ea2>] tgt_enqueue+0x62/0x210 [ptlrpc]
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 20:11:22 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 20:11:22 oak-io2-s1 kernel: 
Feb 23 20:11:22 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519445482.132016
Feb 23 20:11:32 oak-io2-s1 kernel: Lustre: oak-OST0040: haven't heard from client 3e76c9eb-9a6e-7b92-cae4-6e1294a0c730 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881cffa65400, cur 1519445492 expire 1519445342 last 1519445265
Feb 23 20:11:32 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 20:11:35 oak-io2-s1 kernel: Lustre: oak-OST0042: haven't heard from client 3e76c9eb-9a6e-7b92-cae4-6e1294a0c730 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88153b1e0000, cur 1519445495 expire 1519445345 last 1519445268
Feb 23 20:11:35 oak-io2-s1 kernel: LNet: Service thread pid 132016 completed after 212.38s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Feb 23 20:11:40 oak-io2-s1 kernel: Lustre: oak-OST0039: haven't heard from client 3e76c9eb-9a6e-7b92-cae4-6e1294a0c730 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883cac56bc00, cur 1519445500 expire 1519445350 last 1519445273
Feb 23 20:11:40 oak-io2-s1 kernel: Lustre: Skipped 29 previous similar messages
Feb 23 20:16:57 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 20:16:57 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 20:16:57 oak-io2-s1 kernel: Lustre: Skipped 17 previous similar messages
Feb 23 20:16:57 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 23 20:23:33 oak-io2-s1 kernel: Lustre: oak-OST0030: haven't heard from client 6473208f-773b-b394-bc39-e21e7c112935 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883d71402c00, cur 1519446213 expire 1519446063 last 1519445986
Feb 23 20:23:33 oak-io2-s1 kernel: Lustre: Skipped 4 previous similar messages
Feb 23 20:29:00 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 20:29:00 oak-io2-s1 kernel: Lustre: Skipped 23 previous similar messages
Feb 23 20:35:41 oak-io2-s1 kernel: Lustre: oak-OST0052: haven't heard from client 90c8b69b-478a-5303-d4ab-541f49a976aa (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d1230bc00, cur 1519446941 expire 1519446791 last 1519446714
Feb 23 20:35:41 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 20:41:00 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 20:41:00 oak-io2-s1 kernel: Lustre: Skipped 75 previous similar messages
Feb 23 20:41:42 oak-io2-s1 kernel: Lustre: oak-OST0045: haven't heard from client 668312ba-212f-c5b1-7aec-99f47a16bd8d (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881cb497dc00, cur 1519447302 expire 1519447152 last 1519447075
Feb 23 20:41:42 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 20:44:15 oak-io2-s1 kernel: Lustre: 17266:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519447448/real 1519447448]  req@ffff8837b1ac1200 x1592481989729392/t0(0) o106->oak-OST003c@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519447455 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Feb 23 20:44:15 oak-io2-s1 kernel: Lustre: 17266:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
Feb 23 20:44:36 oak-io2-s1 kernel: Lustre: 249330:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519447469/real 1519447469]  req@ffff8812804c5d00 x1592481989729408/t0(0) o106->oak-OST0035@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519447476 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 20:44:36 oak-io2-s1 kernel: Lustre: 249330:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Feb 23 20:45:18 oak-io2-s1 kernel: Lustre: 17266:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519447511/real 1519447511]  req@ffff8837b1ac1200 x1592481989729392/t0(0) o106->oak-OST003c@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519447518 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 20:45:18 oak-io2-s1 kernel: Lustre: 17266:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
Feb 23 20:46:35 oak-io2-s1 kernel: Lustre: 249330:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519447588/real 1519447588]  req@ffff8812804c5d00 x1592481989729408/t0(0) o106->oak-OST0035@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519447595 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 20:46:35 oak-io2-s1 kernel: Lustre: 17266:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519447588/real 1519447588]  req@ffff8837b1ac1200 x1592481989729392/t0(0) o106->oak-OST003c@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519447595 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 20:46:35 oak-io2-s1 kernel: Lustre: 17266:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
Feb 23 20:47:03 oak-io2-s1 kernel: LustreError: 17266:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.112.15@o2ib4) returned error from glimpse AST (req@ffff8837b1ac1200 x1592481989729392 status -107 rc -107), evict it ns: filter-oak-OST003c_UUID lock: ffff880789235200/0x806f959362687467 lrc: 4/0,0 mode: PW/PW res: [0x43b58d:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x40000000000000 nid: 10.9.112.15@o2ib4 remote: 0x74a0d86312df1222 expref: 5 pid: 390784 timeout: 0 lvb_type: 0
Feb 23 20:47:03 oak-io2-s1 kernel: LustreError: 17266:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) Skipped 3 previous similar messages
Feb 23 20:47:03 oak-io2-s1 kernel: LustreError: 138-a: oak-OST0035: A client on nid 10.9.112.15@o2ib4 was evicted due to a lock glimpse callback time out: rc -107
Feb 23 20:47:03 oak-io2-s1 kernel: LustreError: Skipped 3 previous similar messages
Feb 23 20:47:38 oak-io2-s1 kernel: Lustre: oak-OST0040: haven't heard from client 72df71fd-c66e-637d-269e-2f412d3603ee (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88229768ec00, cur 1519447658 expire 1519447508 last 1519447431
Feb 23 20:47:38 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 20:47:42 oak-io2-s1 kernel: Lustre: oak-OST0033: haven't heard from client 72df71fd-c66e-637d-269e-2f412d3603ee (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a029a9000, cur 1519447662 expire 1519447512 last 1519447435
Feb 23 20:47:42 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 23 20:47:47 oak-io2-s1 kernel: Lustre: oak-OST0052: haven't heard from client 72df71fd-c66e-637d-269e-2f412d3603ee (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883563ce8c00, cur 1519447667 expire 1519447517 last 1519447440
Feb 23 20:47:47 oak-io2-s1 kernel: Lustre: Skipped 25 previous similar messages
Feb 23 20:53:07 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 20:53:07 oak-io2-s1 kernel: Lustre: Skipped 70 previous similar messages
Feb 23 20:53:43 oak-io2-s1 kernel: Lustre: oak-OST0039: haven't heard from client 7a5304b5-b571-74f9-bf8a-674cf180b432 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881869b9e800, cur 1519448023 expire 1519447873 last 1519447796
Feb 23 20:53:43 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
Feb 23 20:59:49 oak-io2-s1 kernel: Lustre: oak-OST0038: haven't heard from client 24b17af5-2128-c8c3-c3ed-7493547f8575 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e7b360400, cur 1519448389 expire 1519448239 last 1519448162
Feb 23 20:59:49 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 21:05:09 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 21:05:09 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 21:05:09 oak-io2-s1 kernel: Lustre: Skipped 51 previous similar messages
Feb 23 21:05:09 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 23 21:05:47 oak-io2-s1 kernel: Lustre: oak-OST0043: haven't heard from client e9b7bdc4-9bf6-3312-81d7-dec225b70671 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bfacda000, cur 1519448747 expire 1519448597 last 1519448520
Feb 23 21:05:47 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 21:11:51 oak-io2-s1 kernel: Lustre: oak-OST0033: haven't heard from client ed722225-0e01-df72-d145-529efd0ad48c (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883caae1cc00, cur 1519449111 expire 1519448961 last 1519448884
Feb 23 21:11:51 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 21:17:53 oak-io2-s1 kernel: Lustre: oak-OST0035: haven't heard from client de1e5c6a-3a5b-696c-19b9-25d8c5b308d5 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883cd82cd400, cur 1519449473 expire 1519449323 last 1519449246
Feb 23 21:17:53 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 21:35:17 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 21:35:17 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 21:35:17 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 21:35:17 oak-io2-s1 kernel: Lustre: Skipped 52 previous similar messages
Feb 23 21:35:17 oak-io2-s1 kernel: Lustre: Skipped 52 previous similar messages
Feb 23 21:35:17 oak-io2-s1 kernel: Lustre: Skipped 15 previous similar messages
Feb 23 21:41:15 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 21:41:15 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 21:41:15 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 21:41:15 oak-io2-s1 kernel: Lustre: Skipped 17 previous similar messages
Feb 23 21:41:15 oak-io2-s1 kernel: Lustre: Skipped 18 previous similar messages
Feb 23 21:41:15 oak-io2-s1 kernel: Lustre: Skipped 13 previous similar messages
Feb 23 21:41:59 oak-io2-s1 kernel: Lustre: oak-OST0030: haven't heard from client 28f13573-ae57-18b0-0efb-2f0b23d85238 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883e99d54800, cur 1519450919 expire 1519450769 last 1519450692
Feb 23 21:41:59 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 21:44:33 oak-io2-s1 kernel: Lustre: 390794:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519451066/real 1519451066]  req@ffff88000ff00900 x1592481990160112/t0(0) o106->oak-OST0034@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519451073 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Feb 23 21:44:33 oak-io2-s1 kernel: Lustre: 390794:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
Feb 23 21:45:15 oak-io2-s1 kernel: Lustre: 137164:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519451108/real 1519451108]  req@ffff8800048bfb00 x1592481990160128/t0(0) o106->oak-OST003e@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519451115 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 21:45:15 oak-io2-s1 kernel: Lustre: 137164:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Feb 23 21:46:32 oak-io2-s1 kernel: Lustre: 137164:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519451185/real 1519451185]  req@ffff8800048bfb00 x1592481990160128/t0(0) o106->oak-OST003e@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519451192 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 21:46:32 oak-io2-s1 kernel: Lustre: 137164:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
Feb 23 21:47:20 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 21:47:20 oak-io2-s1 kernel: Lustre: Skipped 34 previous similar messages
Feb 23 21:47:21 oak-io2-s1 kernel: LustreError: 390794:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.112.15@o2ib4) returned error from glimpse AST (req@ffff88000ff00900 x1592481990160112 status -107 rc -107), evict it ns: filter-oak-OST0034_UUID lock: ffff883c89ab3000/0x806f959362778eac lrc: 4/0,0 mode: PW/PW res: [0x41775f:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x40000000000000 nid: 10.9.112.15@o2ib4 remote: 0x4e1f71a39d43afab expref: 5 pid: 21428 timeout: 0 lvb_type: 0
Feb 23 21:47:21 oak-io2-s1 kernel: LustreError: 138-a: oak-OST003e: A client on nid 10.9.112.15@o2ib4 was evicted due to a lock glimpse callback time out: rc -107
Feb 23 21:47:21 oak-io2-s1 kernel: LustreError: Skipped 4 previous similar messages
Feb 23 21:47:21 oak-io2-s1 kernel: LustreError: 390794:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) Skipped 4 previous similar messages
Feb 23 21:47:56 oak-io2-s1 kernel: Lustre: oak-OST0042: haven't heard from client 1997f5e9-9072-a03d-9e54-3126cf681923 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e7beec800, cur 1519451276 expire 1519451126 last 1519451049
Feb 23 21:47:56 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 21:53:50 oak-io2-s1 kernel: Lustre: oak-OST0040: haven't heard from client d8b5411c-c28b-e928-4336-d13d9af3c72c (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d47c36c00, cur 1519451630 expire 1519451480 last 1519451403
Feb 23 21:53:50 oak-io2-s1 kernel: Lustre: Skipped 30 previous similar messages
Feb 23 22:00:04 oak-io2-s1 kernel: Lustre: oak-OST0037: haven't heard from client 4a891d62-cdbb-063d-10a3-fc27f7ee3a74 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882eb9988800, cur 1519452004 expire 1519451854 last 1519451777
Feb 23 22:00:04 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 22:05:20 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 22:05:20 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 22:05:20 oak-io2-s1 kernel: Lustre: Skipped 51 previous similar messages
Feb 23 22:05:20 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
Feb 23 22:12:02 oak-io2-s1 kernel: Lustre: oak-OST0033: haven't heard from client d0deae9a-d1e1-4f0d-9522-2d67f83a5746 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fdeab0c00, cur 1519452722 expire 1519452572 last 1519452495
Feb 23 22:12:02 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 22:14:37 oak-io2-s1 kernel: Lustre: 135933:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519452870/real 1519452870]  req@ffff880012957500 x1592481990366880/t0(0) o106->oak-OST004e@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519452877 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Feb 23 22:14:37 oak-io2-s1 kernel: Lustre: 135933:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
Feb 23 22:14:58 oak-io2-s1 kernel: Lustre: 249494:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519452891/real 1519452891]  req@ffff8819da8d0900 x1592481990366896/t0(0) o106->oak-OST003d@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519452898 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 22:14:58 oak-io2-s1 kernel: Lustre: 249494:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
Feb 23 22:15:40 oak-io2-s1 kernel: Lustre: 249494:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519452933/real 1519452933]  req@ffff8819da8d0900 x1592481990366896/t0(0) o106->oak-OST003d@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519452940 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 22:15:40 oak-io2-s1 kernel: Lustre: 249494:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
Feb 23 22:16:57 oak-io2-s1 kernel: Lustre: 249494:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519453010/real 1519453010]  req@ffff8819da8d0900 x1592481990366896/t0(0) o106->oak-OST003d@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519453017 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 22:16:57 oak-io2-s1 kernel: Lustre: 249494:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 20 previous similar messages
Feb 23 22:17:33 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 22:17:33 oak-io2-s1 kernel: Lustre: Skipped 65 previous similar messages
Feb 23 22:17:39 oak-io2-s1 kernel: LustreError: 249494:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.112.15@o2ib4) returned error from glimpse AST (req@ffff8819da8d0900 x1592481990366896 status -107 rc -107), evict it ns: filter-oak-OST003d_UUID lock: ffff8837c0035400/0x806f9593628123ed lrc: 4/0,0 mode: PW/PW res: [0x438d06:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x40000000000000 nid: 10.9.112.15@o2ib4 remote: 0xd7d0c605e8fec5a expref: 5 pid: 210324 timeout: 0 lvb_type: 0
Feb 23 22:17:39 oak-io2-s1 kernel: LustreError: 138-a: oak-OST004e: A client on nid 10.9.112.15@o2ib4 was evicted due to a lock glimpse callback time out: rc -107
Feb 23 22:17:39 oak-io2-s1 kernel: LustreError: Skipped 4 previous similar messages
Feb 23 22:17:39 oak-io2-s1 kernel: LustreError: 249494:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) Skipped 4 previous similar messages
Feb 23 22:24:15 oak-io2-s1 kernel: Lustre: oak-OST0037: haven't heard from client c6cf18b3-da6c-d9d0-72d7-ca69d6a5b32c (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d017af400, cur 1519453455 expire 1519453305 last 1519453228
Feb 23 22:24:15 oak-io2-s1 kernel: Lustre: Skipped 67 previous similar messages
Feb 23 22:26:42 oak-io2-s1 kernel: Lustre: 17268:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519453595/real 1519453595]  req@ffff881448137b00 x1592481990451312/t0(0) o106->oak-OST0051@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519453602 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Feb 23 22:26:42 oak-io2-s1 kernel: Lustre: 17265:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519453595/real 1519453595]  req@ffff880058676f00 x1592481990451296/t0(0) o106->oak-OST0036@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519453602 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Feb 23 22:26:42 oak-io2-s1 kernel: Lustre: 17265:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
Feb 23 22:29:31 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 22:29:31 oak-io2-s1 kernel: Lustre: Skipped 70 previous similar messages
Feb 23 22:29:37 oak-io2-s1 kernel: LustreError: 17265:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.112.15@o2ib4) returned error from glimpse AST (req@ffff880058676f00 x1592481990451296 status -107 rc -107), evict it ns: filter-oak-OST0036_UUID lock: ffff880b14d1b000/0x806f9593628697c5 lrc: 4/0,0 mode: PW/PW res: [0x40d8cf:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x40000000000000 nid: 10.9.112.15@o2ib4 remote: 0x4754545b3a1b127f expref: 5 pid: 135933 timeout: 0 lvb_type: 0
Feb 23 22:29:37 oak-io2-s1 kernel: LustreError: 138-a: oak-OST0051: A client on nid 10.9.112.15@o2ib4 was evicted due to a lock glimpse callback time out: rc -107
Feb 23 22:29:37 oak-io2-s1 kernel: LustreError: Skipped 3 previous similar messages
Feb 23 22:29:37 oak-io2-s1 kernel: LustreError: 17265:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) Skipped 3 previous similar messages
Feb 23 22:36:13 oak-io2-s1 kernel: Lustre: oak-OST0032: haven't heard from client 492c0ac2-f006-441d-6ac3-becadbe15e91 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881577af3c00, cur 1519454173 expire 1519454023 last 1519453946
Feb 23 22:36:13 oak-io2-s1 kernel: Lustre: Skipped 67 previous similar messages
Feb 23 22:41:33 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 22:41:33 oak-io2-s1 kernel: Lustre: Skipped 32 previous similar messages
Feb 23 22:48:15 oak-io2-s1 kernel: Lustre: oak-OST003b: haven't heard from client fdb6af0d-770d-e835-152d-ab49ab53558f (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d03b35c00, cur 1519454895 expire 1519454745 last 1519454668
Feb 23 22:48:15 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 22:53:36 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to c8ed7df6-e4c7-6d43-dca5-2cc11a7d71fc (at 10.9.112.15@o2ib4)
Feb 23 22:53:36 oak-io2-s1 kernel: Lustre: Skipped 32 previous similar messages
Feb 23 23:00:17 oak-io2-s1 kernel: Lustre: oak-OST0047: haven't heard from client 8e16fa36-2b9e-961f-9969-c62621d2894d (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b251dd000, cur 1519455617 expire 1519455467 last 1519455390
Feb 23 23:00:17 oak-io2-s1 kernel: Lustre: Skipped 35 previous similar messages
Feb 23 23:00:31 oak-io2-s1 kernel: Lustre: 137164:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519455624/real 1519455624]  req@ffff88004b3ed400 x1592481990801536/t0(0) o106->oak-OST0034@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519455631 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Feb 23 23:00:31 oak-io2-s1 kernel: Lustre: 137164:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 49 previous similar messages
Feb 23 23:01:48 oak-io2-s1 kernel: Lustre: 137164:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519455701/real 1519455701]  req@ffff88004b3ed400 x1592481990801536/t0(0) o106->oak-OST0034@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519455708 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 23:01:48 oak-io2-s1 kernel: Lustre: 137164:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
Feb 23 23:01:51 oak-io2-s1 kernel: INFO: task kswapd0:264 blocked for more than 120 seconds.
Feb 23 23:01:51 oak-io2-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 23 23:01:51 oak-io2-s1 kernel: LustreError: 173725:0:(lustre_dlm.h:1097:ldlm_lvbo_fill()) lock ffff880040c4a000: delayed lvb init failed (rc -2)
Feb 23 23:01:51 oak-io2-s1 kernel: kswapd0         D
Feb 23 23:01:51 oak-io2-s1 kernel: ffff880151e49958     0   264      2 0x00000000
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881ffac6f8d0 0000000000000046 ffff881ffb7c2f70 ffff881ffac6ffd8
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881ffac6ffd8 ffff881ffac6ffd8 ffff881ffb7c2f70 ffff880151e49950
Feb 23 23:01:51 oak-io2-s1 kernel: ffff880151e49954 ffff881ffb7c2f70 00000000ffffffff ffff880151e49958
Feb 23 23:01:51 oak-io2-s1 kernel: Call Trace:
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc0949a93>] ? jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81265e8a>] dquot_acquire+0x3a/0x130
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc1004766>] ldiskfs_acquire_dquot+0x66/0xb0 [ldiskfs]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc108d214>] lquota_disk_read+0x124/0x390 [lquota]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc1094bfa>] qsd_refresh_usage+0x6a/0x2b0 [lquota]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc10a3881>] qsd_op_adjust+0x2f1/0x730 [lquota]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc10e7c50>] osd_object_delete+0x230/0x330 [osd_ldiskfs]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc0a27e3d>] lu_object_free.isra.31+0x9d/0x1a0 [obdclass]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc0a28a9e>] lu_site_purge_objects+0x2fe/0x520 [obdclass]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc0a298e9>] lu_cache_shrink+0x259/0x2d0 [obdclass]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81195413>] shrink_slab+0x163/0x330
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff811f7547>] ? vmpressure+0x87/0x90
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81199081>] balance_pgdat+0x4b1/0x5e0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81199323>] kswapd+0x173/0x440
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff811991b0>] ? balance_pgdat+0x5e0/0x5e0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Feb 23 23:01:51 oak-io2-s1 kernel: INFO: task kswapd1:265 blocked for more than 120 seconds.
Feb 23 23:01:51 oak-io2-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 23 23:01:51 oak-io2-s1 kernel: kswapd1         D ffff881ffac73b80     0   265      2 0x00000000
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881ffac73b58 0000000000000046 ffff881ffb7c3f40 ffff881ffac73fd8
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881ffac73fd8 ffff881ffac73fd8 ffff881ffb7c3f40 ffff881ffb7c3f40
Feb 23 23:01:51 oak-io2-s1 kernel: ffffffffc0a826a0 fffffffeffffffff ffffffffc0a826a8 ffff881ffac73b80
Feb 23 23:01:51 oak-io2-s1 kernel: Call Trace:
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816aab1d>] rwsem_down_read_failed+0x10d/0x1a0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81332018>] call_rwsem_down_read_failed+0x18/0x30
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a8780>] down_read+0x20/0x40
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc0a296fa>] lu_cache_shrink+0x6a/0x2d0 [obdclass]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81195359>] shrink_slab+0xa9/0x330
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff811a880b>] ? compaction_suitable+0x5b/0xb0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81199081>] balance_pgdat+0x4b1/0x5e0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81199323>] kswapd+0x173/0x440
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff811991b0>] ? balance_pgdat+0x5e0/0x5e0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Feb 23 23:01:51 oak-io2-s1 kernel: INFO: task multipathd:66017 blocked for more than 120 seconds.
Feb 23 23:01:51 oak-io2-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 23 23:01:51 oak-io2-s1 kernel: multipathd      D ffffffff81a8ae08     0 66017      1 0x00000080
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881fe5357b50 0000000000000082 ffff881fe533cf10 ffff881fe5357fd8
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881fe5357fd8 ffff881fe5357fd8 ffff881fe533cf10 ffffffff81a8ae00
Feb 23 23:01:51 oak-io2-s1 kernel: ffffffff81a8ae04 ffff881fe533cf10 00000000ffffffff ffffffff81a8ae08
Feb 23 23:01:51 oak-io2-s1 kernel: Call Trace:
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8127f9a2>] sysfs_permission+0x32/0x60
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8120c31e>] __inode_permission+0x6e/0xc0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8120c388>] inode_permission+0x18/0x50
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8120e44e>] link_path_walk+0x27e/0x8b0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8120ebdb>] path_lookupat+0x6b/0x7b0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81327f56>] ? plist_del+0x46/0x70
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810f49fc>] ? __unqueue_futex+0x2c/0x60
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810d8278>] ? cpupri_set+0x98/0x100
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff811df555>] ? kmem_cache_alloc+0x35/0x1e0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81211d2f>] ? getname_flags+0x4f/0x1a0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8120f34b>] filename_lookup+0x2b/0xc0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81212ec7>] user_path_at_empty+0x67/0xc0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a8fad>] ? __schedule+0x39d/0x8b0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81212f31>] user_path_at+0x11/0x20
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81206473>] vfs_fstatat+0x63/0xc0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff812069de>] SYSC_newstat+0x2e/0x60
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a84b2>] ? do_nanosleep+0x72/0xf0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8111f5c6>] ? __audit_syscall_exit+0x1e6/0x280
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81206cbe>] SyS_newstat+0xe/0x10
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Feb 23 23:01:51 oak-io2-s1 kernel: INFO: task dsm_sa_datamgrd:327053 blocked for more than 120 seconds.
Feb 23 23:01:51 oak-io2-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 23 23:01:51 oak-io2-s1 kernel: dsm_sa_datamgrd D ffffffff81a8ae08     0 327053      1 0x00000080
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881ff975bb50 0000000000000086 ffff881f9ab28000 ffff881ff975bfd8
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881ff975bfd8 ffff881ff975bfd8 ffff881f9ab28000 ffffffff81a8ae00
Feb 23 23:01:51 oak-io2-s1 kernel: ffffffff81a8ae04 ffff881f9ab28000 00000000ffffffff ffffffff81a8ae08
Feb 23 23:01:51 oak-io2-s1 kernel: Call Trace:
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8127f9a2>] sysfs_permission+0x32/0x60
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8120c31e>] __inode_permission+0x6e/0xc0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8120c388>] inode_permission+0x18/0x50
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8120e44e>] link_path_walk+0x27e/0x8b0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8121b45a>] ? __d_lookup+0x7a/0x160
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8120ebdb>] path_lookupat+0x6b/0x7b0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff811df555>] ? kmem_cache_alloc+0x35/0x1e0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81211d2f>] ? getname_flags+0x4f/0x1a0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8120f34b>] filename_lookup+0x2b/0xc0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81212ec7>] user_path_at_empty+0x67/0xc0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810c76f5>] ? sched_clock_cpu+0x85/0xc0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810c12c8>] ? check_preempt_curr+0x78/0xa0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81212f31>] user_path_at+0x11/0x20
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81206473>] vfs_fstatat+0x63/0xc0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810c4593>] ? try_to_wake_up+0x183/0x340
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff812069de>] SYSC_newstat+0x2e/0x60
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810c4765>] ? wake_up_process+0x15/0x20
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8129c2d7>] ? wake_up_sem_queue_do+0x37/0x60
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8111f5c6>] ? __audit_syscall_exit+0x1e6/0x280
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81206cbe>] SyS_newstat+0xe/0x10
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Feb 23 23:01:51 oak-io2-s1 kernel: INFO: task md30_raid6:340220 blocked for more than 120 seconds.
Feb 23 23:01:51 oak-io2-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 23 23:01:51 oak-io2-s1 kernel: md30_raid6      D ffffffff81a8ae08     0 340220      2 0x00000080
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881fde99bc10 0000000000000046 ffff88015367af70 ffff881fde99bfd8
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881fde99bfd8 ffff881fde99bfd8 ffff88015367af70 ffffffff81a8ae00
Feb 23 23:01:51 oak-io2-s1 kernel: ffffffff81a8ae04 ffff88015367af70 00000000ffffffff ffffffff81a8ae08
Feb 23 23:01:51 oak-io2-s1 kernel: Call Trace:
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8127ffc4>] sysfs_notify+0x24/0x90
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff815150c6>] md_update_sb+0x686/0x690
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a7742>] ? mutex_lock+0x12/0x2f
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81515c5a>] md_check_recovery+0x1ca/0x4c0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc08570f2>] raid5d+0x542/0x7f0 [raid456]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8150e2f5>] md_thread+0x155/0x1a0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8150e1a0>] ? find_pers+0x80/0x80
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Feb 23 23:01:51 oak-io2-s1 kernel: INFO: task md2_raid6:131162 blocked for more than 120 seconds.
Feb 23 23:01:51 oak-io2-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 23 23:01:51 oak-io2-s1 kernel: md2_raid6       D ffffffff81a8ae08     0 131162      2 0x00000080
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881ed4b93c10 0000000000000046 ffff881ed82a8000 ffff881ed4b93fd8
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881ed4b93fd8 ffff881ed4b93fd8 ffff881ed82a8000 ffffffff81a8ae00
Feb 23 23:01:51 oak-io2-s1 kernel: ffffffff81a8ae04 ffff881ed82a8000 00000000ffffffff ffffffff81a8ae08
Feb 23 23:01:51 oak-io2-s1 kernel: Call Trace:
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8127ffc4>] sysfs_notify+0x24/0x90
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff815150c6>] md_update_sb+0x686/0x690
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a7742>] ? mutex_lock+0x12/0x2f
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81515c5a>] md_check_recovery+0x1ca/0x4c0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc0856c06>] raid5d+0x56/0x7f0 [raid456]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81099302>] ? del_timer_sync+0x52/0x60
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a6f3c>] ? schedule_timeout+0x17c/0x2c0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81098b20>] ? internal_add_timer+0x70/0x70
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8150e2f5>] md_thread+0x155/0x1a0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8150e1a0>] ? find_pers+0x80/0x80
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Feb 23 23:01:51 oak-io2-s1 kernel: INFO: task md0_raid6:131228 blocked for more than 120 seconds.
Feb 23 23:01:51 oak-io2-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 23 23:01:51 oak-io2-s1 kernel: md0_raid6       D ffffffff81a8ae08     0 131228      2 0x00000080
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881ebafabc10 0000000000000046 ffff881ed82a9fa0 ffff881ebafabfd8
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881ebafabfd8 ffff881ebafabfd8 ffff881ed82a9fa0 ffffffff81a8ae00
Feb 23 23:01:51 oak-io2-s1 kernel: ffffffff81a8ae04 ffff881ed82a9fa0 00000000ffffffff ffffffff81a8ae08
Feb 23 23:01:51 oak-io2-s1 kernel: Call Trace:
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8127ffc4>] sysfs_notify+0x24/0x90
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff815150c6>] md_update_sb+0x686/0x690
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a7742>] ? mutex_lock+0x12/0x2f
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81515c5a>] md_check_recovery+0x1ca/0x4c0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc0856c06>] raid5d+0x56/0x7f0 [raid456]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81099302>] ? del_timer_sync+0x52/0x60
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a6f3c>] ? schedule_timeout+0x17c/0x2c0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81098b20>] ? internal_add_timer+0x70/0x70
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8150e2f5>] md_thread+0x155/0x1a0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8150e1a0>] ? find_pers+0x80/0x80
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Feb 23 23:01:51 oak-io2-s1 kernel: INFO: task md8_raid6:131281 blocked for more than 120 seconds.
Feb 23 23:01:51 oak-io2-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 23 23:01:51 oak-io2-s1 kernel: md8_raid6       D ffffffff81a8ae08     0 131281      2 0x00000080
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881eabfefc10 0000000000000046 ffff881ebd281fa0 ffff881eabfeffd8
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881eabfeffd8 ffff881eabfeffd8 ffff881ebd281fa0 ffffffff81a8ae00
Feb 23 23:01:51 oak-io2-s1 kernel: ffffffff81a8ae04 ffff881ebd281fa0 00000000ffffffff ffffffff81a8ae08
Feb 23 23:01:51 oak-io2-s1 kernel: Call Trace:
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8127ffc4>] sysfs_notify+0x24/0x90
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff815150c6>] md_update_sb+0x686/0x690
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81515c5a>] md_check_recovery+0x1ca/0x4c0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc0856c06>] raid5d+0x56/0x7f0 [raid456]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81099302>] ? del_timer_sync+0x52/0x60
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a6f3c>] ? schedule_timeout+0x17c/0x2c0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81098b20>] ? internal_add_timer+0x70/0x70
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8150e2f5>] md_thread+0x155/0x1a0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8150e1a0>] ? find_pers+0x80/0x80
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Feb 23 23:01:51 oak-io2-s1 kernel: INFO: task md18_raid6:131341 blocked for more than 120 seconds.
Feb 23 23:01:51 oak-io2-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 23 23:01:51 oak-io2-s1 kernel: md18_raid6      D ffffffff81a8ae08     0 131341      2 0x00000080
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881eaa6e7c10 0000000000000046 ffff881fb0342f70 ffff881eaa6e7fd8
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881eaa6e7fd8 ffff881eaa6e7fd8 ffff881fb0342f70 ffffffff81a8ae00
Feb 23 23:01:51 oak-io2-s1 kernel: ffffffff81a8ae04 ffff881fb0342f70 00000000ffffffff ffffffff81a8ae08
Feb 23 23:01:51 oak-io2-s1 kernel: Call Trace:
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8127ffc4>] sysfs_notify+0x24/0x90
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff815150c6>] md_update_sb+0x686/0x690
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a7742>] ? mutex_lock+0x12/0x2f
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81515c5a>] md_check_recovery+0x1ca/0x4c0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc08570f2>] raid5d+0x542/0x7f0 [raid456]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8150e2f5>] md_thread+0x155/0x1a0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8150e1a0>] ? find_pers+0x80/0x80
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Feb 23 23:01:51 oak-io2-s1 kernel: INFO: task kmmpd-md2:131859 blocked for more than 120 seconds.
Feb 23 23:01:51 oak-io2-s1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 23 23:01:51 oak-io2-s1 kernel: kmmpd-md2       D ffff881e60747cc0     0 131859      2 0x00000080
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881e60747b38 0000000000000046 ffff881fde7b0fd0 ffff881e60747fd8
Feb 23 23:01:51 oak-io2-s1 kernel: ffff881e60747fd8 ffff881e60747fd8 ffff881fde7b0fd0 ffff883f235c0800
Feb 23 23:01:51 oak-io2-s1 kernel: ffff883f235c0a98 ffff883f235c083c ffff8814a9db3500 ffff881e60747cc0
Feb 23 23:01:51 oak-io2-s1 kernel: Call Trace:
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81510fa5>] md_write_start+0xd5/0x220
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc08535d0>] raid5_make_request+0xc0/0xd10 [raid456]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810cd794>] ? update_curr+0x104/0x190
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810cdc7c>] ? dequeue_entity+0x11c/0x5d0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8150a744>] md_make_request+0x104/0x290
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff812f9085>] generic_make_request+0x105/0x310
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff812f9300>] submit_bio+0x70/0x150
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff8123b7c5>] ? bio_alloc_bioset+0x115/0x310
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff81237867>] _submit_bh+0x127/0x160
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff812378b0>] submit_bh+0x10/0x20
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc0ff6c31>] write_mmp_block+0x111/0x170 [ldiskfs]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc0ff6ea8>] kmmpd+0x1a8/0x430 [ldiskfs]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816a8fad>] ? __schedule+0x39d/0x8b0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffffc0ff6d00>] ? __dump_mmp_msg+0x70/0x70 [ldiskfs]
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:01:51 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Feb 23 23:02:33 oak-io2-s1 kernel: LNet: Service thread pid 251689 was inactive for 200.06s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 23 23:02:33 oak-io2-s1 kernel: Pid: 251689, comm: ll_ost_io01_029
Feb 23 23:02:33 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff81267099>] dquot_commit+0x29/0xc0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc100481c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc100487f>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff8126a0f2>] __dquot_alloc_space+0x1f2/0x250
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff81349341>] ? __percpu_counter_add+0x51/0x70
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0fd5b41>] ldiskfs_mb_new_blocks+0xf1/0xb20 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc1019c05>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff811e0623>] ? __kmalloc+0x1e3/0x230
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc101ac9c>] ? ldiskfs_ext_find_extent+0x12c/0x2f0 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc101f186>] ldiskfs_ext_map_blocks+0x496/0xf50 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0fee2b3>] ldiskfs_map_blocks+0x163/0x700 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc110a573>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc110b9b2>] osd_write_commit+0x3b2/0x8d0 [osd_ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc10eeb12>] ? osd_trans_start+0x1f2/0x490 [osd_ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc122ac83>] ofd_commitrw_write+0xfe3/0x1c50 [ofd]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc122e5a9>] ofd_commitrw+0x4b9/0xac0 [ofd]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0ce7eb7>] obd_commitrw+0x2ed/0x330 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0cbad91>] tgt_brw_write+0xff1/0x17c0 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810cd794>] ? update_curr+0x104/0x190
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c0ce60>] ? target_bulk_timeout+0x0/0xb0 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:02:33 oak-io2-s1 kernel: 
Feb 23 23:02:33 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455753.251689
Feb 23 23:02:33 oak-io2-s1 kernel: Pid: 251676, comm: ll_ost_io00_018
Feb 23 23:02:33 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816a7610>] ? bit_wait_io+0x0/0x50
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816a6ff9>] schedule_timeout+0x239/0x2c0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816a7610>] ? bit_wait_io+0x0/0x50
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816a8b6d>] io_schedule_timeout+0xad/0x130
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816a8c08>] io_schedule+0x18/0x20
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816a7621>] bit_wait_io+0x11/0x50
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816a733f>] __wait_on_bit_lock+0x5f/0xc0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816a7610>] ? bit_wait_io+0x0/0x50
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816a7421>] out_of_line_wait_on_bit_lock+0x81/0xb0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810b19d0>] ? wake_bit_function_rh+0x0/0x40
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff81236612>] __lock_buffer+0x32/0x40
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc094ae6f>] do_get_write_access+0x42f/0x4c0 [jbd2]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0fee8f6>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc094af27>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0fce2bb>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc1007306>] ldiskfs_quota_write+0xc6/0x1e0 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff8126b4f9>] qtree_write_dquot+0xc9/0x170
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff8126a2eb>] v2_write_dquot+0x2b/0x30
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff81267127>] dquot_commit+0xb7/0xc0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc100481c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc100487f>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff8126a0f2>] __dquot_alloc_space+0x1f2/0x250
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff812675d7>] ? dqget+0x1a7/0x440
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc1010000>] ? ldiskfs_try_to_write_inline_data+0x1b0/0x5d0 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0fd5b41>] ldiskfs_mb_new_blocks+0xf1/0xb20 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc1015e3b>] ? ldiskfs_es_insert_extent+0xdb/0x1b0 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc108d1dc>] ? lquota_disk_read+0xec/0x390 [lquota]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff811e0623>] ? __kmalloc+0x1e3/0x230
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc101adb9>] ? ldiskfs_ext_find_extent+0x249/0x2f0 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc101adb9>] ? ldiskfs_ext_find_extent+0x249/0x2f0 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc1019393>] ? ldiskfs_ext_put_gap_in_cache+0x83/0x90 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc101f186>] ldiskfs_ext_map_blocks+0x496/0xf50 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0fee2b3>] ldiskfs_map_blocks+0x163/0x700 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc110a573>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc110b9b2>] osd_write_commit+0x3b2/0x8d0 [osd_ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc10eeb12>] ? osd_trans_start+0x1f2/0x490 [osd_ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc122ac83>] ofd_commitrw_write+0xfe3/0x1c50 [ofd]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc122e5a9>] ofd_commitrw+0x4b9/0xac0 [ofd]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0ce7eb7>] obd_commitrw+0x2ed/0x330 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0cbad91>] tgt_brw_write+0xff1/0x17c0 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810cd794>] ? update_curr+0x104/0x190
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810ca29e>] ? account_entity_dequeue+0xae/0xd0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810cdc7c>] ? dequeue_entity+0x11c/0x5d0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c0ce60>] ? target_bulk_timeout+0x0/0xb0 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:02:33 oak-io2-s1 kernel: 
Feb 23 23:02:33 oak-io2-s1 kernel: Pid: 63775, comm: ll_ost_io00_118
Feb 23 23:02:33 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff81267099>] dquot_commit+0x29/0xc0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc100481c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc100487f>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff8126a0f2>] __dquot_alloc_space+0x1f2/0x250
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff81349341>] ? __percpu_counter_add+0x51/0x70
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0fd5b41>] ldiskfs_mb_new_blocks+0xf1/0xb20 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc1019c05>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff811e0623>] ? __kmalloc+0x1e3/0x230
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc101ac9c>] ? ldiskfs_ext_find_extent+0x12c/0x2f0 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc101f186>] ldiskfs_ext_map_blocks+0x496/0xf50 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0fee2b3>] ldiskfs_map_blocks+0x163/0x700 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc110a573>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc110b9b2>] osd_write_commit+0x3b2/0x8d0 [osd_ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc10eeb12>] ? osd_trans_start+0x1f2/0x490 [osd_ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc122ac83>] ofd_commitrw_write+0xfe3/0x1c50 [ofd]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc122e5a9>] ofd_commitrw+0x4b9/0xac0 [ofd]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0ce7eb7>] obd_commitrw+0x2ed/0x330 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0cbad91>] tgt_brw_write+0xff1/0x17c0 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810cd794>] ? update_curr+0x104/0x190
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff811dd579>] ? ___slab_alloc+0x209/0x4f0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c0ce60>] ? target_bulk_timeout+0x0/0xb0 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:02:33 oak-io2-s1 kernel: 
Feb 23 23:02:33 oak-io2-s1 kernel: Pid: 301783, comm: ll_ost_io00_040
Feb 23 23:02:33 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff81267099>] dquot_commit+0x29/0xc0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc100481c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc100487f>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff8126a0f2>] __dquot_alloc_space+0x1f2/0x250
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff81349341>] ? __percpu_counter_add+0x51/0x70
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0fd5b41>] ldiskfs_mb_new_blocks+0xf1/0xb20 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc1019c05>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff811e0623>] ? __kmalloc+0x1e3/0x230
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc101ac9c>] ? ldiskfs_ext_find_extent+0x12c/0x2f0 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc101f186>] ldiskfs_ext_map_blocks+0x496/0xf50 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0fee2b3>] ldiskfs_map_blocks+0x163/0x700 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc110a573>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc110b9b2>] osd_write_commit+0x3b2/0x8d0 [osd_ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc10eeb12>] ? osd_trans_start+0x1f2/0x490 [osd_ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc122ac83>] ofd_commitrw_write+0xfe3/0x1c50 [ofd]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc122e5a9>] ofd_commitrw+0x4b9/0xac0 [ofd]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0ce7eb7>] obd_commitrw+0x2ed/0x330 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0cbad91>] tgt_brw_write+0xff1/0x17c0 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810cd794>] ? update_curr+0x104/0x190
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c0ce60>] ? target_bulk_timeout+0x0/0xb0 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:02:33 oak-io2-s1 kernel: 
Feb 23 23:02:33 oak-io2-s1 kernel: Pid: 184234, comm: ll_ost_io01_066
Feb 23 23:02:33 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc10f4425>] osd_trans_stop+0x205/0x830 [osd_ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0ff4f44>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc122425f>] ofd_trans_stop+0x1f/0x60 [ofd]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc122a484>] ofd_commitrw_write+0x7e4/0x1c50 [ofd]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc122e5a9>] ofd_commitrw+0x4b9/0xac0 [ofd]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0ce7eb7>] obd_commitrw+0x2ed/0x330 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0cbad91>] tgt_brw_write+0xff1/0x17c0 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810cd794>] ? update_curr+0x104/0x190
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c0ce60>] ? target_bulk_timeout+0x0/0xb0 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:02:33 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:02:33 oak-io2-s1 kernel: 
Feb 23 23:02:33 oak-io2-s1 kernel: LNet: Service thread pid 316476 was inactive for 200.14s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 23 23:02:34 oak-io2-s1 kernel: LNet: Service thread pid 251691 was inactive for 200.51s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 23 23:02:34 oak-io2-s1 kernel: LNet: Skipped 2 previous similar messages
Feb 23 23:02:34 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455754.251691
Feb 23 23:02:35 oak-io2-s1 kernel: LNet: Service thread pid 40338 was inactive for 200.14s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 23 23:02:35 oak-io2-s1 kernel: LNet: Skipped 36 previous similar messages
Feb 23 23:02:35 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455755.40338
Feb 23 23:02:36 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455756.301765
Feb 23 23:02:37 oak-io2-s1 kernel: LNet: Service thread pid 322467 was inactive for 200.56s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 23 23:02:37 oak-io2-s1 kernel: LNet: Skipped 70 previous similar messages
Feb 23 23:02:37 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455757.322467
Feb 23 23:02:38 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455758.39072
Feb 23 23:02:39 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455759.183386
Feb 23 23:02:40 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455760.316480
Feb 23 23:02:41 oak-io2-s1 kernel: LNet: Service thread pid 185054 was inactive for 200.24s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 23 23:02:41 oak-io2-s1 kernel: LNet: Skipped 60 previous similar messages
Feb 23 23:02:41 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455761.185054
Feb 23 23:02:43 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455763.184441
Feb 23 23:02:44 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455764.188984
Feb 23 23:02:45 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455765.63777
Feb 23 23:02:47 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455767.362684
Feb 23 23:02:52 oak-io2-s1 kernel: LNet: Service thread pid 249482 was inactive for 200.48s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 23 23:02:52 oak-io2-s1 kernel: LNet: Skipped 7 previous similar messages
Feb 23 23:02:52 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455772.249482
Feb 23 23:02:53 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455773.251662
Feb 23 23:02:54 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455774.301769
Feb 23 23:02:55 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455775.190094
Feb 23 23:03:02 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455782.251692
Feb 23 23:03:03 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455783.40335
Feb 23 23:03:05 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455785.362697
Feb 23 23:03:11 oak-io2-s1 kernel: LNet: Service thread pid 251688 was inactive for 200.47s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 23 23:03:11 oak-io2-s1 kernel: LNet: Skipped 13 previous similar messages
Feb 23 23:03:11 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455791.251688
Feb 23 23:03:18 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455798.210331
Feb 23 23:03:26 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455806.6922
Feb 23 23:03:44 oak-io2-s1 kernel: LNet: Service thread pid 137164 was inactive for 200.73s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 23 23:03:44 oak-io2-s1 kernel: LNet: Skipped 5 previous similar messages
Feb 23 23:03:44 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455824.137164
Feb 23 23:03:51 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455831.249478
Feb 23 23:04:22 oak-io2-s1 kernel: Lustre: 137164:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519455855/real 1519455855]  req@ffff88004b3ed400 x1592481990801536/t0(0) o106->oak-OST0034@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519455862 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 23:04:22 oak-io2-s1 kernel: Lustre: 137164:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
Feb 23 23:04:26 oak-io2-s1 kernel: LustreError: 362684:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519455566, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST0030_UUID lock: ffff88005db96200/0x806f959362a6eaba lrc: 3/0,1 mode: --/PW res: [0x443f8c:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 362684 timeout: 0 lvb_type: 0
Feb 23 23:04:26 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455866.362684
Feb 23 23:04:31 oak-io2-s1 kernel: LustreError: 21426:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519455571, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST0030_UUID lock: ffff88002200c200/0x806f959362a6f579 lrc: 3/0,1 mode: --/PW res: [0x443f8d:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 21426 timeout: 0 lvb_type: 0
Feb 23 23:04:45 oak-io2-s1 kernel: LustreError: 362697:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519455585, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST0032_UUID lock: ffff8817a30da200/0x806f959362a702b5 lrc: 3/0,1 mode: --/PW res: [0x4386ad:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 362697 timeout: 0 lvb_type: 0
Feb 23 23:04:57 oak-io2-s1 kernel: LNet: Service thread pid 132011 was inactive for 238.53s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 23 23:04:57 oak-io2-s1 kernel: LNet: Skipped 1 previous similar message
Feb 23 23:04:57 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455897.132011
Feb 23 23:04:57 oak-io2-s1 kernel: LustreError: 210328:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519455597, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST0038_UUID lock: ffff880022b18800/0x806f959362a70563 lrc: 3/0,1 mode: --/PW res: [0x42f8b9:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 210328 timeout: 0 lvb_type: 0
Feb 23 23:04:57 oak-io2-s1 kernel: LustreError: 210328:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) Skipped 1 previous similar message
Feb 23 23:04:59 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455899.210344
Feb 23 23:05:05 oak-io2-s1 kernel: LustreError: 6922:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519455605, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST0030_UUID lock: ffff88007200ba00/0x806f959362a70873 lrc: 3/0,1 mode: --/PW res: [0x443f90:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 6922 timeout: 0 lvb_type: 0
Feb 23 23:05:05 oak-io2-s1 kernel: LustreError: 6922:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) Skipped 1 previous similar message
Feb 23 23:05:58 oak-io2-s1 kernel: LustreError: 132011:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519455658, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST0041_UUID lock: ffff8817a30de000/0x806f959362a70b91 lrc: 3/0,1 mode: --/PW res: [0x43c75f:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 132011 timeout: 0 lvb_type: 0
Feb 23 23:06:18 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519455978.249476
Feb 23 23:07:02 oak-io2-s1 kernel: LustreError: 132969:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519455722, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST0045_UUID lock: ffff88004c832200/0x806f959362a70e54 lrc: 3/0,1 mode: --/PW res: [0x437798:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 132969 timeout: 0 lvb_type: 0
Feb 23 23:07:41 oak-io2-s1 kernel: LNet: Service thread pid 132969 was inactive for 338.37s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 23 23:07:41 oak-io2-s1 kernel: LNet: Skipped 4 previous similar messages
Feb 23 23:07:41 oak-io2-s1 kernel: Pid: 132969, comm: ll_ost01_011
Feb 23 23:07:41 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffffc0c1e020>] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc]
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffffc0c1e89d>] ldlm_completion_ast+0x63d/0x920 [ptlrpc]
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffffc0c20730>] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc]
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffffc1220b2c>] ofd_destroy_by_fid+0x1cc/0x4a0 [ofd]
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffffc1213307>] ofd_destroy_hdl+0x267/0x970 [ofd]
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:07:41 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:07:41 oak-io2-s1 kernel: 
Feb 23 23:07:41 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519456061.132969
Feb 23 23:08:58 oak-io2-s1 kernel: LNet: Service thread pid 249485 was inactive for 386.56s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 23 23:08:58 oak-io2-s1 kernel: Pid: 249485, comm: ll_ost00_066
Feb 23 23:08:58 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:08:58 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:08:58 oak-io2-s1 kernel: 
Feb 23 23:08:58 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519456138.249485
Feb 23 23:09:07 oak-io2-s1 kernel: Lustre: 39067:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff8814d92c2c50 x1593152211412416/t0(0) o4->664a7c90-84ba-06c2-9654-f9c1cbd5a207@10.8.2.15@o2ib6:532/0 lens 608/448 e 23 to 0 dl 1519456152 ref 2 fl Interpret:H/0/0 rc 0/0
Feb 23 23:09:07 oak-io2-s1 kernel: Lustre: 360834:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff883cad2a9050 x1593220766694640/t4295755399(0) o4->f1347269-cc7b-9c70-cadb-2cc85f7a8a70@10.9.114.10@o2ib4:532/0 lens 608/448 e 23 to 0 dl 1519456152 ref 2 fl Interpret:/0/0 rc 0/0
Feb 23 23:09:07 oak-io2-s1 kernel: Lustre: 360834:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages
Feb 23 23:09:08 oak-io2-s1 kernel: Lustre: 360834:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff883cb8925050 x1593152416260592/t0(0) o4->0d0e28ee-47ab-6a9b-a21b-20d577530272@10.9.102.32@o2ib4:533/0 lens 608/448 e 23 to 0 dl 1519456153 ref 2 fl Interpret:/0/0 rc 0/0
Feb 23 23:09:08 oak-io2-s1 kernel: Lustre: 360834:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 19 previous similar messages
Feb 23 23:09:10 oak-io2-s1 kernel: Lustre: 360834:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff8831c345f450 x1593214014939440/t0(0) o3->3326eb74-fc4c-e39b-1783-8c55bbb22498@10.9.112.6@o2ib4:535/0 lens 608/432 e 23 to 0 dl 1519456155 ref 2 fl Interpret:/0/0 rc 0/0
Feb 23 23:09:10 oak-io2-s1 kernel: Lustre: 360834:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 55 previous similar messages
Feb 23 23:09:13 oak-io2-s1 kernel: Lustre: oak-OST0042: Client 221c25d8-1036-b285-2495-b7639654d19c (at 10.9.102.17@o2ib4) reconnecting
Feb 23 23:09:13 oak-io2-s1 kernel: Lustre: oak-OST0042: Connection restored to 13de3f5f-65b7-a3ba-803f-cddc50d5eb14 (at 10.9.102.17@o2ib4)
Feb 23 23:09:13 oak-io2-s1 kernel: Lustre: Skipped 17 previous similar messages
Feb 23 23:09:13 oak-io2-s1 kernel: Lustre: oak-OST0042: Client 664a7c90-84ba-06c2-9654-f9c1cbd5a207 (at 10.8.2.15@o2ib6) reconnecting
Feb 23 23:09:14 oak-io2-s1 kernel: Lustre: 184236:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff883cad260450 x1593126948529424/t0(0) o3->0dc7dd6f-f224-4477-c533-f0c7ba30d61b@10.9.104.17@o2ib4:539/0 lens 608/432 e 23 to 0 dl 1519456159 ref 2 fl Interpret:/0/0 rc 0/0
Feb 23 23:09:14 oak-io2-s1 kernel: Lustre: 184236:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 91 previous similar messages
Feb 23 23:09:15 oak-io2-s1 kernel: Lustre: oak-OST0034: Client 6c5035e3-5c55-5738-2ecd-8ce793ddb381 (at 10.9.102.21@o2ib4) reconnecting
Feb 23 23:09:15 oak-io2-s1 kernel: Lustre: Skipped 15 previous similar messages
Feb 23 23:09:17 oak-io2-s1 kernel: Lustre: oak-OST0032: Client fadeddec-8594-ff20-3bec-7ad4bb33bece (at 10.8.18.23@o2ib6) reconnecting
Feb 23 23:09:17 oak-io2-s1 kernel: Lustre: Skipped 25 previous similar messages
Feb 23 23:09:22 oak-io2-s1 kernel: Lustre: oak-OST0042: Client oak-MDT0000-mdtlov_UUID (at 10.0.2.52@o2ib5) reconnecting
Feb 23 23:09:22 oak-io2-s1 kernel: Lustre: Skipped 8 previous similar messages
Feb 23 23:09:22 oak-io2-s1 kernel: Lustre: oak-OST0042: deleting orphan objects from 0x0:4333563 to 0x0:4333601
Feb 23 23:09:23 oak-io2-s1 kernel: Lustre: 137164:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1519456156/real 1519456156]  req@ffff88004b3ed400 x1592481990801536/t0(0) o106->oak-OST0034@10.9.112.15@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1519456163 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Feb 23 23:09:23 oak-io2-s1 kernel: Lustre: 137164:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 42 previous similar messages
Feb 23 23:09:26 oak-io2-s1 kernel: Lustre: 210320:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff8839e383a050 x1593042641062864/t0(0) o6->oak-MDT0000-mdtlov_UUID@10.0.2.52@o2ib5:551/0 lens 664/432 e 23 to 0 dl 1519456171 ref 2 fl Interpret:/0/0 rc 0/0
Feb 23 23:09:26 oak-io2-s1 kernel: Lustre: 210320:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 13 previous similar messages
Feb 23 23:09:27 oak-io2-s1 kernel: Lustre: oak-OST0030: deleting orphan objects from 0x0:4472721 to 0x0:4472737
Feb 23 23:09:32 oak-io2-s1 kernel: Lustre: oak-OST0030: Client cd9072f9-71b4-1cfb-a759-fd6823f1a4a9 (at 10.0.2.3@o2ib5) reconnecting
Feb 23 23:09:32 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 23 23:09:42 oak-io2-s1 kernel: Lustre: 39067:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff88073a8d4c50 x1593126948621472/t0(0) o3->0dc7dd6f-f224-4477-c533-f0c7ba30d61b@10.9.104.17@o2ib4:567/0 lens 608/0 e 12 to 0 dl 1519456187 ref 2 fl New:/0/ffffffff rc 0/-1
Feb 23 23:09:42 oak-io2-s1 kernel: Lustre: 39067:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 142 previous similar messages
Feb 23 23:09:46 oak-io2-s1 kernel: Lustre: oak-OST0032: deleting orphan objects from 0x0:4425393 to 0x0:4425409
Feb 23 23:09:49 oak-io2-s1 kernel: Lustre: oak-OST003f: Client 1a8fdd02-e840-cd78-d015-ebdfc48eadda (at 10.9.102.48@o2ib4) reconnecting
Feb 23 23:09:49 oak-io2-s1 kernel: Lustre: Skipped 88 previous similar messages
Feb 23 23:09:53 oak-io2-s1 kernel: LustreError: 362691:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519455893, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST0035_UUID lock: ffff88005e575400/0x806f959362a712de lrc: 3/0,1 mode: --/PW res: [0x428644:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 362691 timeout: 0 lvb_type: 0
Feb 23 23:09:58 oak-io2-s1 kernel: Lustre: oak-OST0038: deleting orphan objects from 0x0:4389051 to 0x0:4389089
Feb 23 23:10:06 oak-io2-s1 kernel: Lustre: oak-OST004d: deleting orphan objects from 0x0:3465762 to 0x0:3465793
Feb 23 23:10:14 oak-io2-s1 kernel: Lustre: 39067:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff8812f8306850 x1593152182170096/t0(0) o4->385ec7a6-34dc-1efa-0d7b-713ce9bf54e6@10.9.101.65@o2ib4:599/0 lens 624/0 e 7 to 0 dl 1519456219 ref 2 fl New:/0/ffffffff rc 0/-1
Feb 23 23:10:14 oak-io2-s1 kernel: Lustre: 39067:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 569 previous similar messages
Feb 23 23:10:21 oak-io2-s1 kernel: Lustre: oak-OST0044: Client aa35134f-8565-3554-fe57-f6d38e77256e (at 10.8.18.21@o2ib6) reconnecting
Feb 23 23:10:21 oak-io2-s1 kernel: Lustre: Skipped 161 previous similar messages
Feb 23 23:10:59 oak-io2-s1 kernel: Lustre: oak-OST0041: deleting orphan objects from 0x0:4441952 to 0x0:4441985
Feb 23 23:11:26 oak-io2-s1 kernel: Lustre: 174518:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff8806eadcbc50 x1591002486533584/t0(0) o19->cd9072f9-71b4-1cfb-a759-fd6823f1a4a9@10.0.2.3@o2ib5:671/0 lens 336/336 e 4 to 0 dl 1519456291 ref 2 fl Interpret:/0/0 rc 0/0
Feb 23 23:11:26 oak-io2-s1 kernel: Lustre: 174518:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 24 previous similar messages
Feb 23 23:11:38 oak-io2-s1 kernel: Lustre: oak-OST0035: Client 98f13d8f-bb6e-bfe1-8000-a39226051fc5 (at 10.8.18.34@o2ib6) reconnecting
Feb 23 23:11:38 oak-io2-s1 kernel: Lustre: Skipped 9 previous similar messages
Feb 23 23:12:03 oak-io2-s1 kernel: Lustre: oak-OST0045: deleting orphan objects from 0x0:4421529 to 0x0:4421569
Feb 23 23:12:29 oak-io2-s1 kernel: LNet: Service thread pid 249332 was inactive for 537.46s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 23 23:12:29 oak-io2-s1 kernel: Pid: 249332, comm: ll_ost00_045
Feb 23 23:12:29 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:12:29 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:12:29 oak-io2-s1 kernel: 
Feb 23 23:12:29 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519456349.249332
Feb 23 23:13:41 oak-io2-s1 kernel: Lustre: 39067:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff881d02798c50 x1591169517243856/t0(0) o4->a7b231a5-a7d7-2f9d-4b5d-464beea873b1@10.8.0.63@o2ib6:51/0 lens 608/0 e 3 to 0 dl 1519456426 ref 2 fl New:/0/ffffffff rc 0/-1
Feb 23 23:13:41 oak-io2-s1 kernel: Lustre: 39067:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 7 previous similar messages
Feb 23 23:13:48 oak-io2-s1 kernel: Lustre: oak-OST003e: Client a7b231a5-a7d7-2f9d-4b5d-464beea873b1 (at 10.8.0.63@o2ib6) reconnecting
Feb 23 23:13:48 oak-io2-s1 kernel: Lustre: Skipped 4 previous similar messages
Feb 23 23:13:59 oak-io2-s1 kernel: LustreError: 210343:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519456139, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST0049_UUID lock: ffff880050855000/0x806f959362a71347 lrc: 3/0,1 mode: --/PW res: [0x34c6f1:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 210343 timeout: 0 lvb_type: 0
Feb 23 23:14:54 oak-io2-s1 kernel: Lustre: oak-OST0035: deleting orphan objects from 0x0:4361536 to 0x0:4361570
Feb 23 23:15:08 oak-io2-s1 kernel: LNet: Service thread pid 132016 was inactive for 637.26s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 23 23:15:08 oak-io2-s1 kernel: Pid: 132016, comm: ll_ost00_005
Feb 23 23:15:08 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffffc097ef56>] ? lnet_select_pathway+0x4d6/0x1150 [lnet]
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:15:08 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:15:08 oak-io2-s1 kernel: 
Feb 23 23:15:08 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519456508.132016
Feb 23 23:16:22 oak-io2-s1 kernel: LNet: Service thread pid 362691 was inactive for 688.96s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 23 23:16:22 oak-io2-s1 kernel: Pid: 362691, comm: ll_ost01_082
Feb 23 23:16:22 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffffc0c1e020>] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc]
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffffc0c1e89d>] ldlm_completion_ast+0x63d/0x920 [ptlrpc]
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffffc0c20730>] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc]
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffffc1220b2c>] ofd_destroy_by_fid+0x1cc/0x4a0 [ofd]
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffffc1213307>] ofd_destroy_hdl+0x267/0x970 [ofd]
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:16:22 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:16:22 oak-io2-s1 kernel: 
Feb 23 23:16:22 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519456582.362691
Feb 23 23:17:47 oak-io2-s1 kernel: LustreError: 137164:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.112.15@o2ib4) returned error from glimpse AST (req@ffff88004b3ed400 x1592481990801536 status -107 rc -107), evict it ns: filter-oak-OST0034_UUID lock: ffff883a8366ae00/0x806f9593628e8f8f lrc: 4/0,0 mode: PW/PW res: [0x3d20d7:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40000080020000 nid: 10.9.112.15@o2ib4 remote: 0x47ee55642915aecc expref: 5 pid: 4910 timeout: 0 lvb_type: 0
Feb 23 23:17:47 oak-io2-s1 kernel: LustreError: 137164:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) Skipped 2 previous similar messages
Feb 23 23:17:47 oak-io2-s1 kernel: LustreError: 138-a: oak-OST0034: A client on nid 10.9.112.15@o2ib4 was evicted due to a lock glimpse callback time out: rc -107
Feb 23 23:17:47 oak-io2-s1 kernel: LustreError: Skipped 3 previous similar messages
Feb 23 23:17:47 oak-io2-s1 kernel: Lustre: 137164:0:(service.c:2112:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:443s); client may timeout.  req@ffff881534771850 x1591002486520240/t0(0) o101->cd9072f9-71b4-1cfb-a759-fd6823f1a4a9@10.0.2.3@o2ib5:604/0 lens 328/368 e 7 to 0 dl 1519456224 ref 1 fl Complete:/0/0 rc 301/301
Feb 23 23:17:47 oak-io2-s1 kernel: LNet: Service thread pid 137164 completed after 1043.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Feb 23 23:17:48 oak-io2-s1 kernel: LNet: Service thread pid 249492 was inactive for 737.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 23 23:17:48 oak-io2-s1 kernel: Pid: 249492, comm: ll_ost00_073
Feb 23 23:17:48 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:17:48 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:17:48 oak-io2-s1 kernel: 
Feb 23 23:17:48 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519456668.249492
Feb 23 23:18:33 oak-io2-s1 kernel: Lustre: oak-OST0030: Export ffff8801d6e14c00 already connecting from 10.9.112.15@o2ib4
Feb 23 23:18:33 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
Feb 23 23:18:39 oak-io2-s1 kernel: Lustre: 39067:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff881d0279fc50 x1591169520451984/t0(0) o4->a7b231a5-a7d7-2f9d-4b5d-464beea873b1@10.8.0.63@o2ib6:349/0 lens 608/0 e 1 to 0 dl 1519456724 ref 2 fl New:/0/ffffffff rc 0/-1
Feb 23 23:18:39 oak-io2-s1 kernel: Lustre: 39067:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages
Feb 23 23:18:46 oak-io2-s1 kernel: Lustre: oak-OST0052: Client a7b231a5-a7d7-2f9d-4b5d-464beea873b1 (at 10.8.0.63@o2ib6) reconnecting
Feb 23 23:18:46 oak-io2-s1 kernel: Lustre: Skipped 9 previous similar messages
Feb 23 23:19:00 oak-io2-s1 kernel: Lustre: oak-OST0049: deleting orphan objects from 0x0:3458802 to 0x0:3458817
Feb 23 23:19:14 oak-io2-s1 kernel: Lustre: oak-OST0042: Connection restored to 9f3e786c-91bd-0b59-f474-23ccea62b4f0 (at 10.8.2.15@o2ib6)
Feb 23 23:19:14 oak-io2-s1 kernel: Lustre: Skipped 364 previous similar messages
Feb 23 23:19:23 oak-io2-s1 kernel: Lustre: oak-OST0030: Export ffff8801d6e14c00 already connecting from 10.9.112.15@o2ib4
Feb 23 23:19:23 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 23 23:19:57 oak-io2-s1 kernel: LustreError: 21429:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519456497, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST003f_UUID lock: ffff880047362000/0x806f959362a717d1 lrc: 3/0,1 mode: --/PW res: [0x4388fd:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 21429 timeout: 0 lvb_type: 0
Feb 23 23:20:07 oak-io2-s1 kernel: Lustre: oak-OST004d: deleting orphan objects from 0x0:3465762 to 0x0:3465825
Feb 23 23:20:13 oak-io2-s1 kernel: Lustre: oak-OST0030: Export ffff8801d6e14c00 already connecting from 10.9.112.15@o2ib4
Feb 23 23:20:13 oak-io2-s1 kernel: Lustre: oak-OST0032: Export ffff883d13fdc400 already connecting from 10.9.112.15@o2ib4
Feb 23 23:20:13 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 23 23:21:00 oak-io2-s1 kernel: Lustre: oak-OST0041: deleting orphan objects from 0x0:4441952 to 0x0:4442017
Feb 23 23:21:19 oak-io2-s1 kernel: LNet: Service thread pid 390779 was inactive for 887.88s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 23 23:21:19 oak-io2-s1 kernel: Pid: 390779, comm: ll_ost00_082
Feb 23 23:21:19 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:21:19 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:21:19 oak-io2-s1 kernel: 
Feb 23 23:21:19 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519456879.390779
Feb 23 23:22:04 oak-io2-s1 kernel: Lustre: oak-OST0045: deleting orphan objects from 0x0:4421529 to 0x0:4421601
Feb 23 23:24:01 oak-io2-s1 kernel: LNet: Service thread pid 174522 was inactive for 989.73s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 23 23:24:01 oak-io2-s1 kernel: Pid: 174522, comm: ll_ost00_115
Feb 23 23:24:01 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:24:01 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:24:01 oak-io2-s1 kernel: 
Feb 23 23:24:01 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457041.174522
Feb 23 23:24:30 oak-io2-s1 kernel: LustreError: 131985:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.112.15@o2ib4) returned error from glimpse AST (req@ffff881d2a687b00 x1592481990838992 status -107 rc -107), evict it ns: filter-oak-OST0034_UUID lock: ffff8835a9404600/0x806f959362a71a94 lrc: 4/0,0 mode: PW/PW res: [0x3d20d7:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40000080000000 nid: 10.9.112.15@o2ib4 remote: 0xe4db7b3653a7cbbf expref: 6 pid: 296683 timeout: 0 lvb_type: 0
Feb 23 23:24:30 oak-io2-s1 kernel: LustreError: 138-a: oak-OST0034: A client on nid 10.9.112.15@o2ib4 was evicted due to a lock glimpse callback time out: rc -107
Feb 23 23:24:35 oak-io2-s1 kernel: Lustre: oak-OST0030: Export ffff881ecaa8ac00 already connecting from 10.9.112.15@o2ib4
Feb 23 23:24:35 oak-io2-s1 kernel: Lustre: oak-OST0032: Export ffff880526fd6400 already connecting from 10.9.112.15@o2ib4
Feb 23 23:24:35 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
Feb 23 23:24:55 oak-io2-s1 kernel: Lustre: oak-OST0035: deleting orphan objects from 0x0:4361572 to 0x0:4361602
Feb 23 23:24:58 oak-io2-s1 kernel: Lustre: oak-OST003f: deleting orphan objects from 0x0:4425982 to 0x0:4426020
Feb 23 23:24:59 oak-io2-s1 kernel: LustreError: 6918:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519456799, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST0035_UUID lock: ffff8800480e6e00/0x806f959362a71c54 lrc: 3/0,1 mode: --/PW res: [0x428d63:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 6918 timeout: 0 lvb_type: 0
Feb 23 23:24:59 oak-io2-s1 kernel: LustreError: 6918:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) Skipped 2 previous similar messages
Feb 23 23:25:25 oak-io2-s1 kernel: Lustre: oak-OST0030: Export ffff881ecaa8ac00 already connecting from 10.9.112.15@o2ib4
Feb 23 23:25:25 oak-io2-s1 kernel: Lustre: Skipped 4 previous similar messages
Feb 23 23:26:04 oak-io2-s1 kernel: Lustre: oak-OST0039: deleting orphan objects from 0x0:4347818 to 0x0:4347841
Feb 23 23:26:15 oak-io2-s1 kernel: Lustre: oak-OST0030: Export ffff881ecaa8ac00 already connecting from 10.9.112.15@o2ib4
Feb 23 23:26:15 oak-io2-s1 kernel: Lustre: Skipped 4 previous similar messages
Feb 23 23:27:00 oak-io2-s1 kernel: Lustre: oak-OST004f: deleting orphan objects from 0x0:175045 to 0x0:175074
Feb 23 23:27:30 oak-io2-s1 kernel: LNet: Service thread pid 131981 was inactive for 1138.63s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 23 23:27:30 oak-io2-s1 kernel: Pid: 131981, comm: ll_ost00_003
Feb 23 23:27:30 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:27:30 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:27:30 oak-io2-s1 kernel: 
Feb 23 23:27:30 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457250.131981
Feb 23 23:27:47 oak-io2-s1 kernel: Lustre: 39067:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff881ebd332450 x1593265787771344/t0(0) o10->f0e3dca9-4f94-7859-aae2-78241dcced78@10.9.112.15@o2ib4:142/0 lens 560/0 e 1 to 0 dl 1519457272 ref 2 fl New:/0/ffffffff rc 0/-1
Feb 23 23:27:47 oak-io2-s1 kernel: Lustre: 39067:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 967 previous similar messages
Feb 23 23:28:04 oak-io2-s1 kernel: Lustre: oak-OST0035: Client 3ee95ef7-4278-ead7-52a3-bdca1c47a323 (at 10.9.112.3@o2ib4) reconnecting
Feb 23 23:28:04 oak-io2-s1 kernel: Lustre: Skipped 335 previous similar messages
Feb 23 23:28:52 oak-io2-s1 kernel: Pid: 210343, comm: ll_ost01_070
Feb 23 23:28:52 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffffc0c1e020>] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc]
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffffc0c1e89d>] ldlm_completion_ast+0x63d/0x920 [ptlrpc]
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffffc0c20730>] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc]
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffffc1220b2c>] ofd_destroy_by_fid+0x1cc/0x4a0 [ofd]
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffffc1213307>] ofd_destroy_hdl+0x267/0x970 [ofd]
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:28:52 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:28:52 oak-io2-s1 kernel: 
Feb 23 23:28:52 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457332.210343
Feb 23 23:29:04 oak-io2-s1 kernel: Lustre: oak-OST0049: deleting orphan objects from 0x0:3458802 to 0x0:3458849
Feb 23 23:29:15 oak-io2-s1 kernel: Lustre: oak-OST0042: Connection restored to 9f3e786c-91bd-0b59-f474-23ccea62b4f0 (at 10.8.2.15@o2ib6)
Feb 23 23:29:15 oak-io2-s1 kernel: Lustre: Skipped 368 previous similar messages
Feb 23 23:29:16 oak-io2-s1 kernel: Pid: 184438, comm: ll_ost_io01_073
Feb 23 23:29:16 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a7610>] ? bit_wait_io+0x0/0x50
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a6ff9>] schedule_timeout+0x239/0x2c0
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0547740>] ? mlx4_ib_post_send+0x510/0xb50 [mlx4_ib]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a7610>] ? bit_wait_io+0x0/0x50
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a8b6d>] io_schedule_timeout+0xad/0x130
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a8c08>] io_schedule+0x18/0x20
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a7621>] bit_wait_io+0x11/0x50
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a733f>] __wait_on_bit_lock+0x5f/0xc0
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff81182094>] __lock_page+0x74/0x90
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810b19d0>] ? wake_bit_function_rh+0x0/0x40
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff81182a34>] __find_lock_page+0x54/0x70
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff81183614>] find_or_create_page+0x34/0xa0
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc110a235>] osd_bufs_get+0x235/0x430 [osd_ldiskfs]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc122d648>] ofd_preprw+0x6a8/0x1150 [ofd]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0c795ad>] ? __req_capsule_get+0x15d/0x700 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0c8bddd>] ? null_alloc_rs+0x15d/0x330 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0cb8ebf>] tgt_brw_read+0x96f/0x1850 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0a0be69>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0c8bdf6>] ? null_alloc_rs+0x176/0x330 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0c53f2f>] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:29:16 oak-io2-s1 kernel: 
Feb 23 23:29:16 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457356.184438
Feb 23 23:29:16 oak-io2-s1 kernel: Pid: 184435, comm: ll_ost_io01_070
Feb 23 23:29:16 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a7610>] ? bit_wait_io+0x0/0x50
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a6ff9>] schedule_timeout+0x239/0x2c0
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0547740>] ? mlx4_ib_post_send+0x510/0xb50 [mlx4_ib]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a7610>] ? bit_wait_io+0x0/0x50
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a8b6d>] io_schedule_timeout+0xad/0x130
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a8c08>] io_schedule+0x18/0x20
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a7621>] bit_wait_io+0x11/0x50
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a733f>] __wait_on_bit_lock+0x5f/0xc0
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff81182094>] __lock_page+0x74/0x90
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810b19d0>] ? wake_bit_function_rh+0x0/0x40
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff81182a34>] __find_lock_page+0x54/0x70
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff81183614>] find_or_create_page+0x34/0xa0
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc110a235>] osd_bufs_get+0x235/0x430 [osd_ldiskfs]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc122c417>] ofd_preprw_write.isra.30+0x1f7/0xd80 [ofd]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc122d3c2>] ofd_preprw+0x422/0x1150 [ofd]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0cba9d4>] tgt_brw_write+0xc34/0x17c0 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810cd794>] ? update_curr+0x104/0x190
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff811dd579>] ? ___slab_alloc+0x209/0x4f0
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a7742>] ? mutex_lock+0x12/0x2f
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:29:16 oak-io2-s1 kernel: 
Feb 23 23:29:16 oak-io2-s1 kernel: Pid: 185057, comm: ll_ost_io01_093
Feb 23 23:29:16 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a7610>] ? bit_wait_io+0x0/0x50
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a6ff9>] schedule_timeout+0x239/0x2c0
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0547740>] ? mlx4_ib_post_send+0x510/0xb50 [mlx4_ib]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810e939c>] ? ktime_get_ts64+0x4c/0xf0
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a7610>] ? bit_wait_io+0x0/0x50
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a8b6d>] io_schedule_timeout+0xad/0x130
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a8c08>] io_schedule+0x18/0x20
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a7621>] bit_wait_io+0x11/0x50
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a733f>] __wait_on_bit_lock+0x5f/0xc0
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff81182094>] __lock_page+0x74/0x90
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810b19d0>] ? wake_bit_function_rh+0x0/0x40
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff81182a34>] __find_lock_page+0x54/0x70
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff81183614>] find_or_create_page+0x34/0xa0
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc110a235>] osd_bufs_get+0x235/0x430 [osd_ldiskfs]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc122c417>] ofd_preprw_write.isra.30+0x1f7/0xd80 [ofd]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc122d3c2>] ofd_preprw+0x422/0x1150 [ofd]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0cba9d4>] tgt_brw_write+0xc34/0x17c0 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810cd794>] ? update_curr+0x104/0x190
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816a7742>] ? mutex_lock+0x12/0x2f
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:29:16 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:29:16 oak-io2-s1 kernel: 
Feb 23 23:29:16 oak-io2-s1 kernel: LNet: Service thread pid 184443 was inactive for 1201.37s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 23 23:29:16 oak-io2-s1 kernel: LNet: Skipped 2 previous similar messages
Feb 23 23:29:20 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457360.301771
Feb 23 23:29:24 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457364.210332
Feb 23 23:29:29 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457369.21430
Feb 23 23:29:33 oak-io2-s1 kernel: LNet: Service thread pid 135933 was inactive for 1200.23s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 23 23:29:33 oak-io2-s1 kernel: LNet: Skipped 48 previous similar messages
Feb 23 23:29:33 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457373.135933
Feb 23 23:29:49 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457389.131960
Feb 23 23:30:01 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457401.6919
Feb 23 23:30:08 oak-io2-s1 kernel: Lustre: oak-OST004d: deleting orphan objects from 0x0:3465827 to 0x0:3465857
Feb 23 23:30:34 oak-io2-s1 kernel: LNet: Service thread pid 249471 was inactive for 1201.85s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 23 23:30:34 oak-io2-s1 kernel: LNet: Skipped 3 previous similar messages
Feb 23 23:30:34 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457434.249471
Feb 23 23:30:35 oak-io2-s1 kernel: Lustre: oak-OST0032: Export ffff883ca70bd800 already connecting from 10.9.112.15@o2ib4
Feb 23 23:30:35 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
Feb 23 23:31:01 oak-io2-s1 kernel: Lustre: oak-OST0041: deleting orphan objects from 0x0:4441952 to 0x0:4442049
Feb 23 23:31:35 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457495.174525
Feb 23 23:32:05 oak-io2-s1 kernel: Lustre: oak-OST0045: deleting orphan objects from 0x0:4421529 to 0x0:4421633
Feb 23 23:32:23 oak-io2-s1 kernel: Lustre: oak-OST004e: deleting orphan objects from 0x0:176029 to 0x0:176065
Feb 23 23:32:23 oak-io2-s1 kernel: Lustre: oak-OST0034: deleting orphan objects from 0x0:4290565 to 0x0:4290593
Feb 23 23:32:33 oak-io2-s1 kernel: LNet: Service thread pid 141214 was inactive for 1201.72s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 23 23:32:33 oak-io2-s1 kernel: LNet: Skipped 4 previous similar messages
Feb 23 23:32:33 oak-io2-s1 kernel: Pid: 141214, comm: ll_ost00_017
Feb 23 23:32:33 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:32:33 oak-io2-s1 kernel: 
Feb 23 23:32:33 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457553.141214
Feb 23 23:32:33 oak-io2-s1 kernel: Pid: 131957, comm: ll_ost00_000
Feb 23 23:32:33 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:32:33 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:32:33 oak-io2-s1 kernel: 
Feb 23 23:33:34 oak-io2-s1 kernel: Pid: 174518, comm: ll_ost00_111
Feb 23 23:33:34 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:33:34 oak-io2-s1 kernel: 
Feb 23 23:33:34 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457614.174518
Feb 23 23:33:34 oak-io2-s1 kernel: Pid: 249487, comm: ll_ost00_068
Feb 23 23:33:34 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:33:34 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:33:34 oak-io2-s1 kernel: 
Feb 23 23:34:36 oak-io2-s1 kernel: Pid: 174527, comm: ll_ost00_119
Feb 23 23:34:36 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:34:36 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:34:36 oak-io2-s1 kernel: 
Feb 23 23:34:36 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457676.174527
Feb 23 23:34:36 oak-io2-s1 kernel: LNet: Service thread pid 390793 was inactive for 1204.55s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 23 23:34:36 oak-io2-s1 kernel: LNet: Skipped 3 previous similar messages
Feb 23 23:34:56 oak-io2-s1 kernel: Lustre: oak-OST0035: deleting orphan objects from 0x0:4361572 to 0x0:4361634
Feb 23 23:34:59 oak-io2-s1 kernel: Lustre: oak-OST003f: deleting orphan objects from 0x0:4425982 to 0x0:4426052
Feb 23 23:35:00 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457700.21429
Feb 23 23:35:33 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457733.17221
Feb 23 23:36:02 oak-io2-s1 kernel: Lustre: oak-OST0031: deleting orphan objects from 0x0:4401615 to 0x0:4401633
Feb 23 23:36:05 oak-io2-s1 kernel: Lustre: oak-OST0039: deleting orphan objects from 0x0:4347818 to 0x0:4347873
Feb 23 23:36:06 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457766.21425
Feb 23 23:36:06 oak-io2-s1 kernel: LustreError: 6923:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519457466, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST004c_UUID lock: ffff880019c05e00/0x806f959362a72cd1 lrc: 3/0,1 mode: --/PW res: [0x34bb90:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 6923 timeout: 0 lvb_type: 0
Feb 23 23:36:06 oak-io2-s1 kernel: LustreError: 6923:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) Skipped 4 previous similar messages
Feb 23 23:36:34 oak-io2-s1 kernel: Lustre: oak-OST0030: Export ffff883f7ce53400 already connecting from 10.9.112.15@o2ib4
Feb 23 23:36:34 oak-io2-s1 kernel: Lustre: Skipped 11 previous similar messages
Feb 23 23:36:35 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457795.133118
Feb 23 23:37:01 oak-io2-s1 kernel: Lustre: oak-OST004f: deleting orphan objects from 0x0:175045 to 0x0:175106
Feb 23 23:37:03 oak-io2-s1 kernel: LNet: Service thread pid 136078 was inactive for 1203.96s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 23 23:37:03 oak-io2-s1 kernel: LNet: Skipped 6 previous similar messages
Feb 23 23:37:03 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457823.136078
Feb 23 23:37:36 oak-io2-s1 kernel: Pid: 174521, comm: ll_ost00_114
Feb 23 23:37:36 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:37:36 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:37:36 oak-io2-s1 kernel: 
Feb 23 23:37:36 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457856.174521
Feb 23 23:37:41 oak-io2-s1 kernel: Lustre: oak-OST003c: deleting orphan objects from 0x0:4437666 to 0x0:4437697
Feb 23 23:37:44 oak-io2-s1 kernel: Pid: 249483, comm: ll_ost00_064
Feb 23 23:37:44 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff813492e0>] ? __percpu_counter_sum+0x70/0x80
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810cdc7c>] ? dequeue_entity+0x11c/0x5d0
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810ce500>] ? dequeue_task_fair+0x3d0/0x660
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff81029557>] ? __switch_to+0xd7/0x510
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:37:44 oak-io2-s1 kernel: 
Feb 23 23:37:44 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457864.249483
Feb 23 23:37:44 oak-io2-s1 kernel: Pid: 249486, comm: ll_ost00_067
Feb 23 23:37:44 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff813492e0>] ? __percpu_counter_sum+0x70/0x80
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:37:44 oak-io2-s1 kernel: 
Feb 23 23:37:44 oak-io2-s1 kernel: Pid: 390794, comm: ll_ost00_097
Feb 23 23:37:44 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff813492e0>] ? __percpu_counter_sum+0x70/0x80
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:37:44 oak-io2-s1 kernel: 
Feb 23 23:37:44 oak-io2-s1 kernel: Pid: 174514, comm: ll_ost00_107
Feb 23 23:37:44 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff813492e0>] ? __percpu_counter_sum+0x70/0x80
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:37:44 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:37:44 oak-io2-s1 kernel: 
Feb 23 23:37:56 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457876.249495
Feb 23 23:37:57 oak-io2-s1 kernel: Lustre: 251666:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff883cbc43f450 x1592239615714592/t0(0) o4->3ee95ef7-4278-ead7-52a3-bdca1c47a323@10.9.112.3@o2ib4:752/0 lens 608/0 e 1 to 0 dl 1519457882 ref 2 fl New:H/0/ffffffff rc 0/-1
Feb 23 23:37:57 oak-io2-s1 kernel: Lustre: 251666:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 968 previous similar messages
Feb 23 23:38:05 oak-io2-s1 kernel: Lustre: oak-OST0035: Client 3ee95ef7-4278-ead7-52a3-bdca1c47a323 (at 10.9.112.3@o2ib4) reconnecting
Feb 23 23:38:05 oak-io2-s1 kernel: Lustre: Skipped 355 previous similar messages
Feb 23 23:38:33 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457913.249472
Feb 23 23:39:05 oak-io2-s1 kernel: Lustre: oak-OST0049: deleting orphan objects from 0x0:3458851 to 0x0:3458881
Feb 23 23:39:16 oak-io2-s1 kernel: Lustre: oak-OST0042: Connection restored to 9f3e786c-91bd-0b59-f474-23ccea62b4f0 (at 10.8.2.15@o2ib6)
Feb 23 23:39:16 oak-io2-s1 kernel: Lustre: Skipped 420 previous similar messages
Feb 23 23:39:35 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457975.132963
Feb 23 23:39:59 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519457999.249473
Feb 23 23:40:03 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458003.6918
Feb 23 23:40:10 oak-io2-s1 kernel: Lustre: oak-OST004d: deleting orphan objects from 0x0:3465827 to 0x0:3465889
Feb 23 23:40:32 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458032.131958
Feb 23 23:41:03 oak-io2-s1 kernel: Lustre: oak-OST0041: deleting orphan objects from 0x0:4441952 to 0x0:4442081
Feb 23 23:41:07 oak-io2-s1 kernel: Lustre: oak-OST004c: deleting orphan objects from 0x0:3455889 to 0x0:3455905
Feb 23 23:41:34 oak-io2-s1 kernel: LNet: Service thread pid 390785 was inactive for 1202.40s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 23 23:41:34 oak-io2-s1 kernel: LNet: Skipped 6 previous similar messages
Feb 23 23:41:34 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458094.390785
Feb 23 23:42:02 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458122.4898
Feb 23 23:42:06 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458126.4896
Feb 23 23:42:06 oak-io2-s1 kernel: Lustre: oak-OST0045: deleting orphan objects from 0x0:4421529 to 0x0:4421665
Feb 23 23:42:10 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458130.17222
Feb 23 23:42:23 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458143.196786
Feb 23 23:42:24 oak-io2-s1 kernel: Lustre: oak-OST0034: deleting orphan objects from 0x0:4290565 to 0x0:4290625
Feb 23 23:42:27 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458147.362707
Feb 23 23:42:35 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458155.249330
Feb 23 23:42:39 oak-io2-s1 kernel: LNet: Service thread pid 210324 was inactive for 1204.10s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 23 23:42:39 oak-io2-s1 kernel: LNet: Skipped 9 previous similar messages
Feb 23 23:42:39 oak-io2-s1 kernel: Pid: 210324, comm: ll_ost01_051
Feb 23 23:42:39 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:42:39 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 23 23:42:39 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 23 23:42:39 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 23 23:42:39 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 23 23:42:39 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 23 23:42:39 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 23 23:42:39 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 23 23:42:39 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:42:39 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:42:39 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:42:39 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:42:39 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:42:39 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:42:39 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:42:39 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:42:39 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:42:39 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:42:39 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:42:39 oak-io2-s1 kernel: 
Feb 23 23:42:39 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458159.210324
Feb 23 23:42:47 oak-io2-s1 kernel: Lustre: oak-OST0048: deleting orphan objects from 0x0:3442464 to 0x0:3442497
Feb 23 23:43:12 oak-io2-s1 kernel: Pid: 249494, comm: ll_ost00_075
Feb 23 23:43:12 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:43:12 oak-io2-s1 kernel: 
Feb 23 23:43:12 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458192.249494
Feb 23 23:43:12 oak-io2-s1 kernel: Pid: 390786, comm: ll_ost00_089
Feb 23 23:43:12 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:43:12 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:43:12 oak-io2-s1 kernel: 
Feb 23 23:43:32 oak-io2-s1 kernel: Pid: 261357, comm: ll_ost00_022
Feb 23 23:43:32 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:43:32 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:43:32 oak-io2-s1 kernel: 
Feb 23 23:43:32 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458212.261357
Feb 23 23:43:45 oak-io2-s1 kernel: Pid: 174516, comm: ll_ost00_109
Feb 23 23:43:45 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffff813492e0>] ? __percpu_counter_sum+0x70/0x80
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:43:45 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:43:45 oak-io2-s1 kernel: 
Feb 23 23:43:45 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458225.174516
Feb 23 23:44:09 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458249.390778
Feb 23 23:44:34 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458274.249468
Feb 23 23:44:57 oak-io2-s1 kernel: Lustre: oak-OST0035: deleting orphan objects from 0x0:4361572 to 0x0:4361666
Feb 23 23:45:00 oak-io2-s1 kernel: Lustre: oak-OST003f: deleting orphan objects from 0x0:4426054 to 0x0:4426084
Feb 23 23:45:11 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458311.17263
Feb 23 23:45:35 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458335.267671
Feb 23 23:46:04 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458364.4899
Feb 23 23:46:05 oak-io2-s1 kernel: Lustre: oak-OST0031: deleting orphan objects from 0x0:4401635 to 0x0:4401665
Feb 23 23:46:06 oak-io2-s1 kernel: Lustre: oak-OST0039: deleting orphan objects from 0x0:4347875 to 0x0:4347905
Feb 23 23:46:08 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458368.390790
Feb 23 23:46:12 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458372.390795
Feb 23 23:46:33 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458393.17242
Feb 23 23:46:45 oak-io2-s1 kernel: Lustre: oak-OST0037: deleting orphan objects from 0x0:4441495 to 0x0:4441538
Feb 23 23:47:02 oak-io2-s1 kernel: Lustre: oak-OST004f: deleting orphan objects from 0x0:175045 to 0x0:175138
Feb 23 23:47:09 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458429.249474
Feb 23 23:47:34 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458454.174524
Feb 23 23:47:42 oak-io2-s1 kernel: Lustre: oak-OST003c: deleting orphan objects from 0x0:4437666 to 0x0:4437729
Feb 23 23:47:42 oak-io2-s1 kernel: Pid: 133753, comm: ll_ost01_015
Feb 23 23:47:42 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffffc0c1e020>] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc]
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffffc0c1e89d>] ldlm_completion_ast+0x63d/0x920 [ptlrpc]
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffffc0c20730>] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc]
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffffc1220b2c>] ofd_destroy_by_fid+0x1cc/0x4a0 [ofd]
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffffc1213307>] ofd_destroy_hdl+0x267/0x970 [ofd]
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:47:42 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:47:42 oak-io2-s1 kernel: 
Feb 23 23:47:42 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458462.133753
Feb 23 23:47:58 oak-io2-s1 kernel: Lustre: 39067:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff881a40848c50 x1592239615714592/t0(0) o4->3ee95ef7-4278-ead7-52a3-bdca1c47a323@10.9.112.3@o2ib4:598/0 lens 608/0 e 0 to 0 dl 1519458483 ref 2 fl New:H/2/ffffffff rc 0/-1
Feb 23 23:47:58 oak-io2-s1 kernel: Lustre: 39067:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 1002 previous similar messages
Feb 23 23:48:06 oak-io2-s1 kernel: Lustre: oak-OST0035: Client 3ee95ef7-4278-ead7-52a3-bdca1c47a323 (at 10.9.112.3@o2ib4) reconnecting
Feb 23 23:48:06 oak-io2-s1 kernel: Lustre: Skipped 369 previous similar messages
Feb 23 23:48:11 oak-io2-s1 kernel: Pid: 17262, comm: ll_ost00_029
Feb 23 23:48:11 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:48:11 oak-io2-s1 kernel: 
Feb 23 23:48:11 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458491.17262
Feb 23 23:48:11 oak-io2-s1 kernel: Pid: 17272, comm: ll_ost00_039
Feb 23 23:48:11 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:48:11 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:48:11 oak-io2-s1 kernel: 
Feb 23 23:48:35 oak-io2-s1 kernel: Pid: 131959, comm: ll_ost00_002
Feb 23 23:48:35 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:48:35 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:48:35 oak-io2-s1 kernel: 
Feb 23 23:48:35 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458515.131959
Feb 23 23:48:58 oak-io2-s1 kernel: Lustre: oak-OST0050: deleting orphan objects from 0x0:175410 to 0x0:175425
Feb 23 23:49:06 oak-io2-s1 kernel: Lustre: oak-OST0049: deleting orphan objects from 0x0:3458851 to 0x0:3458913
Feb 23 23:49:08 oak-io2-s1 kernel: Pid: 174511, comm: ll_ost00_105
Feb 23 23:49:08 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:49:08 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:49:08 oak-io2-s1 kernel: 
Feb 23 23:49:08 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458548.174511
Feb 23 23:49:12 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458552.133661
Feb 23 23:49:17 oak-io2-s1 kernel: Lustre: oak-OST0042: Connection restored to 9f3e786c-91bd-0b59-f474-23ccea62b4f0 (at 10.8.2.15@o2ib6)
Feb 23 23:49:17 oak-io2-s1 kernel: Lustre: Skipped 404 previous similar messages
Feb 23 23:49:29 oak-io2-s1 kernel: Lustre: oak-OST0030: Export ffff880d4614a000 already connecting from 10.9.112.15@o2ib4
Feb 23 23:49:29 oak-io2-s1 kernel: Lustre: Skipped 9 previous similar messages
Feb 23 23:49:33 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458573.137847
Feb 23 23:49:42 oak-io2-s1 kernel: LustreError: 4897:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519458282, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST0053_UUID lock: ffff880019c07600/0x806f959362a74884 lrc: 3/0,1 mode: --/PW res: [0x2b046:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 4897 timeout: 0 lvb_type: 0
Feb 23 23:49:42 oak-io2-s1 kernel: LustreError: 4897:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) Skipped 5 previous similar messages
Feb 23 23:49:49 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458589.362709
Feb 23 23:50:10 oak-io2-s1 kernel: LNet: Service thread pid 174512 was inactive for 1201.32s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 23 23:50:10 oak-io2-s1 kernel: LNet: Skipped 31 previous similar messages
Feb 23 23:50:10 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458610.174512
Feb 23 23:50:11 oak-io2-s1 kernel: Lustre: oak-OST004d: deleting orphan objects from 0x0:3465827 to 0x0:3465921
Feb 23 23:50:34 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458634.174515
Feb 23 23:51:06 oak-io2-s1 kernel: Lustre: oak-OST0041: deleting orphan objects from 0x0:4441952 to 0x0:4442113
Feb 23 23:51:07 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458667.6923
Feb 23 23:51:08 oak-io2-s1 kernel: Lustre: oak-OST004c: deleting orphan objects from 0x0:3455889 to 0x0:3455937
Feb 23 23:51:11 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458671.249470
Feb 23 23:51:32 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458692.390782
Feb 23 23:52:07 oak-io2-s1 kernel: Lustre: oak-OST0045: deleting orphan objects from 0x0:4421667 to 0x0:4421697
Feb 23 23:52:08 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458728.390791
Feb 23 23:52:25 oak-io2-s1 kernel: Lustre: oak-OST0034: deleting orphan objects from 0x0:4290565 to 0x0:4290657
Feb 23 23:52:25 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458745.210347
Feb 23 23:52:33 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458753.17265
Feb 23 23:52:37 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458757.174509
Feb 23 23:52:49 oak-io2-s1 kernel: Lustre: oak-OST0048: deleting orphan objects from 0x0:3442499 to 0x0:3442529
Feb 23 23:52:49 oak-io2-s1 kernel: LNet: Service thread pid 362692 was inactive for 1202.90s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 23 23:52:49 oak-io2-s1 kernel: LNet: Skipped 9 previous similar messages
Feb 23 23:52:49 oak-io2-s1 kernel: Pid: 362692, comm: ll_ost01_083
Feb 23 23:52:49 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffffc0c1e020>] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc]
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffffc0c1e89d>] ldlm_completion_ast+0x63d/0x920 [ptlrpc]
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffffc0c20730>] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc]
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffffc1220b2c>] ofd_destroy_by_fid+0x1cc/0x4a0 [ofd]
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffffc1213307>] ofd_destroy_hdl+0x267/0x970 [ofd]
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:52:49 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:52:49 oak-io2-s1 kernel: 
Feb 23 23:52:49 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458769.362692
Feb 23 23:53:10 oak-io2-s1 kernel: Pid: 17264, comm: ll_ost00_031
Feb 23 23:53:10 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:53:10 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:53:10 oak-io2-s1 kernel: 
Feb 23 23:53:10 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458790.17264
Feb 23 23:53:34 oak-io2-s1 kernel: Pid: 249481, comm: ll_ost00_062
Feb 23 23:53:34 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:53:34 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:53:34 oak-io2-s1 kernel: 
Feb 23 23:53:34 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458814.249481
Feb 23 23:54:07 oak-io2-s1 kernel: Pid: 390789, comm: ll_ost00_092
Feb 23 23:54:07 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:54:07 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:54:07 oak-io2-s1 kernel: 
Feb 23 23:54:07 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458847.390789
Feb 23 23:54:32 oak-io2-s1 kernel: Pid: 25988, comm: ll_ost00_079
Feb 23 23:54:32 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffff81029557>] ? __switch_to+0xd7/0x510
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffff816a8f00>] ? __schedule+0x2f0/0x8b0
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:54:32 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:54:32 oak-io2-s1 kernel: 
Feb 23 23:54:32 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458872.25988
Feb 23 23:54:40 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458880.296683
Feb 23 23:54:44 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458884.390783
Feb 23 23:54:48 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458888.174523
Feb 23 23:54:58 oak-io2-s1 kernel: Lustre: oak-OST0035: deleting orphan objects from 0x0:4361668 to 0x0:4361698
Feb 23 23:55:02 oak-io2-s1 kernel: Lustre: oak-OST003f: deleting orphan objects from 0x0:4426054 to 0x0:4426116
Feb 23 23:55:05 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458905.210340
Feb 23 23:55:06 oak-io2-s1 kernel: Lustre: oak-OST003e: deleting orphan objects from 0x0:4288655 to 0x0:4288673
Feb 23 23:55:09 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458909.249469
Feb 23 23:55:13 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458913.196795
Feb 23 23:55:33 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458933.131985
Feb 23 23:55:45 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458945.362694
Feb 23 23:56:06 oak-io2-s1 kernel: Lustre: oak-OST0031: deleting orphan objects from 0x0:4401667 to 0x0:4401697
Feb 23 23:56:07 oak-io2-s1 kernel: Lustre: oak-OST0039: deleting orphan objects from 0x0:4347907 to 0x0:4347937
Feb 23 23:56:10 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458970.390781
Feb 23 23:56:26 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458986.196783
Feb 23 23:56:35 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519458995.390780
Feb 23 23:56:46 oak-io2-s1 kernel: Lustre: oak-OST0037: deleting orphan objects from 0x0:4441495 to 0x0:4441570
Feb 23 23:56:47 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459007.21427
Feb 23 23:57:03 oak-io2-s1 kernel: Lustre: oak-OST004f: deleting orphan objects from 0x0:175140 to 0x0:175170
Feb 23 23:57:07 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459027.249475
Feb 23 23:57:18 oak-io2-s1 kernel: Lustre: oak-OST0043: deleting orphan objects from 0x0:4403912 to 0x0:4403937
Feb 23 23:57:21 oak-io2-s1 kernel: Lustre: oak-OST0053: deleting orphan objects from 0x0:176199 to 0x0:176258
Feb 23 23:57:32 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459052.249489
Feb 23 23:57:43 oak-io2-s1 kernel: Lustre: oak-OST003c: deleting orphan objects from 0x0:4437666 to 0x0:4437761
Feb 23 23:57:44 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459064.249479
Feb 23 23:57:48 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459068.249493
Feb 23 23:57:59 oak-io2-s1 kernel: Lustre: 132528:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply#012  req@ffff88097d184450 x1591002487081696/t0(0) o101->cd9072f9-71b4-1cfb-a759-fd6823f1a4a9@10.0.2.3@o2ib5:444/0 lens 328/0 e 0 to 0 dl 1519459084 ref 2 fl New:/0/ffffffff rc 0/-1
Feb 23 23:57:59 oak-io2-s1 kernel: Lustre: 132528:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 1017 previous similar messages
Feb 23 23:58:07 oak-io2-s1 kernel: Lustre: oak-OST0035: Client 3ee95ef7-4278-ead7-52a3-bdca1c47a323 (at 10.9.112.3@o2ib4) reconnecting
Feb 23 23:58:07 oak-io2-s1 kernel: Lustre: Skipped 390 previous similar messages
Feb 23 23:58:09 oak-io2-s1 kernel: Pid: 17270, comm: ll_ost00_037
Feb 23 23:58:09 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:58:09 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:58:09 oak-io2-s1 kernel: 
Feb 23 23:58:09 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459089.17270
Feb 23 23:58:25 oak-io2-s1 kernel: Lustre: oak-OST0051: deleting orphan objects from 0x0:176434 to 0x0:176449
Feb 23 23:58:33 oak-io2-s1 kernel: Pid: 17245, comm: ll_ost00_027
Feb 23 23:58:33 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:58:33 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:58:33 oak-io2-s1 kernel: 
Feb 23 23:58:33 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459113.17245
Feb 23 23:58:46 oak-io2-s1 kernel: Pid: 390787, comm: ll_ost00_090
Feb 23 23:58:46 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:58:46 oak-io2-s1 kernel: 
Feb 23 23:58:46 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459126.390787
Feb 23 23:58:46 oak-io2-s1 kernel: Pid: 17269, comm: ll_ost00_036
Feb 23 23:58:46 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:58:46 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:58:46 oak-io2-s1 kernel: 
Feb 23 23:59:07 oak-io2-s1 kernel: Lustre: oak-OST0049: deleting orphan objects from 0x0:3458851 to 0x0:3458945
Feb 23 23:59:10 oak-io2-s1 kernel: Pid: 174520, comm: ll_ost00_113
Feb 23 23:59:10 oak-io2-s1 kernel: #012Call Trace:
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 23 23:59:10 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 23 23:59:10 oak-io2-s1 kernel: 
Feb 23 23:59:10 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459150.174520
Feb 23 23:59:18 oak-io2-s1 kernel: Lustre: oak-OST0042: Connection restored to 9f3e786c-91bd-0b59-f474-23ccea62b4f0 (at 10.8.2.15@o2ib6)
Feb 23 23:59:18 oak-io2-s1 kernel: Lustre: Skipped 423 previous similar messages
Feb 23 23:59:35 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459175.249480
Feb 23 23:59:47 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459187.134046
Feb 24 00:00:08 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459208.17273
Feb 24 00:00:12 oak-io2-s1 kernel: LNet: Service thread pid 249334 was inactive for 1203.62s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 24 00:00:12 oak-io2-s1 kernel: LNet: Skipped 37 previous similar messages
Feb 24 00:00:12 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459212.249334
Feb 24 00:00:12 oak-io2-s1 kernel: Lustre: oak-OST004d: deleting orphan objects from 0x0:3465827 to 0x0:3465953
Feb 24 00:00:32 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459232.174517
Feb 24 00:00:40 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459240.362688
Feb 24 00:00:44 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459244.174505
Feb 24 00:00:49 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459249.137742
Feb 24 00:01:07 oak-io2-s1 kernel: Lustre: oak-OST0041: deleting orphan objects from 0x0:4441952 to 0x0:4442145
Feb 24 00:01:09 oak-io2-s1 kernel: Lustre: oak-OST004c: deleting orphan objects from 0x0:3455939 to 0x0:3455969
Feb 24 00:01:09 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459269.134225
Feb 24 00:01:34 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459294.249537
Feb 24 00:01:34 oak-io2-s1 kernel: Lustre: oak-OST0050: deleting orphan objects from 0x0:175410 to 0x0:175457
Feb 24 00:01:46 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459306.17266
Feb 24 00:02:08 oak-io2-s1 kernel: Lustre: oak-OST0045: deleting orphan objects from 0x0:4421667 to 0x0:4421729
Feb 24 00:02:11 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459331.174519
Feb 24 00:02:26 oak-io2-s1 kernel: Lustre: oak-OST0034: deleting orphan objects from 0x0:4290565 to 0x0:4290689
Feb 24 00:02:35 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459355.141924
Feb 24 00:02:47 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459367.249331
Feb 24 00:02:52 oak-io2-s1 kernel: Lustre: oak-OST0048: deleting orphan objects from 0x0:3442499 to 0x0:3442561
Feb 24 00:03:08 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459388.390788
Feb 24 00:03:32 oak-io2-s1 kernel: LNet: Service thread pid 249490 was inactive for 1200.27s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 24 00:03:32 oak-io2-s1 kernel: LNet: Skipped 9 previous similar messages
Feb 24 00:03:32 oak-io2-s1 kernel: Pid: 249490, comm: ll_ost00_071
Feb 24 00:03:32 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:03:32 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:03:32 oak-io2-s1 kernel: 
Feb 24 00:03:32 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459412.249490
Feb 24 00:03:45 oak-io2-s1 kernel: Pid: 17267, comm: ll_ost00_034
Feb 24 00:03:45 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:03:45 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:03:45 oak-io2-s1 kernel: 
Feb 24 00:03:45 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459425.17267
Feb 24 00:04:09 oak-io2-s1 kernel: Pid: 249335, comm: ll_ost00_048
Feb 24 00:04:09 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:04:09 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:04:09 oak-io2-s1 kernel: 
Feb 24 00:04:09 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459449.249335
Feb 24 00:04:20 oak-io2-s1 kernel: LustreError: 210342:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519459159, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST0039_UUID lock: ffff88258777d200/0x806f959362a77dc8 lrc: 3/0,1 mode: --/PW res: [0x4257e2:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 210342 timeout: 0 lvb_type: 0
Feb 24 00:04:20 oak-io2-s1 kernel: LustreError: 210342:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) Skipped 5 previous similar messages
Feb 24 00:04:34 oak-io2-s1 kernel: Pid: 174507, comm: ll_ost00_101
Feb 24 00:04:34 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffffc090a2e7>] ? kiblnd_send+0x357/0xa10 [ko2iblnd]
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffffc097bf0b>] ? lnet_ni_send+0x3b/0xd0 [lnet]
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffffc1004753>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffffc1004753>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffff81267814>] dqget+0x3e4/0x440
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffff812685f4>] dquot_get_dqblk+0x14/0x1f0
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffffc111e8df>] osd_acct_index_lookup+0x22f/0x470 [osd_ldiskfs]
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffffc108a436>] lquotactl_slv+0x286/0xac0 [lquota]
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffffc121073c>] ofd_quotactl+0x13c/0x380 [ofd]
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:04:34 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:04:34 oak-io2-s1 kernel: 
Feb 24 00:04:34 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459474.174507
Feb 24 00:04:46 oak-io2-s1 kernel: Pid: 132143, comm: ll_ost01_009
Feb 24 00:04:46 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffffc0c1e020>] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc]
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffffc0c1e89d>] ldlm_completion_ast+0x63d/0x920 [ptlrpc]
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffffc0c20730>] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc]
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffffc1220b2c>] ofd_destroy_by_fid+0x1cc/0x4a0 [ofd]
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffffc1213307>] ofd_destroy_hdl+0x267/0x970 [ofd]
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:04:46 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:04:46 oak-io2-s1 kernel: 
Feb 24 00:04:46 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459486.132143
Feb 24 00:04:59 oak-io2-s1 kernel: Lustre: oak-OST0035: deleting orphan objects from 0x0:4361668 to 0x0:4361730
Feb 24 00:05:03 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459503.21431
Feb 24 00:05:03 oak-io2-s1 kernel: Lustre: oak-OST003f: deleting orphan objects from 0x0:4426054 to 0x0:4426148
Feb 24 00:05:07 oak-io2-s1 kernel: Lustre: oak-OST003e: deleting orphan objects from 0x0:4288655 to 0x0:4288705
Feb 24 00:05:07 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459507.4890
Feb 24 00:05:11 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459511.390796
Feb 24 00:05:35 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459535.362705
Feb 24 00:05:48 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459548.196785
Feb 24 00:05:52 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459552.362693
Feb 24 00:06:08 oak-io2-s1 kernel: Lustre: oak-OST0031: deleting orphan objects from 0x0:4401667 to 0x0:4401729
Feb 24 00:06:08 oak-io2-s1 kernel: Lustre: oak-OST0039: deleting orphan objects from 0x0:4347907 to 0x0:4347969
Feb 24 00:06:48 oak-io2-s1 kernel: Lustre: oak-OST0037: deleting orphan objects from 0x0:4441495 to 0x0:4441602
Feb 24 00:07:04 oak-io2-s1 kernel: Lustre: oak-OST004f: deleting orphan objects from 0x0:175140 to 0x0:175202
Feb 24 00:07:14 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459634.210337
Feb 24 00:07:22 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459642.132017
Feb 24 00:07:38 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459658.4910
Feb 24 00:07:44 oak-io2-s1 kernel: Lustre: oak-OST003c: deleting orphan objects from 0x0:4437666 to 0x0:4437793
Feb 24 00:07:50 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459670.196796
Feb 24 00:07:59 oak-io2-s1 kernel: Lustre: 132528:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply#012  req@ffff8806ca79a850 x1591169533815232/t0(0) o101->a7b231a5-a7d7-2f9d-4b5d-464beea873b1@10.8.0.63@o2ib6:289/0 lens 328/0 e 0 to 0 dl 1519459684 ref 2 fl New:/0/ffffffff rc 0/-1
Feb 24 00:07:59 oak-io2-s1 kernel: Lustre: 132528:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 1041 previous similar messages
Feb 24 00:08:08 oak-io2-s1 kernel: Lustre: oak-OST0035: Client 3ee95ef7-4278-ead7-52a3-bdca1c47a323 (at 10.9.112.3@o2ib4) reconnecting
Feb 24 00:08:08 oak-io2-s1 kernel: Lustre: Skipped 390 previous similar messages
Feb 24 00:08:44 oak-io2-s1 kernel: Pid: 362696, comm: ll_ost01_087
Feb 24 00:08:44 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff813492e0>] ? __percpu_counter_sum+0x70/0x80
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810cdc7c>] ? dequeue_entity+0x11c/0x5d0
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810ce500>] ? dequeue_task_fair+0x3d0/0x660
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff81029557>] ? __switch_to+0xd7/0x510
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:08:44 oak-io2-s1 kernel: 
Feb 24 00:08:44 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459724.362696
Feb 24 00:08:44 oak-io2-s1 kernel: Pid: 132313, comm: ll_ost01_010
Feb 24 00:08:44 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff813492e0>] ? __percpu_counter_sum+0x70/0x80
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:08:44 oak-io2-s1 kernel: 
Feb 24 00:08:44 oak-io2-s1 kernel: Pid: 210330, comm: ll_ost01_057
Feb 24 00:08:44 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff813492e0>] ? __percpu_counter_sum+0x70/0x80
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:08:44 oak-io2-s1 kernel: 
Feb 24 00:08:44 oak-io2-s1 kernel: Pid: 135199, comm: ll_ost01_021
Feb 24 00:08:44 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff813492e0>] ? __percpu_counter_sum+0x70/0x80
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810cdc7c>] ? dequeue_entity+0x11c/0x5d0
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810ce500>] ? dequeue_task_fair+0x3d0/0x660
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff81029557>] ? __switch_to+0xd7/0x510
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:08:44 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:08:44 oak-io2-s1 kernel: 
Feb 24 00:09:08 oak-io2-s1 kernel: Lustre: oak-OST0049: deleting orphan objects from 0x0:3458851 to 0x0:3458977
Feb 24 00:09:12 oak-io2-s1 kernel: Pid: 362702, comm: ll_ost01_093
Feb 24 00:09:12 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffffc0c1e020>] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc]
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffffc0c1e89d>] ldlm_completion_ast+0x63d/0x920 [ptlrpc]
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffffc0c20730>] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc]
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffffc1220b2c>] ofd_destroy_by_fid+0x1cc/0x4a0 [ofd]
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffffc1213307>] ofd_destroy_hdl+0x267/0x970 [ofd]
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:09:12 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:09:12 oak-io2-s1 kernel: 
Feb 24 00:09:12 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519459752.362702
Feb 24 00:09:19 oak-io2-s1 kernel: Lustre: oak-OST0042: Connection restored to 9f3e786c-91bd-0b59-f474-23ccea62b4f0 (at 10.8.2.15@o2ib6)
Feb 24 00:09:19 oak-io2-s1 kernel: Lustre: Skipped 388 previous similar messages
Feb 24 00:09:54 oak-io2-s1 kernel: Lustre: oak-OST0043: deleting orphan objects from 0x0:4403912 to 0x0:4403969
Feb 24 00:09:57 oak-io2-s1 kernel: Lustre: oak-OST0053: deleting orphan objects from 0x0:176199 to 0x0:176290
Feb 24 00:10:13 oak-io2-s1 kernel: Lustre: oak-OST004d: deleting orphan objects from 0x0:3465827 to 0x0:3465985
Feb 24 00:11:01 oak-io2-s1 kernel: Lustre: oak-OST0051: deleting orphan objects from 0x0:176434 to 0x0:176481
Feb 24 00:11:09 oak-io2-s1 kernel: Lustre: oak-OST0041: deleting orphan objects from 0x0:4441952 to 0x0:4442177
Feb 24 00:11:10 oak-io2-s1 kernel: Lustre: oak-OST004c: deleting orphan objects from 0x0:3455939 to 0x0:3456001
Feb 24 00:12:09 oak-io2-s1 kernel: Lustre: oak-OST0045: deleting orphan objects from 0x0:4421667 to 0x0:4421761
Feb 24 00:12:27 oak-io2-s1 kernel: Lustre: oak-OST0034: deleting orphan objects from 0x0:4290565 to 0x0:4290721
Feb 24 00:12:42 oak-io2-s1 kernel: Lustre: oak-OST0032: Export ffff880612719800 already connecting from 10.9.112.15@o2ib4
Feb 24 00:12:42 oak-io2-s1 kernel: Lustre: Skipped 6 previous similar messages
Feb 24 00:12:53 oak-io2-s1 kernel: Lustre: oak-OST0048: deleting orphan objects from 0x0:3442499 to 0x0:3442593
Feb 24 00:13:55 oak-io2-s1 kernel: LNet: Service thread pid 196789 was inactive for 1203.30s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 24 00:13:55 oak-io2-s1 kernel: LNet: Skipped 9 previous similar messages
Feb 24 00:13:55 oak-io2-s1 kernel: Pid: 196789, comm: ll_ost01_030
Feb 24 00:13:55 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff813492e0>] ? __percpu_counter_sum+0x70/0x80
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:13:55 oak-io2-s1 kernel: 
Feb 24 00:13:55 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519460035.196789
Feb 24 00:13:55 oak-io2-s1 kernel: Pid: 196797, comm: ll_ost01_035
Feb 24 00:13:55 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff813492e0>] ? __percpu_counter_sum+0x70/0x80
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:13:55 oak-io2-s1 kernel: 
Feb 24 00:13:55 oak-io2-s1 kernel: Pid: 4887, comm: ll_ost01_101
Feb 24 00:13:55 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff813492e0>] ? __percpu_counter_sum+0x70/0x80
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:13:55 oak-io2-s1 kernel: 
Feb 24 00:13:55 oak-io2-s1 kernel: Pid: 21434, comm: ll_ost01_046
Feb 24 00:13:55 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff813492e0>] ? __percpu_counter_sum+0x70/0x80
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:13:55 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:13:55 oak-io2-s1 kernel: 
Feb 24 00:13:58 oak-io2-s1 kernel: Lustre: oak-OST0033: deleting orphan objects from 0x0:4424462 to 0x0:4424481
Feb 24 00:14:10 oak-io2-s1 kernel: Lustre: oak-OST0050: deleting orphan objects from 0x0:175410 to 0x0:175489
Feb 24 00:15:01 oak-io2-s1 kernel: Lustre: oak-OST0035: deleting orphan objects from 0x0:4361668 to 0x0:4361762
Feb 24 00:15:04 oak-io2-s1 kernel: Lustre: oak-OST003f: deleting orphan objects from 0x0:4426054 to 0x0:4426180
Feb 24 00:15:08 oak-io2-s1 kernel: Lustre: oak-OST003e: deleting orphan objects from 0x0:4288655 to 0x0:4288737
Feb 24 00:16:09 oak-io2-s1 kernel: Lustre: oak-OST0031: deleting orphan objects from 0x0:4401667 to 0x0:4401761
Feb 24 00:16:10 oak-io2-s1 kernel: Lustre: oak-OST0039: deleting orphan objects from 0x0:4347907 to 0x0:4348001
Feb 24 00:16:49 oak-io2-s1 kernel: Lustre: oak-OST0037: deleting orphan objects from 0x0:4441495 to 0x0:4441634
Feb 24 00:17:05 oak-io2-s1 kernel: Lustre: oak-OST004f: deleting orphan objects from 0x0:175140 to 0x0:175234
Feb 24 00:17:40 oak-io2-s1 kernel: Pid: 362695, comm: ll_ost01_086
Feb 24 00:17:40 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:17:40 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 00:17:40 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 00:17:40 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 00:17:40 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 00:17:40 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 00:17:40 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 00:17:40 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 00:17:40 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:17:40 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:17:40 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:17:40 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:17:40 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:17:40 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:17:40 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:17:40 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:17:40 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:17:40 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:17:40 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:17:40 oak-io2-s1 kernel: 
Feb 24 00:17:40 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519460260.362695
Feb 24 00:17:45 oak-io2-s1 kernel: Lustre: oak-OST003c: deleting orphan objects from 0x0:4437666 to 0x0:4437825
Feb 24 00:18:01 oak-io2-s1 kernel: Lustre: 251666:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff882a7d196450 x1592239615714592/t0(0) o4->3ee95ef7-4278-ead7-52a3-bdca1c47a323@10.9.112.3@o2ib4:136/0 lens 608/0 e 0 to 0 dl 1519460286 ref 2 fl New:H/2/ffffffff rc 0/-1
Feb 24 00:18:01 oak-io2-s1 kernel: Lustre: 251666:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 1041 previous similar messages
Feb 24 00:18:09 oak-io2-s1 kernel: Lustre: oak-OST0035: Client 3ee95ef7-4278-ead7-52a3-bdca1c47a323 (at 10.9.112.3@o2ib4) reconnecting
Feb 24 00:18:09 oak-io2-s1 kernel: Lustre: Skipped 403 previous similar messages
Feb 24 00:19:09 oak-io2-s1 kernel: Lustre: oak-OST0049: deleting orphan objects from 0x0:3458851 to 0x0:3459009
Feb 24 00:19:20 oak-io2-s1 kernel: Lustre: oak-OST0042: Connection restored to 9f3e786c-91bd-0b59-f474-23ccea62b4f0 (at 10.8.2.15@o2ib6)
Feb 24 00:19:20 oak-io2-s1 kernel: Lustre: Skipped 430 previous similar messages
Feb 24 00:19:23 oak-io2-s1 kernel: Pid: 210342, comm: ll_ost01_069
Feb 24 00:19:23 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffffc0c1e020>] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc]
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffffc0c1e89d>] ldlm_completion_ast+0x63d/0x920 [ptlrpc]
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffffc0c20730>] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc]
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffffc1220b2c>] ofd_destroy_by_fid+0x1cc/0x4a0 [ofd]
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffffc1213307>] ofd_destroy_hdl+0x267/0x970 [ofd]
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:19:23 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:19:23 oak-io2-s1 kernel: 
Feb 24 00:19:23 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519460363.210342
Feb 24 00:19:51 oak-io2-s1 kernel: Pid: 134397, comm: ll_ost01_016
Feb 24 00:19:51 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:19:51 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:19:51 oak-io2-s1 kernel: 
Feb 24 00:19:51 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519460391.134397
Feb 24 00:19:55 oak-io2-s1 kernel: Pid: 210336, comm: ll_ost01_063
Feb 24 00:19:55 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:19:55 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 00:19:55 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 00:19:55 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 00:19:55 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 00:19:55 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 00:19:55 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 00:19:55 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 00:19:55 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:19:55 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:19:55 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:19:55 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:19:55 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:19:55 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:19:55 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:19:55 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:19:55 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:19:55 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:19:55 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:19:55 oak-io2-s1 kernel: 
Feb 24 00:19:55 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519460395.210336
Feb 24 00:20:14 oak-io2-s1 kernel: Lustre: oak-OST004d: deleting orphan objects from 0x0:3465827 to 0x0:3466017
Feb 24 00:20:16 oak-io2-s1 kernel: Pid: 210333, comm: ll_ost01_060
Feb 24 00:20:16 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:20:16 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 00:20:16 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 00:20:16 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 00:20:16 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 00:20:16 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 00:20:16 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 00:20:16 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 00:20:16 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:20:16 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:20:16 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:20:16 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:20:16 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:20:16 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:20:16 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:20:16 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:20:16 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:20:16 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:20:16 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:20:16 oak-io2-s1 kernel: 
Feb 24 00:20:16 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519460416.210333
Feb 24 00:20:28 oak-io2-s1 kernel: Pid: 210329, comm: ll_ost01_056
Feb 24 00:20:28 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:20:28 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 00:20:28 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 00:20:28 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 00:20:28 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 00:20:28 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 00:20:28 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 00:20:28 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 00:20:28 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:20:28 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:20:28 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:20:28 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:20:28 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:20:28 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:20:28 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:20:28 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:20:28 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:20:28 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:20:28 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:20:28 oak-io2-s1 kernel: 
Feb 24 00:20:28 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519460428.210329
Feb 24 00:21:10 oak-io2-s1 kernel: Lustre: oak-OST0041: deleting orphan objects from 0x0:4442179 to 0x0:4442209
Feb 24 00:21:13 oak-io2-s1 kernel: Lustre: oak-OST004c: deleting orphan objects from 0x0:3455939 to 0x0:3456033
Feb 24 00:21:21 oak-io2-s1 kernel: LNet: Service thread pid 133446 was inactive for 1201.99s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 24 00:21:21 oak-io2-s1 kernel: LNet: Skipped 25 previous similar messages
Feb 24 00:21:21 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519460481.133446
Feb 24 00:22:10 oak-io2-s1 kernel: Lustre: oak-OST0045: deleting orphan objects from 0x0:4421667 to 0x0:4421793
Feb 24 00:22:28 oak-io2-s1 kernel: Lustre: oak-OST0034: deleting orphan objects from 0x0:4290565 to 0x0:4290753
Feb 24 00:22:30 oak-io2-s1 kernel: Lustre: oak-OST0043: deleting orphan objects from 0x0:4403912 to 0x0:4404001
Feb 24 00:22:33 oak-io2-s1 kernel: Lustre: oak-OST0053: deleting orphan objects from 0x0:176199 to 0x0:176322
Feb 24 00:22:55 oak-io2-s1 kernel: Lustre: oak-OST0048: deleting orphan objects from 0x0:3442499 to 0x0:3442625
Feb 24 00:23:37 oak-io2-s1 kernel: Lustre: oak-OST0051: deleting orphan objects from 0x0:176434 to 0x0:176513
Feb 24 00:24:49 oak-io2-s1 kernel: Lustre: oak-OST0030: Export ffff8835775b9000 already connecting from 10.9.112.15@o2ib4
Feb 24 00:24:49 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 24 00:25:02 oak-io2-s1 kernel: Lustre: oak-OST0035: deleting orphan objects from 0x0:4361668 to 0x0:4361794
Feb 24 00:25:05 oak-io2-s1 kernel: Lustre: oak-OST003f: deleting orphan objects from 0x0:4426054 to 0x0:4426212
Feb 24 00:25:09 oak-io2-s1 kernel: Lustre: oak-OST003e: deleting orphan objects from 0x0:4288655 to 0x0:4288769
Feb 24 00:25:39 oak-io2-s1 kernel: Lustre: oak-OST0030: Export ffff8835775b9000 already connecting from 10.9.112.15@o2ib4
Feb 24 00:25:39 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
Feb 24 00:26:10 oak-io2-s1 kernel: Lustre: oak-OST0031: deleting orphan objects from 0x0:4401667 to 0x0:4401793
Feb 24 00:26:11 oak-io2-s1 kernel: Lustre: oak-OST0039: deleting orphan objects from 0x0:4347907 to 0x0:4348033
Feb 24 00:26:29 oak-io2-s1 kernel: Lustre: oak-OST0030: Export ffff8835775b9000 already connecting from 10.9.112.15@o2ib4
Feb 24 00:26:29 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 24 00:26:34 oak-io2-s1 kernel: Lustre: oak-OST0033: deleting orphan objects from 0x0:4424462 to 0x0:4424513
Feb 24 00:26:47 oak-io2-s1 kernel: Lustre: oak-OST0050: deleting orphan objects from 0x0:175410 to 0x0:175521
Feb 24 00:26:50 oak-io2-s1 kernel: Lustre: oak-OST0037: deleting orphan objects from 0x0:4441495 to 0x0:4441666
Feb 24 00:27:06 oak-io2-s1 kernel: Lustre: oak-OST004f: deleting orphan objects from 0x0:175140 to 0x0:175266
Feb 24 00:27:46 oak-io2-s1 kernel: Lustre: oak-OST003c: deleting orphan objects from 0x0:4437666 to 0x0:4437857
Feb 24 00:28:02 oak-io2-s1 kernel: Lustre: 132528:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply#012  req@ffff8816e486e450 x1591002487469040/t0(0) o19->cd9072f9-71b4-1cfb-a759-fd6823f1a4a9@10.0.2.3@o2ib5:737/0 lens 336/0 e 0 to 0 dl 1519460887 ref 2 fl New:/0/ffffffff rc 0/-1
Feb 24 00:28:02 oak-io2-s1 kernel: Lustre: 132528:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 1055 previous similar messages
Feb 24 00:28:10 oak-io2-s1 kernel: Lustre: oak-OST0035: Client 3ee95ef7-4278-ead7-52a3-bdca1c47a323 (at 10.9.112.3@o2ib4) reconnecting
Feb 24 00:28:10 oak-io2-s1 kernel: Lustre: Skipped 409 previous similar messages
Feb 24 00:28:57 oak-io2-s1 kernel: LustreError: 196788:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519460637, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST004b_UUID lock: ffff8800402bd200/0x806f959362a7a79e lrc: 3/0,1 mode: --/PW res: [0x34cace:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 196788 timeout: 0 lvb_type: 0
Feb 24 00:28:57 oak-io2-s1 kernel: LustreError: 196788:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) Skipped 1 previous similar message
Feb 24 00:29:10 oak-io2-s1 kernel: Lustre: oak-OST0049: deleting orphan objects from 0x0:3458851 to 0x0:3459041
Feb 24 00:29:21 oak-io2-s1 kernel: Lustre: oak-OST0033: Connection restored to 3ee95ef7-4278-ead7-52a3-bdca1c47a323 (at 10.9.112.3@o2ib4)
Feb 24 00:29:21 oak-io2-s1 kernel: Lustre: Skipped 439 previous similar messages
Feb 24 00:30:15 oak-io2-s1 kernel: Lustre: oak-OST004d: deleting orphan objects from 0x0:3465827 to 0x0:3466049
Feb 24 00:30:18 oak-io2-s1 kernel: LNet: Service thread pid 147130 was inactive for 1204.37s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 24 00:30:18 oak-io2-s1 kernel: LNet: Skipped 9 previous similar messages
Feb 24 00:30:18 oak-io2-s1 kernel: Pid: 147130, comm: ll_ost01_018
Feb 24 00:30:18 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:30:18 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:30:18 oak-io2-s1 kernel: 
Feb 24 00:30:18 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519461018.147130
Feb 24 00:31:12 oak-io2-s1 kernel: Lustre: oak-OST0041: deleting orphan objects from 0x0:4442179 to 0x0:4442241
Feb 24 00:31:14 oak-io2-s1 kernel: Lustre: oak-OST004c: deleting orphan objects from 0x0:3455939 to 0x0:3456065
Feb 24 00:31:56 oak-io2-s1 kernel: Pid: 196790, comm: ll_ost01_031
Feb 24 00:31:56 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810cdc7c>] ? dequeue_entity+0x11c/0x5d0
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810ce500>] ? dequeue_task_fair+0x3d0/0x660
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff81029557>] ? __switch_to+0xd7/0x510
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:31:56 oak-io2-s1 kernel: 
Feb 24 00:31:56 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519461116.196790
Feb 24 00:31:56 oak-io2-s1 kernel: Pid: 362685, comm: ll_ost01_076
Feb 24 00:31:56 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:31:56 oak-io2-s1 kernel: 
Feb 24 00:31:56 oak-io2-s1 kernel: Pid: 210345, comm: ll_ost01_072
Feb 24 00:31:56 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:31:56 oak-io2-s1 kernel: 
Feb 24 00:31:56 oak-io2-s1 kernel: Pid: 196781, comm: ll_ost01_023
Feb 24 00:31:56 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff811dd579>] ? ___slab_alloc+0x209/0x4f0
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:31:56 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:31:56 oak-io2-s1 kernel: 
Feb 24 00:32:11 oak-io2-s1 kernel: Lustre: oak-OST0045: deleting orphan objects from 0x0:4421667 to 0x0:4421825
Feb 24 00:32:29 oak-io2-s1 kernel: Lustre: oak-OST0034: deleting orphan objects from 0x0:4290565 to 0x0:4290785
Feb 24 00:32:29 oak-io2-s1 kernel: LNet: Service thread pid 362708 was inactive for 1202.59s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 24 00:32:29 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519461149.362708
Feb 24 00:32:33 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519461153.362701
Feb 24 00:32:54 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519461174.362690
Feb 24 00:32:57 oak-io2-s1 kernel: Lustre: oak-OST0048: deleting orphan objects from 0x0:3442499 to 0x0:3442657
Feb 24 00:33:06 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519461186.21428
Feb 24 00:33:59 oak-io2-s1 kernel: Lustre: oak-OST004b: deleting orphan objects from 0x0:3459791 to 0x0:3459819
Feb 24 00:35:03 oak-io2-s1 kernel: Lustre: oak-OST0035: deleting orphan objects from 0x0:4361668 to 0x0:4361826
Feb 24 00:35:06 oak-io2-s1 kernel: Lustre: oak-OST0043: deleting orphan objects from 0x0:4403912 to 0x0:4404033
Feb 24 00:35:06 oak-io2-s1 kernel: Lustre: oak-OST003f: deleting orphan objects from 0x0:4426054 to 0x0:4426244
Feb 24 00:35:10 oak-io2-s1 kernel: Lustre: oak-OST003e: deleting orphan objects from 0x0:4288655 to 0x0:4288801
Feb 24 00:35:10 oak-io2-s1 kernel: Lustre: oak-OST0053: deleting orphan objects from 0x0:176199 to 0x0:176354
Feb 24 00:36:11 oak-io2-s1 kernel: Lustre: oak-OST0031: deleting orphan objects from 0x0:4401667 to 0x0:4401825
Feb 24 00:36:12 oak-io2-s1 kernel: Lustre: oak-OST0039: deleting orphan objects from 0x0:4347907 to 0x0:4348065
Feb 24 00:36:13 oak-io2-s1 kernel: Lustre: oak-OST0051: deleting orphan objects from 0x0:176434 to 0x0:176545
Feb 24 00:36:51 oak-io2-s1 kernel: Lustre: oak-OST0037: deleting orphan objects from 0x0:4441495 to 0x0:4441698
Feb 24 00:37:07 oak-io2-s1 kernel: Lustre: oak-OST004f: deleting orphan objects from 0x0:175140 to 0x0:175298
Feb 24 00:37:47 oak-io2-s1 kernel: Lustre: oak-OST003c: deleting orphan objects from 0x0:4437666 to 0x0:4437889
Feb 24 00:38:03 oak-io2-s1 kernel: Lustre: 251666:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff882883a71450 x1592239615714592/t0(0) o4->3ee95ef7-4278-ead7-52a3-bdca1c47a323@10.9.112.3@o2ib4:583/0 lens 608/0 e 0 to 0 dl 1519461488 ref 2 fl New:H/2/ffffffff rc 0/-1
Feb 24 00:38:03 oak-io2-s1 kernel: Lustre: 251666:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 1063 previous similar messages
Feb 24 00:38:11 oak-io2-s1 kernel: Lustre: oak-OST0035: Client 3ee95ef7-4278-ead7-52a3-bdca1c47a323 (at 10.9.112.3@o2ib4) reconnecting
Feb 24 00:38:11 oak-io2-s1 kernel: Lustre: Skipped 404 previous similar messages
Feb 24 00:39:10 oak-io2-s1 kernel: Lustre: oak-OST0033: deleting orphan objects from 0x0:4424462 to 0x0:4424545
Feb 24 00:39:12 oak-io2-s1 kernel: Lustre: oak-OST0049: deleting orphan objects from 0x0:3458851 to 0x0:3459073
Feb 24 00:39:22 oak-io2-s1 kernel: Lustre: oak-OST0033: Connection restored to 3ee95ef7-4278-ead7-52a3-bdca1c47a323 (at 10.9.112.3@o2ib4)
Feb 24 00:39:22 oak-io2-s1 kernel: Lustre: Skipped 407 previous similar messages
Feb 24 00:39:25 oak-io2-s1 kernel: Lustre: oak-OST0050: deleting orphan objects from 0x0:175410 to 0x0:175553
Feb 24 00:40:16 oak-io2-s1 kernel: Lustre: oak-OST004d: deleting orphan objects from 0x0:3465827 to 0x0:3466081
Feb 24 00:41:13 oak-io2-s1 kernel: Lustre: oak-OST0041: deleting orphan objects from 0x0:4442179 to 0x0:4442273
Feb 24 00:41:15 oak-io2-s1 kernel: Lustre: oak-OST004c: deleting orphan objects from 0x0:3455939 to 0x0:3456097
Feb 24 00:42:12 oak-io2-s1 kernel: Lustre: oak-OST0045: deleting orphan objects from 0x0:4421667 to 0x0:4421857
Feb 24 00:42:31 oak-io2-s1 kernel: Lustre: oak-OST0034: deleting orphan objects from 0x0:4290565 to 0x0:4290817
Feb 24 00:42:52 oak-io2-s1 kernel: LNet: Service thread pid 210322 was inactive for 1201.04s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 24 00:42:52 oak-io2-s1 kernel: LNet: Skipped 4 previous similar messages
Feb 24 00:42:52 oak-io2-s1 kernel: Pid: 210322, comm: ll_ost01_050
Feb 24 00:42:52 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:42:52 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 00:42:52 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 00:42:52 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 00:42:52 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 00:42:52 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 00:42:52 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 00:42:52 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 00:42:52 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:42:52 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:42:52 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:42:52 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:42:52 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:42:52 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:42:52 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:42:52 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:42:52 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:42:52 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:42:52 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:42:52 oak-io2-s1 kernel: 
Feb 24 00:42:52 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519461772.210322
Feb 24 00:42:58 oak-io2-s1 kernel: Lustre: oak-OST0048: deleting orphan objects from 0x0:3442499 to 0x0:3442689
Feb 24 00:43:31 oak-io2-s1 kernel: LustreError: 131982:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519461511, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST0041_UUID lock: ffff88003bfc9c00/0x806f959362a7c3a5 lrc: 3/0,1 mode: --/PW res: [0x43c842:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 131982 timeout: 0 lvb_type: 0
Feb 24 00:44:01 oak-io2-s1 kernel: Lustre: oak-OST004b: deleting orphan objects from 0x0:3459791 to 0x0:3459851
Feb 24 00:44:01 oak-io2-s1 kernel: Pid: 4902, comm: ll_ost01_113
Feb 24 00:44:01 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0885f83>] ? cfs_hash_buckets_realloc+0x1b3/0x660 [libcfs]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:44:01 oak-io2-s1 kernel: 
Feb 24 00:44:01 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519461841.4902
Feb 24 00:44:01 oak-io2-s1 kernel: Pid: 133457, comm: ll_ost01_013
Feb 24 00:44:01 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0885f83>] ? cfs_hash_buckets_realloc+0x1b3/0x660 [libcfs]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc09767e5>] ? lnet_md_alloc.isra.5+0x145/0x300 [lnet]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0976d94>] ? LNetMDAttach+0x3f4/0x450 [lnet]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:44:01 oak-io2-s1 kernel: 
Feb 24 00:44:01 oak-io2-s1 kernel: Pid: 196788, comm: ll_ost01_029
Feb 24 00:44:01 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c1e020>] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c1e89d>] ldlm_completion_ast+0x63d/0x920 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c20730>] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc1220b2c>] ofd_destroy_by_fid+0x1cc/0x4a0 [ofd]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc1213307>] ofd_destroy_hdl+0x267/0x970 [ofd]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:44:01 oak-io2-s1 kernel: 
Feb 24 00:44:01 oak-io2-s1 kernel: Pid: 196787, comm: ll_ost01_028
Feb 24 00:44:01 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810c8f18>] ? __enqueue_entity+0x78/0x80
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810cf90c>] ? enqueue_entity+0x26c/0xb60
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:44:01 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:44:01 oak-io2-s1 kernel: 
Feb 24 00:44:01 oak-io2-s1 kernel: LNet: Service thread pid 131961 was inactive for 1202.75s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 24 00:44:01 oak-io2-s1 kernel: LNet: Skipped 3 previous similar messages
Feb 24 00:45:04 oak-io2-s1 kernel: Lustre: oak-OST0035: deleting orphan objects from 0x0:4361668 to 0x0:4361858
Feb 24 00:45:07 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519461907.159218
Feb 24 00:45:07 oak-io2-s1 kernel: Lustre: oak-OST003f: deleting orphan objects from 0x0:4426054 to 0x0:4426276
Feb 24 00:45:11 oak-io2-s1 kernel: Lustre: oak-OST003e: deleting orphan objects from 0x0:4288655 to 0x0:4288833
Feb 24 00:45:11 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519461911.131984
Feb 24 00:45:31 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519461931.362706
Feb 24 00:45:44 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519461944.4889
Feb 24 00:46:12 oak-io2-s1 kernel: Lustre: oak-OST0031: deleting orphan objects from 0x0:4401667 to 0x0:4401857
Feb 24 00:46:13 oak-io2-s1 kernel: Lustre: oak-OST0039: deleting orphan objects from 0x0:4347907 to 0x0:4348097
Feb 24 00:46:52 oak-io2-s1 kernel: Lustre: oak-OST0037: deleting orphan objects from 0x0:4441495 to 0x0:4441730
Feb 24 00:47:08 oak-io2-s1 kernel: Lustre: oak-OST004f: deleting orphan objects from 0x0:175140 to 0x0:175330
Feb 24 00:47:43 oak-io2-s1 kernel: Lustre: oak-OST0043: deleting orphan objects from 0x0:4403912 to 0x0:4404065
Feb 24 00:47:46 oak-io2-s1 kernel: Lustre: oak-OST0053: deleting orphan objects from 0x0:176199 to 0x0:176386
Feb 24 00:47:48 oak-io2-s1 kernel: Lustre: oak-OST003c: deleting orphan objects from 0x0:4437666 to 0x0:4437921
Feb 24 00:48:04 oak-io2-s1 kernel: Lustre: 132528:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply#012  req@ffff8800223c1450 x1591002487779920/t0(0) o19->cd9072f9-71b4-1cfb-a759-fd6823f1a4a9@10.0.2.3@o2ib5:429/0 lens 336/0 e 0 to 0 dl 1519462089 ref 2 fl New:/0/ffffffff rc 0/-1
Feb 24 00:48:04 oak-io2-s1 kernel: Lustre: 132528:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 1036 previous similar messages
Feb 24 00:48:12 oak-io2-s1 kernel: Lustre: oak-OST0035: Client 3ee95ef7-4278-ead7-52a3-bdca1c47a323 (at 10.9.112.3@o2ib4) reconnecting
Feb 24 00:48:12 oak-io2-s1 kernel: Lustre: Skipped 404 previous similar messages
Feb 24 00:48:49 oak-io2-s1 kernel: Lustre: oak-OST0051: deleting orphan objects from 0x0:176434 to 0x0:176577
Feb 24 00:49:13 oak-io2-s1 kernel: Lustre: oak-OST0049: deleting orphan objects from 0x0:3458851 to 0x0:3459105
Feb 24 00:49:23 oak-io2-s1 kernel: Lustre: oak-OST0042: Connection restored to 9f3e786c-91bd-0b59-f474-23ccea62b4f0 (at 10.8.2.15@o2ib6)
Feb 24 00:49:23 oak-io2-s1 kernel: Lustre: Skipped 407 previous similar messages
Feb 24 00:50:17 oak-io2-s1 kernel: Lustre: oak-OST004d: deleting orphan objects from 0x0:3465827 to 0x0:3466113
Feb 24 00:51:14 oak-io2-s1 kernel: Lustre: oak-OST0041: deleting orphan objects from 0x0:4442179 to 0x0:4442305
Feb 24 00:51:16 oak-io2-s1 kernel: Lustre: oak-OST004c: deleting orphan objects from 0x0:3455939 to 0x0:3456129
Feb 24 00:51:46 oak-io2-s1 kernel: Lustre: oak-OST0033: deleting orphan objects from 0x0:4424547 to 0x0:4424577
Feb 24 00:52:01 oak-io2-s1 kernel: Lustre: oak-OST0050: deleting orphan objects from 0x0:175410 to 0x0:175585
Feb 24 00:52:14 oak-io2-s1 kernel: Lustre: oak-OST0045: deleting orphan objects from 0x0:4421667 to 0x0:4421889
Feb 24 00:52:32 oak-io2-s1 kernel: Lustre: oak-OST0034: deleting orphan objects from 0x0:4290565 to 0x0:4290849
Feb 24 00:53:02 oak-io2-s1 kernel: Lustre: oak-OST0048: deleting orphan objects from 0x0:3442499 to 0x0:3442721
Feb 24 00:54:02 oak-io2-s1 kernel: Lustre: oak-OST004b: deleting orphan objects from 0x0:3459791 to 0x0:3459883
Feb 24 00:54:26 oak-io2-s1 kernel: LustreError: 362689:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519462166, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST0046_UUID lock: ffff88004720d200/0x806f959362a7de32 lrc: 3/0,1 mode: --/PW res: [0x44152a:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 362689 timeout: 0 lvb_type: 0
Feb 24 00:55:05 oak-io2-s1 kernel: Lustre: oak-OST0035: deleting orphan objects from 0x0:4361668 to 0x0:4361890
Feb 24 00:55:09 oak-io2-s1 kernel: Lustre: oak-OST003f: deleting orphan objects from 0x0:4426054 to 0x0:4426308
Feb 24 00:55:12 oak-io2-s1 kernel: Lustre: oak-OST003e: deleting orphan objects from 0x0:4288655 to 0x0:4288865
Feb 24 00:55:29 oak-io2-s1 kernel: LNet: Service thread pid 131983 was inactive for 1201.80s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 24 00:55:29 oak-io2-s1 kernel: LNet: Skipped 4 previous similar messages
Feb 24 00:55:29 oak-io2-s1 kernel: Pid: 131983, comm: ll_ost01_004
Feb 24 00:55:29 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:55:29 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 00:55:29 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 00:55:29 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 00:55:29 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 00:55:29 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 00:55:29 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 00:55:29 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 00:55:29 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:55:29 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:55:29 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:55:29 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:55:29 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:55:29 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:55:29 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:55:29 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:55:29 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:55:29 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:55:29 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:55:29 oak-io2-s1 kernel: 
Feb 24 00:55:29 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519462529.131983
Feb 24 00:56:14 oak-io2-s1 kernel: Lustre: oak-OST0031: deleting orphan objects from 0x0:4401667 to 0x0:4401889
Feb 24 00:56:14 oak-io2-s1 kernel: Lustre: oak-OST0039: deleting orphan objects from 0x0:4347907 to 0x0:4348129
Feb 24 00:56:53 oak-io2-s1 kernel: Lustre: oak-OST0037: deleting orphan objects from 0x0:4441495 to 0x0:4441762
Feb 24 00:57:09 oak-io2-s1 kernel: Lustre: oak-OST004f: deleting orphan objects from 0x0:175140 to 0x0:175362
Feb 24 00:57:45 oak-io2-s1 kernel: Pid: 362703, comm: ll_ost01_094
Feb 24 00:57:45 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:57:45 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 00:57:45 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 00:57:45 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 00:57:45 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 00:57:45 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 00:57:45 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 00:57:45 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 00:57:45 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:57:45 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:57:45 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:57:45 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:57:45 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:57:45 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:57:45 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:57:45 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:57:45 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:57:45 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:57:45 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:57:45 oak-io2-s1 kernel: 
Feb 24 00:57:45 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519462665.362703
Feb 24 00:57:49 oak-io2-s1 kernel: Pid: 6921, comm: ll_ost01_121
Feb 24 00:57:49 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:57:49 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 00:57:49 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 00:57:49 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 00:57:49 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 00:57:49 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 00:57:49 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 00:57:49 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 00:57:49 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:57:49 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:57:49 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:57:49 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:57:49 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:57:49 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:57:49 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:57:49 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:57:49 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:57:49 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:57:49 oak-io2-s1 kernel: 
Feb 24 00:57:49 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519462669.6921
Feb 24 00:57:49 oak-io2-s1 kernel: Lustre: oak-OST003c: deleting orphan objects from 0x0:4437666 to 0x0:4437953
Feb 24 00:58:05 oak-io2-s1 kernel: Lustre: 251666:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff883974e4ac50 x1592239615714592/t0(0) o4->3ee95ef7-4278-ead7-52a3-bdca1c47a323@10.9.112.3@o2ib4:275/0 lens 608/0 e 0 to 0 dl 1519462690 ref 2 fl New:H/2/ffffffff rc 0/-1
Feb 24 00:58:05 oak-io2-s1 kernel: Lustre: 251666:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 1059 previous similar messages
Feb 24 00:58:09 oak-io2-s1 kernel: Pid: 210339, comm: ll_ost01_066
Feb 24 00:58:09 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:58:09 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 00:58:09 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 00:58:09 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 00:58:09 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 00:58:09 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 00:58:09 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 00:58:09 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 00:58:09 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:58:09 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:58:09 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:58:09 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:58:09 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:58:09 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:58:09 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:58:09 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:58:09 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:58:09 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:58:09 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:58:09 oak-io2-s1 kernel: 
Feb 24 00:58:09 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519462689.210339
Feb 24 00:58:13 oak-io2-s1 kernel: Lustre: oak-OST0035: Client 3ee95ef7-4278-ead7-52a3-bdca1c47a323 (at 10.9.112.3@o2ib4) reconnecting
Feb 24 00:58:13 oak-io2-s1 kernel: Lustre: Skipped 403 previous similar messages
Feb 24 00:58:17 oak-io2-s1 kernel: Pid: 4907, comm: ll_ost01_114
Feb 24 00:58:17 oak-io2-s1 kernel: #012Call Trace:
Feb 24 00:58:17 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 00:58:17 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 00:58:17 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 00:58:17 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 00:58:17 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 00:58:17 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 00:58:17 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 00:58:17 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 00:58:17 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 00:58:17 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 00:58:17 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 00:58:17 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 00:58:17 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 00:58:17 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 00:58:17 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 00:58:17 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:58:17 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 00:58:17 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 00:58:17 oak-io2-s1 kernel: 
Feb 24 00:58:17 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519462697.4907
Feb 24 00:58:34 oak-io2-s1 kernel: LNet: Service thread pid 131982 was inactive for 1202.33s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 24 00:58:34 oak-io2-s1 kernel: LNet: Skipped 4 previous similar messages
Feb 24 00:58:34 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519462714.131982
Feb 24 00:59:17 oak-io2-s1 kernel: Lustre: oak-OST0049: deleting orphan objects from 0x0:3458851 to 0x0:3459137
Feb 24 00:59:24 oak-io2-s1 kernel: Lustre: oak-OST0042: Connection restored to 9f3e786c-91bd-0b59-f474-23ccea62b4f0 (at 10.8.2.15@o2ib6)
Feb 24 00:59:24 oak-io2-s1 kernel: Lustre: Skipped 402 previous similar messages
Feb 24 01:00:18 oak-io2-s1 kernel: Lustre: oak-OST004d: deleting orphan objects from 0x0:3465827 to 0x0:3466145
Feb 24 01:00:19 oak-io2-s1 kernel: Lustre: oak-OST0043: deleting orphan objects from 0x0:4403912 to 0x0:4404097
Feb 24 01:00:22 oak-io2-s1 kernel: Lustre: oak-OST0053: deleting orphan objects from 0x0:176199 to 0x0:176418
Feb 24 01:01:15 oak-io2-s1 kernel: Lustre: oak-OST0041: deleting orphan objects from 0x0:4442179 to 0x0:4442337
Feb 24 01:01:17 oak-io2-s1 kernel: Lustre: oak-OST004c: deleting orphan objects from 0x0:3455939 to 0x0:3456161
Feb 24 01:01:25 oak-io2-s1 kernel: Lustre: oak-OST0051: deleting orphan objects from 0x0:176434 to 0x0:176609
Feb 24 01:02:02 oak-io2-s1 kernel: Lustre: oak-OST0046: deleting orphan objects from 0x0:4461867 to 0x0:4461889
Feb 24 01:02:15 oak-io2-s1 kernel: Lustre: oak-OST0045: deleting orphan objects from 0x0:4421667 to 0x0:4421921
Feb 24 01:02:33 oak-io2-s1 kernel: Lustre: oak-OST0034: deleting orphan objects from 0x0:4290565 to 0x0:4290881
Feb 24 01:03:03 oak-io2-s1 kernel: Lustre: oak-OST0048: deleting orphan objects from 0x0:3442499 to 0x0:3442753
Feb 24 01:04:03 oak-io2-s1 kernel: Lustre: oak-OST004b: deleting orphan objects from 0x0:3459791 to 0x0:3459915
Feb 24 01:04:23 oak-io2-s1 kernel: Lustre: oak-OST0033: deleting orphan objects from 0x0:4424580 to 0x0:4424609
Feb 24 01:04:25 oak-io2-s1 kernel: LustreError: 4900:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519462765, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST0033_UUID lock: ffff880003197e00/0x806f959362a80935 lrc: 3/0,1 mode: --/PW res: [0x438362:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 4900 timeout: 0 lvb_type: 0
Feb 24 01:04:38 oak-io2-s1 kernel: Lustre: oak-OST0050: deleting orphan objects from 0x0:175410 to 0x0:175617
Feb 24 01:05:06 oak-io2-s1 kernel: Lustre: oak-OST0035: deleting orphan objects from 0x0:4361668 to 0x0:4361922
Feb 24 01:05:10 oak-io2-s1 kernel: Lustre: oak-OST003f: deleting orphan objects from 0x0:4426054 to 0x0:4426340
Feb 24 01:05:13 oak-io2-s1 kernel: Lustre: oak-OST003e: deleting orphan objects from 0x0:4288655 to 0x0:4288897
Feb 24 01:06:15 oak-io2-s1 kernel: Lustre: oak-OST0039: deleting orphan objects from 0x0:4347907 to 0x0:4348161
Feb 24 01:06:18 oak-io2-s1 kernel: Lustre: oak-OST0031: deleting orphan objects from 0x0:4401667 to 0x0:4401921
Feb 24 01:06:29 oak-io2-s1 kernel: LustreError: 210321:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519462889, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST0033_UUID lock: ffff88066706bc00/0x806f959362a80ebb lrc: 3/0,1 mode: --/PW res: [0x438382:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 210321 timeout: 0 lvb_type: 0
Feb 24 01:06:54 oak-io2-s1 kernel: Lustre: oak-OST0037: deleting orphan objects from 0x0:4441495 to 0x0:4441794
Feb 24 01:07:10 oak-io2-s1 kernel: Lustre: oak-OST004f: deleting orphan objects from 0x0:175140 to 0x0:175394
Feb 24 01:07:45 oak-io2-s1 kernel: Lustre: oak-OST0032: Export ffff88227aea4800 already connecting from 10.9.112.15@o2ib4
Feb 24 01:07:45 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
Feb 24 01:07:50 oak-io2-s1 kernel: Lustre: oak-OST003c: deleting orphan objects from 0x0:4437666 to 0x0:4437985
Feb 24 01:08:06 oak-io2-s1 kernel: Lustre: 251666:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff88378efa6050 x1592239615714592/t0(0) o4->3ee95ef7-4278-ead7-52a3-bdca1c47a323@10.9.112.3@o2ib4:121/0 lens 608/0 e 0 to 0 dl 1519463291 ref 2 fl New:H/2/ffffffff rc 0/-1
Feb 24 01:08:06 oak-io2-s1 kernel: Lustre: 251666:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 1064 previous similar messages
Feb 24 01:08:07 oak-io2-s1 kernel: LNet: Service thread pid 210326 was inactive for 1202.56s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 24 01:08:07 oak-io2-s1 kernel: LNet: Skipped 4 previous similar messages
Feb 24 01:08:07 oak-io2-s1 kernel: Pid: 210326, comm: ll_ost01_053
Feb 24 01:08:07 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:08:07 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 01:08:07 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 01:08:07 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 01:08:07 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 01:08:07 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 01:08:07 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 01:08:07 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 01:08:07 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 01:08:07 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:08:07 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:08:07 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 01:08:07 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 01:08:07 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:08:07 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:08:07 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:08:07 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:08:07 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:08:07 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:08:07 oak-io2-s1 kernel: 
Feb 24 01:08:07 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519463287.210326
Feb 24 01:08:14 oak-io2-s1 kernel: Lustre: oak-OST0035: Client 3ee95ef7-4278-ead7-52a3-bdca1c47a323 (at 10.9.112.3@o2ib4) reconnecting
Feb 24 01:08:14 oak-io2-s1 kernel: Lustre: Skipped 409 previous similar messages
Feb 24 01:08:25 oak-io2-s1 kernel: LustreError: 4892:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519463005, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST0047_UUID lock: ffff880022b19600/0x806f959362a8121f lrc: 3/0,1 mode: --/PW res: [0x4331b4:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 4892 timeout: 0 lvb_type: 0
Feb 24 01:08:35 oak-io2-s1 kernel: Lustre: oak-OST0030: Export ffff882e8fc84800 already connecting from 10.9.112.15@o2ib4
Feb 24 01:08:35 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 24 01:09:18 oak-io2-s1 kernel: Lustre: oak-OST0049: deleting orphan objects from 0x0:3459139 to 0x0:3459169
Feb 24 01:09:25 oak-io2-s1 kernel: Lustre: oak-OST0042: Connection restored to 9f3e786c-91bd-0b59-f474-23ccea62b4f0 (at 10.8.2.15@o2ib6)
Feb 24 01:09:25 oak-io2-s1 kernel: Lustre: Skipped 432 previous similar messages
Feb 24 01:09:29 oak-io2-s1 kernel: Pid: 362689, comm: ll_ost01_080
Feb 24 01:09:29 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffffc0c1e020>] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc]
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffffc0c1e89d>] ldlm_completion_ast+0x63d/0x920 [ptlrpc]
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffffc0c20730>] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc]
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffffc1220b2c>] ofd_destroy_by_fid+0x1cc/0x4a0 [ofd]
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffffc1213307>] ofd_destroy_hdl+0x267/0x970 [ofd]
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:09:29 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:09:29 oak-io2-s1 kernel: 
Feb 24 01:09:29 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519463369.362689
Feb 24 01:10:18 oak-io2-s1 kernel: Pid: 210327, comm: ll_ost01_054
Feb 24 01:10:18 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:10:18 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 01:10:18 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 01:10:18 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 01:10:18 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 01:10:18 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 01:10:18 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 01:10:18 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 01:10:18 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 01:10:18 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:10:18 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:10:18 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 01:10:18 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 01:10:18 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:10:18 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:10:18 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:10:18 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:10:18 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:10:18 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:10:18 oak-io2-s1 kernel: 
Feb 24 01:10:18 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519463418.210327
Feb 24 01:10:19 oak-io2-s1 kernel: Lustre: oak-OST004d: deleting orphan objects from 0x0:3465827 to 0x0:3466177
Feb 24 01:10:27 oak-io2-s1 kernel: Pid: 4895, comm: ll_ost01_106
Feb 24 01:10:27 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:10:27 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 01:10:27 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 01:10:27 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 01:10:27 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 01:10:27 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 01:10:27 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 01:10:27 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 01:10:27 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 01:10:27 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:10:27 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:10:27 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 01:10:27 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 01:10:27 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:10:27 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:10:27 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:10:27 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:10:27 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:10:27 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:10:27 oak-io2-s1 kernel: 
Feb 24 01:10:27 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519463427.4895
Feb 24 01:10:29 oak-io2-s1 kernel: LustreError: 210338:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1519463129, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-oak-OST0033_UUID lock: ffff8815a477d000/0x806f959362a81774 lrc: 3/0,1 mode: --/PW res: [0x438383:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 210338 timeout: 0 lvb_type: 0
Feb 24 01:10:43 oak-io2-s1 kernel: Pid: 4914, comm: ll_ost01_117
Feb 24 01:10:43 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:10:43 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 01:10:43 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 01:10:43 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 01:10:43 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 01:10:43 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 01:10:43 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 01:10:43 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 01:10:43 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 01:10:43 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:10:43 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:10:43 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 01:10:43 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 01:10:43 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:10:43 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:10:43 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:10:43 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:10:43 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:10:43 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:10:43 oak-io2-s1 kernel: 
Feb 24 01:10:43 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519463443.4914
Feb 24 01:10:55 oak-io2-s1 kernel: LNet: Service thread pid 210335 was inactive for 1201.22s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 24 01:10:55 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519463455.210335
Feb 24 01:11:16 oak-io2-s1 kernel: Lustre: oak-OST0041: deleting orphan objects from 0x0:4442179 to 0x0:4442369
Feb 24 01:11:18 oak-io2-s1 kernel: Lustre: oak-OST004c: deleting orphan objects from 0x0:3455939 to 0x0:3456193
Feb 24 01:12:18 oak-io2-s1 kernel: Lustre: oak-OST0045: deleting orphan objects from 0x0:4421667 to 0x0:4421953
Feb 24 01:12:34 oak-io2-s1 kernel: Lustre: oak-OST0034: deleting orphan objects from 0x0:4290565 to 0x0:4290913
Feb 24 01:12:55 oak-io2-s1 kernel: Lustre: oak-OST0043: deleting orphan objects from 0x0:4403912 to 0x0:4404129
Feb 24 01:12:59 oak-io2-s1 kernel: Lustre: oak-OST0053: deleting orphan objects from 0x0:176199 to 0x0:176450
Feb 24 01:13:04 oak-io2-s1 kernel: Lustre: oak-OST0048: deleting orphan objects from 0x0:3442499 to 0x0:3442785
Feb 24 01:14:01 oak-io2-s1 kernel: Lustre: oak-OST0051: deleting orphan objects from 0x0:176434 to 0x0:176641
Feb 24 01:14:04 oak-io2-s1 kernel: Lustre: oak-OST004b: deleting orphan objects from 0x0:3459791 to 0x0:3459947
Feb 24 01:14:38 oak-io2-s1 kernel: Lustre: oak-OST0046: deleting orphan objects from 0x0:4461867 to 0x0:4461921
Feb 24 01:15:07 oak-io2-s1 kernel: Lustre: oak-OST0035: deleting orphan objects from 0x0:4361668 to 0x0:4361954
Feb 24 01:15:11 oak-io2-s1 kernel: Lustre: oak-OST003f: deleting orphan objects from 0x0:4426054 to 0x0:4426372
Feb 24 01:15:14 oak-io2-s1 kernel: Lustre: oak-OST003e: deleting orphan objects from 0x0:4288655 to 0x0:4288929
Feb 24 01:18:07 oak-io2-s1 kernel: Lustre: 39067:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff88012009ec50 x1592239615714592/t0(0) o4->3ee95ef7-4278-ead7-52a3-bdca1c47a323@10.9.112.3@o2ib4:722/0 lens 608/0 e 0 to 0 dl 1519463892 ref 2 fl New:H/2/ffffffff rc 0/-1
Feb 24 01:18:07 oak-io2-s1 kernel: Lustre: 39067:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 1050 previous similar messages
Feb 24 01:18:09 oak-io2-s1 kernel: LNet: Service thread pid 21433 was inactive for 1202.67s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 24 01:18:09 oak-io2-s1 kernel: LNet: Skipped 4 previous similar messages
Feb 24 01:18:09 oak-io2-s1 kernel: Pid: 21433, comm: ll_ost01_045
Feb 24 01:18:09 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:18:09 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 01:18:09 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 01:18:09 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 01:18:09 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 01:18:09 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 01:18:09 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 01:18:09 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 01:18:09 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 01:18:09 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:18:09 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:18:09 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 01:18:09 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 01:18:09 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:18:09 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:18:09 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:18:09 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:18:09 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:18:09 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:18:09 oak-io2-s1 kernel: 
Feb 24 01:18:09 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519463889.21433
Feb 24 01:18:15 oak-io2-s1 kernel: Lustre: oak-OST0035: Client 3ee95ef7-4278-ead7-52a3-bdca1c47a323 (at 10.9.112.3@o2ib4) reconnecting
Feb 24 01:18:15 oak-io2-s1 kernel: Lustre: Skipped 417 previous similar messages
Feb 24 01:19:26 oak-io2-s1 kernel: Lustre: oak-OST0042: Connection restored to 9f3e786c-91bd-0b59-f474-23ccea62b4f0 (at 10.8.2.15@o2ib6)
Feb 24 01:19:26 oak-io2-s1 kernel: Lustre: Skipped 417 previous similar messages
Feb 24 01:19:27 oak-io2-s1 kernel: Pid: 4900, comm: ll_ost01_111
Feb 24 01:19:27 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffffc0c1e020>] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc]
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffffc0c1e89d>] ldlm_completion_ast+0x63d/0x920 [ptlrpc]
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffffc0c20730>] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc]
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffffc1220b2c>] ofd_destroy_by_fid+0x1cc/0x4a0 [ofd]
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffffc1213307>] ofd_destroy_hdl+0x267/0x970 [ofd]
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:19:27 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:19:27 oak-io2-s1 kernel: 
Feb 24 01:19:27 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519463967.4900
Feb 24 01:21:30 oak-io2-s1 kernel: Pid: 210321, comm: ll_ost01_049
Feb 24 01:21:30 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffffc0c1e020>] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc]
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffffc0c1e89d>] ldlm_completion_ast+0x63d/0x920 [ptlrpc]
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffffc0c20730>] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc]
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffffc1220b2c>] ofd_destroy_by_fid+0x1cc/0x4a0 [ofd]
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffffc1213307>] ofd_destroy_hdl+0x267/0x970 [ofd]
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:21:30 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:21:30 oak-io2-s1 kernel: 
Feb 24 01:21:30 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519464090.210321
Feb 24 01:22:56 oak-io2-s1 kernel: Pid: 21424, comm: ll_ost01_036
Feb 24 01:22:56 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:22:56 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:22:56 oak-io2-s1 kernel: 
Feb 24 01:22:56 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519464176.21424
Feb 24 01:23:00 oak-io2-s1 kernel: Pid: 362687, comm: ll_ost01_078
Feb 24 01:23:00 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:23:00 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 01:23:00 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 01:23:00 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 01:23:00 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 01:23:00 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 01:23:00 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 01:23:00 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 01:23:00 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 01:23:00 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:23:00 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:23:00 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 01:23:00 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 01:23:00 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:23:00 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:23:00 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:23:00 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:23:00 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:23:00 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:23:00 oak-io2-s1 kernel: 
Feb 24 01:23:00 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519464180.362687
Feb 24 01:23:21 oak-io2-s1 kernel: Pid: 362700, comm: ll_ost01_091
Feb 24 01:23:21 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:23:21 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 01:23:21 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 01:23:21 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 01:23:21 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 01:23:21 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 01:23:21 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 01:23:21 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 01:23:21 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 01:23:21 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:23:21 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:23:21 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 01:23:21 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 01:23:21 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:23:21 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:23:21 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:23:21 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:23:21 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:23:21 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:23:21 oak-io2-s1 kernel: 
Feb 24 01:23:21 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519464201.362700
Feb 24 01:23:29 oak-io2-s1 kernel: Pid: 4892, comm: ll_ost01_105
Feb 24 01:23:29 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffffc0c1e020>] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc]
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffffc0c1e89d>] ldlm_completion_ast+0x63d/0x920 [ptlrpc]
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffffc0c20730>] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc]
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffffc1220b2c>] ofd_destroy_by_fid+0x1cc/0x4a0 [ofd]
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffffc1213307>] ofd_destroy_hdl+0x267/0x970 [ofd]
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:23:29 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:23:29 oak-io2-s1 kernel: 
Feb 24 01:23:29 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519464209.4892
Feb 24 01:23:33 oak-io2-s1 kernel: Pid: 132004, comm: ll_ost01_006
Feb 24 01:23:33 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:23:33 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 01:23:33 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 01:23:33 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 01:23:33 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 01:23:33 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 01:23:33 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 01:23:33 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 01:23:33 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 01:23:33 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:23:33 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:23:33 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 01:23:33 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 01:23:33 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:23:33 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:23:33 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:23:33 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:23:33 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:23:33 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:23:33 oak-io2-s1 kernel: 
Feb 24 01:23:33 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519464213.132004
Feb 24 01:25:32 oak-io2-s1 kernel: Pid: 210338, comm: ll_ost01_065
Feb 24 01:25:32 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffffc0c1e020>] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc]
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffffc0c1e89d>] ldlm_completion_ast+0x63d/0x920 [ptlrpc]
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffffc0c20730>] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc]
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffffc1220b2c>] ofd_destroy_by_fid+0x1cc/0x4a0 [ofd]
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffffc0c23e00>] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc]
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffffc0c1e260>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffffc1213307>] ofd_destroy_hdl+0x267/0x970 [ofd]
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:25:32 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:25:32 oak-io2-s1 kernel: 
Feb 24 01:25:32 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519464332.210338
Feb 24 01:26:58 oak-io2-s1 kernel: Pid: 131962, comm: ll_ost01_002
Feb 24 01:26:58 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffff816a94e9>] schedule+0x29/0x70
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffffc0949085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffff810b1910>] ? autoremove_wake_function+0x0/0x40
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffffc0949368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffff813492e0>] ? __percpu_counter_sum+0x70/0x80
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffffc09495e1>] start_this_handle+0x1a1/0x430 [jbd2]
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffffc111fd10>] ? osd_declare_qid+0x1f0/0x480 [osd_ldiskfs]
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffff811df6da>] ? kmem_cache_alloc+0x1ba/0x1e0
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffffc0949a93>] jbd2__journal_start+0xf3/0x1e0 [jbd2]
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffffc10eead4>] ? osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffffc0fce069>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffffc10eead4>] osd_trans_start+0x1b4/0x490 [osd_ldiskfs]
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffffc0cab9ad>] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc]
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffffc0cac0ab>] tgt_client_new+0x41b/0x610 [ptlrpc]
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffffc12204b3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffffc0c13bb6>] target_handle_connect+0x1146/0x2a70 [ptlrpc]
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffff810cdc7c>] ? dequeue_entity+0x11c/0x5d0
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffff810ce500>] ? dequeue_task_fair+0x3d0/0x660
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffff81029557>] ? __switch_to+0xd7/0x510
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffffc0cb6882>] tgt_request_handle+0x402/0x1370 [ptlrpc]
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffff810c4810>] ? default_wake_function+0x0/0x20
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:26:58 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:26:58 oak-io2-s1 kernel: 
Feb 24 01:26:58 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519464418.131962
Feb 24 01:26:58 oak-io2-s1 kernel: LNet: Service thread pid 362699 was inactive for 1202.49s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Feb 24 01:28:07 oak-io2-s1 kernel: Lustre: 210334:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply#012  req@ffff88229eede850 x1593042643189984/t0(0) o5->oak-MDT0000-mdtlov_UUID@10.0.2.52@o2ib5:567/0 lens 432/432 e 0 to 0 dl 1519464492 ref 2 fl Interpret:/0/0 rc 0/0
Feb 24 01:28:07 oak-io2-s1 kernel: Lustre: 210334:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 1067 previous similar messages
Feb 24 01:28:16 oak-io2-s1 kernel: Lustre: oak-OST0035: Client 3ee95ef7-4278-ead7-52a3-bdca1c47a323 (at 10.9.112.3@o2ib4) reconnecting
Feb 24 01:28:16 oak-io2-s1 kernel: Lustre: Skipped 414 previous similar messages
Feb 24 01:29:27 oak-io2-s1 kernel: Lustre: oak-OST0042: Connection restored to 9f3e786c-91bd-0b59-f474-23ccea62b4f0 (at 10.8.2.15@o2ib6)
Feb 24 01:29:27 oak-io2-s1 kernel: Lustre: Skipped 416 previous similar messages
Feb 24 01:30:47 oak-io2-s1 kernel: LNet: Service thread pid 4911 was inactive for 1203.43s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Feb 24 01:30:47 oak-io2-s1 kernel: LNet: Skipped 9 previous similar messages
Feb 24 01:30:47 oak-io2-s1 kernel: Pid: 4911, comm: ll_ost01_116
Feb 24 01:30:47 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:30:47 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 01:30:47 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 01:30:47 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 01:30:47 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 01:30:47 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 01:30:47 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 01:30:47 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 01:30:47 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 01:30:47 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:30:47 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:30:47 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 01:30:47 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 01:30:47 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:30:47 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:30:47 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:30:47 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:30:47 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:30:47 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:30:47 oak-io2-s1 kernel: 
Feb 24 01:30:47 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519464647.4911
Feb 24 01:35:34 oak-io2-s1 kernel: Pid: 196794, comm: ll_ost01_032
Feb 24 01:35:34 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:35:34 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 01:35:34 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 01:35:34 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 01:35:34 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 01:35:34 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 01:35:34 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 01:35:34 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 01:35:34 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 01:35:34 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:35:34 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:35:34 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 01:35:34 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 01:35:34 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:35:34 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:35:34 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:35:34 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:35:34 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:35:34 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:35:34 oak-io2-s1 kernel: 
Feb 24 01:35:34 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519464934.196794
Feb 24 01:35:38 oak-io2-s1 kernel: Pid: 210325, comm: ll_ost01_052
Feb 24 01:35:38 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:35:38 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 01:35:38 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 01:35:38 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 01:35:38 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 01:35:38 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 01:35:38 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 01:35:38 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 01:35:38 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 01:35:38 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:35:38 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:35:38 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 01:35:38 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 01:35:38 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:35:38 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:35:38 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:35:38 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:35:38 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:35:38 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:35:38 oak-io2-s1 kernel: 
Feb 24 01:35:38 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519464938.210325
Feb 24 01:35:58 oak-io2-s1 kernel: Pid: 21432, comm: ll_ost01_044
Feb 24 01:35:58 oak-io2-s1 kernel: #012Call Trace:
Feb 24 01:35:58 oak-io2-s1 kernel: [<ffffffff816aa409>] schedule_preempt_disabled+0x29/0x70
Feb 24 01:35:58 oak-io2-s1 kernel: [<ffffffff816a8337>] __mutex_lock_slowpath+0xc7/0x1d0
Feb 24 01:35:58 oak-io2-s1 kernel: [<ffffffff816a774f>] mutex_lock+0x1f/0x2f
Feb 24 01:35:58 oak-io2-s1 kernel: [<ffffffffc12147db>] ofd_create_hdl+0xdcb/0x2090 [ofd]
Feb 24 01:35:58 oak-io2-s1 kernel: [<ffffffffc0c53a37>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Feb 24 01:35:58 oak-io2-s1 kernel: [<ffffffffc0c53d8f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Feb 24 01:35:58 oak-io2-s1 kernel: [<ffffffffc0c540b1>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Feb 24 01:35:58 oak-io2-s1 kernel: [<ffffffffc0cb6da5>] tgt_request_handle+0x925/0x1370 [ptlrpc]
Feb 24 01:35:58 oak-io2-s1 kernel: [<ffffffffc0c5fb16>] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc]
Feb 24 01:35:58 oak-io2-s1 kernel: [<ffffffffc0c5c148>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc]
Feb 24 01:35:58 oak-io2-s1 kernel: [<ffffffff810c4822>] ? default_wake_function+0x12/0x20
Feb 24 01:35:58 oak-io2-s1 kernel: [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
Feb 24 01:35:58 oak-io2-s1 kernel: [<ffffffffc0c63252>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
Feb 24 01:35:58 oak-io2-s1 kernel: [<ffffffffc0c627c0>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
Feb 24 01:35:58 oak-io2-s1 kernel: [<ffffffff810b098f>] kthread+0xcf/0xe0
Feb 24 01:35:58 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:35:58 oak-io2-s1 kernel: [<ffffffff816b4f58>] ret_from_fork+0x58/0x90
Feb 24 01:35:58 oak-io2-s1 kernel: [<ffffffff810b08c0>] ? kthread+0x0/0xe0
Feb 24 01:35:58 oak-io2-s1 kernel: 
Feb 24 01:35:58 oak-io2-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1519464958.21432
Feb 24 01:38:09 oak-io2-s1 kernel: Lustre: 251666:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff8826bea0cc50 x1592239615714592/t0(0) o4->3ee95ef7-4278-ead7-52a3-bdca1c47a323@10.9.112.3@o2ib4:414/0 lens 608/0 e 0 to 0 dl 1519465094 ref 2 fl New:H/2/ffffffff rc 0/-1
Feb 24 01:38:09 oak-io2-s1 kernel: Lustre: 251666:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 1087 previous similar messages
Feb 24 01:38:17 oak-io2-s1 kernel: Lustre: oak-OST0035: Client 3ee95ef7-4278-ead7-52a3-bdca1c47a323 (at 10.9.112.3@o2ib4) reconnecting
Feb 24 01:38:17 oak-io2-s1 kernel: Lustre: Skipped 401 previous similar messages
Feb 24 01:39:28 oak-io2-s1 kernel: Lustre: oak-OST0042: Connection restored to 9f3e786c-91bd-0b59-f474-23ccea62b4f0 (at 10.8.2.15@o2ib6)
Feb 24 01:39:28 oak-io2-s1 kernel: Lustre: Skipped 404 previous similar messages
Feb 24 01:48:10 oak-io2-s1 kernel: Lustre: 39067:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff8837b4e10050 x1592239615714592/t0(0) o4->3ee95ef7-4278-ead7-52a3-bdca1c47a323@10.9.112.3@o2ib4:260/0 lens 608/0 e 0 to 0 dl 1519465695 ref 2 fl New:H/2/ffffffff rc 0/-1
Feb 24 01:48:10 oak-io2-s1 kernel: Lustre: 39067:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 1077 previous similar messages
Feb 24 01:48:17 oak-io2-s1 kernel: Lustre: oak-OST003b: Client 3326eb74-fc4c-e39b-1783-8c55bbb22498 (at 10.9.112.6@o2ib4) reconnecting
Feb 24 01:48:17 oak-io2-s1 kernel: Lustre: Skipped 411 previous similar messages
Feb 24 01:49:29 oak-io2-s1 kernel: Lustre: oak-OST0042: Connection restored to 9f3e786c-91bd-0b59-f474-23ccea62b4f0 (at 10.8.2.15@o2ib6)
Feb 24 01:49:29 oak-io2-s1 kernel: Lustre: Skipped 410 previous similar messages
Feb 24 01:58:11 oak-io2-s1 kernel: Lustre: 251666:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff882b61b06c50 x1592239615714592/t0(0) o4->3ee95ef7-4278-ead7-52a3-bdca1c47a323@10.9.112.3@o2ib4:106/0 lens 608/0 e 0 to 0 dl 1519466296 ref 2 fl New:H/2/ffffffff rc 0/-1
Feb 24 01:58:11 oak-io2-s1 kernel: Lustre: 251666:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 1083 previous similar messages
Feb 24 01:58:19 oak-io2-s1 kernel: Lustre: oak-OST0035: Client 3ee95ef7-4278-ead7-52a3-bdca1c47a323 (at 10.9.112.3@o2ib4) reconnecting
Feb 24 01:58:19 oak-io2-s1 kernel: Lustre: Skipped 421 previous similar messages
Feb 24 01:59:30 oak-io2-s1 kernel: Lustre: oak-OST0042: Connection restored to 9f3e786c-91bd-0b59-f474-23ccea62b4f0 (at 10.8.2.15@o2ib6)
Feb 24 01:59:30 oak-io2-s1 kernel: Lustre: Skipped 417 previous similar messages