Jun 19 04:01:39 oak-io1-s2 kernel: Lustre: oak-OST0003: haven't heard from client 15ff5fa6-1dda-1b77-7689-0f45e67e26d3 (at 10.210.47.121@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883d4652cc00, cur 1497870099 expire 1497869949 last 1497869872 Jun 19 04:01:39 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 19 04:16:03 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to 15ff5fa6-1dda-1b77-7689-0f45e67e26d3 (at 10.210.47.121@o2ib3) Jun 19 04:16:03 oak-io1-s2 kernel: Lustre: Skipped 14 previous similar messages Jun 19 09:23:41 oak-io1-s2 kernel: Lustre: oak-OST0001: haven't heard from client 12026e20-f0f2-5703-1078-c1a349ee4566 (at 10.9.101.48@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883d40415000, cur 1497889421 expire 1497889271 last 1497889194 Jun 19 09:23:41 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 19 09:23:41 oak-io1-s2 kernel: Lustre: oak-OST0011: haven't heard from client 12026e20-f0f2-5703-1078-c1a349ee4566 (at 10.9.101.48@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883d3b790000, cur 1497889421 expire 1497889271 last 1497889194 Jun 19 09:23:41 oak-io1-s2 kernel: Lustre: Skipped 27 previous similar messages Jun 19 09:32:08 oak-io1-s2 kernel: Lustre: oak-OST001d: haven't heard from client 760cee34-c3ce-b50d-dae6-867f8e12fcd4 (at 10.12.4.36@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d25ec6800, cur 1497889928 expire 1497889778 last 1497889701 Jun 19 09:32:08 oak-io1-s2 kernel: Lustre: Skipped 7 previous similar messages Jun 19 09:33:24 oak-io1-s2 kernel: Lustre: oak-OST0011: haven't heard from client fc2a3fda-1eba-7ea7-09ba-6c1b1dd28fd0 (at 10.12.4.49@o2ib) in 222 seconds. I think it's dead, and I am evicting it. exp ffff883d3b76fc00, cur 1497890004 expire 1497889854 last 1497889782 Jun 19 09:33:24 oak-io1-s2 kernel: Lustre: Skipped 125 previous similar messages Jun 19 09:35:26 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to (at 10.9.101.48@o2ib4) Jun 19 09:35:26 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 19 09:35:26 oak-io1-s2 kernel: Lustre: oak-OST0015: Connection restored to (at 10.9.101.48@o2ib4) Jun 19 09:36:19 oak-io1-s2 kernel: Lustre: oak-OST0003: Connection restored to (at 10.9.101.55@o2ib4) Jun 19 09:36:19 oak-io1-s2 kernel: Lustre: Skipped 16 previous similar messages Jun 19 09:59:21 oak-io1-s2 kernel: Lustre: oak-OST0005: Connection restored to a8021b60-575f-f6bd-5e29-972e8cd71dae (at 10.12.4.29@o2ib) Jun 19 09:59:21 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to a8021b60-575f-f6bd-5e29-972e8cd71dae (at 10.12.4.29@o2ib) Jun 19 09:59:21 oak-io1-s2 kernel: Lustre: Skipped 15 previous similar messages Jun 19 10:13:54 oak-io1-s2 kernel: Lustre: oak-OST000d: Connection restored to 282fcab8-ace1-1fa0-935a-c755717a1c83 (at 10.12.4.26@o2ib) Jun 19 10:13:54 oak-io1-s2 kernel: Lustre: Skipped 11 previous similar messages Jun 19 10:28:51 oak-io1-s2 kernel: Lustre: oak-OST000b: Connection restored to 760cee34-c3ce-b50d-dae6-867f8e12fcd4 (at 10.12.4.36@o2ib) Jun 19 10:28:51 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 19 11:01:10 oak-io1-s2 kernel: Lustre: oak-OST0007: Connection restored to 92921068-c24c-6515-b6d6-f605500e2b0d (at 10.12.4.40@o2ib) Jun 19 11:01:10 oak-io1-s2 kernel: Lustre: Skipped 5 previous similar messages Jun 19 11:11:17 oak-io1-s2 kernel: Lustre: oak-OST0015: Connection restored to 657e12ab-e8cd-dbe8-604e-55778c3efb90 (at 10.12.4.45@o2ib) Jun 19 11:11:17 oak-io1-s2 kernel: Lustre: Skipped 22 previous similar messages Jun 19 11:39:02 oak-io1-s2 kernel: nr_pdflush_threads exported in /proc is scheduled for removal Jun 19 11:39:59 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to 3103ddc7-20c4-2f67-94b7-da1602948f53 (at 10.12.4.56@o2ib) Jun 19 11:39:59 oak-io1-s2 kernel: Lustre: oak-OST0003: Connection restored to 3103ddc7-20c4-2f67-94b7-da1602948f53 (at 10.12.4.56@o2ib) Jun 19 11:39:59 oak-io1-s2 kernel: Lustre: Skipped 5 previous similar messages Jun 19 11:39:59 oak-io1-s2 kernel: Lustre: Skipped 15 previous similar messages Jun 19 11:40:00 oak-io1-s2 kernel: Lustre: oak-OST000f: haven't heard from client 42a586d2-3b08-f7d9-669a-6fdabc788e64 (at 10.12.4.72@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883e050ca400, cur 1497897600 expire 1497897450 last 1497897373 Jun 19 11:40:00 oak-io1-s2 kernel: Lustre: Skipped 35 previous similar messages Jun 19 11:40:00 oak-io1-s2 kernel: Lustre: oak-OST0005: haven't heard from client 9b40696c-affe-0205-20f2-c1ce52e70ca6 (at 10.12.4.65@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883d4110ec00, cur 1497897600 expire 1497897450 last 1497897373 Jun 19 11:40:00 oak-io1-s2 kernel: Lustre: Skipped 27 previous similar messages Jun 19 11:41:14 oak-io1-s2 kernel: LustreError: 137-5: oak-OST0000_UUID: not available for connect from 10.12.4.56@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 19 11:41:14 oak-io1-s2 kernel: LustreError: Skipped 477 previous similar messages Jun 19 11:41:16 oak-io1-s2 kernel: Lustre: oak-OST0003: Connection restored to db311064-a0d6-f916-d16a-20bbdf0b4a04 (at 10.12.4.63@o2ib) Jun 19 11:41:16 oak-io1-s2 kernel: Lustre: Skipped 16 previous similar messages Jun 19 11:42:31 oak-io1-s2 kernel: LustreError: 137-5: oak-OST0000_UUID: not available for connect from 10.12.4.63@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 19 11:42:31 oak-io1-s2 kernel: LustreError: Skipped 1 previous similar message Jun 19 11:43:31 oak-io1-s2 kernel: Lustre: oak-OST000b: haven't heard from client 55ce9471-631b-80da-f3ed-95e9f07eb66f (at 10.210.44.249@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883e09c31000, cur 1497897811 expire 1497897661 last 1497897584 Jun 19 11:43:31 oak-io1-s2 kernel: Lustre: Skipped 25 previous similar messages Jun 19 11:43:44 oak-io1-s2 kernel: LustreError: 137-5: oak-OST0000_UUID: not available for connect from 10.12.4.56@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 19 11:43:44 oak-io1-s2 kernel: LustreError: Skipped 7 previous similar messages Jun 19 11:45:01 oak-io1-s2 kernel: LustreError: 137-5: oak-OST0002_UUID: not available for connect from 10.12.4.63@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 19 11:45:01 oak-io1-s2 kernel: LustreError: Skipped 4 previous similar messages Jun 19 11:46:14 oak-io1-s2 kernel: LustreError: 137-5: oak-OST0002_UUID: not available for connect from 10.12.4.56@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 19 11:46:14 oak-io1-s2 kernel: LustreError: Skipped 4 previous similar messages Jun 19 11:47:31 oak-io1-s2 kernel: LustreError: 137-5: oak-OST0000_UUID: not available for connect from 10.12.4.63@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 19 11:47:31 oak-io1-s2 kernel: LustreError: Skipped 4 previous similar messages Jun 19 11:48:10 oak-io1-s2 kernel: Lustre: oak-OST0015: Client 8918f908-fa3d-ce3a-2125-68fbdaa4ad98 (at 10.210.44.179@o2ib3) reconnecting Jun 19 11:48:10 oak-io1-s2 kernel: Lustre: oak-OST0019: Connection restored to 8918f908-fa3d-ce3a-2125-68fbdaa4ad98 (at 10.210.44.179@o2ib3) Jun 19 11:48:10 oak-io1-s2 kernel: Lustre: Skipped 140 previous similar messages Jun 19 11:48:11 oak-io1-s2 kernel: LustreError: 244024:0:(ldlm_lib.c:3230:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff883fe57b6450 x1570574442249280/t0(0) o3->23e37277-a86e-8c15-c880-4b2585f688cb@10.210.44.248@o2ib3:-1/-1 lens 488/432 e 1 to 0 dl 1497898121 ref 1 fl Interpret:/0/0 rc 0/0 Jun 19 11:48:11 oak-io1-s2 kernel: Lustre: oak-OST0001: Bulk IO read error with 23e37277-a86e-8c15-c880-4b2585f688cb (at 10.210.44.248@o2ib3), client will retry: rc -110 Jun 19 11:48:11 oak-io1-s2 kernel: LustreError: 244024:0:(ldlm_lib.c:3230:target_bulk_io()) Skipped 15 previous similar messages Jun 19 11:48:22 oak-io1-s2 kernel: Lustre: oak-OST0011: Client 16c54162-c341-4fba-f314-0be950a00b07 (at 10.9.101.55@o2ib4) reconnecting Jun 19 11:48:22 oak-io1-s2 kernel: Lustre: oak-OST0013: Connection restored to (at 10.9.101.55@o2ib4) Jun 19 11:48:22 oak-io1-s2 kernel: Lustre: Skipped 14427 previous similar messages Jun 19 11:48:22 oak-io1-s2 kernel: Lustre: Skipped 14509 previous similar messages Jun 19 11:48:23 oak-io1-s2 kernel: LustreError: 130921:0:(ldlm_lib.c:3230:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff881fdfb22c50 x1568853701081232/t0(0) o4->1b1c6319-5ec1-dbcc-f6e7-f85575c95c4c@10.9.112.3@o2ib4:-1/-1 lens 608/448 e 0 to 0 dl 1497898108 ref 1 fl Interpret:/0/0 rc 0/0 Jun 19 11:48:23 oak-io1-s2 kernel: Lustre: oak-OST000b: Bulk IO write error with 1b1c6319-5ec1-dbcc-f6e7-f85575c95c4c (at 10.9.112.3@o2ib4), client will retry: rc -110 Jun 19 11:48:23 oak-io1-s2 kernel: Lustre: Skipped 21 previous similar messages Jun 19 11:48:23 oak-io1-s2 kernel: Lustre: oak-OST0019: Bulk IO read error with 1b1c6319-5ec1-dbcc-f6e7-f85575c95c4c (at 10.9.112.3@o2ib4), client will retry: rc -110 Jun 19 11:48:23 oak-io1-s2 kernel: Lustre: Skipped 32 previous similar messages Jun 19 11:48:23 oak-io1-s2 kernel: LustreError: 130921:0:(ldlm_lib.c:3230:target_bulk_io()) Skipped 41 previous similar messages Jun 19 11:48:27 oak-io1-s2 kernel: Lustre: oak-OST0003: Connection restored to 6c06a1a9-2069-17d6-2e02-6d784c5aaf2e (at 10.210.46.2@o2ib3) Jun 19 11:48:27 oak-io1-s2 kernel: Lustre: Skipped 3995 previous similar messages Jun 19 11:48:35 oak-io1-s2 kernel: Lustre: oak-OST0019: Connection restored to 6b4cb54a-5373-d347-6637-89247cb9f354 (at 10.9.101.69@o2ib4) Jun 19 11:48:35 oak-io1-s2 kernel: Lustre: Skipped 177 previous similar messages Jun 19 11:48:35 oak-io1-s2 kernel: LustreError: 137-5: oak-OST0000_UUID: not available for connect from 10.9.103.8@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 19 11:48:35 oak-io1-s2 kernel: LustreError: Skipped 955 previous similar messages Jun 19 11:48:38 oak-io1-s2 kernel: Lustre: oak-OST0001: Client 481d2032-c582-eba0-3961-2ab0094a7d96 (at 10.210.47.36@o2ib3) reconnecting Jun 19 11:48:38 oak-io1-s2 kernel: Lustre: Skipped 3976 previous similar messages Jun 19 11:48:51 oak-io1-s2 kernel: Lustre: oak-OST0003: Connection restored to 7df2edb4-f5ea-8ea5-8029-3afad9527ea5 (at 10.210.44.99@o2ib3) Jun 19 11:48:51 oak-io1-s2 kernel: Lustre: Skipped 5745 previous similar messages Jun 19 11:49:11 oak-io1-s2 kernel: Lustre: oak-OST0001: Client 43ece2b6-3455-6435-636d-7eeafb46172d (at 10.9.102.11@o2ib4) reconnecting Jun 19 11:49:11 oak-io1-s2 kernel: Lustre: Skipped 15833 previous similar messages Jun 19 11:49:23 oak-io1-s2 kernel: Lustre: oak-OST000d: Connection restored to 19609051-171f-6e29-7fcf-8e767e18c3e9 (at 10.210.47.240@o2ib3) Jun 19 11:49:23 oak-io1-s2 kernel: Lustre: Skipped 11165 previous similar messages Jun 19 11:50:43 oak-io1-s2 kernel: LustreError: 137-5: oak-OST0016_UUID: not available for connect from 10.9.113.6@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 19 11:50:43 oak-io1-s2 kernel: LustreError: Skipped 7334 previous similar messages Jun 19 11:54:29 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to 9b40696c-affe-0205-20f2-c1ce52e70ca6 (at 10.12.4.65@o2ib) Jun 19 11:54:29 oak-io1-s2 kernel: Lustre: Skipped 62 previous similar messages Jun 19 11:54:59 oak-io1-s2 kernel: LustreError: 137-5: oak-OST0000_UUID: not available for connect from 10.9.104.4@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. Jun 19 11:54:59 oak-io1-s2 kernel: LustreError: Skipped 20658 previous similar messages Jun 19 12:08:49 oak-io1-s2 kernel: Lustre: oak-OST0003: Connection restored to 42a586d2-3b08-f7d9-669a-6fdabc788e64 (at 10.12.4.72@o2ib) Jun 19 12:08:49 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 19 12:18:33 oak-io1-s2 kernel: Lustre: oak-OST0017: haven't heard from client 05ae0637-c573-a682-81c8-0fb016ffc6a0 (at 10.210.47.54@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883df0069c00, cur 1497899913 expire 1497899763 last 1497899686 Jun 19 12:18:33 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 19 12:21:00 oak-io1-s2 kernel: Lustre: oak-OST0005: Connection restored to d80af44e-768f-4052-8e51-0582577c0662 (at 10.12.4.73@o2ib) Jun 19 12:21:00 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to d80af44e-768f-4052-8e51-0582577c0662 (at 10.12.4.73@o2ib) Jun 19 12:21:00 oak-io1-s2 kernel: Lustre: Skipped 1 previous similar message Jun 19 12:21:00 oak-io1-s2 kernel: Lustre: Skipped 15 previous similar messages Jun 19 12:22:42 oak-io1-s2 systemd: Binding to IPv6 address not available since kernel does not support IPv6. Jun 19 12:22:59 oak-io1-s2 systemd: Binding to IPv6 address not available since kernel does not support IPv6. Jun 19 12:26:36 oak-io1-s2 kernel: Lustre: oak-OST0015: Connection restored to 55ce9471-631b-80da-f3ed-95e9f07eb66f (at 10.210.44.249@o2ib3) Jun 19 12:26:36 oak-io1-s2 kernel: Lustre: Skipped 8 previous similar messages Jun 19 12:27:04 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to 50f915f1-0494-6217-ad81-34c86b06a358 (at 10.12.4.62@o2ib) Jun 19 12:27:04 oak-io1-s2 kernel: Lustre: Skipped 23 previous similar messages Jun 19 12:29:49 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to 52f241fa-e558-a46a-7dfc-bb8c0d8d2fac (at 10.12.4.38@o2ib) Jun 19 12:29:49 oak-io1-s2 kernel: Lustre: Skipped 18 previous similar messages Jun 19 12:30:40 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to 4a89c86c-798b-6365-298b-e2b714e84497 (at 10.210.46.160@o2ib3) Jun 19 12:30:40 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 19 12:34:23 oak-io1-s2 kernel: Lustre: oak-OST0003: Connection restored to 05ae0637-c573-a682-81c8-0fb016ffc6a0 (at 10.210.47.54@o2ib3) Jun 19 12:34:23 oak-io1-s2 kernel: Lustre: Skipped 33 previous similar messages Jun 19 12:36:04 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to 58c8ea3d-8ae5-7534-6967-edb021eb4c01 (at 10.210.47.18@o2ib3) Jun 19 12:36:04 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 19 13:47:40 oak-io1-s2 kernel: Lustre: oak-OST0013: haven't heard from client c83d9db0-eed9-3e10-b974-ee38d75923a6 (at 10.12.4.22@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883ded584000, cur 1497905260 expire 1497905110 last 1497905033 Jun 19 13:47:40 oak-io1-s2 kernel: Lustre: Skipped 53 previous similar messages Jun 19 13:47:40 oak-io1-s2 kernel: Lustre: oak-OST0001: haven't heard from client a917e282-3d5a-b5ea-4ed2-3aef729f9927 (at 10.12.4.21@o2ib) in 226 seconds. I think it's dead, and I am evicting it. exp ffff883d4044c400, cur 1497905260 expire 1497905110 last 1497905034 Jun 19 13:47:40 oak-io1-s2 kernel: Lustre: Skipped 99 previous similar messages Jun 19 13:48:56 oak-io1-s2 kernel: Lustre: oak-OST0019: haven't heard from client 967023d2-464a-4422-881e-67a9d9ef1a98 (at 10.12.4.45@o2ib) in 224 seconds. I think it's dead, and I am evicting it. exp ffff883d42749800, cur 1497905336 expire 1497905186 last 1497905112 Jun 19 13:48:56 oak-io1-s2 kernel: Lustre: Skipped 61 previous similar messages Jun 19 14:02:52 oak-io1-s2 kernel: Lustre: 275625:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497906165/real 1497906165] req@ffff881d26b7e900 x1570579276608976/t0(0) o601->oak-MDT0000-lwp-OST000b@10.0.2.51@o2ib5:23/10 lens 336/336 e 0 to 1 dl 1497906172 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 19 14:02:52 oak-io1-s2 kernel: Lustre: 275625:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 36 previous similar messages Jun 19 14:02:52 oak-io1-s2 kernel: Lustre: oak-MDT0000-lwp-OST000b: Connection to oak-MDT0000 (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jun 19 14:02:52 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 19 14:03:47 oak-io1-s2 kernel: Lustre: 103394:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497906197/real 1497906197] req@ffff881d15c3aa00 x1570579276629376/t0(0) o400->MGC10.0.2.51@o2ib5@10.0.2.51@o2ib5:26/25 lens 224/224 e 0 to 1 dl 1497906204 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 19 14:03:47 oak-io1-s2 kernel: Lustre: 103394:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jun 19 14:03:47 oak-io1-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail Jun 19 14:04:15 oak-io1-s2 kernel: LNetError: 103335:0:(o2iblnd_cb.c:3126:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 8 seconds Jun 19 14:04:15 oak-io1-s2 kernel: LNetError: 103335:0:(o2iblnd_cb.c:3189:kiblnd_check_conns()) Timed out RDMA with 10.0.2.51@o2ib5 (58): c: 0, oc: 0, rc: 8 Jun 19 14:04:15 oak-io1-s2 kernel: Lustre: oak-MDT0000-lwp-OST0001: Connection to oak-MDT0000 (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jun 19 14:04:15 oak-io1-s2 kernel: Lustre: Skipped 7 previous similar messages Jun 19 14:05:54 oak-io1-s2 kernel: Lustre: 103388:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1497906197/real 1497906354] req@ffff881d15c3bc00 x1570579276629616/t0(0) o400->oak-MDT0000-lwp-OST0019@10.0.2.51@o2ib5:12/10 lens 224/224 e 0 to 1 dl 1497906865 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Jun 19 14:05:54 oak-io1-s2 kernel: Lustre: oak-MDT0000-lwp-OST0015: Connection to oak-MDT0000 (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Jun 19 14:05:54 oak-io1-s2 kernel: Lustre: 103388:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 88 previous similar messages Jun 19 14:06:05 oak-io1-s2 kernel: LNet: Service thread pid 244872 was inactive for 200.44s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 19 14:06:05 oak-io1-s2 kernel: Pid: 244872, comm: ll_ost_io00_059 Jun 19 14:06:05 oak-io1-s2 kernel: #012Call Trace: Jun 19 14:06:05 oak-io1-s2 kernel: [] schedule+0x29/0x70 Jun 19 14:06:05 oak-io1-s2 kernel: [] schedule_timeout+0x174/0x2c0 Jun 19 14:06:05 oak-io1-s2 kernel: [] ? process_timeout+0x0/0x10 Jun 19 14:06:05 oak-io1-s2 kernel: [] ptlrpc_set_wait+0x4c0/0x900 [ptlrpc] Jun 19 14:06:05 oak-io1-s2 kernel: [] ? default_wake_function+0x0/0x20 Jun 19 14:06:05 oak-io1-s2 kernel: [] ? qsd_req_completion+0x0/0xb20 [lquota] Jun 19 14:06:05 oak-io1-s2 kernel: [] ptlrpc_queue_wait+0x7d/0x220 [ptlrpc] Jun 19 14:06:05 oak-io1-s2 kernel: [] qsd_send_dqacq+0x2e8/0x340 [lquota] Jun 19 14:06:05 oak-io1-s2 kernel: [] qsd_acquire+0x983/0xc90 [lquota] Jun 19 14:06:05 oak-io1-s2 kernel: [] qsd_op_begin0+0x181/0x930 [lquota] Jun 19 14:06:05 oak-io1-s2 kernel: [] ? lu_obj_hop_keycmp+0x1d/0x30 [obdclass] Jun 19 14:06:05 oak-io1-s2 kernel: [] qsd_op_begin+0x174/0x460 [lquota] Jun 19 14:06:05 oak-io1-s2 kernel: [] osd_declare_qid+0x1f9/0x4d0 [osd_ldiskfs] Jun 19 14:06:05 oak-io1-s2 kernel: [] osd_declare_inode_qid+0x1ae/0x290 [osd_ldiskfs] Jun 19 14:06:05 oak-io1-s2 kernel: [] osd_declare_write_commit+0x2db/0x4d0 [osd_ldiskfs] Jun 19 14:06:05 oak-io1-s2 kernel: [] ofd_commitrw_write+0x72d/0x1c20 [ofd] Jun 19 14:06:05 oak-io1-s2 kernel: [] ofd_commitrw+0x51d/0xa40 [ofd] Jun 19 14:06:05 oak-io1-s2 kernel: [] obd_commitrw+0x2ec/0x32f [ptlrpc] Jun 19 14:06:05 oak-io1-s2 kernel: [] tgt_brw_write+0xea1/0x1640 [ptlrpc] Jun 19 14:06:05 oak-io1-s2 kernel: [] ? update_curr+0x104/0x190 Jun 19 14:06:05 oak-io1-s2 kernel: [] ? cpumask_next_and+0x35/0x50 Jun 19 14:06:05 oak-io1-s2 kernel: [] ? target_bulk_timeout+0x0/0xb0 [ptlrpc] Jun 19 14:06:05 oak-io1-s2 kernel: [] tgt_request_handle+0x915/0x1320 [ptlrpc] Jun 19 14:06:05 oak-io1-s2 kernel: [] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc] Jun 19 14:06:05 oak-io1-s2 kernel: [] ? lc_watchdog_touch+0x68/0x180 [libcfs] Jun 19 14:06:05 oak-io1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jun 19 14:06:05 oak-io1-s2 kernel: [] ? default_wake_function+0x12/0x20 Jun 19 14:06:05 oak-io1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Jun 19 14:06:05 oak-io1-s2 kernel: [] ptlrpc_main+0xaa0/0x1de0 [ptlrpc] Jun 19 14:06:05 oak-io1-s2 kernel: [] ? ptlrpc_main+0x0/0x1de0 [ptlrpc] Jun 19 14:06:05 oak-io1-s2 kernel: [] kthread+0xcf/0xe0 Jun 19 14:06:05 oak-io1-s2 kernel: [] ? kthread+0x0/0xe0 Jun 19 14:06:05 oak-io1-s2 kernel: [] ret_from_fork+0x58/0x90 Jun 19 14:06:05 oak-io1-s2 kernel: [] ? kthread+0x0/0xe0 Jun 19 14:06:05 oak-io1-s2 kernel: Jun 19 14:06:05 oak-io1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1497906365.244872 Jun 19 14:06:28 oak-io1-s2 kernel: Lustre: oak-OST0019: haven't heard from client oak-MDT0000-mdtlov_UUID (at 10.0.2.51@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883d40b83c00, cur 1497906388 expire 1497906238 last 1497906161 Jun 19 14:06:28 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 19 14:06:29 oak-io1-s2 kernel: Lustre: oak-OST0017: haven't heard from client oak-MDT0000-mdtlov_UUID (at 10.0.2.51@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883fb135d400, cur 1497906389 expire 1497906239 last 1497906162 Jun 19 14:06:29 oak-io1-s2 kernel: Lustre: Skipped 16 previous similar messages Jun 19 14:06:37 oak-io1-s2 kernel: LNet: Service thread pid 244872 completed after 232.05s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jun 19 14:08:20 oak-io1-s2 kernel: Lustre: oak-OST0003: Connection restored to oak-MDT0000-mdtlov_UUID (at 10.0.2.52@o2ib5) Jun 19 14:08:20 oak-io1-s2 kernel: Lustre: Skipped 2 previous similar messages Jun 19 14:08:20 oak-io1-s2 kernel: Lustre: 103377:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497906479/real 1497906479] req@ffff881db1793f00 x1570579276887984/t0(0) o250->MGC10.0.2.51@o2ib5@10.0.2.51@o2ib5:26/25 lens 520/544 e 0 to 1 dl 1497906500 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 19 14:08:20 oak-io1-s2 kernel: Lustre: 103377:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 80 previous similar messages Jun 19 14:08:49 oak-io1-s2 kernel: Lustre: Evicted from MGS (at MGC10.0.2.51@o2ib5_0) after server handle changed from 0xf0d4e97d03ca3937 to 0xbfb40de81fe96cca Jun 19 14:08:49 oak-io1-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to MGC10.0.2.51@o2ib5_0 (at 10.0.2.51@o2ib5) Jun 19 14:08:49 oak-io1-s2 kernel: Lustre: Skipped 15 previous similar messages Jun 19 14:09:14 oak-io1-s2 kernel: LustreError: 167-0: oak-MDT0000-lwp-OST0017: This client was evicted by oak-MDT0000; in progress operations using this service will fail. Jun 19 14:09:14 oak-io1-s2 kernel: LustreError: Skipped 9 previous similar messages Jun 19 14:09:14 oak-io1-s2 kernel: LustreError: 11-0: oak-MDT0000-lwp-OST000b: operation quota_acquire to node 10.0.2.52@o2ib5 failed: rc = -11 Jun 19 14:09:14 oak-io1-s2 kernel: LustreError: Skipped 11 previous similar messages Jun 19 14:09:16 oak-io1-s2 kernel: LustreError: 11-0: oak-MDT0000-lwp-OST000b: operation quota_acquire to node 10.0.2.52@o2ib5 failed: rc = -11 Jun 19 14:09:16 oak-io1-s2 kernel: LustreError: Skipped 109 previous similar messages Jun 19 14:09:20 oak-io1-s2 kernel: LustreError: 11-0: oak-MDT0000-lwp-OST000b: operation quota_acquire to node 10.0.2.52@o2ib5 failed: rc = -11 Jun 19 14:09:20 oak-io1-s2 kernel: LustreError: Skipped 209 previous similar messages Jun 19 14:09:28 oak-io1-s2 kernel: LustreError: 11-0: oak-MDT0000-lwp-OST000b: operation quota_acquire to node 10.0.2.52@o2ib5 failed: rc = -11 Jun 19 14:09:28 oak-io1-s2 kernel: LustreError: Skipped 400 previous similar messages Jun 19 14:09:44 oak-io1-s2 kernel: LustreError: 11-0: oak-MDT0000-lwp-OST000b: operation quota_acquire to node 10.0.2.52@o2ib5 failed: rc = -11 Jun 19 14:09:44 oak-io1-s2 kernel: LustreError: Skipped 843 previous similar messages Jun 19 14:10:16 oak-io1-s2 kernel: LustreError: 11-0: oak-MDT0000-lwp-OST000b: operation quota_acquire to node 10.0.2.52@o2ib5 failed: rc = -11 Jun 19 14:10:16 oak-io1-s2 kernel: LustreError: Skipped 1686 previous similar messages Jun 19 14:11:20 oak-io1-s2 kernel: LustreError: 11-0: oak-MDT0000-lwp-OST000b: operation quota_acquire to node 10.0.2.52@o2ib5 failed: rc = -11 Jun 19 14:11:20 oak-io1-s2 kernel: LustreError: Skipped 3372 previous similar messages Jun 19 14:13:28 oak-io1-s2 kernel: LustreError: 11-0: oak-MDT0000-lwp-OST000b: operation quota_acquire to node 10.0.2.52@o2ib5 failed: rc = -11 Jun 19 14:13:28 oak-io1-s2 kernel: LustreError: Skipped 6727 previous similar messages Jun 19 14:17:44 oak-io1-s2 kernel: LustreError: 11-0: oak-MDT0000-lwp-OST000b: operation quota_acquire to node 10.0.2.52@o2ib5 failed: rc = -11 Jun 19 14:17:44 oak-io1-s2 kernel: LustreError: Skipped 13708 previous similar messages Jun 19 14:18:49 oak-io1-s2 kernel: Lustre: oak-OST0015: haven't heard from client 885baf8c-870a-8435-ca79-8ae574ed2fb6 (at 10.12.4.29@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bac5f2800, cur 1497907129 expire 1497906979 last 1497906902 Jun 19 14:22:35 oak-io1-s2 kernel: Lustre: oak-OST0001: deleting orphan objects from 0x0:2549971 to 0x0:2550049 Jun 19 14:22:35 oak-io1-s2 kernel: Lustre: oak-OST0003: deleting orphan objects from 0x0:2606977 to 0x0:2607009 Jun 19 14:22:35 oak-io1-s2 kernel: Lustre: oak-OST0005: deleting orphan objects from 0x0:2580700 to 0x0:2580737 Jun 19 14:22:35 oak-io1-s2 kernel: Lustre: oak-OST0009: deleting orphan objects from 0x0:2528702 to 0x0:2528737 Jun 19 14:22:35 oak-io1-s2 kernel: Lustre: oak-OST0007: deleting orphan objects from 0x0:2500749 to 0x0:2500769 Jun 19 14:22:35 oak-io1-s2 kernel: Lustre: oak-OST000b: deleting orphan objects from 0x0:2536249 to 0x0:2536289 Jun 19 14:22:35 oak-io1-s2 kernel: Lustre: oak-OST000d: deleting orphan objects from 0x0:2545681 to 0x0:2545697 Jun 19 14:22:35 oak-io1-s2 kernel: Lustre: oak-OST0017: deleting orphan objects from 0x0:2513866 to 0x0:2513921 Jun 19 14:22:35 oak-io1-s2 kernel: Lustre: oak-OST0013: deleting orphan objects from 0x0:2506451 to 0x0:2506497 Jun 19 14:22:35 oak-io1-s2 kernel: Lustre: oak-OST0019: deleting orphan objects from 0x0:2469024 to 0x0:2469057 Jun 19 14:22:35 oak-io1-s2 kernel: Lustre: oak-OST001f: deleting orphan objects from 0x0:1938644 to 0x0:1938689 Jun 19 14:22:35 oak-io1-s2 kernel: Lustre: oak-OST0023: deleting orphan objects from 0x0:1929116 to 0x0:1929153 Jun 19 14:22:35 oak-io1-s2 kernel: Lustre: oak-OST0011: deleting orphan objects from 0x0:2515481 to 0x0:2515521 Jun 19 14:22:35 oak-io1-s2 kernel: Lustre: oak-OST000f: deleting orphan objects from 0x0:2515592 to 0x0:2515617 Jun 19 14:22:35 oak-io1-s2 kernel: Lustre: oak-OST0015: deleting orphan objects from 0x0:2486200 to 0x0:2486241 Jun 19 14:22:35 oak-io1-s2 kernel: Lustre: oak-OST001b: deleting orphan objects from 0x0:2472174 to 0x0:2472193 Jun 19 14:22:35 oak-io1-s2 kernel: Lustre: oak-OST0021: deleting orphan objects from 0x0:1922853 to 0x0:1922881 Jun 19 14:22:35 oak-io1-s2 kernel: Lustre: oak-OST001d: deleting orphan objects from 0x0:2464472 to 0x0:2464545 Jun 19 15:10:16 oak-io1-s2 kernel: Lustre: oak-OST0005: Connection restored to a8021b60-575f-f6bd-5e29-972e8cd71dae (at 10.12.4.29@o2ib) Jun 19 15:10:16 oak-io1-s2 kernel: Lustre: oak-OST0003: Connection restored to a8021b60-575f-f6bd-5e29-972e8cd71dae (at 10.12.4.29@o2ib) Jun 19 15:10:16 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to a8021b60-575f-f6bd-5e29-972e8cd71dae (at 10.12.4.29@o2ib) Jun 19 15:10:16 oak-io1-s2 kernel: Lustre: Skipped 18 previous similar messages Jun 19 15:10:16 oak-io1-s2 kernel: Lustre: Skipped 18 previous similar messages Jun 19 15:10:16 oak-io1-s2 kernel: Lustre: Skipped 15 previous similar messages Jun 19 15:10:32 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to 657e12ab-e8cd-dbe8-604e-55778c3efb90 (at 10.12.4.45@o2ib) Jun 19 15:10:32 oak-io1-s2 kernel: Lustre: oak-OST0005: Connection restored to 657e12ab-e8cd-dbe8-604e-55778c3efb90 (at 10.12.4.45@o2ib) Jun 19 15:10:32 oak-io1-s2 kernel: Lustre: Skipped 16 previous similar messages Jun 19 15:20:34 oak-io1-s2 kernel: Lustre: oak-OST0019: haven't heard from client b0933a62-6f9c-4d5a-d12a-9155a2b027c0 (at 10.12.4.45@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88299dc76800, cur 1497910834 expire 1497910684 last 1497910607 Jun 19 15:20:34 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 19 15:27:02 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to c83d9db0-eed9-3e10-b974-ee38d75923a6 (at 10.12.4.22@o2ib) Jun 19 15:27:02 oak-io1-s2 kernel: Lustre: oak-OST0003: Connection restored to c83d9db0-eed9-3e10-b974-ee38d75923a6 (at 10.12.4.22@o2ib) Jun 19 15:27:02 oak-io1-s2 kernel: Lustre: Skipped 1 previous similar message Jun 19 15:27:02 oak-io1-s2 kernel: Lustre: Skipped 15 previous similar messages Jun 19 15:27:07 oak-io1-s2 kernel: Lustre: oak-OST0005: Connection restored to 308a3d6e-8e88-3302-85e5-874fbc2b524f (at 10.12.4.25@o2ib) Jun 19 15:27:07 oak-io1-s2 kernel: Lustre: Skipped 31 previous similar messages Jun 19 15:27:42 oak-io1-s2 kernel: Lustre: oak-OST0005: Connection restored to 9dc628f4-9e44-4c31-c06b-175574204bce (at 10.12.4.27@o2ib) Jun 19 15:27:42 oak-io1-s2 kernel: Lustre: oak-OST000b: Connection restored to 9dc628f4-9e44-4c31-c06b-175574204bce (at 10.12.4.27@o2ib) Jun 19 15:27:42 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to 9dc628f4-9e44-4c31-c06b-175574204bce (at 10.12.4.27@o2ib) Jun 19 15:27:42 oak-io1-s2 kernel: Lustre: oak-OST0007: Connection restored to 9dc628f4-9e44-4c31-c06b-175574204bce (at 10.12.4.27@o2ib) Jun 19 15:27:42 oak-io1-s2 kernel: Lustre: Skipped 21 previous similar messages Jun 19 15:27:42 oak-io1-s2 kernel: Lustre: Skipped 21 previous similar messages Jun 19 15:27:42 oak-io1-s2 kernel: Lustre: Skipped 21 previous similar messages Jun 19 15:27:42 oak-io1-s2 kernel: Lustre: Skipped 12 previous similar messages Jun 19 15:54:34 oak-io1-s2 kernel: Lustre: oak-OST0003: Connection restored to f6175fed-7a02-6b6b-0fb5-d34e379ad30c (at 10.12.4.30@o2ib) Jun 19 15:54:34 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to f6175fed-7a02-6b6b-0fb5-d34e379ad30c (at 10.12.4.30@o2ib) Jun 19 15:54:34 oak-io1-s2 kernel: Lustre: Skipped 1 previous similar message Jun 19 15:54:34 oak-io1-s2 kernel: Lustre: Skipped 15 previous similar messages Jun 19 15:54:47 oak-io1-s2 kernel: Lustre: oak-OST0003: Connection restored to 79120b39-eaf6-8355-f583-74d2a5813bf1 (at 10.12.4.31@o2ib) Jun 19 15:54:47 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to 79120b39-eaf6-8355-f583-74d2a5813bf1 (at 10.12.4.31@o2ib) Jun 19 15:54:47 oak-io1-s2 kernel: Lustre: Skipped 1 previous similar message Jun 19 15:54:47 oak-io1-s2 kernel: Lustre: Skipped 15 previous similar messages Jun 19 16:11:35 oak-io1-s2 kernel: Lustre: oak-OST0003: Connection restored to 1669659d-1855-4888-3385-ed2607b9d6b8 (at 10.12.4.32@o2ib) Jun 19 16:11:35 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to 1669659d-1855-4888-3385-ed2607b9d6b8 (at 10.12.4.32@o2ib) Jun 19 16:11:35 oak-io1-s2 kernel: Lustre: Skipped 16 previous similar messages Jun 19 16:13:46 oak-io1-s2 kernel: Lustre: oak-OST0005: Connection restored to 657e12ab-e8cd-dbe8-604e-55778c3efb90 (at 10.12.4.45@o2ib) Jun 19 16:13:46 oak-io1-s2 kernel: Lustre: Skipped 16 previous similar messages Jun 19 16:18:48 oak-io1-s2 kernel: Lustre: oak-OST000f: haven't heard from client 8ccfe725-ae60-cd08-20a8-5ec62878d68c (at 10.12.4.45@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883f79a09c00, cur 1497914328 expire 1497914178 last 1497914101 Jun 19 16:18:48 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 19 16:20:17 oak-io1-s2 kernel: Lustre: oak-OST0015: haven't heard from client 50f915f1-0494-6217-ad81-34c86b06a358 (at 10.12.4.62@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804b9a58c00, cur 1497914417 expire 1497914267 last 1497914190 Jun 19 16:20:17 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 19 16:20:54 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to 50f915f1-0494-6217-ad81-34c86b06a358 (at 10.12.4.62@o2ib) Jun 19 16:20:54 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 19 16:33:16 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to 657e12ab-e8cd-dbe8-604e-55778c3efb90 (at 10.12.4.45@o2ib) Jun 19 16:33:16 oak-io1-s2 kernel: Lustre: oak-OST0003: Connection restored to 657e12ab-e8cd-dbe8-604e-55778c3efb90 (at 10.12.4.45@o2ib) Jun 19 16:33:16 oak-io1-s2 kernel: Lustre: Skipped 16 previous similar messages Jun 19 16:36:29 oak-io1-s2 kernel: Lustre: oak-OST0015: haven't heard from client 6cb0747e-2ffa-5bb1-1a03-c7a638ae57a3 (at 10.12.4.27@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883d2ed9d800, cur 1497915389 expire 1497915239 last 1497915162 Jun 19 16:36:29 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 19 16:52:07 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to e53fda25-e3eb-5eda-5682-e8cf14afbde7 (at 10.9.102.48@o2ib4) Jun 19 16:52:07 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 19 16:52:08 oak-io1-s2 kernel: Lustre: oak-OST0003: Connection restored to 30161511-42cb-0fce-a091-91d5e219a8ac (at 10.9.102.47@o2ib4) Jun 19 16:52:08 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to 30161511-42cb-0fce-a091-91d5e219a8ac (at 10.9.102.47@o2ib4) Jun 19 16:52:08 oak-io1-s2 kernel: Lustre: Skipped 18 previous similar messages Jun 19 16:52:08 oak-io1-s2 kernel: Lustre: Skipped 16 previous similar messages Jun 19 16:53:19 oak-io1-s2 kernel: Lustre: oak-OST0001: haven't heard from client a3a52d7c-5838-49a0-abe8-4baf0041def5 (at 10.9.102.46@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883d40410c00, cur 1497916399 expire 1497916249 last 1497916172 Jun 19 16:53:19 oak-io1-s2 kernel: Lustre: Skipped 53 previous similar messages Jun 19 17:06:11 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to 9dc628f4-9e44-4c31-c06b-175574204bce (at 10.12.4.27@o2ib) Jun 19 17:06:11 oak-io1-s2 kernel: Lustre: Skipped 14 previous similar messages Jun 19 17:12:08 oak-io1-s2 kernel: Lustre: oak-OST0005: Connection restored to 1669659d-1855-4888-3385-ed2607b9d6b8 (at 10.12.4.32@o2ib) Jun 19 17:12:08 oak-io1-s2 kernel: Lustre: Skipped 16 previous similar messages Jun 19 17:13:43 oak-io1-s2 kernel: Lustre: oak-OST0015: haven't heard from client a4c8b633-2c4c-fbc3-51bb-43aa9d91776a (at 10.12.4.27@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88299dc72c00, cur 1497917623 expire 1497917473 last 1497917396 Jun 19 17:13:43 oak-io1-s2 kernel: Lustre: Skipped 53 previous similar messages Jun 19 17:18:25 oak-io1-s2 kernel: Lustre: oak-OST0013: haven't heard from client 45512dba-adee-9a4b-f30d-486561e40dc1 (at 10.12.4.32@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883f82b12800, cur 1497917905 expire 1497917755 last 1497917678 Jun 19 17:18:25 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 19 18:13:06 oak-io1-s2 kernel: LustreError: 168-f: BAD WRITE CHECKSUM: oak-OST000d from 12345-10.0.2.240@o2ib5 inode [0x200001a94:0x1037f:0x0] object 0x0:2549098 extent [38689308672-38690357247]: client csum 58c715e3, server csum d646217b Jun 19 18:13:06 oak-io1-s2 kernel: LustreError: Skipped 16 previous similar messages Jun 19 18:14:59 oak-io1-s2 kernel: LustreError: 168-f: BAD WRITE CHECKSUM: oak-OST000d from 12345-10.0.2.240@o2ib5 inode [0x200001a94:0x1037f:0x0] object 0x0:2549098 extent [58463354880-58464403455]: client csum 58c715e3, server csum 9073185f Jun 19 18:14:59 oak-io1-s2 kernel: LustreError: Skipped 10 previous similar messages Jun 20 00:05:02 oak-io1-s2 kernel: LustreError: 168-f: BAD WRITE CHECKSUM: oak-OST001f from 12345-10.0.2.240@o2ib5 inode [0x200001a94:0x115c7:0x0] object 0x0:1947413 extent [382730240-383778815]: client csum 58c715e3, server csum b5756f8f Jun 20 00:05:02 oak-io1-s2 kernel: LustreError: Skipped 21 previous similar messages Jun 20 00:05:23 oak-io1-s2 kernel: LustreError: 168-f: BAD WRITE CHECKSUM: oak-OST001f from 12345-10.0.2.240@o2ib5 inode [0x200001a94:0x115c7:0x0] object 0x0:1947413 extent [382730240-383778815]: client csum 58c715e3, server csum b5756f8f Jun 20 00:05:23 oak-io1-s2 kernel: LustreError: Skipped 5 previous similar messages Jun 20 00:06:01 oak-io1-s2 kernel: LustreError: 168-f: BAD WRITE CHECKSUM: oak-OST000d from 12345-10.0.2.240@o2ib5 inode [0x200001a94:0x115d8:0x0] object 0x0:2553932 extent [2682519552-2683568127]: client csum bda75a83, server csum 867a8a5c Jun 20 00:06:01 oak-io1-s2 kernel: LustreError: Skipped 7 previous similar messages Jun 20 00:07:18 oak-io1-s2 kernel: LustreError: 168-f: BAD WRITE CHECKSUM: oak-OST000d from 12345-10.0.2.240@o2ib5 inode [0x200001a94:0x115d8:0x0] object 0x0:2553932 extent [5927862272-5928124415]: client csum d28efa3f, server csum 3e2b83df Jun 20 00:07:18 oak-io1-s2 kernel: LustreError: Skipped 21 previous similar messages Jun 20 00:09:48 oak-io1-s2 kernel: LustreError: 168-f: BAD WRITE CHECKSUM: oak-OST001f from 12345-10.0.2.240@o2ib5 inode [0x200001a94:0x115c7:0x0] object 0x0:1947413 extent [41489006592-41490055167]: client csum 58c715e3, server csum 850741e Jun 20 00:09:48 oak-io1-s2 kernel: LustreError: Skipped 21 previous similar messages Jun 20 00:17:19 oak-io1-s2 kernel: LustreError: 168-f: BAD WRITE CHECKSUM: oak-OST001f from 12345-10.0.2.240@o2ib5 inode [0x200001a94:0x115c7:0x0] object 0x0:1947413 extent [68138565632-68139614207]: client csum 58c715e3, server csum 85cd17e6 Jun 20 00:17:19 oak-io1-s2 kernel: LustreError: Skipped 7 previous similar messages Jun 20 00:42:03 oak-io1-s2 kernel: Lustre: oak-OST001d: haven't heard from client da0a30db-5fc3-2864-2e42-7f0f39526700 (at 10.12.4.45@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883d4118e400, cur 1497944523 expire 1497944373 last 1497944296 Jun 20 00:42:03 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 20 00:42:03 oak-io1-s2 kernel: Lustre: oak-OST000f: haven't heard from client da0a30db-5fc3-2864-2e42-7f0f39526700 (at 10.12.4.45@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883f93be1c00, cur 1497944523 expire 1497944373 last 1497944296 Jun 20 00:42:03 oak-io1-s2 kernel: Lustre: Skipped 16 previous similar messages Jun 20 05:33:35 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to 2485b616-54a7-a574-7377-95e7a94de902 (at 10.9.102.67@o2ib4) Jun 20 05:33:35 oak-io1-s2 kernel: Lustre: Skipped 11 previous similar messages Jun 20 05:34:11 oak-io1-s2 kernel: Lustre: oak-OST000b: haven't heard from client 2485b616-54a7-a574-7377-95e7a94de902 (at 10.9.102.67@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883e0a27e800, cur 1497962051 expire 1497961901 last 1497961824 Jun 20 05:34:11 oak-io1-s2 kernel: Lustre: oak-OST0003: haven't heard from client 2485b616-54a7-a574-7377-95e7a94de902 (at 10.9.102.67@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883d45d1d000, cur 1497962051 expire 1497961901 last 1497961824 Jun 20 05:34:11 oak-io1-s2 kernel: Lustre: Skipped 16 previous similar messages Jun 20 06:03:11 oak-io1-s2 kernel: LustreError: 168-f: BAD WRITE CHECKSUM: oak-OST001d from 12345-10.0.2.240@o2ib5 inode [0x200001a94:0x1281f:0x0] object 0x0:2479014 extent [1043333120-1044381695]: client csum 58c715e3, server csum 77790522 Jun 20 06:03:11 oak-io1-s2 kernel: LustreError: Skipped 32 previous similar messages Jun 20 06:04:48 oak-io1-s2 kernel: LustreError: 168-f: BAD WRITE CHECKSUM: oak-OST001d from 12345-10.0.2.240@o2ib5 inode [0x200001a94:0x1281f:0x0] object 0x0:2479014 extent [16224616448-16225665023]: client csum 58c715e3, server csum 93b15a82 Jun 20 06:04:48 oak-io1-s2 kernel: LustreError: Skipped 10 previous similar messages Jun 20 06:08:37 oak-io1-s2 kernel: LustreError: 168-f: BAD WRITE CHECKSUM: oak-OST001d from 12345-10.0.2.240@o2ib5 inode [0x200001a94:0x1281f:0x0] object 0x0:2479014 extent [34297872384-34298920959]: client csum 58c715e3, server csum cf02f0cc Jun 20 06:08:37 oak-io1-s2 kernel: LustreError: Skipped 21 previous similar messages Jun 20 06:13:42 oak-io1-s2 kernel: LustreError: 168-f: BAD WRITE CHECKSUM: oak-OST001d from 12345-10.0.2.240@o2ib5 inode [0x200001a94:0x1281f:0x0] object 0x0:2479014 extent [75267833856-75268882431]: client csum 58c715e3, server csum 76489bbe Jun 20 06:13:42 oak-io1-s2 kernel: LustreError: Skipped 31 previous similar messages Jun 20 08:44:09 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to a917e282-3d5a-b5ea-4ed2-3aef729f9927 (at 10.12.4.21@o2ib) Jun 20 08:44:09 oak-io1-s2 kernel: Lustre: Skipped 17 previous similar messages Jun 20 08:44:20 oak-io1-s2 kernel: Lustre: oak-OST0007: Connection restored to 9dc628f4-9e44-4c31-c06b-175574204bce (at 10.12.4.27@o2ib) Jun 20 08:44:20 oak-io1-s2 kernel: Lustre: oak-OST0009: Connection restored to 9dc628f4-9e44-4c31-c06b-175574204bce (at 10.12.4.27@o2ib) Jun 20 08:44:20 oak-io1-s2 kernel: Lustre: oak-OST0005: Connection restored to 9dc628f4-9e44-4c31-c06b-175574204bce (at 10.12.4.27@o2ib) Jun 20 08:44:20 oak-io1-s2 kernel: Lustre: oak-OST0003: Connection restored to 9dc628f4-9e44-4c31-c06b-175574204bce (at 10.12.4.27@o2ib) Jun 20 08:44:20 oak-io1-s2 kernel: Lustre: Skipped 3 previous similar messages Jun 20 08:44:20 oak-io1-s2 kernel: Lustre: Skipped 3 previous similar messages Jun 20 08:44:20 oak-io1-s2 kernel: Lustre: Skipped 3 previous similar messages Jun 20 08:44:20 oak-io1-s2 kernel: Lustre: Skipped 13 previous similar messages Jun 20 08:49:40 oak-io1-s2 kernel: Lustre: oak-OST0007: Connection restored to 1669659d-1855-4888-3385-ed2607b9d6b8 (at 10.12.4.32@o2ib) Jun 20 08:49:40 oak-io1-s2 kernel: Lustre: oak-OST0009: Connection restored to 1669659d-1855-4888-3385-ed2607b9d6b8 (at 10.12.4.32@o2ib) Jun 20 08:49:40 oak-io1-s2 kernel: Lustre: oak-OST0001: Connection restored to 1669659d-1855-4888-3385-ed2607b9d6b8 (at 10.12.4.32@o2ib) Jun 20 08:49:40 oak-io1-s2 kernel: Lustre: oak-OST0005: Connection restored to 1669659d-1855-4888-3385-ed2607b9d6b8 (at 10.12.4.32@o2ib) Jun 20 08:49:40 oak-io1-s2 kernel: Lustre: oak-OST0003: Connection restored to 1669659d-1855-4888-3385-ed2607b9d6b8 (at 10.12.4.32@o2ib) Jun 20 08:49:40 oak-io1-s2 kernel: Lustre: Skipped 13 previous similar messages Jun 20 09:02:36 oak-io1-s2 kernel: Lustre: oak-OST0015: haven't heard from client ba626e51-4272-69a8-f9b4-f954748c04b3 (at 10.12.4.33@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff883d38daa800, cur 1497974556 expire 1497974406 last 1497974329 Jun 20 09:02:36 oak-io1-s2 kernel: Lustre: oak-OST0021: haven't heard from client d19eadf6-792a-22b6-3b36-e89bda7860eb (at 10.12.4.55@o2ib) in 152 seconds. I think it's dead, and I am evicting it. exp ffff883de22f9000, cur 1497974556 expire 1497974406 last 1497974404 Jun 20 09:02:36 oak-io1-s2 kernel: Lustre: Skipped 63 previous similar messages Jun 20 09:02:37 oak-io1-s2 kernel: Lustre: oak-OST0001: haven't heard from client b7969c00-00e4-d096-9882-c8ac06884dbb (at 10.12.4.42@o2ib) in 219 seconds. I think it's dead, and I am evicting it. exp ffff883d40edf000, cur 1497974557 expire 1497974407 last 1497974338 Jun 20 09:02:37 oak-io1-s2 kernel: Lustre: Skipped 113 previous similar messages Jun 20 09:03:52 oak-io1-s2 kernel: Lustre: oak-OST0015: haven't heard from client 5c7fb246-5dad-a170-73b4-818ee19874d2 (at 10.12.4.48@o2ib) in 220 seconds. I think it's dead, and I am evicting it. exp ffff883d38556c00, cur 1497974632 expire 1497974482 last 1497974412 Jun 20 09:03:52 oak-io1-s2 kernel: Lustre: Skipped 19 previous similar messages Jun 20 09:05:08 oak-io1-s2 kernel: Lustre: oak-OST0015: haven't heard from client 23ec3d48-7f0a-a3b7-3633-1e0d0ee3aebb (at 10.12.4.61@o2ib) in 221 seconds. I think it's dead, and I am evicting it. exp ffff883d3852f800, cur 1497974708 expire 1497974558 last 1497974487 Jun 20 09:05:08 oak-io1-s2 kernel: Lustre: Skipped 107 previous similar messages