Aug 25 18:44:12 oak-gw06 kernel: Lustre: DEBUG MARKER: Fri Aug 25 18:44:12 2017 Aug 31 11:27:05 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b6a:0x0] object 0x0:4413301 extent [726925312-727973887] Aug 31 11:27:05 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) original client csum 4ecd330 (type 1), server csum 5610e5e5 (type 1), client csum now 4ecd330 Aug 31 11:27:05 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88001b654600 x1566273879372336/t60132831973(60132831973) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504204071 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:27:06 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b6a:0x0] object 0x0:4413301 extent [726925312-727973887] Aug 31 11:27:06 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum 4ecd330 (type 1), server csum 5610e5e5 (type 1), client csum now 4ecd330 Aug 31 11:27:06 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802b770ed00 x1566273879372416/t60132831982(60132831982) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504204073 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:27:08 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b6a:0x0] object 0x0:4413301 extent [726925312-727973887] Aug 31 11:27:08 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum 4ecd330 (type 1), server csum 5610e5e5 (type 1), client csum now 4ecd330 Aug 31 11:27:08 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800683c0300 x1566273879373872/t60132832002(60132832002) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504204075 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:27:11 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b6a:0x0] object 0x0:4413301 extent [726925312-727973887] Aug 31 11:27:11 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum 4ecd330 (type 1), server csum 5610e5e5 (type 1), client csum now 4ecd330 Aug 31 11:27:11 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800683c0600 x1566273879375136/t60132832026(60132832026) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504204078 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:27:21 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b6a:0x0] object 0x0:4413301 extent [726925312-727973887] Aug 31 11:27:21 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 31 11:27:21 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) original client csum 4ecd330 (type 1), server csum 5610e5e5 (type 1), client csum now 4ecd330 Aug 31 11:27:21 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) Skipped 1 previous similar message Aug 31 11:27:21 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803b9792100 x1566273879378544/t60132832103(60132832103) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504204087 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:27:21 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 1 previous similar message Aug 31 11:27:34 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b6a:0x0] object 0x0:4413301 extent [726925312-727973887] Aug 31 11:27:34 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 31 11:27:34 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum 4ecd330 (type 1), server csum 5610e5e5 (type 1), client csum now 4ecd330 Aug 31 11:27:34 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 1 previous similar message Aug 31 11:27:34 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88041b857000 x1566273879385120/t60132832218(60132832218) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504204101 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:27:34 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 1 previous similar message Aug 31 11:27:51 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b6a:0x0] object 0x0:4413301 extent [726925312-727973887] Aug 31 11:27:51 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 31 11:27:51 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum 4ecd330 (type 1), server csum 5610e5e5 (type 1), client csum now 4ecd330 Aug 31 11:27:51 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 1 previous similar message Aug 31 11:27:51 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88001b655b00 x1566273879393600/t60132832445(60132832445) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504204118 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:27:51 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 1 previous similar message Aug 31 11:28:01 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST001a-osc-ffff88041b99c000: too many resent retries for object: 0:4413301, rc = -11. Aug 31 11:32:35 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b6a:0x0] object 0x0:4413301 extent [2963800064-2967470079] Aug 31 11:32:35 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 31 11:32:35 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum 46fb7986 (type 1), server csum 557743ef (type 1), client csum now 46fb7986 Aug 31 11:32:35 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) Skipped 1 previous similar message Aug 31 11:32:35 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880367f21500 x1566273879618992/t60132835521(60132835521) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504204402 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:33:27 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST001a-osc-ffff88041b99c000: too many resent retries for object: 0:4413301, rc = -11. Aug 31 11:37:42 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4413301 extent [1581252608-1582301183] Aug 31 11:37:42 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client d181f15f, server fb46c5e2, cksum_type 1 Aug 31 11:37:42 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880331a91800 x1566273879895088/t0(0) o3->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504204710 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 11:37:42 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 9 previous similar messages Aug 31 11:37:43 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4413301 extent [1581252608-1582301183] Aug 31 11:37:43 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client d181f15f, server fb46c5e2, cksum_type 1 Aug 31 11:37:45 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4413301 extent [1581252608-1582301183] Aug 31 11:37:45 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client d181f15f, server fb46c5e2, cksum_type 1 Aug 31 11:37:48 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4413301 extent [1581252608-1582301183] Aug 31 11:37:48 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client d181f15f, server fb46c5e2, cksum_type 1 Aug 31 11:37:52 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4413301 extent [1581252608-1582301183] Aug 31 11:37:52 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client d181f15f, server fb46c5e2, cksum_type 1 Aug 31 11:38:03 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4413301 extent [1581252608-1582301183] Aug 31 11:38:03 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 31 11:38:03 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client d181f15f, server fb46c5e2, cksum_type 1 Aug 31 11:38:03 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 1 previous similar message Aug 31 11:38:27 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4413301 extent [1581252608-1582301183] Aug 31 11:38:27 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 11:38:27 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client d181f15f, server fb46c5e2, cksum_type 1 Aug 31 11:38:27 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Aug 31 11:38:37 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001a-osc-ffff88041b99c000: too many resent retries for object: 0:4413301, rc = -11. Aug 31 11:39:05 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4413301 extent [1581252608-1582301183] Aug 31 11:39:05 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Aug 31 11:39:05 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client d181f15f, server fb46c5e2, cksum_type 1 Aug 31 11:39:05 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 8 previous similar messages Aug 31 11:39:32 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST001a-osc-ffff88041b99c000: too many resent retries for object: 0:4413301, rc = -11. Aug 31 11:39:55 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880331a92d00 x1566273880509696/t0(0) o3->oak-OST000a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504204842 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 11:39:55 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 19 previous similar messages Aug 31 11:40:10 oak-gw06 kernel: LustreError: 133-1: oak-OST000a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4409505 extent [29903290368-29904338943] Aug 31 11:40:10 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Aug 31 11:40:10 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 5fbc7f37, server 46e0c8d, cksum_type 1 Aug 31 11:40:10 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 8 previous similar messages Aug 31 11:40:50 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST000a-osc-ffff88041b99c000: too many resent retries for object: 0:4409505, rc = -11. Aug 31 11:41:45 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST000a-osc-ffff88041b99c000: too many resent retries for object: 0:4409505, rc = -11. Aug 31 11:42:21 oak-gw06 kernel: LustreError: 133-1: oak-OST000a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4409505 extent [29903290368-29904338943] Aug 31 11:42:21 oak-gw06 kernel: LustreError: Skipped 24 previous similar messages Aug 31 11:42:21 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client 5fbc7f37, server 46e0c8d, cksum_type 1 Aug 31 11:42:21 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 24 previous similar messages Aug 31 11:42:40 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST000a-osc-ffff88041b99c000: too many resent retries for object: 0:4409505, rc = -11. Aug 31 11:58:48 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b28:0x0] object 0x0:3755960 extent [61770301440-61771350015] Aug 31 11:58:48 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Aug 31 11:58:48 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) original client csum ddce27b3 (type 1), server csum 34f06c50 (type 1), client csum now ddce27b3 Aug 31 11:58:48 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Aug 31 11:58:48 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880136f32700 x1566273880943232/t51542778710(51542778710) o4->oak-OST001e-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504205939 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:58:48 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 29 previous similar messages Aug 31 11:58:58 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b28:0x0] object 0x0:3755960 extent [61770301440-61771350015] Aug 31 11:58:58 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Aug 31 11:58:58 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) original client csum ddce27b3 (type 1), server csum 34f06c50 (type 1), client csum now ddce27b3 Aug 31 11:58:58 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) Skipped 3 previous similar messages Aug 31 11:59:16 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b28:0x0] object 0x0:3755960 extent [61770301440-61771350015] Aug 31 11:59:16 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 11:59:16 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum ddce27b3 (type 1), server csum 34f06c50 (type 1), client csum now ddce27b3 Aug 31 11:59:16 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Aug 31 11:59:24 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88012821e100 x1566273880956800/t51542778871(51542778871) o4->oak-OST001e-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504206013 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 11:59:24 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 7 previous similar messages Aug 31 11:59:43 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001e-osc-ffff88041b99c000: too many resent retries for object: 0:3755960, rc = -11. Aug 31 12:00:26 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b28:0x0] object 0x0:3755960 extent [62205722624-62206771199] Aug 31 12:00:26 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Aug 31 12:00:26 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum b00ad904 (type 1), server csum af5a1186 (type 1), client csum now b00ad904 Aug 31 12:00:26 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 3 previous similar messages Aug 31 12:00:29 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880125c4d800 x1566273880998512/t51542779207(51542779207) o4->oak-OST001e-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504206078 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 12:00:29 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 3 previous similar messages Aug 31 12:01:21 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST001e-osc-ffff88041b99c000: too many resent retries for object: 0:3755960, rc = -11. Aug 31 12:01:51 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6b56:0x0] object 0x0:4397797 extent [8915517440-8917876735] Aug 31 12:01:51 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Aug 31 12:01:51 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) original client csum 6e00acc (type 1), server csum 1bd6cfbe (type 1), client csum now 6e00acc Aug 31 12:01:51 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Aug 31 12:02:46 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0006-osc-ffff88041b99c000: too many resent retries for object: 0:4397797, rc = -11. Aug 31 12:43:04 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f83:0x15488:0x0] object 0x0:3991245 extent [15734931456-15737028607] Aug 31 12:43:04 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Aug 31 12:43:04 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum c1563535 (type 1), server csum dbb78289 (type 1), client csum now c1563535 Aug 31 12:43:04 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Aug 31 12:43:04 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880397cdd200 x1566273885951680/t51542723909(51542723909) o4->oak-OST0023-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504208631 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 12:43:04 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 17 previous similar messages Aug 31 12:43:25 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f83:0x15488:0x0] object 0x0:3991245 extent [15734931456-15737028607] Aug 31 12:43:25 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Aug 31 12:43:25 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum c1563535 (type 1), server csum dbb78289 (type 1), client csum now c1563535 Aug 31 12:43:25 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 5 previous similar messages Aug 31 12:43:25 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803c22dd800 x1566273885958448/t51542724081(51542724081) o4->oak-OST0023-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504208652 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 12:43:25 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 5 previous similar messages Aug 31 12:43:59 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f83:0x15488:0x0] object 0x0:3991245 extent [15734931456-15737028607] Aug 31 12:43:59 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Aug 31 12:43:59 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum c1563535 (type 1), server csum dbb78289 (type 1), client csum now c1563535 Aug 31 12:43:59 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) Skipped 3 previous similar messages Aug 31 12:43:59 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST0023-osc-ffff88041b99c000: too many resent retries for object: 0:3991245, rc = -11. Aug 31 12:44:22 oak-gw06 kernel: LustreError: 133-1: oak-OST0004-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:552911 extent [39845888-40894463] Aug 31 12:44:22 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 12:44:22 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client a5db2e8a, server 608d44ea, cksum_type 1 Aug 31 12:44:22 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Aug 31 12:44:22 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801aeb23900 x1566273886000624/t0(0) o3->oak-OST0004-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504208709 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 12:44:22 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 3 previous similar messages Aug 31 12:44:58 oak-gw06 kernel: LustreError: 133-1: oak-OST0004-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:552911 extent [39845888-40894463] Aug 31 12:44:58 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Aug 31 12:44:58 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client a5db2e8a, server 608d44ea, cksum_type 1 Aug 31 12:44:58 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Aug 31 12:45:17 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0004-osc-ffff88041b99c000: too many resent retries for object: 0:552911, rc = -11. Aug 31 12:45:27 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88039fc8b900 x1566273886044896/t0(0) o3->oak-OST0004-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504208774 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 12:45:27 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 13 previous similar messages Aug 31 12:46:12 oak-gw06 kernel: LustreError: 133-1: oak-OST0004-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:552911 extent [39845888-40894463] Aug 31 12:46:12 oak-gw06 kernel: LustreError: Skipped 12 previous similar messages Aug 31 12:46:12 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client a5db2e8a, server 608d44ea, cksum_type 1 Aug 31 12:46:12 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 12 previous similar messages Aug 31 12:46:12 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0004-osc-ffff88041b99c000: too many resent retries for object: 0:552911, rc = -11. Aug 31 12:47:07 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0004-osc-ffff88041b99c000: too many resent retries for object: 0:552911, rc = -11. Aug 31 12:47:43 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88020afe7600 x1566273886122000/t0(0) o3->oak-OST0004-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504208874 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 12:47:43 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 23 previous similar messages Aug 31 12:48:02 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0004-osc-ffff88041b99c000: too many resent retries for object: 0:552911, rc = -11. Aug 31 12:48:23 oak-gw06 kernel: LustreError: 133-1: oak-OST0004-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:552911 extent [39845888-40894463] Aug 31 12:48:23 oak-gw06 kernel: LustreError: Skipped 28 previous similar messages Aug 31 12:48:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client a5db2e8a, server 608d44ea, cksum_type 1 Aug 31 12:48:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 28 previous similar messages Aug 31 12:48:57 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0004-osc-ffff88041b99c000: too many resent retries for object: 0:552911, rc = -11. Aug 31 13:02:24 oak-gw06 kernel: LustreError: 133-1: oak-OST001d-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:239355 extent [27262976-28311551] Aug 31 13:02:24 oak-gw06 kernel: LustreError: Skipped 4 previous similar messages Aug 31 13:02:24 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client 2bd21bed, server 7ca5c1f2, cksum_type 1 Aug 31 13:02:24 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 4 previous similar messages Aug 31 13:02:24 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802b770e700 x1566273887994352/t0(0) o3->oak-OST001d-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504209753 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 13:02:24 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 11 previous similar messages Aug 31 13:03:00 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8804138d0000 x1566273888009536/t0(0) o3->oak-OST001d-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504209789 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 13:03:00 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 7 previous similar messages Aug 31 13:03:16 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST001d-osc-ffff88041b99c000: too many resent retries for object: 0:239355, rc = -11. Aug 31 13:04:08 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST001d-osc-ffff88041b99c000: too many resent retries for object: 0:239355, rc = -11. Aug 31 13:04:08 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801e7da0300 x1566273888066080/t0(0) o3->oak-OST001d-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504209857 ref 2 fl Interpret:RM/0/0 rc 700416/700416 Aug 31 13:04:08 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 11 previous similar messages Aug 31 13:05:00 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST001d-osc-ffff88041b99c000: too many resent retries for object: 0:239355, rc = -11. Aug 31 13:05:52 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001d-osc-ffff88041b99c000: too many resent retries for object: 0:239355, rc = -11. Aug 31 13:06:21 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88022563cc00 x1566273888132288/t0(0) o3->oak-OST001d-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 624/400 e 0 to 0 dl 1504209990 ref 2 fl Interpret:RM/0/0 rc 704512/704512 Aug 31 13:06:21 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 26 previous similar messages Aug 31 13:07:37 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST001d-osc-ffff88041b99c000: too many resent retries for object: 0:239355, rc = -11. Aug 31 13:07:37 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 13:52:51 oak-gw06 kernel: LustreError: 133-1: oak-OST0019-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:239998 extent [812646400-813694975] Aug 31 13:52:51 oak-gw06 kernel: LustreError: Skipped 65 previous similar messages Aug 31 13:52:51 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 48e11765, server f3e3cbdc, cksum_type 1 Aug 31 13:52:51 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 65 previous similar messages Aug 31 13:52:51 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880101731800 x1566273895440048/t0(0) o3->oak-OST0019-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504212823 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 13:52:51 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 12 previous similar messages Aug 31 13:53:27 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8804138d1500 x1566273895483904/t0(0) o3->oak-OST0019-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504212859 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 13:53:27 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 7 previous similar messages Aug 31 13:53:46 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0019-osc-ffff88041b99c000: too many resent retries for object: 0:239998, rc = -11. Aug 31 13:53:56 oak-gw06 kernel: LustreError: 133-1: oak-OST0019-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:239998 extent [812646400-813694975] Aug 31 13:53:56 oak-gw06 kernel: LustreError: Skipped 14 previous similar messages Aug 31 13:53:56 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 48e11765, server f3e3cbdc, cksum_type 1 Aug 31 13:53:56 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 14 previous similar messages Aug 31 13:54:31 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880044d02100 x1566273895762752/t0(0) o3->oak-OST0019-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504212923 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 13:54:31 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 10 previous similar messages Aug 31 13:54:41 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0019-osc-ffff88041b99c000: too many resent retries for object: 0:239998, rc = -11. Aug 31 13:55:36 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0019-osc-ffff88041b99c000: too many resent retries for object: 0:239998, rc = -11. Aug 31 13:57:05 oak-gw06 kernel: LustreError: 133-1: oak-OST0011-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371203 extent [940572672-941621247] Aug 31 13:57:05 oak-gw06 kernel: LustreError: Skipped 17 previous similar messages Aug 31 13:57:05 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client 9c853d33, server ec07fbc5, cksum_type 1 Aug 31 13:57:05 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 17 previous similar messages Aug 31 13:57:05 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88029be14000 x1566273896455872/t0(0) o3->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504213036 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 13:57:05 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 10 previous similar messages Aug 31 13:58:00 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST0011-osc-ffff88041b99c000: too many resent retries for object: 0:371203, rc = -11. Aug 31 14:18:07 oak-gw06 kernel: LustreError: 133-1: oak-OST0012-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321391 extent [1389363200-1390411775] Aug 31 14:18:07 oak-gw06 kernel: LustreError: Skipped 32 previous similar messages Aug 31 14:18:07 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client 829abbfd, server 1291d7d7, cksum_type 1 Aug 31 14:18:07 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 32 previous similar messages Aug 31 14:18:07 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801be08c000 x1566273899891344/t0(0) o3->oak-OST0012-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504214334 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 14:18:07 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 29 previous similar messages Aug 31 14:18:43 oak-gw06 kernel: LustreError: 133-1: oak-OST0012-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321391 extent [1389363200-1390411775] Aug 31 14:18:43 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Aug 31 14:18:43 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 829abbfd, server 1291d7d7, cksum_type 1 Aug 31 14:18:43 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Aug 31 14:18:43 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801d2f1d500 x1566273899988832/t0(0) o3->oak-OST0012-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504214370 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 14:18:43 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 7 previous similar messages Aug 31 14:19:02 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST0012-osc-ffff88041b99c000: too many resent retries for object: 0:321391, rc = -11. Aug 31 14:19:02 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 14:19:47 oak-gw06 kernel: LustreError: 133-1: oak-OST0012-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321391 extent [1389363200-1390411775] Aug 31 14:19:47 oak-gw06 kernel: LustreError: Skipped 11 previous similar messages Aug 31 14:19:47 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 829abbfd, server 1291d7d7, cksum_type 1 Aug 31 14:19:47 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 11 previous similar messages Aug 31 14:19:47 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880399575b00 x1566273900306256/t0(0) o3->oak-OST0012-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504214434 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 14:19:47 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 10 previous similar messages Aug 31 14:19:57 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0012-osc-ffff88041b99c000: too many resent retries for object: 0:321391, rc = -11. Aug 31 14:20:52 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0012-osc-ffff88041b99c000: too many resent retries for object: 0:321391, rc = -11. Aug 31 14:21:59 oak-gw06 kernel: LustreError: 133-1: oak-OST0012-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321391 extent [493879296-494927871] Aug 31 14:21:59 oak-gw06 kernel: LustreError: Skipped 27 previous similar messages Aug 31 14:21:59 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client db927ccb, server d71cb979, cksum_type 1 Aug 31 14:21:59 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 27 previous similar messages Aug 31 14:21:59 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88012cc8b600 x1566273900660736/t0(0) o3->oak-OST0012-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504214566 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 14:21:59 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 24 previous similar messages Aug 31 14:22:44 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0012-osc-ffff88041b99c000: too many resent retries for object: 0:321391, rc = -11. Aug 31 14:22:44 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 14:38:19 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321360 extent [864026624-865075199] Aug 31 14:38:19 oak-gw06 kernel: LustreError: Skipped 17 previous similar messages Aug 31 14:38:19 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 8ce62d65, server fda437a3, cksum_type 1 Aug 31 14:38:19 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 17 previous similar messages Aug 31 14:38:19 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800697c9800 x1566273904699184/t0(0) o3->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504215506 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 14:38:19 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Aug 31 14:38:54 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321360 extent [864026624-865075199] Aug 31 14:38:54 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Aug 31 14:38:54 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client 8ce62d65, server fda437a3, cksum_type 1 Aug 31 14:38:54 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Aug 31 14:38:54 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800697cb300 x1566273904717632/t0(0) o3->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504215542 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 14:38:54 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 7 previous similar messages Aug 31 14:39:08 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321360, rc = -11. Aug 31 14:39:08 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 14:39:57 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321360, rc = -11. Aug 31 15:14:48 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321374 extent [208666624-209715199] Aug 31 15:14:48 oak-gw06 kernel: LustreError: Skipped 13 previous similar messages Aug 31 15:14:48 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 5920f5c6, server b566e40b, cksum_type 1 Aug 31 15:14:48 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 13 previous similar messages Aug 31 15:14:48 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801b9cf3600 x1566273911281312/t0(0) o3->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504217699 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 15:14:48 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 11 previous similar messages Aug 31 15:14:58 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321374 extent [208666624-209715199] Aug 31 15:14:58 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Aug 31 15:14:58 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 5920f5c6, server b566e40b, cksum_type 1 Aug 31 15:14:58 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 3 previous similar messages Aug 31 15:14:58 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88041364ad00 x1566273911284064/t0(0) o3->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504217709 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 15:14:58 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 3 previous similar messages Aug 31 15:15:16 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321374 extent [208666624-209715199] Aug 31 15:15:16 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 15:15:16 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client 5920f5c6, server b566e40b, cksum_type 1 Aug 31 15:15:16 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Aug 31 15:15:16 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801d8886a00 x1566273911291008/t0(0) o3->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504217727 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 15:15:16 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 2 previous similar messages Aug 31 15:15:43 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321374, rc = -11. Aug 31 15:15:49 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321374 extent [208666624-209715199] Aug 31 15:15:49 oak-gw06 kernel: LustreError: Skipped 6 previous similar messages Aug 31 15:15:49 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 5920f5c6, server b566e40b, cksum_type 1 Aug 31 15:15:49 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 6 previous similar messages Aug 31 15:15:49 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802ad776a00 x1566273911455248/t0(0) o3->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504217760 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 15:15:49 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 5 previous similar messages Aug 31 15:16:38 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321374, rc = -11. Aug 31 15:16:53 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321374 extent [208666624-209715199] Aug 31 15:16:53 oak-gw06 kernel: LustreError: Skipped 12 previous similar messages Aug 31 15:16:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 5920f5c6, server b566e40b, cksum_type 1 Aug 31 15:16:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 12 previous similar messages Aug 31 15:16:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803f62cea00 x1566273911581200/t0(0) o3->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504217824 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 15:16:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 11 previous similar messages Aug 31 15:17:33 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321374, rc = -11. Aug 31 15:18:30 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321374, rc = -11. Aug 31 15:19:06 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321374 extent [208666624-209387519] Aug 31 15:19:06 oak-gw06 kernel: LustreError: Skipped 24 previous similar messages Aug 31 15:19:06 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 5e93980e, server d9790f9d, cksum_type 1 Aug 31 15:19:06 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 24 previous similar messages Aug 31 15:19:06 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88038a226400 x1566273911828496/t0(0) o3->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504217957 ref 2 fl Interpret:RM/0/0 rc 720896/720896 Aug 31 15:19:06 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 22 previous similar messages Aug 31 15:19:25 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321374, rc = -11. Aug 31 15:25:15 oak-gw06 kernel: LustreError: 133-1: oak-OST001d-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240031 extent [455081984-456130559] Aug 31 15:25:15 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 15:25:15 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client 9ba42b9d, server 64849ffc, cksum_type 1 Aug 31 15:25:15 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Aug 31 15:25:15 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800a2f6e100 x1566273912549776/t0(0) o3->oak-OST001d-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504218326 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 15:25:15 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 1 previous similar message Aug 31 15:26:10 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001d-osc-ffff88041b99c000: too many resent retries for object: 0:240031, rc = -11. Aug 31 15:28:56 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST001d-osc-ffff88041b99c000: too many resent retries for object: 0:240031, rc = -11. Aug 31 15:28:56 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 15:38:20 oak-gw06 kernel: LustreError: 133-1: oak-OST001d-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240033 extent [52428800-53477375] Aug 31 15:38:21 oak-gw06 kernel: LustreError: Skipped 65 previous similar messages Aug 31 15:38:21 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 107e0d8d, server d06f4de0, cksum_type 1 Aug 31 15:38:21 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 65 previous similar messages Aug 31 15:38:21 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880087db7900 x1566273914409648/t0(0) o3->oak-OST001d-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504219111 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 15:38:21 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 59 previous similar messages Aug 31 15:39:16 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST001d-osc-ffff88041b99c000: too many resent retries for object: 0:240033, rc = -11. Aug 31 15:39:16 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 15:44:32 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6c99:0x0] object 0x0:2281084 extent [11516772352-11517820927] Aug 31 15:44:32 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum 4c6ff3eb (type 1), server csum 671924f5 (type 1), client csum now 4c6ff3eb Aug 31 15:44:42 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6c99:0x0] object 0x0:2281084 extent [11516772352-11517820927] Aug 31 15:44:42 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Aug 31 15:44:42 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) original client csum 4c6ff3eb (type 1), server csum 671924f5 (type 1), client csum now 4c6ff3eb Aug 31 15:44:42 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) Skipped 3 previous similar messages Aug 31 15:45:00 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6c99:0x0] object 0x0:2281084 extent [11516772352-11517820927] Aug 31 15:45:00 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 15:45:00 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) original client csum 4c6ff3eb (type 1), server csum 671924f5 (type 1), client csum now 4c6ff3eb Aug 31 15:45:00 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Aug 31 15:48:05 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:553509, rc = -11. Aug 31 15:48:05 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Aug 31 15:48:21 oak-gw06 kernel: LustreError: 133-1: oak-OST0001-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:553509 extent [20971520-22020095] Aug 31 15:48:21 oak-gw06 kernel: LustreError: Skipped 59 previous similar messages Aug 31 15:48:21 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client ee489532, server d6c6690a, cksum_type 1 Aug 31 15:48:21 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 59 previous similar messages Aug 31 15:48:21 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880406f4c600 x1566273915646768/t0(0) o3->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504219713 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 15:48:21 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 64 previous similar messages Aug 31 16:12:31 oak-gw06 kernel: LustreError: 133-1: oak-OST0001-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:553516 extent [428867584-429916159] Aug 31 16:12:31 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Aug 31 16:12:31 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client e47a92c6, server c71a4ec1, cksum_type 1 Aug 31 16:12:31 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Aug 31 16:12:31 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88024212bc00 x1566273920953264/t0(0) o3->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504221198 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 16:12:31 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 4 previous similar messages Aug 31 16:13:26 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:553516, rc = -11. Aug 31 16:13:26 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 16:13:47 oak-gw06 kernel: LustreError: 133-1: oak-OST0001-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:553516 extent [428867584-429916159] Aug 31 16:13:47 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Aug 31 16:13:47 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client e47a92c6, server c71a4ec1, cksum_type 1 Aug 31 16:13:47 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 16 previous similar messages Aug 31 16:13:47 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802fe512a00 x1566273921133664/t0(0) o3->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504221274 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 16:13:47 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Aug 31 16:15:16 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:553516, rc = -11. Aug 31 16:15:16 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 16:16:21 oak-gw06 kernel: LustreError: 133-1: oak-OST0001-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:553516 extent [133169152-134217727] Aug 31 16:16:21 oak-gw06 kernel: LustreError: Skipped 30 previous similar messages Aug 31 16:16:21 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client 93f3c597, server 84a62819, cksum_type 1 Aug 31 16:16:21 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 30 previous similar messages Aug 31 16:16:21 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880118ee1500 x1566273921589680/t0(0) o3->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504221428 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 16:16:21 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 27 previous similar messages Aug 31 16:18:01 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:553516, rc = -11. Aug 31 16:18:01 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 16:21:26 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:4381453 extent [2012217344-2013265919] Aug 31 16:21:26 oak-gw06 kernel: LustreError: Skipped 27 previous similar messages Aug 31 16:21:26 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client cf8c733, server 843ede42, cksum_type 1 Aug 31 16:21:26 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 27 previous similar messages Aug 31 16:21:26 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880062d9fc00 x1566273922064976/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504221697 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 16:21:26 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 25 previous similar messages Aug 31 16:23:16 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST001b-osc-ffff88041b99c000: too many resent retries for object: 0:4381453, rc = -11. Aug 31 16:23:16 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 16:35:33 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:239955 extent [186646528-187695103] Aug 31 16:35:33 oak-gw06 kernel: LustreError: Skipped 22 previous similar messages Aug 31 16:35:33 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client 55cace85, server 267d4825, cksum_type 1 Aug 31 16:35:33 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 22 previous similar messages Aug 31 16:35:33 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800259e8900 x1566273926180160/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504222580 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 16:35:33 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 19 previous similar messages Aug 31 16:36:28 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST001b-osc-ffff88041b99c000: too many resent retries for object: 0:239955, rc = -11. Aug 31 16:48:09 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:239960 extent [169869312-170917887] Aug 31 16:48:09 oak-gw06 kernel: LustreError: Skipped 65 previous similar messages Aug 31 16:48:09 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client 8857b49c, server 291f7e3a, cksum_type 1 Aug 31 16:48:09 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 65 previous similar messages Aug 31 16:48:09 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800a2f6e400 x1566273928663216/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504223300 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 16:48:09 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 59 previous similar messages Aug 31 16:49:04 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001b-osc-ffff88041b99c000: too many resent retries for object: 0:239960, rc = -11. Aug 31 16:49:04 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) Skipped 5 previous similar messages Aug 31 16:58:52 oak-gw06 kernel: LustreError: 133-1: oak-OST0011-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371483 extent [1670381568-1671430143] Aug 31 16:58:52 oak-gw06 kernel: LustreError: Skipped 54 previous similar messages Aug 31 16:58:52 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client 17c9517, server 80c262, cksum_type 1 Aug 31 16:58:52 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 54 previous similar messages Aug 31 16:58:52 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88030f8c0c00 x1566273931461680/t0(0) o3->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504223979 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 16:58:52 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 49 previous similar messages Aug 31 16:59:47 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0011-osc-ffff88041b99c000: too many resent retries for object: 0:371483, rc = -11. Aug 31 16:59:47 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Aug 31 17:11:22 oak-gw06 kernel: LustreError: 133-1: oak-OST0011-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371478 extent [163577856-164626431] Aug 31 17:11:22 oak-gw06 kernel: LustreError: Skipped 32 previous similar messages Aug 31 17:11:22 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client a31d70ca, server ad590d5f, cksum_type 1 Aug 31 17:11:22 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 32 previous similar messages Aug 31 17:11:22 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800ba229b00 x1566273934115664/t0(0) o3->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504224694 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 17:11:22 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 29 previous similar messages Aug 31 17:12:17 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST0011-osc-ffff88041b99c000: too many resent retries for object: 0:371478, rc = -11. Aug 31 17:12:17 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 17:24:39 oak-gw06 kernel: LustreError: 133-1: oak-OST0011-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371485 extent [187695104-188743679] Aug 31 17:24:39 oak-gw06 kernel: LustreError: Skipped 32 previous similar messages Aug 31 17:24:39 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client f53a1d0f, server 9b5948f8, cksum_type 1 Aug 31 17:24:39 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 32 previous similar messages Aug 31 17:24:39 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88010eba0600 x1566273936942048/t0(0) o3->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504225491 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 17:24:39 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 29 previous similar messages Aug 31 17:25:34 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0011-osc-ffff88041b99c000: too many resent retries for object: 0:371485, rc = -11. Aug 31 17:25:34 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 17:43:15 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:239963 extent [233832448-234881023] Aug 31 17:43:15 oak-gw06 kernel: LustreError: Skipped 65 previous similar messages Aug 31 17:43:15 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client 2b3a1356, server 7aec27ac, cksum_type 1 Aug 31 17:43:15 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 65 previous similar messages Aug 31 17:43:15 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802159fd500 x1566273942693856/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504226607 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 17:43:15 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 59 previous similar messages Aug 31 17:44:10 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST001b-osc-ffff88041b99c000: too many resent retries for object: 0:239963, rc = -11. Aug 31 17:44:10 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) Skipped 5 previous similar messages Aug 31 18:04:24 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240001 extent [191889408-192937983] Aug 31 18:04:24 oak-gw06 kernel: LustreError: Skipped 54 previous similar messages Aug 31 18:04:24 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 6a683663, server e9390f02, cksum_type 1 Aug 31 18:04:24 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 54 previous similar messages Aug 31 18:04:24 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802cc5aa700 x1566273947601424/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504227896 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 18:04:24 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 49 previous similar messages Aug 31 18:05:19 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST001b-osc-ffff88041b99c000: too many resent retries for object: 0:240001, rc = -11. Aug 31 18:05:19 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Aug 31 18:05:40 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240001 extent [191889408-192557055] Aug 31 18:05:40 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Aug 31 18:05:40 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client 8cabe409, server c8c3057f, cksum_type 1 Aug 31 18:05:40 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 16 previous similar messages Aug 31 18:05:40 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88033a3e1800 x1566273947872320/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504227972 ref 2 fl Interpret:RM/0/0 rc 667648/667648 Aug 31 18:05:40 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Aug 31 18:07:09 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST001b-osc-ffff88041b99c000: too many resent retries for object: 0:240001, rc = -11. Aug 31 18:07:09 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 18:08:11 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240001 extent [191889408-192937983] Aug 31 18:08:11 oak-gw06 kernel: LustreError: Skipped 29 previous similar messages Aug 31 18:08:11 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client 6a683663, server e9390f02, cksum_type 1 Aug 31 18:08:11 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 29 previous similar messages Aug 31 18:08:11 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880118623c00 x1566273948389072/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504228123 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 18:08:11 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 26 previous similar messages Aug 31 18:13:15 oak-gw06 kernel: LustreError: 133-1: oak-OST0011-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371527 extent [68157440-69206015] Aug 31 18:13:15 oak-gw06 kernel: LustreError: Skipped 17 previous similar messages Aug 31 18:13:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client fd3c2e16, server 26920f47, cksum_type 1 Aug 31 18:13:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 17 previous similar messages Aug 31 18:13:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0011-osc-ffff88041b99c000: too many resent retries for object: 0:371527, rc = -11. Aug 31 18:13:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 18:13:15 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880193933600 x1566273949963840/t0(0) o3->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504228418 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 18:13:15 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 16 previous similar messages Aug 31 18:21:17 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0011-osc-ffff88041b99c000: too many resent retries for object: 0:371531, rc = -11. Aug 31 18:21:17 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Aug 31 18:23:18 oak-gw06 kernel: LustreError: 133-1: oak-OST0011-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371531 extent [286261248-287309823] Aug 31 18:23:18 oak-gw06 kernel: LustreError: Skipped 81 previous similar messages Aug 31 18:23:18 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 64058ff4, server 3319c8df, cksum_type 1 Aug 31 18:23:18 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 81 previous similar messages Aug 31 18:23:18 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800855fad00 x1566273951269200/t0(0) o3->oak-OST0011-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504229009 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 18:23:18 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 73 previous similar messages Aug 31 18:40:39 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6905:0x0] object 0x0:3989770 extent [800849920-801898495] Aug 31 18:40:39 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Aug 31 18:40:39 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum e8f92351 (type 1), server csum 7403364b (type 1), client csum now e8f92351 Aug 31 18:40:39 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 3 previous similar messages Aug 31 18:40:39 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880425e39200 x1566273954643936/t51542881734(51542881734) o4->oak-OST0023-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504230051 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 18:40:39 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Aug 31 18:40:45 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6905:0x0] object 0x0:3989770 extent [800849920-801898495] Aug 31 18:40:45 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 18:40:45 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum e8f92351 (type 1), server csum 7403364b (type 1), client csum now e8f92351 Aug 31 18:40:45 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Aug 31 18:40:54 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6905:0x0] object 0x0:3989770 extent [800849920-801898495] Aug 31 18:40:54 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 31 18:40:54 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum e8f92351 (type 1), server csum 7403364b (type 1), client csum now e8f92351 Aug 31 18:40:54 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 1 previous similar message Aug 31 18:41:15 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6905:0x0] object 0x0:3989770 extent [800849920-801898495] Aug 31 18:41:15 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 18:41:15 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum e8f92351 (type 1), server csum 7403364b (type 1), client csum now e8f92351 Aug 31 18:41:15 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Aug 31 18:41:34 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0023-osc-ffff88041b99c000: too many resent retries for object: 0:3989770, rc = -11. Aug 31 18:41:34 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Aug 31 18:42:04 oak-gw06 kernel: LustreError: 133-1: oak-OST0019-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240120 extent [891289600-892338175] Aug 31 18:42:04 oak-gw06 kernel: LustreError: Skipped 17 previous similar messages Aug 31 18:42:04 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client 56aaab50, server 38349f19, cksum_type 1 Aug 31 18:42:04 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 17 previous similar messages Aug 31 18:42:59 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST0019-osc-ffff88041b99c000: too many resent retries for object: 0:240120, rc = -11. Aug 31 18:45:46 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST0019-osc-ffff88041b99c000: too many resent retries for object: 0:240120, rc = -11. Aug 31 18:45:46 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 18:54:53 oak-gw06 kernel: LustreError: 133-1: oak-OST0017-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:321555 extent [1815085056-1816133631] Aug 31 18:54:53 oak-gw06 kernel: LustreError: Skipped 54 previous similar messages Aug 31 18:54:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 9d48c84a, server 35359b11, cksum_type 1 Aug 31 18:54:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 54 previous similar messages Aug 31 18:54:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880252796700 x1566273957383536/t0(0) o3->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504230904 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 18:54:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 59 previous similar messages Aug 31 18:55:48 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0017-osc-ffff88041b99c000: too many resent retries for object: 0:321555, rc = -11. Aug 31 18:55:48 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 19:06:12 oak-gw06 kernel: LustreError: 133-1: oak-OST0014-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321526 extent [1475346432-1476395007] Aug 31 19:06:12 oak-gw06 kernel: LustreError: Skipped 32 previous similar messages Aug 31 19:06:12 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 2e9f5d89, server 23ecd3d1, cksum_type 1 Aug 31 19:06:12 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 32 previous similar messages Aug 31 19:06:12 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801b9deb600 x1566273959345360/t0(0) o3->oak-OST0014-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504231619 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 19:06:12 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 29 previous similar messages Aug 31 19:07:07 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0014-osc-ffff88041b99c000: too many resent retries for object: 0:321526, rc = -11. Aug 31 19:07:07 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 19:20:15 oak-gw06 kernel: LustreError: 133-1: oak-OST0000-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:553502 extent [420478976-421527551] Aug 31 19:20:15 oak-gw06 kernel: LustreError: Skipped 32 previous similar messages Aug 31 19:20:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 614a0826, server 451c4910, cksum_type 1 Aug 31 19:20:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 32 previous similar messages Aug 31 19:20:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800563c7300 x1566273962459072/t0(0) o3->oak-OST0000-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504232426 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 19:20:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 29 previous similar messages Aug 31 19:21:10 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0000-osc-ffff88041b99c000: too many resent retries for object: 0:553502, rc = -11. Aug 31 19:21:10 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 19:32:33 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240001 extent [164626432-165675007] Aug 31 19:32:33 oak-gw06 kernel: LustreError: Skipped 54 previous similar messages Aug 31 19:32:33 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client bbd7793b, server 1c573201, cksum_type 1 Aug 31 19:32:33 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 54 previous similar messages Aug 31 19:32:33 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803146af300 x1566273965313440/t0(0) o3->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504233165 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 19:32:33 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 49 previous similar messages Aug 31 19:33:28 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST001a-osc-ffff88041b99c000: too many resent retries for object: 0:240001, rc = -11. Aug 31 19:33:28 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Aug 31 20:10:43 oak-gw06 kernel: LustreError: 133-1: oak-OST0007-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371582 extent [1054867456-1055916031] Aug 31 20:10:43 oak-gw06 kernel: LustreError: Skipped 65 previous similar messages Aug 31 20:10:43 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client a0a17d83, server b6d0d647, cksum_type 1 Aug 31 20:10:43 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 65 previous similar messages Aug 31 20:10:43 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8804147de700 x1566273971034784/t0(0) o3->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504235490 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 20:10:43 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 59 previous similar messages Aug 31 20:11:38 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0007-osc-ffff88041b99c000: too many resent retries for object: 0:371582, rc = -11. Aug 31 20:11:38 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 5 previous similar messages Aug 31 20:11:59 oak-gw06 kernel: LustreError: 133-1: oak-OST0007-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371582 extent [1054867456-1055916031] Aug 31 20:11:59 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Aug 31 20:11:59 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client a0a17d83, server b6d0d647, cksum_type 1 Aug 31 20:11:59 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 16 previous similar messages Aug 31 20:11:59 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801b712b900 x1566273971272112/t0(0) o3->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504235531 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 20:11:59 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Aug 31 20:13:28 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST0007-osc-ffff88041b99c000: too many resent retries for object: 0:371582, rc = -11. Aug 31 20:13:28 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 20:30:02 oak-gw06 kernel: LustreError: 133-1: oak-OST0007-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371577 extent [143654912-144703487] Aug 31 20:30:02 oak-gw06 kernel: LustreError: Skipped 15 previous similar messages Aug 31 20:30:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client 305d2dcf, server f98e3eb, cksum_type 1 Aug 31 20:30:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 15 previous similar messages Aug 31 20:30:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802b770f600 x1566273973054240/t0(0) o3->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504236613 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 20:30:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 13 previous similar messages Aug 31 20:30:23 oak-gw06 kernel: LustreError: 133-1: oak-OST0007-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371577 extent [143654912-144703487] Aug 31 20:30:23 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Aug 31 20:30:23 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client 305d2dcf, server f98e3eb, cksum_type 1 Aug 31 20:30:23 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Aug 31 20:30:23 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801e0260c00 x1566273973122256/t0(0) o3->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504236634 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 20:30:23 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 5 previous similar messages Aug 31 20:30:57 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0007-osc-ffff88041b99c000: too many resent retries for object: 0:371577, rc = -11. Aug 31 20:31:03 oak-gw06 kernel: LustreError: 133-1: oak-OST0007-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371577 extent [143654912-144703487] Aug 31 20:31:03 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Aug 31 20:31:03 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 305d2dcf, server f98e3eb, cksum_type 1 Aug 31 20:31:03 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Aug 31 20:31:03 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880044d02700 x1566273973294016/t0(0) o3->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504236672 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 20:31:03 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 6 previous similar messages Aug 31 20:31:49 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0007-osc-ffff88041b99c000: too many resent retries for object: 0:371577, rc = -11. Aug 31 20:32:25 oak-gw06 kernel: LustreError: 133-1: oak-OST0007-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371577 extent [143654912-144703487] Aug 31 20:32:25 oak-gw06 kernel: LustreError: Skipped 15 previous similar messages Aug 31 20:32:25 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 305d2dcf, server f98e3eb, cksum_type 1 Aug 31 20:32:25 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 15 previous similar messages Aug 31 20:32:25 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801b7128f00 x1566273973677184/t0(0) o3->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504236754 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 20:32:25 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 14 previous similar messages Aug 31 20:32:41 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0007-osc-ffff88041b99c000: too many resent retries for object: 0:371577, rc = -11. Aug 31 20:34:26 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0007-osc-ffff88041b99c000: too many resent retries for object: 0:371577, rc = -11. Aug 31 20:34:26 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 20:41:38 oak-gw06 kernel: LustreError: 133-1: oak-OST0007-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371586 extent [174063616-175112191] Aug 31 20:41:38 oak-gw06 kernel: LustreError: Skipped 24 previous similar messages Aug 31 20:41:38 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 1ee0afad, server b1cc68a9, cksum_type 1 Aug 31 20:41:38 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 24 previous similar messages Aug 31 20:41:38 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802772b2100 x1566273975335648/t0(0) o3->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504237345 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 20:41:38 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 21 previous similar messages Aug 31 20:42:33 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0007-osc-ffff88041b99c000: too many resent retries for object: 0:371586, rc = -11. Aug 31 20:46:41 oak-gw06 kernel: LustreError: 133-1: oak-OST0007-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371586 extent [98566144-99614719] Aug 31 20:46:41 oak-gw06 kernel: LustreError: Skipped 61 previous similar messages Aug 31 20:46:41 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client 49ee9224, server d1df3fec, cksum_type 1 Aug 31 20:46:41 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 61 previous similar messages Aug 31 20:46:41 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800a8e95b00 x1566273976069184/t0(0) o3->oak-OST0007-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504237648 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 20:46:41 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 56 previous similar messages Aug 31 20:56:46 oak-gw06 kernel: LustreError: 133-1: oak-OST0013-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:321756 extent [123731968-124780543] Aug 31 20:56:46 oak-gw06 kernel: LustreError: Skipped 13 previous similar messages Aug 31 20:56:46 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client e67bbf75, server 599ac697, cksum_type 1 Aug 31 20:56:46 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 13 previous similar messages Aug 31 20:56:46 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0013-osc-ffff88041b99c000: too many resent retries for object: 0:321756, rc = -11. Aug 31 20:56:46 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 5 previous similar messages Aug 31 20:56:46 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803b12c3000 x1566273977943440/t0(0) o3->oak-OST0013-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504238217 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 20:56:46 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 12 previous similar messages Aug 31 21:11:53 oak-gw06 kernel: LustreError: 133-1: oak-OST0013-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:321763 extent [355467264-356515839] Aug 31 21:11:53 oak-gw06 kernel: LustreError: Skipped 22 previous similar messages Aug 31 21:11:53 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client ef870ec7, server 313b9835, cksum_type 1 Aug 31 21:11:53 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 22 previous similar messages Aug 31 21:11:53 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800a2f6ea00 x1566273980167328/t0(0) o3->oak-OST0013-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504239160 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 21:11:53 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 19 previous similar messages Aug 31 21:12:48 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0013-osc-ffff88041b99c000: too many resent retries for object: 0:321763, rc = -11. Aug 31 21:12:48 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 21:21:54 oak-gw06 kernel: LustreError: 133-1: oak-OST0013-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:321753 extent [214958080-216006655] Aug 31 21:21:54 oak-gw06 kernel: LustreError: Skipped 59 previous similar messages Aug 31 21:21:54 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client 52411125, server befad77d, cksum_type 1 Aug 31 21:21:54 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 59 previous similar messages Aug 31 21:21:54 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88038123de00 x1566273981854848/t0(0) o3->oak-OST0013-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504239736 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 21:21:54 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 54 previous similar messages Aug 31 22:00:30 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cd0:0x0] object 0x0:4394922 extent [2126774272-2127822847] Aug 31 22:00:30 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 22:00:30 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum 23c21036 (type 1), server csum 180dec39 (type 1), client csum now 23c21036 Aug 31 22:00:30 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Aug 31 22:00:30 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88028bb20f00 x1566273991385888/t60132674342(60132674342) o4->oak-OST001d-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504242076 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 22:00:30 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 4 previous similar messages Aug 31 22:00:36 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cd0:0x0] object 0x0:4394922 extent [2126774272-2127822847] Aug 31 22:00:36 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 22:00:36 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum 23c21036 (type 1), server csum 180dec39 (type 1), client csum now 23c21036 Aug 31 22:00:36 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Aug 31 22:00:45 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cd0:0x0] object 0x0:4394922 extent [2126774272-2127822847] Aug 31 22:00:45 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Aug 31 22:00:45 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) original client csum 23c21036 (type 1), server csum 180dec39 (type 1), client csum now 23c21036 Aug 31 22:00:45 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) Skipped 1 previous similar message Aug 31 22:01:06 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cd0:0x0] object 0x0:4394922 extent [2126774272-2127822847] Aug 31 22:01:06 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 22:01:06 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum 23c21036 (type 1), server csum 180dec39 (type 1), client csum now 23c21036 Aug 31 22:01:06 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Aug 31 22:01:25 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST001d-osc-ffff88041b99c000: too many resent retries for object: 0:4394922, rc = -11. Aug 31 22:01:25 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) Skipped 5 previous similar messages Aug 31 22:01:53 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cb0:0x0] object 0x0:4189801 extent [4477681664-4479516671] Aug 31 22:01:53 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 22:01:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) original client csum feb62adb (type 1), server csum 1d65d4a (type 1), client csum now feb62adb Aug 31 22:01:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Aug 31 22:01:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88028bba3600 x1566273991659296/t81607388970(81607388970) o4->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504242137 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 22:01:53 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 9 previous similar messages Aug 31 22:02:48 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:4189801, rc = -11. Aug 31 22:03:15 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cd0:0x0] object 0x0:4394922 extent [2886991872-2888040447] Aug 31 22:03:15 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Aug 31 22:03:15 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum ede57b30 (type 1), server csum bdb44afd (type 1), client csum now ede57b30 Aug 31 22:03:15 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Aug 31 22:04:34 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880227bcc300 x1566273991927664/t81607390867(81607390867) o4->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 624/416 e 0 to 0 dl 1504242293 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 22:04:34 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 19 previous similar messages Aug 31 22:05:29 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cb0:0x0] object 0x0:4189801 extent [5413011456-5414322175] Aug 31 22:05:29 oak-gw06 kernel: LustreError: Skipped 20 previous similar messages Aug 31 22:05:29 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum f426da61 (type 1), server csum cf1920 (type 1), client csum now f426da61 Aug 31 22:05:29 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 20 previous similar messages Aug 31 22:05:29 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:4189801, rc = -11. Aug 31 22:05:29 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 22:09:59 oak-gw06 kernel: LustreError: 133-1: oak-OST0008-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:371498 extent [118489088-119537663] Aug 31 22:09:59 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Aug 31 22:09:59 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client ff456f38, server d147f8b2, cksum_type 1 Aug 31 22:09:59 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Aug 31 22:09:59 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803029a4000 x1566273992868416/t0(0) o3->oak-OST0008-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504242646 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 22:09:59 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 39 previous similar messages Aug 31 22:10:54 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST0008-osc-ffff88041b99c000: too many resent retries for object: 0:371498, rc = -11. Aug 31 22:10:54 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 3 previous similar messages Aug 31 22:11:15 oak-gw06 kernel: LustreError: 133-1: oak-OST0008-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:371498 extent [118489088-119537663] Aug 31 22:11:15 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Aug 31 22:11:15 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client ff456f38, server d147f8b2, cksum_type 1 Aug 31 22:11:15 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 16 previous similar messages Aug 31 22:42:23 oak-gw06 kernel: LustreError: 133-1: oak-OST0015-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:4328394 extent [8388608-9437183] Aug 31 22:42:23 oak-gw06 kernel: LustreError: Skipped 15 previous similar messages Aug 31 22:42:23 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client a813f1e1, server d674848b, cksum_type 1 Aug 31 22:42:23 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 15 previous similar messages Aug 31 22:42:23 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802b770fc00 x1566274001083600/t0(0) o3->oak-OST0015-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504244555 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 22:42:23 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 29 previous similar messages Aug 31 22:42:44 oak-gw06 kernel: LustreError: 133-1: oak-OST0015-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:4328394 extent [8388608-9437183] Aug 31 22:42:44 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Aug 31 22:42:44 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client a813f1e1, server d674848b, cksum_type 1 Aug 31 22:42:44 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Aug 31 22:43:18 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0015-osc-ffff88041b99c000: too many resent retries for object: 0:4328394, rc = -11. Aug 31 22:43:18 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Aug 31 23:13:06 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cb0:0x0] object 0x0:4189801 extent [30926700544-30929321983] Aug 31 23:13:06 oak-gw06 kernel: LustreError: Skipped 33 previous similar messages Aug 31 23:13:06 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) original client csum 9f9353d3 (type 1), server csum 62151155 (type 1), client csum now 9f9353d3 Aug 31 23:13:06 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) Skipped 33 previous similar messages Aug 31 23:13:06 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802b2e1e400 x1566274011568592/t81607436328(81607436328) o4->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504246397 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 23:13:06 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 9 previous similar messages Aug 31 23:13:21 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803bbd3c300 x1566274011636864/t81607436499(81607436499) o4->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504246412 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 23:13:21 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 4 previous similar messages Aug 31 23:13:42 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cb0:0x0] object 0x0:4189801 extent [30926700544-30929321983] Aug 31 23:13:42 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Aug 31 23:13:42 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) original client csum 9f9353d3 (type 1), server csum 62151155 (type 1), client csum now 9f9353d3 Aug 31 23:13:42 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) Skipped 7 previous similar messages Aug 31 23:13:42 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880096c4c000 x1566274011769376/t81607436661(81607436661) o4->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504246433 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 23:13:42 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 2 previous similar messages Aug 31 23:14:01 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:4189801, rc = -11. Aug 31 23:14:22 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880149c87900 x1566274012115616/t81606955923(81606955923) o4->oak-OST0005-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504246474 ref 3 fl Interpret:R/4/0 rc 0/0 Aug 31 23:14:22 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 4 previous similar messages Aug 31 23:14:52 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cd6:0x0] object 0x0:4311074 extent [1254359040-1255407615] Aug 31 23:14:52 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Aug 31 23:14:52 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1355:check_write_checksum()) original client csum f3ecc038 (type 1), server csum 1e5033cc (type 1), client csum now f3ecc038 Aug 31 23:14:52 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Aug 31 23:15:11 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST0005-osc-ffff88041b99c000: too many resent retries for object: 0:4311074, rc = -11. Aug 31 23:15:29 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240059 extent [741343232-742391807] Aug 31 23:15:29 oak-gw06 kernel: LustreError: Skipped 4 previous similar messages Aug 31 23:15:29 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client a433398, server c52b2d1, cksum_type 1 Aug 31 23:15:29 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 4 previous similar messages Aug 31 23:15:35 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240059 extent [741343232-742391807] Aug 31 23:15:35 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 23:15:35 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client a433398, server c52b2d1, cksum_type 1 Aug 31 23:15:35 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Aug 31 23:15:39 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880044d00f00 x1566274012666240/t0(0) o3->oak-OST0018-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504246586 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 23:15:39 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 10 previous similar messages Aug 31 23:15:50 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240059 extent [741343232-742391807] Aug 31 23:15:50 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 23:15:50 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client a433398, server c52b2d1, cksum_type 1 Aug 31 23:15:50 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Aug 31 23:16:14 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240059 extent [741343232-742391807] Aug 31 23:16:14 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Aug 31 23:16:14 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client a433398, server c52b2d1, cksum_type 1 Aug 31 23:16:14 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Aug 31 23:16:24 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST0018-osc-ffff88041b99c000: too many resent retries for object: 0:240059, rc = -11. Aug 31 23:16:52 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240059 extent [741343232-742391807] Aug 31 23:16:52 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Aug 31 23:16:52 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client a433398, server c52b2d1, cksum_type 1 Aug 31 23:16:52 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 8 previous similar messages Aug 31 23:17:19 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0018-osc-ffff88041b99c000: too many resent retries for object: 0:240059, rc = -11. Aug 31 23:18:14 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240059 extent [741343232-742391807] Aug 31 23:18:14 oak-gw06 kernel: LustreError: Skipped 13 previous similar messages Aug 31 23:18:14 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client a433398, server c52b2d1, cksum_type 1 Aug 31 23:18:14 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 13 previous similar messages Aug 31 23:44:59 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240106 extent [208666624-209715199] Aug 31 23:44:59 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 5294ba90, server 13653e9a, cksum_type 1 Aug 31 23:44:59 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880275da7900 x1566274020285504/t0(0) o3->oak-OST0018-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504248310 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 23:44:59 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 25 previous similar messages Aug 31 23:45:20 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240106 extent [208666624-209715199] Aug 31 23:45:20 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Aug 31 23:45:20 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client 5294ba90, server 13653e9a, cksum_type 1 Aug 31 23:45:20 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Aug 31 23:45:20 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802076b8300 x1566274020374048/t0(0) o3->oak-OST0018-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504248331 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 23:45:20 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 5 previous similar messages Aug 31 23:45:52 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST0018-osc-ffff88041b99c000: too many resent retries for object: 0:240106, rc = -11. Aug 31 23:45:52 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Aug 31 23:45:58 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240106 extent [208666624-209715199] Aug 31 23:45:58 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Aug 31 23:45:58 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 5294ba90, server 13653e9a, cksum_type 1 Aug 31 23:45:58 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Aug 31 23:45:58 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880040a98300 x1566274020471104/t0(0) o3->oak-OST0018-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504248367 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 23:45:58 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 6 previous similar messages Aug 31 23:46:44 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0018-osc-ffff88041b99c000: too many resent retries for object: 0:240106, rc = -11. Aug 31 23:47:20 oak-gw06 kernel: LustreError: 133-1: oak-OST0018-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240106 extent [208666624-209715199] Aug 31 23:47:20 oak-gw06 kernel: LustreError: Skipped 15 previous similar messages Aug 31 23:47:20 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 5294ba90, server 13653e9a, cksum_type 1 Aug 31 23:47:20 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 15 previous similar messages Aug 31 23:47:20 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880112378600 x1566274020962928/t0(0) o3->oak-OST0018-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504248449 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Aug 31 23:47:20 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 14 previous similar messages Aug 31 23:47:36 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0018-osc-ffff88041b99c000: too many resent retries for object: 0:240106, rc = -11. Sep 1 00:02:02 oak-gw06 kernel: LustreError: 133-1: oak-OST000f-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371719 extent [121634816-122683391] Sep 1 00:02:02 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Sep 1 00:02:02 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client 657d44e9, server 42565e70, cksum_type 1 Sep 1 00:02:02 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Sep 1 00:02:02 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88035f979200 x1566274023561152/t0(0) o3->oak-OST000f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504249369 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 00:02:02 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 1 previous similar message Sep 1 00:02:23 oak-gw06 kernel: LustreError: 133-1: oak-OST000f-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371719 extent [121634816-122683391] Sep 1 00:02:23 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 00:02:23 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client 657d44e9, server 42565e70, cksum_type 1 Sep 1 00:02:23 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Sep 1 00:02:23 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800a95fea00 x1566274023573696/t0(0) o3->oak-OST000f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504249390 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 00:02:23 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 5 previous similar messages Sep 1 00:02:57 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST000f-osc-ffff88041b99c000: too many resent retries for object: 0:371719, rc = -11. Sep 1 00:26:54 oak-gw06 kernel: LustreError: 133-1: oak-OST0009-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371687 extent [2382364672-2383413247] Sep 1 00:26:54 oak-gw06 kernel: LustreError: Skipped 4 previous similar messages Sep 1 00:26:54 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client c29055ec, server 1f7cad1f, cksum_type 1 Sep 1 00:26:54 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 4 previous similar messages Sep 1 00:26:54 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88030969de00 x1566274030389760/t0(0) o3->oak-OST0009-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504250825 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 00:26:54 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 3 previous similar messages Sep 1 00:27:00 oak-gw06 kernel: LustreError: 133-1: oak-OST0009-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371687 extent [2382364672-2383413247] Sep 1 00:27:00 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Sep 1 00:27:00 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client c29055ec, server 1f7cad1f, cksum_type 1 Sep 1 00:27:00 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Sep 1 00:27:00 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803dd3f9b00 x1566274030434416/t0(0) o3->oak-OST0009-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504250831 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 00:27:00 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 2 previous similar messages Sep 1 00:27:15 oak-gw06 kernel: LustreError: 133-1: oak-OST0009-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371687 extent [2382364672-2383413247] Sep 1 00:27:15 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Sep 1 00:27:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client c29055ec, server 1f7cad1f, cksum_type 1 Sep 1 00:27:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Sep 1 00:27:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880038afa700 x1566274030572096/t0(0) o3->oak-OST0009-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504250846 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 00:27:15 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 2 previous similar messages Sep 1 00:27:39 oak-gw06 kernel: LustreError: 133-1: oak-OST0009-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371687 extent [2382364672-2383413247] Sep 1 00:27:39 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Sep 1 00:27:39 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client c29055ec, server 1f7cad1f, cksum_type 1 Sep 1 00:27:39 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 2 previous similar messages Sep 1 00:27:39 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802321f3000 x1566274030617184/t0(0) o3->oak-OST0009-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504250870 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 00:27:39 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 2 previous similar messages Sep 1 00:27:49 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST0009-osc-ffff88041b99c000: too many resent retries for object: 0:371687, rc = -11. Sep 1 00:28:17 oak-gw06 kernel: LustreError: 133-1: oak-OST0009-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:371687 extent [2382364672-2383413247] Sep 1 00:28:17 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Sep 1 00:28:17 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client c29055ec, server 1f7cad1f, cksum_type 1 Sep 1 00:28:17 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 8 previous similar messages Sep 1 00:28:17 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880028d84600 x1566274030741312/t0(0) o3->oak-OST0009-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504250908 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 00:28:17 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 7 previous similar messages Sep 1 00:28:44 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0009-osc-ffff88041b99c000: too many resent retries for object: 0:371687, rc = -11. Sep 1 00:29:09 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cde:0x0] object 0x0:4425060 extent [20119814144-20120862719] Sep 1 00:29:09 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Sep 1 00:29:09 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum bfea48a0 (type 1), server csum b4387a93 (type 1), client csum now bfea48a0 Sep 1 00:29:09 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Sep 1 00:29:30 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cde:0x0] object 0x0:4425060 extent [20119814144-20120862719] Sep 1 00:29:30 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 00:29:30 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum bfea48a0 (type 1), server csum b4387a93 (type 1), client csum now bfea48a0 Sep 1 00:29:30 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 5 previous similar messages Sep 1 00:29:37 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880062d9db00 x1566274030976528/t64427637390(64427637390) o4->oak-OST0009-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504250988 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 00:29:37 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 9 previous similar messages Sep 1 00:30:04 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cde:0x0] object 0x0:4425060 extent [20119814144-20120862719] Sep 1 00:30:04 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Sep 1 00:30:04 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum bfea48a0 (type 1), server csum b4387a93 (type 1), client csum now bfea48a0 Sep 1 00:30:04 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 3 previous similar messages Sep 1 00:30:04 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0009-osc-ffff88041b99c000: too many resent retries for object: 0:4425060, rc = -11. Sep 1 00:31:14 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cde:0x0] object 0x0:4425060 extent [20755251200-20757348351] Sep 1 00:31:14 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Sep 1 00:31:14 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum 18f2c218 (type 1), server csum eeb58db7 (type 1), client csum now 18f2c218 Sep 1 00:31:14 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 8 previous similar messages Sep 1 00:31:33 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST0009-osc-ffff88041b99c000: too many resent retries for object: 0:4425060, rc = -11. Sep 1 00:32:07 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88008e424f00 x1566274031044992/t47248303493(47248303493) o4->oak-OST0024-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504251174 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 00:32:07 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 17 previous similar messages Sep 1 00:32:48 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0024-osc-ffff88041b99c000: too many resent retries for object: 0:2269993, rc = -11. Sep 1 00:33:39 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6ce3:0x0] object 0x0:2269993 extent [1731461120-1732509695] Sep 1 00:33:39 oak-gw06 kernel: LustreError: Skipped 13 previous similar messages Sep 1 00:33:39 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) original client csum 5f6a302f (type 1), server csum 4160f791 (type 1), client csum now 5f6a302f Sep 1 00:33:39 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) Skipped 13 previous similar messages Sep 1 00:34:31 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0024-osc-ffff88041b99c000: too many resent retries for object: 0:2269993, rc = -11. Sep 1 00:35:06 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321618 extent [221249536-222298111] Sep 1 00:35:06 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Sep 1 00:35:06 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client dd8c41f7, server 3dbd43c5, cksum_type 1 Sep 1 00:35:06 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 3 previous similar messages Sep 1 00:35:58 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321618, rc = -11. Sep 1 00:36:50 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321618, rc = -11. Sep 1 00:37:11 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801c3e52a00 x1566274032094640/t0(0) o3->oak-OST0016-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504251440 ref 2 fl Interpret:RM/0/0 rc 782336/782336 Sep 1 00:37:11 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 40 previous similar messages Sep 1 00:37:42 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:321618 extent [221515776-222298111] Sep 1 00:37:42 oak-gw06 kernel: LustreError: Skipped 31 previous similar messages Sep 1 00:37:42 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client 4b15a28d, server ab24a0bf, cksum_type 1 Sep 1 00:37:42 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 31 previous similar messages Sep 1 00:38:37 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0016-osc-ffff88041b99c000: too many resent retries for object: 0:321618, rc = -11. Sep 1 00:38:37 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 00:47:33 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240204 extent [423624704-424673279] Sep 1 00:47:33 oak-gw06 kernel: LustreError: Skipped 22 previous similar messages Sep 1 00:47:33 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client f4e9ac32, server 93b1391f, cksum_type 1 Sep 1 00:47:33 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 22 previous similar messages Sep 1 00:47:33 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801b0726700 x1566274034277536/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504252100 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 00:47:33 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 23 previous similar messages Sep 1 00:48:28 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001b-osc-ffff88041b99c000: too many resent retries for object: 0:240204, rc = -11. Sep 1 00:48:28 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 00:53:58 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST001b-osc-ffff88041b99c000: too many resent retries for object: 0:240204, rc = -11. Sep 1 00:53:58 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 5 previous similar messages Sep 1 01:13:02 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240200 extent [504365056-505413631] Sep 1 01:13:02 oak-gw06 kernel: LustreError: Skipped 76 previous similar messages Sep 1 01:13:02 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 58c31509, server 656db1e4, cksum_type 1 Sep 1 01:13:02 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 76 previous similar messages Sep 1 01:13:02 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880163b23000 x1566274041407104/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504253593 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 01:13:02 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 69 previous similar messages Sep 1 01:13:57 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST001b-osc-ffff88041b99c000: too many resent retries for object: 0:240200, rc = -11. Sep 1 01:14:18 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240200 extent [504365056-505413631] Sep 1 01:14:18 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Sep 1 01:14:18 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 58c31509, server 656db1e4, cksum_type 1 Sep 1 01:14:18 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 16 previous similar messages Sep 1 01:14:18 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88008d387000 x1566274041567744/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504253669 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 01:14:18 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Sep 1 01:16:50 oak-gw06 kernel: LustreError: 133-1: oak-OST001b-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:240200 extent [375390208-376438783] Sep 1 01:16:50 oak-gw06 kernel: LustreError: Skipped 29 previous similar messages Sep 1 01:16:50 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client 736cdc82, server a9125fb1, cksum_type 1 Sep 1 01:16:50 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 29 previous similar messages Sep 1 01:16:50 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88014a3d9800 x1566274042135504/t0(0) o3->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504253859 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 01:16:50 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 26 previous similar messages Sep 1 02:12:13 oak-gw06 kernel: LustreError: 133-1: oak-OST0001-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:553792 extent [6291456-7340031] Sep 1 02:12:13 oak-gw06 kernel: LustreError: Skipped 18 previous similar messages Sep 1 02:12:13 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client b89ee35, server afe48e38, cksum_type 1 Sep 1 02:12:13 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 18 previous similar messages Sep 1 02:12:13 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88026f768c00 x1566274049384432/t0(0) o3->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504257144 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 02:12:13 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 16 previous similar messages Sep 1 02:12:58 oak-gw06 kernel: LustreError: 133-1: oak-OST0001-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:553792 extent [6291456-7340031] Sep 1 02:12:58 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Sep 1 02:12:58 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client b89ee35, server afe48e38, cksum_type 1 Sep 1 02:12:58 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 8 previous similar messages Sep 1 02:12:58 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801fb756d00 x1566274049402032/t0(0) o3->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504257189 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 02:12:58 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 8 previous similar messages Sep 1 02:13:08 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:553792, rc = -11. Sep 1 02:13:08 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) Skipped 5 previous similar messages Sep 1 02:14:13 oak-gw06 kernel: LustreError: 133-1: oak-OST0001-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:553792 extent [7147520-7340031] Sep 1 02:14:13 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Sep 1 02:14:13 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 42363151, server e65b515c, cksum_type 1 Sep 1 02:14:13 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 16 previous similar messages Sep 1 02:14:13 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88030664f000 x1566274049439408/t0(0) o3->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504257264 ref 2 fl Interpret:RM/0/0 rc 192512/192512 Sep 1 02:14:13 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 14 previous similar messages Sep 1 02:14:58 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:553792, rc = -11. Sep 1 02:14:58 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 02:38:12 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f83:0x154ea:0x0] object 0x0:4072530 extent [24669323264-24673255423] Sep 1 02:38:12 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Sep 1 02:38:12 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) original client csum 75275f24 (type 1), server csum 5085e4fa (type 1), client csum now 75275f24 Sep 1 02:38:12 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Sep 1 02:38:12 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88015cc71200 x1566274051064016/t51543335279(51543335279) o4->oak-OST001f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504258739 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 02:38:12 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Sep 1 02:38:33 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802c3389500 x1566274051076416/t51543335568(51543335568) o4->oak-OST001f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504258724 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 02:38:33 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 5 previous similar messages Sep 1 02:38:48 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f83:0x154ea:0x0] object 0x0:4072530 extent [24669323264-24673255423] Sep 1 02:38:48 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Sep 1 02:38:48 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) original client csum 75275f24 (type 1), server csum 5085e4fa (type 1), client csum now 75275f24 Sep 1 02:38:48 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) Skipped 7 previous similar messages Sep 1 02:39:07 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001f-osc-ffff88041b99c000: too many resent retries for object: 0:4072530, rc = -11. Sep 1 02:39:07 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 02:39:43 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802076bbc00 x1566274051159824/t60133309618(60133309618) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 656/424 e 0 to 0 dl 1504258832 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 02:39:43 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 3 previous similar messages Sep 1 02:39:53 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6cf4:0x0] object 0x0:4468452 extent [23768334336-23770955775] Sep 1 02:39:53 oak-gw06 kernel: LustreError: Skipped 6 previous similar messages Sep 1 02:39:53 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum fb081e85 (type 1), server csum 45890a3 (type 1), client csum now fb081e85 Sep 1 02:39:53 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 6 previous similar messages Sep 1 02:40:38 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001a-osc-ffff88041b99c000: too many resent retries for object: 0:4468452, rc = -11. Sep 1 02:42:23 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6cee:0x0] object 0x0:4032247 extent [2145648640-2146697215] Sep 1 02:42:23 oak-gw06 kernel: LustreError: Skipped 6 previous similar messages Sep 1 02:42:23 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum 6dbd19d5 (type 1), server csum 8611e0ec (type 1), client csum now 6dbd19d5 Sep 1 02:42:23 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) Skipped 6 previous similar messages Sep 1 02:42:23 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880137f88600 x1566274051303536/t51542904695(51542904695) o4->oak-OST0021-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504258990 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 02:42:23 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 9 previous similar messages Sep 1 02:43:18 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0021-osc-ffff88041b99c000: too many resent retries for object: 0:4032247, rc = -11. Sep 1 02:45:00 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88030189db00 x1566274051411600/t51543342146(51543342146) o4->oak-OST001f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504259147 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 02:45:00 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 18 previous similar messages Sep 1 02:45:10 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST001f-osc-ffff88041b99c000: too many resent retries for object: 0:4072530, rc = -11. Sep 1 02:46:34 oak-gw06 kernel: LustreError: 133-1: oak-OST0021-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:4032247 extent [715128832-716177407] Sep 1 02:46:34 oak-gw06 kernel: LustreError: Skipped 17 previous similar messages Sep 1 02:46:34 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client dbf8dbd9, server 96a89361, cksum_type 1 Sep 1 02:46:34 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 17 previous similar messages Sep 1 02:46:55 oak-gw06 kernel: LustreError: 133-1: oak-OST0021-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:4032247 extent [715128832-716177407] Sep 1 02:46:55 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 02:46:55 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client dbf8dbd9, server 96a89361, cksum_type 1 Sep 1 02:46:55 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Sep 1 02:47:35 oak-gw06 kernel: LustreError: 133-1: oak-OST0021-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:4032247 extent [715128832-716177407] Sep 1 02:47:35 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Sep 1 02:47:35 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client dbf8dbd9, server 96a89361, cksum_type 1 Sep 1 02:47:35 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Sep 1 02:48:24 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST0021-osc-ffff88041b99c000: too many resent retries for object: 0:4032247, rc = -11. Sep 1 02:48:24 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 02:48:52 oak-gw06 kernel: LustreError: 133-1: oak-OST0021-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:4032247 extent [715128832-716177407] Sep 1 02:48:52 oak-gw06 kernel: LustreError: Skipped 14 previous similar messages Sep 1 02:48:52 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client dbf8dbd9, server 96a89361, cksum_type 1 Sep 1 02:48:52 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 14 previous similar messages Sep 1 02:50:24 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6cf4:0x0] object 0x0:4468452 extent [27769176064-27772321791] Sep 1 02:50:24 oak-gw06 kernel: LustreError: Skipped 21 previous similar messages Sep 1 02:50:24 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1355:check_write_checksum()) original client csum 774b03a7 (type 1), server csum d4292b0c (type 1), client csum now 774b03a7 Sep 1 02:50:24 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1355:check_write_checksum()) Skipped 21 previous similar messages Sep 1 02:50:24 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88002818f000 x1566274051825840/t60133318270(60133318270) o4->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504259435 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 02:50:24 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 30 previous similar messages Sep 1 02:54:19 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0021-osc-ffff88041b99c000: too many resent retries for object: 0:4043342, rc = -11. Sep 1 02:54:19 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 3 previous similar messages Sep 1 02:54:35 oak-gw06 kernel: LustreError: 133-1: oak-OST0021-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:4043342 extent [28311552-29360127] Sep 1 02:54:35 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Sep 1 02:54:35 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client b6464356, server 9c31066e, cksum_type 1 Sep 1 02:54:35 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 3 previous similar messages Sep 1 03:41:53 oak-gw06 kernel: LustreError: 133-1: oak-OST0008-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:371892 extent [41943040-42991615] Sep 1 03:41:53 oak-gw06 kernel: LustreError: Skipped 32 previous similar messages Sep 1 03:41:53 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client f14ae0e3, server a1807288, cksum_type 1 Sep 1 03:41:53 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 32 previous similar messages Sep 1 03:41:53 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880122004300 x1566274058047728/t0(0) o3->oak-OST0008-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504262525 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 03:41:53 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 59 previous similar messages Sep 1 03:42:38 oak-gw06 kernel: LustreError: 133-1: oak-OST0008-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:371892 extent [41943040-42991615] Sep 1 03:42:38 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Sep 1 03:42:38 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client f14ae0e3, server a1807288, cksum_type 1 Sep 1 03:42:38 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 8 previous similar messages Sep 1 03:42:48 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0008-osc-ffff88041b99c000: too many resent retries for object: 0:371892, rc = -11. Sep 1 03:42:48 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 3 previous similar messages Sep 1 03:43:09 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88020dc7a700 x1566274058115328/t0(0) o3->oak-OST0008-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504262601 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 03:43:09 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Sep 1 03:43:58 oak-gw06 kernel: LustreError: 133-1: oak-OST0008-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:371892 extent [41943040-42991615] Sep 1 03:43:58 oak-gw06 kernel: LustreError: Skipped 17 previous similar messages Sep 1 03:43:58 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client f14ae0e3, server a1807288, cksum_type 1 Sep 1 03:43:58 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 17 previous similar messages Sep 1 03:44:38 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0008-osc-ffff88041b99c000: too many resent retries for object: 0:371892, rc = -11. Sep 1 03:44:38 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 03:59:28 oak-gw06 kernel: LustreError: 133-1: oak-OST0015-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322077 extent [50331648-51380223] Sep 1 03:59:28 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 03:59:28 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 9f9534b4, server 75e2b9f6, cksum_type 1 Sep 1 03:59:28 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Sep 1 03:59:28 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88010164e100 x1566274059046800/t0(0) o3->oak-OST0015-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504263579 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 03:59:28 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 13 previous similar messages Sep 1 03:59:49 oak-gw06 kernel: LustreError: 133-1: oak-OST0015-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322077 extent [50331648-51380223] Sep 1 03:59:49 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 03:59:49 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client 9f9534b4, server 75e2b9f6, cksum_type 1 Sep 1 03:59:49 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Sep 1 03:59:49 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880297544f00 x1566274059063200/t0(0) o3->oak-OST0015-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504263600 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 03:59:49 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 5 previous similar messages Sep 1 04:00:23 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0015-osc-ffff88041b99c000: too many resent retries for object: 0:322077, rc = -11. Sep 1 04:00:29 oak-gw06 kernel: LustreError: 133-1: oak-OST0015-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322077 extent [50331648-51380223] Sep 1 04:00:29 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Sep 1 04:00:29 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client 9f9534b4, server 75e2b9f6, cksum_type 1 Sep 1 04:00:29 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Sep 1 04:00:29 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800afd26700 x1566274059104080/t0(0) o3->oak-OST0015-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504263640 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 04:00:29 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 6 previous similar messages Sep 1 04:01:18 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0015-osc-ffff88041b99c000: too many resent retries for object: 0:322077, rc = -11. Sep 1 04:01:46 oak-gw06 kernel: LustreError: 133-1: oak-OST0015-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322077 extent [50331648-51380223] Sep 1 04:01:46 oak-gw06 kernel: LustreError: Skipped 14 previous similar messages Sep 1 04:01:46 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 9f9534b4, server 75e2b9f6, cksum_type 1 Sep 1 04:01:46 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 14 previous similar messages Sep 1 04:01:46 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88014b6f5b00 x1566274059173712/t0(0) o3->oak-OST0015-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504263717 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 04:01:46 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 13 previous similar messages Sep 1 04:02:13 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0015-osc-ffff88041b99c000: too many resent retries for object: 0:322077, rc = -11. Sep 1 04:07:57 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f83:0x154ff:0x0] object 0x0:4359803 extent [1909981184-1914961919] Sep 1 04:07:57 oak-gw06 kernel: LustreError: Skipped 32 previous similar messages Sep 1 04:07:57 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum 3cf13ab3 (type 1), server csum 8a5b1af7 (type 1), client csum now 3cf13ab3 Sep 1 04:07:57 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) Skipped 32 previous similar messages Sep 1 04:07:57 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880139554c00 x1566274059452272/t60132792991(60132792991) o4->oak-OST0015-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504264089 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 04:07:57 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 2 previous similar messages Sep 1 04:08:52 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0015-osc-ffff88041b99c000: too many resent retries for object: 0:4359803, rc = -11. Sep 1 04:09:23 oak-gw06 kernel: LustreError: 133-1: oak-OST0012-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:322042 extent [37748736-38797311] Sep 1 04:09:23 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Sep 1 04:09:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client 9d433ddf, server 25aeece8, cksum_type 1 Sep 1 04:09:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 3 previous similar messages Sep 1 04:12:08 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST0012-osc-ffff88041b99c000: too many resent retries for object: 0:322042, rc = -11. Sep 1 04:12:08 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Sep 1 04:30:17 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d03:0x0] object 0x0:2344976 extent [18496618496-18500288511] Sep 1 04:30:17 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Sep 1 04:30:17 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) original client csum acc2f25f (type 1), server csum 64afa94e (type 1), client csum now acc2f25f Sep 1 04:30:17 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Sep 1 04:30:17 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880044d03c00 x1566274060818064/t34363704994(34363704994) o4->oak-OST0025-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504265466 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 04:30:17 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 39 previous similar messages Sep 1 04:30:27 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d03:0x0] object 0x0:2344976 extent [18496618496-18500288511] Sep 1 04:30:27 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Sep 1 04:30:27 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum acc2f25f (type 1), server csum 64afa94e (type 1), client csum now acc2f25f Sep 1 04:30:27 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 3 previous similar messages Sep 1 04:30:45 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d03:0x0] object 0x0:2344976 extent [18496618496-18500288511] Sep 1 04:30:45 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Sep 1 04:30:45 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum acc2f25f (type 1), server csum 64afa94e (type 1), client csum now acc2f25f Sep 1 04:30:45 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Sep 1 04:31:02 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801483d3000 x1566274060859264/t34363705541(34363705541) o4->oak-OST0025-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504265511 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 04:31:02 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 8 previous similar messages Sep 1 04:31:10 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0025-osc-ffff88041b99c000: too many resent retries for object: 0:2344976, rc = -11. Sep 1 04:31:38 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d03:0x0] object 0x0:2344976 extent [18991808512-18993643519] Sep 1 04:31:38 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Sep 1 04:31:38 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum ed95c992 (type 1), server csum 424e4ff1 (type 1), client csum now ed95c992 Sep 1 04:31:38 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 3 previous similar messages Sep 1 04:32:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801483d3000 x1566274060931680/t34363706454(34363706454) o4->oak-OST0025-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504265590 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 04:32:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 9 previous similar messages Sep 1 04:32:33 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0025-osc-ffff88041b99c000: too many resent retries for object: 0:2344976, rc = -11. Sep 1 04:32:47 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f83:0x154d0:0x0] object 0x0:4468441 extent [31287672832-31289507839] Sep 1 04:32:47 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Sep 1 04:32:47 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) original client csum a9073043 (type 1), server csum 9efde536 (type 1), client csum now a9073043 Sep 1 04:32:47 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Sep 1 04:35:02 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f83:0x154d0:0x0] object 0x0:4468441 extent [31815630848-31819825151] Sep 1 04:35:02 oak-gw06 kernel: LustreError: Skipped 20 previous similar messages Sep 1 04:35:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum 3904b96c (type 1), server csum 4f2d26f (type 1), client csum now 3904b96c Sep 1 04:35:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 20 previous similar messages Sep 1 04:35:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0018-osc-ffff88041b99c000: too many resent retries for object: 0:4468441, rc = -11. Sep 1 04:35:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 04:43:57 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f83:0x154d0:0x0] object 0x0:4468441 extent [35473063936-35474636799] Sep 1 04:43:57 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum 9fa16172 (type 1), server csum d5cff9e1 (type 1), client csum now 9fa16172 Sep 1 04:43:57 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801483d0c00 x1566274061765104/t60133891875(60133891875) o4->oak-OST0018-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504266284 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 04:43:57 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 20 previous similar messages Sep 1 04:44:52 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0018-osc-ffff88041b99c000: too many resent retries for object: 0:4468441, rc = -11. Sep 1 04:45:31 oak-gw06 kernel: LustreError: 133-1: oak-OST0017-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322239 extent [5242880-6291455] Sep 1 04:45:31 oak-gw06 kernel: LustreError: Skipped 32 previous similar messages Sep 1 04:45:31 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client 93c7acf3, server bb497b77, cksum_type 1 Sep 1 04:45:31 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 32 previous similar messages Sep 1 04:46:16 oak-gw06 kernel: LustreError: 133-1: oak-OST0017-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322239 extent [5242880-6291455] Sep 1 04:46:16 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Sep 1 04:46:16 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 93c7acf3, server bb497b77, cksum_type 1 Sep 1 04:46:16 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 8 previous similar messages Sep 1 04:47:36 oak-gw06 kernel: LustreError: 133-1: oak-OST0017-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322239 extent [5242880-6291455] Sep 1 04:47:36 oak-gw06 kernel: LustreError: Skipped 17 previous similar messages Sep 1 04:47:36 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client 93c7acf3, server bb497b77, cksum_type 1 Sep 1 04:47:36 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 17 previous similar messages Sep 1 04:49:01 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880218263900 x1566274062208816/t0(0) o3->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504266552 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 04:49:01 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 48 previous similar messages Sep 1 04:50:06 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0017-osc-ffff88041b99c000: too many resent retries for object: 0:322239, rc = -11. Sep 1 04:50:06 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Sep 1 04:57:44 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f83:0x154d0:0x0] object 0x0:4468441 extent [39790051328-39792672767] Sep 1 04:57:44 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Sep 1 04:57:44 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1355:check_write_checksum()) original client csum 664817f3 (type 1), server csum b5ab4f22 (type 1), client csum now 664817f3 Sep 1 04:57:44 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Sep 1 04:58:54 oak-gw06 kernel: LustreError: 133-1: oak-OST000f-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:372207 extent [19922944-20971519] Sep 1 04:58:54 oak-gw06 kernel: LustreError: Skipped 27 previous similar messages Sep 1 04:58:54 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client c1dd87c4, server 3c0e1d97, cksum_type 1 Sep 1 04:58:54 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 27 previous similar messages Sep 1 04:59:04 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88025a46ea00 x1566274062729888/t0(0) o3->oak-OST000f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504267155 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 04:59:04 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 24 previous similar messages Sep 1 05:00:44 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST000f-osc-ffff88041b99c000: too many resent retries for object: 0:372207, rc = -11. Sep 1 05:00:44 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Sep 1 05:33:35 oak-gw06 kernel: LustreError: 133-1: oak-OST000f-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:372265 extent [33554432-34603007] Sep 1 05:33:35 oak-gw06 kernel: LustreError: Skipped 54 previous similar messages Sep 1 05:33:35 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client dc989456, server 88e902e4, cksum_type 1 Sep 1 05:33:35 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 54 previous similar messages Sep 1 05:33:35 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880134760900 x1566274066295776/t0(0) o3->oak-OST000f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504269226 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 05:33:35 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 45 previous similar messages Sep 1 05:34:20 oak-gw06 kernel: LustreError: 133-1: oak-OST000f-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:372265 extent [33554432-34603007] Sep 1 05:34:20 oak-gw06 kernel: LustreError: Skipped 8 previous similar messages Sep 1 05:34:20 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client dc989456, server 88e902e4, cksum_type 1 Sep 1 05:34:20 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 8 previous similar messages Sep 1 05:34:30 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST000f-osc-ffff88041b99c000: too many resent retries for object: 0:372265, rc = -11. Sep 1 05:34:30 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 3 previous similar messages Sep 1 05:34:51 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801f62b3c00 x1566274066378720/t0(0) o3->oak-OST000f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504269302 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 05:34:51 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Sep 1 05:35:35 oak-gw06 kernel: LustreError: 133-1: oak-OST000f-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:372265 extent [33554432-34603007] Sep 1 05:35:35 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Sep 1 05:35:35 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client dc989456, server 88e902e4, cksum_type 1 Sep 1 05:35:35 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 16 previous similar messages Sep 1 05:36:17 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST000f-osc-ffff88041b99c000: too many resent retries for object: 0:372265, rc = -11. Sep 1 05:36:17 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 05:37:27 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6ca6:0x0] object 0x0:4387979 extent [10193469440-10197139455] Sep 1 05:37:27 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Sep 1 05:37:27 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum 1bab083a (type 1), server csum 9dd06d77 (type 1), client csum now 1bab083a Sep 1 05:37:27 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Sep 1 05:37:27 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88030189db00 x1566274066507008/t60133410877(60133410877) o4->oak-OST001b-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 656/424 e 0 to 0 dl 1504269456 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 05:37:27 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 23 previous similar messages Sep 1 05:38:47 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f83:0x154d2:0x0] object 0x0:830081 extent [13551009792-13553631231] Sep 1 05:38:47 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Sep 1 05:38:47 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) original client csum 5142110c (type 1), server csum a7d59ee4 (type 1), client csum now 5142110c Sep 1 05:38:47 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Sep 1 05:39:42 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST002e-osc-ffff88041b99c000: too many resent retries for object: 0:830081, rc = -11. Sep 1 05:39:42 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Sep 1 05:41:13 oak-gw06 kernel: LustreError: 133-1: oak-OST000a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:372182 extent [40894464-41943039] Sep 1 05:41:13 oak-gw06 kernel: LustreError: Skipped 17 previous similar messages Sep 1 05:41:13 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 5954e9e, server e427e432, cksum_type 1 Sep 1 05:41:13 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 17 previous similar messages Sep 1 05:42:29 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880362a7d500 x1566274066826720/t0(0) o3->oak-OST000a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504269761 ref 2 fl Interpret:RM/0/0 rc 503808/503808 Sep 1 05:42:29 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 45 previous similar messages Sep 1 05:44:53 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST000a-osc-ffff88041b99c000: too many resent retries for object: 0:372182, rc = -11. Sep 1 05:44:53 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Sep 1 05:53:06 oak-gw06 kernel: LustreError: 133-1: oak-OST0027-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:2395485 extent [438304768-439353343] Sep 1 05:53:06 oak-gw06 kernel: LustreError: Skipped 54 previous similar messages Sep 1 05:53:06 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 21b6fc2d, server 1164d438, cksum_type 1 Sep 1 05:53:06 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 54 previous similar messages Sep 1 05:53:06 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802cbdb9800 x1566274067432992/t0(0) o3->oak-OST0027-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504270434 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 05:53:06 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 33 previous similar messages Sep 1 05:54:56 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0027-osc-ffff88041b99c000: too many resent retries for object: 0:2395485, rc = -11. Sep 1 05:54:56 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Sep 1 06:20:45 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d1a:0x0] object 0x0:905059 extent [1316487168-1317535743] Sep 1 06:20:45 oak-gw06 kernel: LustreError: Skipped 21 previous similar messages Sep 1 06:20:45 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum d04fcbeb (type 1), server csum 9cca509b (type 1), client csum now d04fcbeb Sep 1 06:20:45 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 21 previous similar messages Sep 1 06:20:45 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88000ad54900 x1566274071269728/t21480499916(21480499916) o4->oak-OST002f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504272057 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 06:20:45 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 29 previous similar messages Sep 1 06:21:06 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d1a:0x0] object 0x0:905059 extent [1316487168-1317535743] Sep 1 06:21:06 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 06:21:06 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) original client csum d04fcbeb (type 1), server csum 9cca509b (type 1), client csum now d04fcbeb Sep 1 06:21:06 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1355:check_write_checksum()) Skipped 5 previous similar messages Sep 1 06:21:40 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST002f-osc-ffff88041b99c000: too many resent retries for object: 0:905059, rc = -11. Sep 1 06:21:40 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 06:22:31 oak-gw06 kernel: LustreError: 133-1: oak-OST002f-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:905059 extent [785383424-786431999] Sep 1 06:22:31 oak-gw06 kernel: LustreError: Skipped 32 previous similar messages Sep 1 06:22:31 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client 198d09d3, server 3006e943, cksum_type 1 Sep 1 06:22:31 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 32 previous similar messages Sep 1 06:22:31 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800afd27c00 x1566274071408288/t0(0) o3->oak-OST002f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504272162 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 06:22:31 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 9 previous similar messages Sep 1 06:23:26 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST002f-osc-ffff88041b99c000: too many resent retries for object: 0:905059, rc = -11. Sep 1 06:23:47 oak-gw06 kernel: LustreError: 133-1: oak-OST002f-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:905059 extent [785383424-786431999] Sep 1 06:23:47 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Sep 1 06:23:47 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 198d09d3, server 3006e943, cksum_type 1 Sep 1 06:23:47 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 16 previous similar messages Sep 1 06:25:06 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88014d669200 x1566274071564064/t0(0) o3->oak-OST002f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504272318 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 06:25:06 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 28 previous similar messages Sep 1 06:40:27 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d1d:0x0] object 0x0:2399337 extent [1261699072-1262747647] Sep 1 06:40:27 oak-gw06 kernel: LustreError: Skipped 4 previous similar messages Sep 1 06:40:27 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum f8de5a5a (type 1), server csum 583ed300 (type 1), client csum now f8de5a5a Sep 1 06:40:27 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) Skipped 4 previous similar messages Sep 1 06:40:27 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880060ed3c00 x1566274072568272/t34364223935(34364223935) o4->oak-OST0027-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504273237 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 06:40:33 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d1d:0x0] object 0x0:2399337 extent [1261699072-1262747647] Sep 1 06:40:33 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Sep 1 06:40:33 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum f8de5a5a (type 1), server csum 583ed300 (type 1), client csum now f8de5a5a Sep 1 06:40:33 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Sep 1 06:40:42 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d1d:0x0] object 0x0:2399337 extent [1261699072-1262747647] Sep 1 06:40:42 oak-gw06 kernel: LustreError: Skipped 1 previous similar message Sep 1 06:40:42 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum f8de5a5a (type 1), server csum 583ed300 (type 1), client csum now f8de5a5a Sep 1 06:40:42 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 1 previous similar message Sep 1 06:41:03 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d1d:0x0] object 0x0:2399337 extent [1261699072-1262747647] Sep 1 06:41:03 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Sep 1 06:41:03 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum f8de5a5a (type 1), server csum 583ed300 (type 1), client csum now f8de5a5a Sep 1 06:41:03 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Sep 1 06:41:12 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88023210a700 x1566274072606144/t34364224241(34364224241) o4->oak-OST0027-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504273283 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 06:41:12 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 8 previous similar messages Sep 1 06:41:22 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST0027-osc-ffff88041b99c000: too many resent retries for object: 0:2399337, rc = -11. Sep 1 06:41:22 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Sep 1 06:42:32 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d16:0x0] object 0x0:4105432 extent [18479316992-18481151999] Sep 1 06:42:32 oak-gw06 kernel: LustreError: Skipped 2 previous similar messages Sep 1 06:42:32 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum 2d980cf7 (type 1), server csum 3eccabbe (type 1), client csum now 2d980cf7 Sep 1 06:42:32 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 2 previous similar messages Sep 1 06:42:32 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88034b2a6a00 x1566274072676400/t51543508943(51543508943) o4->oak-OST001f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 640/424 e 0 to 0 dl 1504273363 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 06:43:27 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1643:brw_interpret()) oak-OST001f-osc-ffff88041b99c000: too many resent retries for object: 0:4105432, rc = -11. Sep 1 06:43:57 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.102@o2ib5 inode [0x200002f84:0x6d16:0x0] object 0x0:4105432 extent [19054460928-19055509503] Sep 1 06:43:57 oak-gw06 kernel: LustreError: Skipped 10 previous similar messages Sep 1 06:43:57 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) original client csum 15aa1380 (type 1), server csum ba8d4784 (type 1), client csum now 15aa1380 Sep 1 06:43:57 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1355:check_write_checksum()) Skipped 10 previous similar messages Sep 1 06:44:52 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST001f-osc-ffff88041b99c000: too many resent retries for object: 0:4105432, rc = -11. Sep 1 06:45:18 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88030189f300 x1566274072815696/t17188960666(17188960666) o4->oak-OST002e-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504273530 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 06:45:18 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 19 previous similar messages Sep 1 06:46:14 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST002e-osc-ffff88041b99c000: too many resent retries for object: 0:830081, rc = -11. Sep 1 06:46:47 oak-gw06 kernel: LustreError: 133-1: oak-OST001c-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240954 extent [44040192-45088767] Sep 1 06:46:47 oak-gw06 kernel: LustreError: Skipped 15 previous similar messages Sep 1 06:46:47 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client 345074ed, server a22bb01f, cksum_type 1 Sep 1 06:46:47 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 15 previous similar messages Sep 1 06:47:08 oak-gw06 kernel: LustreError: 133-1: oak-OST001c-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240954 extent [44040192-45088767] Sep 1 06:47:08 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 06:47:08 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client 345074ed, server a22bb01f, cksum_type 1 Sep 1 06:47:08 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Sep 1 06:47:48 oak-gw06 kernel: LustreError: 133-1: oak-OST001c-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240954 extent [44040192-45088767] Sep 1 06:47:48 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Sep 1 06:47:48 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 345074ed, server a22bb01f, cksum_type 1 Sep 1 06:47:48 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Sep 1 06:49:05 oak-gw06 kernel: LustreError: 133-1: oak-OST001c-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:240954 extent [44040192-45088767] Sep 1 06:49:05 oak-gw06 kernel: LustreError: Skipped 14 previous similar messages Sep 1 06:49:05 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 345074ed, server a22bb01f, cksum_type 1 Sep 1 06:49:05 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 14 previous similar messages Sep 1 06:49:32 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST001c-osc-ffff88041b99c000: too many resent retries for object: 0:240954, rc = -11. Sep 1 06:49:32 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Sep 1 06:50:27 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88029d63e400 x1566274073146544/t0(0) o3->oak-OST001c-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504273838 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 06:50:27 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 49 previous similar messages Sep 1 07:03:30 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6d19:0x0] object 0x0:4464990 extent [1492123648-1493172223] Sep 1 07:03:30 oak-gw06 kernel: LustreError: Skipped 21 previous similar messages Sep 1 07:03:30 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) original client csum 47bb6920 (type 1), server csum 130ec260 (type 1), client csum now 47bb6920 Sep 1 07:03:30 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1355:check_write_checksum()) Skipped 21 previous similar messages Sep 1 07:03:30 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88014dd22400 x1566274074780000/t64427852518(64427852518) o4->oak-OST0012-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/416 e 0 to 0 dl 1504274621 ref 3 fl Interpret:R/4/0 rc 0/0 Sep 1 07:03:30 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 9 previous similar messages Sep 1 07:03:51 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6d19:0x0] object 0x0:4464990 extent [1492123648-1493172223] Sep 1 07:03:51 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 07:03:51 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) original client csum 47bb6920 (type 1), server csum 130ec260 (type 1), client csum now 47bb6920 Sep 1 07:03:51 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1355:check_write_checksum()) Skipped 5 previous similar messages Sep 1 07:04:26 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST0012-osc-ffff88041b99c000: too many resent retries for object: 0:4464990, rc = -11. Sep 1 07:04:26 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Sep 1 07:04:44 oak-gw06 kernel: LustreError: 132-0: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 10.0.2.101@o2ib5 inode [0x200002f84:0x6d19:0x0] object 0x0:4464990 extent [1915486208-1917059071] Sep 1 07:04:44 oak-gw06 kernel: LustreError: Skipped 4 previous similar messages Sep 1 07:04:44 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) original client csum a78ed1bc (type 1), server csum cbefcd60 (type 1), client csum now a78ed1bc Sep 1 07:04:44 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1355:check_write_checksum()) Skipped 4 previous similar messages Sep 1 07:05:57 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4410214 extent [21417164800-21418213375] Sep 1 07:05:57 oak-gw06 kernel: LustreError: Skipped 25 previous similar messages Sep 1 07:05:57 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client 8c30af7, server d9559abf, cksum_type 1 Sep 1 07:05:57 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 25 previous similar messages Sep 1 07:06:18 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4410214 extent [21417164800-21418213375] Sep 1 07:06:18 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 07:06:18 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 8c30af7, server d9559abf, cksum_type 1 Sep 1 07:06:18 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Sep 1 07:06:58 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4410214 extent [21417164800-21418213375] Sep 1 07:06:58 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Sep 1 07:06:58 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client 8c30af7, server d9559abf, cksum_type 1 Sep 1 07:06:58 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Sep 1 07:08:15 oak-gw06 kernel: LustreError: 133-1: oak-OST0016-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:4410214 extent [21417164800-21418213375] Sep 1 07:08:15 oak-gw06 kernel: LustreError: Skipped 14 previous similar messages Sep 1 07:08:15 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 8c30af7, server d9559abf, cksum_type 1 Sep 1 07:08:15 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 14 previous similar messages Sep 1 07:15:48 oak-gw06 kernel: LustreError: 133-1: oak-OST000c-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:372555 extent [19922944-20971519] Sep 1 07:15:48 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Sep 1 07:15:48 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client b2eb8dd3, server edad80ea, cksum_type 1 Sep 1 07:15:48 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 3 previous similar messages Sep 1 07:15:48 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801b35ef900 x1566274076259392/t0(0) o3->oak-OST000c-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504275397 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 07:15:48 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 49 previous similar messages Sep 1 07:16:43 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST000c-osc-ffff88041b99c000: too many resent retries for object: 0:372555, rc = -11. Sep 1 07:16:43 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Sep 1 07:28:07 oak-gw06 kernel: LustreError: 133-1: oak-OST0010-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:372522 extent [4194304-5242879] Sep 1 07:28:07 oak-gw06 kernel: LustreError: Skipped 54 previous similar messages Sep 1 07:28:07 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 79baa8e4, server de516ac0, cksum_type 1 Sep 1 07:28:07 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 54 previous similar messages Sep 1 07:28:07 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801b35efc00 x1566274076893792/t0(0) o3->oak-OST0010-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504276134 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 07:28:07 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 49 previous similar messages Sep 1 07:29:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST0010-osc-ffff88041b99c000: too many resent retries for object: 0:372522, rc = -11. Sep 1 07:29:02 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Sep 1 07:49:12 oak-gw06 kernel: LustreError: 133-1: oak-OST0001-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:554782 extent [37748736-38797311] Sep 1 07:49:12 oak-gw06 kernel: LustreError: Skipped 21 previous similar messages Sep 1 07:49:12 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) client b8f2fc54, server dd8a2526, cksum_type 1 Sep 1 07:49:12 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 21 previous similar messages Sep 1 07:49:12 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88007cba3900 x1566274077333952/t0(0) o3->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504277401 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 07:49:12 oak-gw06 kernel: LustreError: 1766:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 19 previous similar messages Sep 1 07:50:07 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:554782, rc = -11. Sep 1 07:50:07 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 07:50:28 oak-gw06 kernel: LustreError: 133-1: oak-OST0001-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:554782 extent [37748736-38797311] Sep 1 07:50:28 oak-gw06 kernel: LustreError: Skipped 16 previous similar messages Sep 1 07:50:28 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) client b8f2fc54, server dd8a2526, cksum_type 1 Sep 1 07:50:28 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 16 previous similar messages Sep 1 07:50:28 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88007cba0c00 x1566274077347696/t0(0) o3->oak-OST0001-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504277477 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 07:50:28 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 15 previous similar messages Sep 1 07:51:57 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0001-osc-ffff88041b99c000: too many resent retries for object: 0:554782, rc = -11. Sep 1 07:51:57 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 08:08:11 oak-gw06 kernel: LustreError: 133-1: oak-OST0017-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322872 extent [50331648-51380223] Sep 1 08:08:11 oak-gw06 kernel: LustreError: Skipped 15 previous similar messages Sep 1 08:08:11 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client f63029b5, server af54972a, cksum_type 1 Sep 1 08:08:11 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 15 previous similar messages Sep 1 08:08:11 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802bee6ea00 x1566274077711296/t0(0) o3->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504278538 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 08:08:11 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 13 previous similar messages Sep 1 08:08:32 oak-gw06 kernel: LustreError: 133-1: oak-OST0017-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322872 extent [50331648-51380223] Sep 1 08:08:32 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 08:08:32 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client f63029b5, server af54972a, cksum_type 1 Sep 1 08:08:32 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Sep 1 08:08:32 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88015f082a00 x1566274077715824/t0(0) o3->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504278559 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 08:08:32 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 5 previous similar messages Sep 1 08:09:06 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0017-osc-ffff88041b99c000: too many resent retries for object: 0:322872, rc = -11. Sep 1 08:09:12 oak-gw06 kernel: LustreError: 133-1: oak-OST0017-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322872 extent [50331648-51380223] Sep 1 08:09:12 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Sep 1 08:09:12 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client f63029b5, server af54972a, cksum_type 1 Sep 1 08:09:12 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Sep 1 08:09:12 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801b35ef600 x1566274077728992/t0(0) o3->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504278599 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 08:09:12 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 6 previous similar messages Sep 1 08:10:01 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0017-osc-ffff88041b99c000: too many resent retries for object: 0:322872, rc = -11. Sep 1 08:10:29 oak-gw06 kernel: LustreError: 133-1: oak-OST0017-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:322872 extent [50331648-51380223] Sep 1 08:10:29 oak-gw06 kernel: LustreError: Skipped 14 previous similar messages Sep 1 08:10:29 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client f63029b5, server af54972a, cksum_type 1 Sep 1 08:10:29 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 14 previous similar messages Sep 1 08:10:29 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802bee6c300 x1566274077743968/t0(0) o3->oak-OST0017-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504278676 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 08:10:29 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 13 previous similar messages Sep 1 08:10:56 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST0017-osc-ffff88041b99c000: too many resent retries for object: 0:322872, rc = -11. Sep 1 08:20:41 oak-gw06 kernel: bash (11172): drop_caches: 1 Sep 1 08:21:19 oak-gw06 kernel: Lustre: DEBUG MARKER: Fri Sep 1 08:21:19 2017 Sep 1 09:01:02 oak-gw06 kernel: LustreError: 133-1: oak-OST0002-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:554917 extent [47185920-48234495] Sep 1 09:01:02 oak-gw06 kernel: LustreError: Skipped 3 previous similar messages Sep 1 09:01:02 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client 11a1a5b9, server dfab14ae, cksum_type 1 Sep 1 09:01:02 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 3 previous similar messages Sep 1 09:01:02 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802b3e66400 x1566274080373632/t0(0) o3->oak-OST0002-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504281713 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 09:01:02 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 2 previous similar messages Sep 1 09:01:23 oak-gw06 kernel: LustreError: 133-1: oak-OST0002-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:554917 extent [47185920-48234495] Sep 1 09:01:23 oak-gw06 kernel: LustreError: Skipped 5 previous similar messages Sep 1 09:01:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client 11a1a5b9, server dfab14ae, cksum_type 1 Sep 1 09:01:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 5 previous similar messages Sep 1 09:01:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88007dadd500 x1566274080374432/t0(0) o3->oak-OST0002-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504281734 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 09:01:23 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 5 previous similar messages Sep 1 09:01:57 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST0002-osc-ffff88041b99c000: too many resent retries for object: 0:554917, rc = -11. Sep 1 09:02:03 oak-gw06 kernel: LustreError: 133-1: oak-OST0002-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:554917 extent [47185920-48234495] Sep 1 09:02:03 oak-gw06 kernel: LustreError: Skipped 7 previous similar messages Sep 1 09:02:03 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) client 11a1a5b9, server dfab14ae, cksum_type 1 Sep 1 09:02:03 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 7 previous similar messages Sep 1 09:02:03 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88007dadcc00 x1566274080401200/t0(0) o3->oak-OST0002-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504281774 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 09:02:03 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 6 previous similar messages Sep 1 09:02:52 oak-gw06 kernel: LustreError: 1768:0:(osc_request.c:1643:brw_interpret()) oak-OST0002-osc-ffff88041b99c000: too many resent retries for object: 0:554917, rc = -11. Sep 1 09:03:20 oak-gw06 kernel: LustreError: 133-1: oak-OST0002-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:554917 extent [47185920-48234495] Sep 1 09:03:20 oak-gw06 kernel: LustreError: Skipped 14 previous similar messages Sep 1 09:03:20 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) client 11a1a5b9, server dfab14ae, cksum_type 1 Sep 1 09:03:20 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 14 previous similar messages Sep 1 09:03:20 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88005bcef600 x1566274080464752/t0(0) o3->oak-OST0002-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504281851 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 09:03:20 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 13 previous similar messages Sep 1 09:03:47 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1643:brw_interpret()) oak-OST0002-osc-ffff88041b99c000: too many resent retries for object: 0:554917, rc = -11. Sep 1 09:04:42 oak-gw06 kernel: LustreError: 1769:0:(osc_request.c:1643:brw_interpret()) oak-OST0002-osc-ffff88041b99c000: too many resent retries for object: 0:554917, rc = -11. Sep 1 09:06:10 oak-gw06 kernel: LustreError: 133-1: oak-OST001c-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:241454 extent [253755392-254803967] Sep 1 09:06:10 oak-gw06 kernel: LustreError: Skipped 25 previous similar messages Sep 1 09:06:10 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) client 44aec564, server 5d8a82d2, cksum_type 1 Sep 1 09:06:10 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 25 previous similar messages Sep 1 09:06:10 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88010fe69e00 x1566274080558272/t0(0) o3->oak-OST001c-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504282021 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 09:06:10 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 22 previous similar messages Sep 1 09:07:05 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001c-osc-ffff88041b99c000: too many resent retries for object: 0:241454, rc = -11. Sep 1 09:07:05 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 09:09:51 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) oak-OST001c-osc-ffff88041b99c000: too many resent retries for object: 0:241454, rc = -11. Sep 1 09:09:51 oak-gw06 kernel: LustreError: 1767:0:(osc_request.c:1643:brw_interpret()) Skipped 2 previous similar messages Sep 1 09:16:50 oak-gw06 kernel: LustreError: 133-1: oak-OST001a-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:241499 extent [245366784-246415359] Sep 1 09:16:50 oak-gw06 kernel: LustreError: Skipped 54 previous similar messages Sep 1 09:16:50 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) client 649604cc, server 52051923, cksum_type 1 Sep 1 09:16:50 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 54 previous similar messages Sep 1 09:16:50 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803b5e58300 x1566274081271888/t0(0) o3->oak-OST001a-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504282659 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 09:16:50 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 49 previous similar messages Sep 1 09:17:45 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) oak-OST001a-osc-ffff88041b99c000: too many resent retries for object: 0:241499, rc = -11. Sep 1 09:17:45 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1643:brw_interpret()) Skipped 1 previous similar message Sep 1 09:26:51 oak-gw06 kernel: LustreError: 133-1: oak-OST000e-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.101@o2ib5 inode [0x0:0x0:0x0] object 0x0:372990 extent [193986560-195035135] Sep 1 09:26:51 oak-gw06 kernel: LustreError: Skipped 81 previous similar messages Sep 1 09:26:51 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) client b04e92f3, server 9ec63b25, cksum_type 1 Sep 1 09:26:51 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 81 previous similar messages Sep 1 09:26:51 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880314654c00 x1566274081637888/t0(0) o3->oak-OST000e-osc-ffff88041b99c000@10.0.2.101@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504283258 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 09:26:51 oak-gw06 kernel: LustreError: 1763:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 74 previous similar messages Sep 1 09:34:09 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) oak-OST000f-osc-ffff88041b99c000: too many resent retries for object: 0:373084, rc = -11. Sep 1 09:34:09 oak-gw06 kernel: LustreError: 1765:0:(osc_request.c:1643:brw_interpret()) Skipped 7 previous similar messages Sep 1 09:41:42 oak-gw06 kernel: LustreError: 133-1: oak-OST000f-osc-ffff88041b99c000: BAD READ CHECKSUM: from 10.0.2.102@o2ib5 inode [0x0:0x0:0x0] object 0x0:373078 extent [338690048-339738623] Sep 1 09:41:42 oak-gw06 kernel: LustreError: Skipped 38 previous similar messages Sep 1 09:41:42 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) client b8369de1, server 38679a6e, cksum_type 1 Sep 1 09:41:42 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1483:osc_brw_fini_request()) Skipped 38 previous similar messages Sep 1 09:41:42 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8803800ef000 x1566274082074704/t0(0) o3->oak-OST000f-osc-ffff88041b99c000@10.0.2.102@o2ib5:6/4 lens 608/400 e 0 to 0 dl 1504284149 ref 2 fl Interpret:RM/0/0 rc 1048576/1048576 Sep 1 09:41:42 oak-gw06 kernel: LustreError: 1762:0:(osc_request.c:1519:osc_brw_redo_request()) Skipped 34 previous similar messages Sep 1 09:44:27 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) oak-OST000f-osc-ffff88041b99c000: too many resent retries for object: 0:373078, rc = -11. Sep 1 09:44:27 oak-gw06 kernel: LustreError: 1764:0:(osc_request.c:1643:brw_interpret()) Skipped 4 previous similar messages Sep 1 10:10:33 oak-gw06 kernel: Initializing cgroup subsys cpuset