./dc2mds01/logs/kern:Mar 3 11:02:00 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=189.63.69.143 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=8767 DF PROTO=TCP SPT=41914 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:03:10 dc2mds01 kernel: : LustreError: 28998:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230230818095, new: 230230818094 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 11:03:10 dc2mds01 kernel: : LustreError: 28998:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 6 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:03:30 dc2mds01 kernel: : Lustre: 12379:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202c2b71c:0x76bd:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 11:03:30 dc2mds01 kernel: : Lustre: 12379:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 140 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:03:40 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=183.87.51.183 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=689 DF PROTO=TCP SPT=41236 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:04:34 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=218.77.79.34 DST=149.165.235.171 LEN=40 TOS=0x00 PREC=0x00 TTL=241 ID=54321 PROTO=TCP SPT=52641 DPT=23 WINDOW=65535 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:04:35 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=108.30.25.211 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=47 ID=59435 DF PROTO=TCP SPT=34187 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:09:14 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 11:09:14 dc2mds01 kernel: : Lustre: Skipped 85 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:09:49 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 11:09:49 dc2mds01 kernel: : Lustre: Skipped 85 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:11:37 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=125.62.210.5 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=43 ID=13221 DF PROTO=TCP SPT=3715 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:13:24 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=117.218.25.214 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=54726 DF PROTO=TCP SPT=54438 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:16:17 dc2mds01 kernel: : Lustre: 12496:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202d821c4:0x1d3c:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 11:16:17 dc2mds01 kernel: : Lustre: 12496:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 49 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:17:30 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=202.191.227.202 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=58748 DF PROTO=TCP SPT=43451 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:19:03 dc2mds01 kernel: : Lustre: dc2-MDT0000: nosquash_nids is set to 149.165.235.42@tcp0 149.165.235.30@tcp0 149.165.235.51@tcp0 149.165.235.18@tcp0 10.10.0.181@o2ib 10.10.0.156@o2ib 10.10.0.157@o2ib 10.10.0.158@o2ib ./dc2mds01/logs/kern:Mar 3 11:19:03 dc2mds01 kernel: : ./dc2mds01/logs/kern:Mar 3 11:19:16 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 11:19:16 dc2mds01 kernel: : Lustre: Skipped 85 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:19:51 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 11:19:51 dc2mds01 kernel: : Lustre: Skipped 85 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:22:07 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=175.141.133.41 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=46043 DF PROTO=TCP SPT=40775 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:24:46 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=95.231.122.206 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=42 ID=30348 DF PROTO=TCP SPT=4494 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:25:07 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=187.20.187.1 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=57750 DF PROTO=TCP SPT=56141 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:26:23 dc2mds01 kernel: : Lustre: 12496:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202e675ea:0xbe9a:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 11:26:23 dc2mds01 kernel: : Lustre: 12496:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 11 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:27:41 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=186.204.157.158 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=650 DF PROTO=TCP SPT=34559 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:27:41 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=189.18.105.51 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=46 ID=459 DF PROTO=TCP SPT=44903 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:27:58 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=187.104.156.40 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=41709 DF PROTO=TCP SPT=51410 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:29:04 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=93.189.150.203 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=42 ID=41593 DF PROTO=TCP SPT=2469 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:29:10 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=179.210.196.14 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=30779 DF PROTO=TCP SPT=42537 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:29:18 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 11:29:18 dc2mds01 kernel: : Lustre: Skipped 85 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:29:53 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 11:29:53 dc2mds01 kernel: : Lustre: Skipped 85 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:31:39 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=208.126.228.235 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=5188 DF PROTO=TCP SPT=4197 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:31:40 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=78.187.88.155 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=42 ID=9465 DF PROTO=TCP SPT=49518 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:31:42 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=78.189.26.193 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=41 ID=22697 DF PROTO=TCP SPT=50496 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:31:42 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=88.247.25.171 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=41 ID=27699 DF PROTO=TCP SPT=60607 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:31:42 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=27.219.121.212 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=12453 DF PROTO=TCP SPT=33758 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:31:54 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=60.25.114.46 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=10651 DF PROTO=TCP SPT=58282 DPT=23 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:31:57 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=60.25.114.46 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=10652 DF PROTO=TCP SPT=58282 DPT=23 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:32:03 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=60.25.114.46 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=10653 DF PROTO=TCP SPT=58282 DPT=23 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:32:24 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=177.16.241.196 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=1197 DF PROTO=TCP SPT=37306 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:32:25 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=109.124.101.102 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=9894 DF PROTO=TCP SPT=56293 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:32:51 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=190.141.228.41 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=497 DF PROTO=TCP SPT=3239 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:33:22 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=201.193.208.142 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=46 ID=26546 DF PROTO=TCP SPT=4940 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:33:25 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=94.72.224.84 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=43 ID=17185 DF PROTO=TCP SPT=47749 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:39:20 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 11:39:20 dc2mds01 kernel: : Lustre: Skipped 85 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:39:55 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 11:39:55 dc2mds01 kernel: : Lustre: Skipped 85 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:43:28 dc2mds01 kernel: : Lustre: 12356:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202e675ea:0xbec1:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 11:43:28 dc2mds01 kernel: : Lustre: 12356:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 12 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:48:52 dc2mds01 kernel: : LustreError: 11054:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230234849252, new: 230234849251 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 11:48:52 dc2mds01 kernel: : LustreError: 11054:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 2 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:49:22 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 11:49:22 dc2mds01 kernel: : Lustre: Skipped 85 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:49:57 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 11:49:57 dc2mds01 kernel: : Lustre: Skipped 85 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:50:12 dc2mds01 kernel: : LustreError: 28998:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230235032118, new: 230235032117 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 11:50:12 dc2mds01 kernel: : LustreError: 28998:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 2 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:53:06 dc2mds01 kernel: : LustreError: 11054:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230235532436, new: 230235532435 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 11:53:06 dc2mds01 kernel: : LustreError: 11054:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 4 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:53:39 dc2mds01 kernel: : Lustre: 13836:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from 1c8096ec-a81e-a36e-7686-c5ecb14fb387@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2mds01/logs/kern:Mar 3 11:53:39 dc2mds01 kernel: : Lustre: 12369:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2mds01/logs/kern:Mar 3 11:53:56 dc2mds01 kernel: : Lustre: 20642:0:(debug.c:326:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. ./dc2mds01/logs/kern:Mar 3 11:54:15 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=151.250.218.55 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=37 ID=14071 DF PROTO=TCP SPT=49992 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 11:55:29 dc2mds01 kernel: : Lustre: 20668:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202e675ea:0xbeed:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 11:55:29 dc2mds01 kernel: : Lustre: 20668:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 5 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : Lustre: Service thread pid 12325 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : Pid: 12325, comm: mdt_368 ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : Call Trace: ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] schedule_timeout+0x192/0x2e0 ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] ? process_timeout+0x0/0x10 ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] ? ldlm_expired_completion_wait+0x0/0x260 [ptlrpc] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] cfs_waitq_timedwait+0x11/0x20 [libcfs] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] ldlm_completion_ast+0x48d/0x720 [ptlrpc] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] ? default_wake_function+0x0/0x20 ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] ldlm_cli_enqueue_local+0x1e6/0x560 [ptlrpc] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] ? ldlm_completion_ast+0x0/0x720 [ptlrpc] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] ? mdt_blocking_ast+0x0/0x2a0 [mdt] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] mdt_object_lock+0x320/0xb70 [mdt] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] ? mdt_blocking_ast+0x0/0x2a0 [mdt] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] ? ldlm_completion_ast+0x0/0x720 [ptlrpc] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] mdt_getattr_name_lock+0xe22/0x1880 [mdt] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] ? lustre_msg_buf+0x5d/0x60 [ptlrpc] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] ? __req_capsule_get+0x176/0x750 [ptlrpc] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] mdt_intent_getattr+0x2cd/0x4a0 [mdt] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] mdt_intent_policy+0x379/0x690 [mdt] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] ldlm_lock_enqueue+0x361/0x8f0 [ptlrpc] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] ldlm_handle_enqueue0+0x48d/0xf50 [ptlrpc] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] mdt_enqueue+0x46/0x130 [mdt] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] mdt_handle_common+0x932/0x1750 [mdt] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] mdt_regular_handle+0x15/0x20 [mdt] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] ptlrpc_main+0xc4e/0x1a40 [ptlrpc] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] ? ptlrpc_main+0x0/0x1a40 [ptlrpc] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] child_rip+0xa/0x20 ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] ? ptlrpc_main+0x0/0x1a40 [ptlrpc] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] ? ptlrpc_main+0x0/0x1a40 [ptlrpc] ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : [] ? child_rip+0x0/0x20 ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : LustreError: dumping log to /tmp/lustre-log.1393865836.12325 ./dc2mds01/logs/kern:Mar 3 11:57:16 dc2mds01 kernel: : LustreError: 12325:0:(ldlm_request.c:91:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1393865636, 200s ago); not entering recovery in server code, just going back to sleep ns: mdt-ffff881843d1a000 lock: ffff8824c4e8c000/0xcf86cae155f50d3e lrc: 3/1,0 mode: --/PR res: 8628332593/26766 bits 0x3 rrc: 21 type: IBT flags: 0x4004000 remote: 0x0 expref: -99 pid: 12325 timeout: 0 ./dc2mds01/logs/kern:Mar 3 11:58:20 dc2mds01 kernel: : LustreError: 11054:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230236485326, new: 230236485325 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 11:58:20 dc2mds01 kernel: : LustreError: 11054:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 15 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:59:24 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 11:59:24 dc2mds01 kernel: : Lustre: Skipped 85 previous similar messages ./dc2mds01/logs/kern:Mar 3 11:59:59 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 11:59:59 dc2mds01 kernel: : Lustre: Skipped 85 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:00:35 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=94.155.101.23 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=44 ID=53765 DF PROTO=TCP SPT=45497 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 12:03:05 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=60.189.169.131 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=20453 DF PROTO=TCP SPT=14224 DPT=23 WINDOW=16384 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 12:03:12 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=60.189.169.131 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=20963 DF PROTO=TCP SPT=14224 DPT=23 WINDOW=16384 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 12:06:11 dc2mds01 kernel: : Lustre: 12479:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202e675ea:0xbf3f:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 12:06:11 dc2mds01 kernel: : Lustre: 12479:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 20 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:06:55 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=5.39.223.68 DST=149.165.235.171 LEN=52 TOS=0x10 PREC=0x00 TTL=54 ID=9496 DF PROTO=TCP SPT=4935 DPT=3389 WINDOW=65535 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 12:07:23 dc2mds01 kernel: : Lustre: 12390:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-207), not sending early reply ./dc2mds01/logs/kern:Mar 3 12:07:23 dc2mds01 kernel: : req@ffff8808e5525c00 x1461574032163495/t0(0) o101->a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp:0/0 lens 568/4808 e 5 to 0 dl 1393866448 ref 2 fl Interpret:/0/0 rc 0/0 ./dc2mds01/logs/kern:Mar 3 12:08:32 dc2mds01 kernel: : LustreError: 28998:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230238269399, new: 230238269398 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 12:08:32 dc2mds01 kernel: : LustreError: 28998:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 33 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:09:25 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client a69ef170-baaf-a6ea-059d-2e445759c567 (at 149.165.235.42@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 12:09:25 dc2mds01 kernel: : Lustre: Skipped 86 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:10:00 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client a69ef170-baaf-a6ea-059d-2e445759c567 (at 149.165.235.42@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 12:10:00 dc2mds01 kernel: : Lustre: Skipped 91 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:16:49 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=211.143.106.187 DST=149.165.235.171 LEN=40 TOS=0x00 PREC=0x00 TTL=102 ID=256 PROTO=TCP SPT=6000 DPT=3306 WINDOW=16384 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 12:16:50 dc2mds01 kernel: : Lustre: 12496:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202aa2798:0x1e630:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 12:16:50 dc2mds01 kernel: : Lustre: 12496:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 13 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:19:24 dc2mds01 kernel: : LustreError: 20941:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230240189155, new: 230240189154 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 12:19:24 dc2mds01 kernel: : LustreError: 20941:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 22 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:19:28 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client a69ef170-baaf-a6ea-059d-2e445759c567 (at 149.165.235.42@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 12:19:28 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:20:03 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client a69ef170-baaf-a6ea-059d-2e445759c567 (at 149.165.235.42@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 12:20:03 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:22:40 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=117.27.254.28 DST=149.165.235.171 LEN=40 TOS=0x00 PREC=0x00 TTL=242 ID=54321 PROTO=TCP SPT=49531 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 12:27:09 dc2mds01 kernel: : Lustre: 12351:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202b55740:0xaa:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 12:27:09 dc2mds01 kernel: : Lustre: 12351:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 85 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:29:30 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client a69ef170-baaf-a6ea-059d-2e445759c567 (at 149.165.235.42@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 12:29:30 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:30:05 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client a69ef170-baaf-a6ea-059d-2e445759c567 (at 149.165.235.42@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 12:30:05 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:30:06 dc2mds01 kernel: : LustreError: 28998:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230241848927, new: 230241848926 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 12:30:06 dc2mds01 kernel: : LustreError: 28998:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 22 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:32:26 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=183.60.110.225 DST=149.165.235.171 LEN=40 TOS=0x00 PREC=0x00 TTL=104 ID=256 PROTO=TCP SPT=6000 DPT=3128 WINDOW=16384 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 12:36:10 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=121.63.200.34 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=27547 DF PROTO=TCP SPT=58065 DPT=5000 WINDOW=5440 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 12:36:13 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=121.63.200.34 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=27548 DF PROTO=TCP SPT=58065 DPT=5000 WINDOW=5440 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 12:36:19 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=121.63.200.34 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=27549 DF PROTO=TCP SPT=58065 DPT=5000 WINDOW=5440 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 12:37:10 dc2mds01 kernel: : Lustre: 12279:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202b55740:0xb0:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 12:37:10 dc2mds01 kernel: : Lustre: 12279:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 125 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:39:30 dc2mds01 kernel: : Lustre: 12274:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from f985b777-a8a6-f691-3072-8f9cb68d9fb0@149.165.229.20@tcp t0 exp ffff8822784a4c00 cur 1393868370 last 1393868369 ./dc2mds01/logs/kern:Mar 3 12:39:30 dc2mds01 kernel: : LustreError: 12274:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation ./dc2mds01/logs/kern:Mar 3 12:39:32 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client a69ef170-baaf-a6ea-059d-2e445759c567 (at 149.165.235.42@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 12:39:32 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:40:06 dc2mds01 kernel: : LustreError: 28998:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230242961801, new: 230242961800 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 12:40:06 dc2mds01 kernel: : LustreError: 28998:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 26 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:40:07 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client a69ef170-baaf-a6ea-059d-2e445759c567 (at 149.165.235.42@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 12:40:07 dc2mds01 kernel: : Lustre: Skipped 172 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:47:13 dc2mds01 kernel: : Lustre: 12402:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2025d6d02:0x4369:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 12:47:13 dc2mds01 kernel: : Lustre: 12402:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 163 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:49:34 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client a69ef170-baaf-a6ea-059d-2e445759c567 (at 149.165.235.42@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 12:49:34 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:50:09 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client a69ef170-baaf-a6ea-059d-2e445759c567 (at 149.165.235.42@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 12:50:09 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:50:49 dc2mds01 kernel: : LustreError: 28998:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230244127854, new: 230244127853 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 12:50:49 dc2mds01 kernel: : LustreError: 28998:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 26 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:52:33 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=81.218.141.230 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=17868 DF PROTO=TCP SPT=39078 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 12:54:26 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=81.213.140.149 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=41 ID=37056 DF PROTO=TCP SPT=47063 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 12:54:45 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=84.80.137.205 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=36039 DF PROTO=TCP SPT=41250 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 12:54:46 dc2mds01 kernel: : Lustre: 16497:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from 8d88eb3e-80ba-520e-a9ba-f73069c1bc96@149.165.229.22@tcp t0 exp ffff8829552a7800 cur 1393869286 last 1393869286 ./dc2mds01/logs/kern:Mar 3 12:54:46 dc2mds01 kernel: : LustreError: 16497:0:(obd_class.h:503:obd_set_info_async()) obd_set_info_async: dev 0 no operation ./dc2mds01/logs/kern:Mar 3 12:55:23 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=89.35.131.238 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=46 ID=32086 DF PROTO=TCP SPT=53829 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 12:57:42 dc2mds01 kernel: : Lustre: 12349:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2025c7375:0x29d9:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 12:57:42 dc2mds01 kernel: : Lustre: 12349:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 170 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:57:49 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=184.175.37.148 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=53 ID=99 DF PROTO=TCP SPT=35726 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 12:59:36 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client a69ef170-baaf-a6ea-059d-2e445759c567 (at 149.165.235.42@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 12:59:36 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 12:59:49 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=109.103.38.155 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=22713 DF PROTO=TCP SPT=38366 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 13:00:11 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client a69ef170-baaf-a6ea-059d-2e445759c567 (at 149.165.235.42@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 13:00:11 dc2mds01 kernel: : Lustre: Skipped 172 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:01:25 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=87.6.181.252 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=43 ID=14671 DF PROTO=TCP SPT=3547 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 13:01:48 dc2mds01 kernel: : LustreError: 11054:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230245538654, new: 230245538653 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 13:01:48 dc2mds01 kernel: : LustreError: 11054:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 26 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:04:56 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=213.172.8.46 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=64144 DF PROTO=TCP SPT=2777 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 13:05:55 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=178.36.177.57 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=39200 DF PROTO=TCP SPT=37947 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 13:08:41 dc2mds01 kernel: : Lustre: 12321:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2025ced8e:0x60dd:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 13:08:41 dc2mds01 kernel: : Lustre: 12321:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 134 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:09:38 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client a69ef170-baaf-a6ea-059d-2e445759c567 (at 149.165.235.42@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 13:09:38 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:10:13 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client a69ef170-baaf-a6ea-059d-2e445759c567 (at 149.165.235.42@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 13:10:13 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:12:11 dc2mds01 kernel: : LustreError: 11054:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230246905036, new: 230246905035 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 13:12:11 dc2mds01 kernel: : LustreError: 11054:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 29 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:15:42 dc2mds01 kernel: : Lustre: dc2-MDT0000: haven't heard from client 7097ef4b-b519-de72-4f21-b6c880acdc47 (at 149.165.230.179@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8823f2722800, cur 1393870542 expire 1393870392 last 1393870315 ./dc2mds01/logs/kern:Mar 3 13:15:42 dc2mds01 kernel: : Lustre: Skipped 1 previous similar message ./dc2mds01/logs/kern:Mar 3 13:17:53 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=31.148.219.9 DST=149.165.235.171 LEN=40 TOS=0x10 PREC=0x00 TTL=245 ID=54321 PROTO=TCP SPT=49006 DPT=21320 WINDOW=65535 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 13:18:50 dc2mds01 kernel: : Lustre: 12292:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202e4f8d8:0x3b46:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 13:18:50 dc2mds01 kernel: : Lustre: 12292:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 43 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:19:40 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client a69ef170-baaf-a6ea-059d-2e445759c567 (at 149.165.235.42@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 13:19:40 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:19:47 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=88.250.21.122 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=41 ID=58927 DF PROTO=TCP SPT=40244 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 13:20:15 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client a69ef170-baaf-a6ea-059d-2e445759c567 (at 149.165.235.42@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 13:20:15 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:20:24 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=2.229.99.45 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=22254 DF PROTO=TCP SPT=2211 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 13:22:42 dc2mds01 kernel: : LustreError: 6473:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230248729792, new: 230248729791 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 13:22:42 dc2mds01 kernel: : LustreError: 6473:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 16 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:25:31 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=175.143.213.122 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=34702 DF PROTO=TCP SPT=57404 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 13:29:36 dc2mds01 kernel: : Lustre: 20668:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202e675ea:0xc0df:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 13:29:36 dc2mds01 kernel: : Lustre: 20668:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 29 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:29:42 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 13:29:42 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:30:17 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 13:30:17 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:31:19 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=177.100.200.6 DST=149.165.235.171 LEN=52 TOS=0x08 PREC=0x40 TTL=46 ID=36075 DF PROTO=TCP SPT=59288 DPT=5000 WINDOW=14600 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 13:32:46 dc2mds01 kernel: : LustreError: 20941:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230250534884, new: 230250534883 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 13:32:46 dc2mds01 kernel: : LustreError: 20941:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 19 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:33:36 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=93.174.93.51 DST=149.165.235.171 LEN=40 TOS=0x08 PREC=0x20 TTL=240 ID=54321 PROTO=TCP SPT=46625 DPT=21679 WINDOW=65535 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 13:36:04 dc2mds01 kernel: : Lustre: 13837:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from 83bf69ae-57f3-9496-1853-8c284ff10198@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2mds01/logs/kern:Mar 3 13:36:04 dc2mds01 kernel: : Lustre: 13980:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2mds01/logs/kern:Mar 3 13:38:29 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=188.209.228.228 DST=149.165.235.171 LEN=52 TOS=0x00 PREC=0x00 TTL=49 ID=61626 DF PROTO=TCP SPT=44372 DPT=5000 WINDOW=14600 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 13:39:44 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 13:39:44 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:40:19 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 13:40:19 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:41:10 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=211.142.52.11 DST=149.165.235.171 LEN=40 TOS=0x00 PREC=0x00 TTL=103 ID=256 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 13:43:40 dc2mds01 kernel: : LustreError: 22680:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230252598101, new: 230252598100 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 13:43:40 dc2mds01 kernel: : LustreError: 22680:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 28 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:44:42 dc2mds01 kernel: : Lustre: 13956:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2025cdda1:0x4ca3:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 13:44:42 dc2mds01 kernel: : Lustre: 13956:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 8 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:49:46 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 13:49:46 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:50:21 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 13:50:21 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:54:27 dc2mds01 kernel: : LustreError: 28998:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230254608262, new: 230254608261 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 13:54:27 dc2mds01 kernel: : LustreError: 28998:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 17 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:54:50 dc2mds01 kernel: : Lustre: 12402:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202e4f8d8:0x3bc1:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 13:54:50 dc2mds01 kernel: : Lustre: 12402:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 19 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:57:30 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=80.34.64.177 DST=149.165.235.171 LEN=52 TOS=0x08 PREC=0x40 TTL=44 ID=39172 DF PROTO=TCP SPT=39098 DPT=5000 WINDOW=14600 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 13:58:44 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=93.146.225.101 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=58437 DF PROTO=TCP SPT=57432 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 13:59:48 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 13:59:48 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 13:59:49 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=111.118.248.106 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x20 TTL=42 ID=10528 DF PROTO=TCP SPT=3623 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:00:23 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 14:00:23 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:01:13 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=218.77.79.34 DST=149.165.235.171 LEN=40 TOS=0x00 PREC=0x00 TTL=241 ID=54321 PROTO=TCP SPT=44369 DPT=21 WINDOW=65535 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:03:12 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=83.25.32.57 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=44 ID=17776 DF PROTO=TCP SPT=44817 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:04:32 dc2mds01 kernel: : LustreError: 11054:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230256015954, new: 230256015953 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 14:04:32 dc2mds01 kernel: : LustreError: 11054:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 19 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:04:51 dc2mds01 kernel: : Lustre: 12292:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202c9ae8d:0x10184:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 14:04:51 dc2mds01 kernel: : Lustre: 12292:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 19 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:08:44 dc2mds01 kernel: : Lustre: 12459:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from d67cf4e1-6b8e-d7a0-a53b-a82a2a7b2cc0@149.165.229.5@tcp t227482008153 exp ffff8823a21d5c00 cur 1393873724 last 1393873724 ./dc2mds01/logs/kern:Mar 3 14:09:50 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 14:09:50 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:10:25 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 14:10:25 dc2mds01 kernel: : Lustre: Skipped 172 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:15:04 dc2mds01 kernel: : Lustre: 13980:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202e675ea:0xc1ad:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 14:15:04 dc2mds01 kernel: : Lustre: 13980:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 19 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:16:11 dc2mds01 kernel: : LustreError: 13085:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230257551778, new: 230257551777 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 14:16:11 dc2mds01 kernel: : LustreError: 13085:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 20 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:16:27 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=93.174.93.51 DST=149.165.235.171 LEN=40 TOS=0x08 PREC=0x20 TTL=240 ID=54321 PROTO=TCP SPT=43407 DPT=48499 WINDOW=65535 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:19:52 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 14:19:52 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:20:27 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 14:20:27 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:21:10 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=186.235.165.100 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x20 TTL=48 ID=21930 DF PROTO=TCP SPT=36265 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:26:16 dc2mds01 kernel: : LustreError: 20941:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230258807559, new: 230258807558 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 14:26:16 dc2mds01 kernel: : LustreError: 20941:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 19 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:27:02 dc2mds01 kernel: : Lustre: 12309:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202e675ea:0xc1c4:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 14:27:02 dc2mds01 kernel: : Lustre: 12309:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 24 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:27:29 dc2mds01 kernel: : UDP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=89.248.168.200 DST=149.165.235.171 LEN=64 TOS=0x08 PREC=0x20 TTL=49 ID=0 DF PROTO=UDP SPT=39940 DPT=53 LEN=44 ./dc2mds01/logs/kern:Mar 3 14:29:54 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 14:29:54 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:30:29 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 14:30:29 dc2mds01 kernel: : Lustre: Skipped 171 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:31:13 dc2mds01 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 9437s: evicting client at 149.165.235.42@tcp ns: mdt-ffff881843d1a000 lock: ffff882afe438000/0xcf86cae155f4b253 lrc: 3/0,0 mode: PR/PR res: 35127297/2218887223 bits 0x3 rrc: 382 type: IBT flags: 0x4000020 remote: 0x8a1cc956fcd9690 expref: 53 pid: 12292 timeout: 8892268113 ./dc2mds01/logs/kern:Mar 3 14:31:13 dc2mds01 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 592711s: evicting client at 149.165.229.118@tcp ns: mdt-ffff881843d1a000 lock: ffff882bd0d266c0/0xcf86cada4b752134 lrc: 3/0,0 mode: PR/PR res: 35127297/2218887223 bits 0x3 rrc: 384 type: IBT flags: 0x4000020 remote: 0x2b7b59a67bf7bbc7 expref: 60 pid: 4234 timeout: 8892268516 ./dc2mds01/logs/kern:Mar 3 14:31:18 dc2mds01 kernel: : Lustre: 12472:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t230235724090 exp (null) cur 1393875078 last 0 ./dc2mds01/logs/kern:Mar 3 14:31:28 dc2mds01 kernel: : LustreError: 4244:0:(mdt_handler.c:2817:mdt_recovery()) operation 400 on unconnected MDS from 12345-149.165.229.118@tcp ./dc2mds01/logs/kern:Mar 3 14:31:28 dc2mds01 kernel: : Lustre: 12350:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from 42562829-489a-9976-4c1e-24a62bef99ac@149.165.229.118@tcp t230245173891 exp (null) cur 1393875088 last 0 ./dc2mds01/logs/kern:Mar 3 14:31:30 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=187.20.235.175 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=14268 DF PROTO=TCP SPT=49820 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:31:48 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=177.189.210.209 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=46 ID=63336 DF PROTO=TCP SPT=43639 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:33:29 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=181.164.12.48 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=45343 DF PROTO=TCP SPT=34141 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:35:36 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=201.9.250.205 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=35941 DF PROTO=TCP SPT=33991 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:35:57 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=187.65.18.238 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=29903 DF PROTO=TCP SPT=59318 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:37:36 dc2mds01 kernel: : LustreError: 13085:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230260166689, new: 230260166688 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 14:37:36 dc2mds01 kernel: : LustreError: 13085:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 24 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:38:18 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=177.19.218.12 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=23446 DF PROTO=TCP SPT=54402 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:38:26 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=190.246.171.188 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=37534 DF PROTO=TCP SPT=59904 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:38:46 dc2mds01 kernel: : Lustre: 10023:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202c93dd7:0x1f12:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 14:38:46 dc2mds01 kernel: : Lustre: 10023:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 26 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:39:56 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 14:39:56 dc2mds01 kernel: : Lustre: Skipped 97 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:40:31 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 14:40:31 dc2mds01 kernel: : Lustre: Skipped 92 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:40:57 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=174.17.95.187 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=55 ID=59604 DF PROTO=TCP SPT=49071 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:42:04 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=223.4.212.106 DST=149.165.235.171 LEN=40 TOS=0x00 PREC=0x00 TTL=104 ID=256 PROTO=TCP SPT=6000 DPT=3306 WINDOW=16384 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:44:46 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=201.83.164.27 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=28853 DF PROTO=TCP SPT=59037 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:44:54 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=121.34.223.181 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=52515 DF PROTO=TCP SPT=58296 DPT=23 WINDOW=5440 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:44:57 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=121.34.223.181 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=52516 DF PROTO=TCP SPT=58296 DPT=23 WINDOW=5440 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:45:03 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=121.34.223.181 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=52517 DF PROTO=TCP SPT=58296 DPT=23 WINDOW=5440 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:46:17 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=181.95.138.40 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=53 ID=13663 DF PROTO=TCP SPT=36065 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:47:40 dc2mds01 kernel: : LustreError: 4298:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230261391154, new: 230261391153 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 14:47:40 dc2mds01 kernel: : LustreError: 4298:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 34 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:48:50 dc2mds01 kernel: : Lustre: 22680:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202b8ff17:0x10438:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 14:48:50 dc2mds01 kernel: : Lustre: 22680:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 17 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:49:58 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) refused reconnection, still busy with 1 active RPCs ./dc2mds01/logs/kern:Mar 3 14:49:58 dc2mds01 kernel: : Lustre: Skipped 85 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:50:32 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=115.135.192.57 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=62422 DF PROTO=TCP SPT=3172 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:50:33 dc2mds01 kernel: : Lustre: dc2-MDT0000: Client ec52d4cd-f4f9-6b4c-ef6d-731b2cbdf25e (at 149.165.229.92@tcp) reconnecting ./dc2mds01/logs/kern:Mar 3 14:50:33 dc2mds01 kernel: : Lustre: Skipped 85 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:51:12 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=179.25.178.217 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=18916 DF PROTO=TCP SPT=3409 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:53:15 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=84.95.88.120 DST=149.165.235.171 LEN=60 TOS=0x1C PREC=0x00 TTL=52 ID=25305 DF PROTO=TCP SPT=32944 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:53:30 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=5.66.70.185 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=41509 DF PROTO=TCP SPT=51981 DPT=5000 WINDOW=14600 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:53:46 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=201.190.0.236 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=47 ID=37157 DF PROTO=TCP SPT=49500 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:54:05 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=60.189.156.115 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=49032 DF PROTO=TCP SPT=29400 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:54:21 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=149.156.163.225 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=55392 DF PROTO=TCP SPT=40744 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:54:46 dc2mds01 kernel: : ib_send_cm_mra: cm_id_priv->id.state: 0x6 ./dc2mds01/logs/kern:Mar 3 14:54:46 dc2mds01 kernel: : Lustre: 16496:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from abc1f77b-367f-23a7-1940-0e4a8d4dfe2e@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2mds01/logs/kern:Mar 3 14:54:46 dc2mds01 kernel: : Lustre: 10225:0:(ldlm_lib.c:952:target_handle_connect()) dc2-MDT0000: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2mds01/logs/kern:Mar 3 14:55:44 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=187.126.125.103 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=32933 DF PROTO=TCP SPT=53390 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:56:37 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=201.227.89.66 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=3221 DF PROTO=TCP SPT=3282 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:57:01 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=117.24.1.116 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=33516 DF PROTO=TCP SPT=46930 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:57:03 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=201.143.177.64 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=12506 DF PROTO=TCP SPT=36886 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:57:16 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=123.142.77.66 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=17013 DF PROTO=TCP SPT=54272 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:57:28 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=88.102.151.202 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=2471 DF PROTO=TCP SPT=47996 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:57:29 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=78.173.13.254 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=41 ID=42619 DF PROTO=TCP SPT=36442 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:57:29 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=88.101.77.161 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=44250 DF PROTO=TCP SPT=58581 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:57:29 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=90.179.24.32 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=3390 DF PROTO=TCP SPT=59007 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:57:32 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=182.74.78.170 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=4807 DF PROTO=TCP SPT=47906 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:57:33 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=121.96.83.91 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=51990 DF PROTO=TCP SPT=1307 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:57:35 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=82.154.62.224 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=55161 DF PROTO=TCP SPT=3098 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:57:46 dc2mds01 kernel: : LustreError: 11054:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 230262510154, new: 230262510153 replay: 0. see LU-617. ./dc2mds01/logs/kern:Mar 3 14:57:46 dc2mds01 kernel: : LustreError: 11054:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Skipped 42 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:58:17 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=190.147.105.167 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=18723 DF PROTO=TCP SPT=4076 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:58:22 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=82.59.85.207 DST=149.165.235.171 LEN=60 TOS=0x08 PREC=0x40 TTL=42 ID=40807 DF PROTO=TCP SPT=2678 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:58:36 dc2mds01 kernel: : UDP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=98.143.175.92 DST=149.165.235.171 LEN=445 TOS=0x00 PREC=0x00 TTL=56 ID=0 DF PROTO=UDP SPT=5064 DPT=5061 LEN=425 ./dc2mds01/logs/kern:Mar 3 14:58:54 dc2mds01 kernel: : Lustre: 5788:0:(mdt_xattr.c:379:mdt_reint_setxattr()) dc2-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x202c9ae8d:0x112bc:0x0]] [valid 68719476736] ./dc2mds01/logs/kern:Mar 3 14:58:54 dc2mds01 kernel: : Lustre: 5788:0:(mdt_xattr.c:379:mdt_reint_setxattr()) Skipped 26 previous similar messages ./dc2mds01/logs/kern:Mar 3 14:59:03 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=85.245.233.179 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=54654 DF PROTO=TCP SPT=3851 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2mds01/logs/kern:Mar 3 14:59:11 dc2mds01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=90:b1:1c:18:c3:d1:00:24:38:a8:dd:00:08:00 SRC=177.141.36.226 DST=149.165.235.171 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=63064 DF PROTO=TCP SPT=61331 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:02:01 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=189.63.69.143 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=13039 DF PROTO=TCP SPT=43498 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:03:42 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=183.87.51.183 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=46454 DF PROTO=TCP SPT=38685 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:04:37 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=108.30.25.211 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x40 TTL=47 ID=52840 DF PROTO=TCP SPT=44472 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:17:32 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=202.191.227.202 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=27304 DF PROTO=TCP SPT=34844 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:22:09 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=175.141.133.41 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=23115 DF PROTO=TCP SPT=58395 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:24:49 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=95.231.122.206 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x40 TTL=42 ID=28481 DF PROTO=TCP SPT=2850 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:25:09 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=187.20.187.1 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=55794 DF PROTO=TCP SPT=47500 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:27:42 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=186.204.157.158 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=51049 DF PROTO=TCP SPT=36754 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:27:42 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=189.18.105.51 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x40 TTL=46 ID=47826 DF PROTO=TCP SPT=47450 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:28:00 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=187.104.156.40 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=34228 DF PROTO=TCP SPT=59644 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:29:06 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=93.189.150.203 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=62157 DF PROTO=TCP SPT=2697 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:29:11 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=179.210.196.14 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=26110 DF PROTO=TCP SPT=54801 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:31:42 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=78.187.88.155 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x40 TTL=42 ID=63661 DF PROTO=TCP SPT=54951 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:31:42 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=208.126.228.235 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=44360 DF PROTO=TCP SPT=4822 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:31:43 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=78.189.26.193 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x40 TTL=41 ID=54571 DF PROTO=TCP SPT=42800 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:31:43 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=88.247.25.171 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x40 TTL=41 ID=17360 DF PROTO=TCP SPT=60343 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:31:43 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=27.219.121.212 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=28089 DF PROTO=TCP SPT=57434 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:31:44 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=60.25.114.46 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=23354 DF PROTO=TCP SPT=53958 DPT=23 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:31:53 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=60.25.114.46 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=23356 DF PROTO=TCP SPT=53958 DPT=23 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:32:25 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=177.16.241.196 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=5403 DF PROTO=TCP SPT=50941 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:32:27 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=109.124.101.102 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=36762 DF PROTO=TCP SPT=35445 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:32:53 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=190.141.228.41 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=32855 DF PROTO=TCP SPT=4900 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:33:24 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=201.193.208.142 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x40 TTL=45 ID=11106 DF PROTO=TCP SPT=2544 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:33:27 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=94.72.224.84 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x40 TTL=43 ID=24747 DF PROTO=TCP SPT=45237 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:47:33 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=93.174.93.51 DST=149.165.235.1 LEN=40 TOS=0x08 PREC=0x20 TTL=240 ID=54321 PROTO=TCP SPT=39891 DPT=21679 WINDOW=65535 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 11:53:39 dc2oss01 kernel: : Lustre: 20387:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0003: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss01/logs/kern:Mar 3 11:53:39 dc2oss01 kernel: : Lustre: 20387:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 10 previous similar messages ./dc2oss01/logs/kern:Mar 3 11:53:39 dc2oss01 kernel: : Lustre: 20495:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0009: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss01/logs/kern:Mar 3 11:53:39 dc2oss01 kernel: : Lustre: 20495:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 5 previous similar messages ./dc2oss01/logs/kern:Mar 3 11:54:16 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=151.250.218.55 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x40 TTL=37 ID=53428 DF PROTO=TCP SPT=40823 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 12:00:37 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=94.155.101.23 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x40 TTL=44 ID=427 DF PROTO=TCP SPT=60247 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 12:38:31 dc2oss01 kernel: : Lustre: dc2-OST0000: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss01/logs/kern:Mar 3 12:38:31 dc2oss01 kernel: : Lustre: 20471:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0001: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8817e362b800 cur 1393868311 last 1393868311 ./dc2oss01/logs/kern:Mar 3 12:38:31 dc2oss01 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss01/logs/kern:Mar 3 12:38:31 dc2oss01 kernel: : Lustre: 20471:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 3 previous similar messages ./dc2oss01/logs/kern:Mar 3 12:38:31 dc2oss01 kernel: : Lustre: dc2-OST0002: Client 1951e547-ef58-f071-b440-91732bd33942 (at 149.165.229.191@tcp) reconnecting ./dc2oss01/logs/kern:Mar 3 12:38:31 dc2oss01 kernel: : Lustre: Skipped 6 previous similar messages ./dc2oss01/logs/kern:Mar 3 12:38:31 dc2oss01 kernel: : Lustre: 21085:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0002: connection from 1951e547-ef58-f071-b440-91732bd33942@149.165.229.191@tcp t0 exp ffff8817f2286800 cur 1393868311 last 1393868311 ./dc2oss01/logs/kern:Mar 3 12:38:31 dc2oss01 kernel: : Lustre: 21085:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 6 previous similar messages ./dc2oss01/logs/kern:Mar 3 12:38:32 dc2oss01 kernel: : Lustre: dc2-OST0003: Client 32ba5f84-0b55-ee15-0c5f-d3d8fd2d40a9 (at 149.165.229.23@tcp) reconnecting ./dc2oss01/logs/kern:Mar 3 12:38:32 dc2oss01 kernel: : Lustre: Skipped 4 previous similar messages ./dc2oss01/logs/kern:Mar 3 12:38:32 dc2oss01 kernel: : Lustre: 19935:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0003: connection from 32ba5f84-0b55-ee15-0c5f-d3d8fd2d40a9@149.165.229.23@tcp t0 exp ffff8817d3b77400 cur 1393868312 last 1393868312 ./dc2oss01/logs/kern:Mar 3 12:38:32 dc2oss01 kernel: : Lustre: 19935:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 4 previous similar messages ./dc2oss01/logs/kern:Mar 3 12:38:35 dc2oss01 kernel: : Lustre: dc2-OST0004: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss01/logs/kern:Mar 3 12:38:35 dc2oss01 kernel: : Lustre: Skipped 2 previous similar messages ./dc2oss01/logs/kern:Mar 3 12:38:35 dc2oss01 kernel: : Lustre: 19830:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0004: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8817d22bb800 cur 1393868315 last 1393868315 ./dc2oss01/logs/kern:Mar 3 12:38:35 dc2oss01 kernel: : Lustre: 19830:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss01/logs/kern:Mar 3 12:38:36 dc2oss01 kernel: : Lustre: dc2-OST0006: Client c238dd2d-69ca-3594-9f8a-c529fc6e9b2b (at 149.165.231.13@tcp) refused reconnection, still busy with 3 active RPCs ./dc2oss01/logs/kern:Mar 3 12:38:36 dc2oss01 kernel: : LustreError: 20055:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880be90ba800 x1459112897852374/t0(0) o3->c238dd2d-69ca-3594-9f8a-c529fc6e9b2b@149.165.231.13@tcp:0/0 lens 448/400 e 1 to 0 dl 1393868344 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss01/logs/kern:Mar 3 12:38:36 dc2oss01 kernel: : LustreError: 20055:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message ./dc2oss01/logs/kern:Mar 3 12:38:36 dc2oss01 kernel: : LustreError: 21468:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bf9fef050 x1459112897852383/t0(0) o3->c238dd2d-69ca-3594-9f8a-c529fc6e9b2b@149.165.231.13@tcp:0/0 lens 448/400 e 1 to 0 dl 1393868345 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss01/logs/kern:Mar 3 12:38:36 dc2oss01 kernel: : LustreError: 21468:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message ./dc2oss01/logs/kern:Mar 3 12:38:41 dc2oss01 kernel: : Lustre: dc2-OST0006: Bulk IO read error with c238dd2d-69ca-3594-9f8a-c529fc6e9b2b (at 149.165.231.13@tcp), client will retry: rc -110 ./dc2oss01/logs/kern:Mar 3 12:38:41 dc2oss01 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss01/logs/kern:Mar 3 12:38:50 dc2oss01 kernel: : Lustre: dc2-OST0006: Client c238dd2d-69ca-3594-9f8a-c529fc6e9b2b (at 149.165.231.13@tcp) reconnecting ./dc2oss01/logs/kern:Mar 3 12:38:50 dc2oss01 kernel: : Lustre: Skipped 3 previous similar messages ./dc2oss01/logs/kern:Mar 3 12:38:50 dc2oss01 kernel: : Lustre: 20377:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0006: connection from c238dd2d-69ca-3594-9f8a-c529fc6e9b2b@149.165.231.13@tcp t0 exp ffff8817ee976800 cur 1393868330 last 1393868330 ./dc2oss01/logs/kern:Mar 3 12:38:50 dc2oss01 kernel: : Lustre: 20377:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss01/logs/kern:Mar 3 12:52:36 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=81.218.141.230 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=37316 DF PROTO=TCP SPT=33785 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 12:54:29 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=81.213.140.149 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x40 TTL=40 ID=63208 DF PROTO=TCP SPT=39652 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 12:54:46 dc2oss01 kernel: : Lustre: dc2-OST0001: Client 3792c245-21cd-8eb3-61bd-37df8472501c (at 149.165.229.22@tcp) reconnecting ./dc2oss01/logs/kern:Mar 3 12:54:46 dc2oss01 kernel: : Lustre: 20471:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0001: connection from 3792c245-21cd-8eb3-61bd-37df8472501c@149.165.229.22@tcp t0 exp ffff880bcb535400 cur 1393869286 last 1393869286 ./dc2oss01/logs/kern:Mar 3 12:54:47 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=84.80.137.205 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=24740 DF PROTO=TCP SPT=60052 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 12:55:26 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=89.35.131.238 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=47 ID=58544 DF PROTO=TCP SPT=38766 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 12:57:48 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=184.175.37.148 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=53 ID=39841 DF PROTO=TCP SPT=50507 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 12:59:51 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=109.103.38.155 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=34022 DF PROTO=TCP SPT=52271 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 13:01:27 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=87.6.181.252 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x40 TTL=43 ID=8654 DF PROTO=TCP SPT=4584 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 13:04:57 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=213.172.8.46 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=44821 DF PROTO=TCP SPT=3814 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 13:05:57 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=178.36.177.57 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=61436 DF PROTO=TCP SPT=35414 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 13:15:42 dc2oss01 kernel: : Lustre: dc2-OST0000: haven't heard from client 7097ef4b-b519-de72-4f21-b6c880acdc47 (at 149.165.230.179@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880be8fab800, cur 1393870542 expire 1393870392 last 1393870315 ./dc2oss01/logs/kern:Mar 3 13:15:42 dc2oss01 kernel: : Lustre: Skipped 10 previous similar messages ./dc2oss01/logs/kern:Mar 3 13:20:26 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=2.229.99.45 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=314 DF PROTO=TCP SPT=3658 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 13:25:33 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=175.143.213.122 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=35809 DF PROTO=TCP SPT=59644 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 13:31:22 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=177.100.200.6 DST=149.165.235.1 LEN=52 TOS=0x08 PREC=0x40 TTL=46 ID=28463 DF PROTO=TCP SPT=43020 DPT=5000 WINDOW=14600 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 13:36:04 dc2oss01 kernel: : Lustre: 20480:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0004: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss01/logs/kern:Mar 3 13:36:04 dc2oss01 kernel: : Lustre: 19923:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0003: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss01/logs/kern:Mar 3 13:36:04 dc2oss01 kernel: : Lustre: 19857:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0007: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss01/logs/kern:Mar 3 13:36:04 dc2oss01 kernel: : Lustre: 19923:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 5 previous similar messages ./dc2oss01/logs/kern:Mar 3 13:36:04 dc2oss01 kernel: : Lustre: 19857:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 5 previous similar messages ./dc2oss01/logs/kern:Mar 3 13:38:31 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=188.209.228.228 DST=149.165.235.1 LEN=52 TOS=0x00 PREC=0x00 TTL=49 ID=39201 DF PROTO=TCP SPT=35042 DPT=5000 WINDOW=14600 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 13:41:09 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=211.142.52.11 DST=149.165.235.1 LEN=40 TOS=0x00 PREC=0x00 TTL=103 ID=256 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 13:46:33 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=93.174.93.51 DST=149.165.235.1 LEN=40 TOS=0x08 PREC=0x20 TTL=240 ID=54321 PROTO=TCP SPT=31268 DPT=48499 WINDOW=65535 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 13:54:41 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=186.205.180.67 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=20777 DF PROTO=TCP SPT=37933 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 13:57:33 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=80.34.64.177 DST=149.165.235.1 LEN=52 TOS=0x08 PREC=0x40 TTL=44 ID=59865 DF PROTO=TCP SPT=43681 DPT=5000 WINDOW=14600 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 13:58:46 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=93.146.225.101 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=29215 DF PROTO=TCP SPT=51995 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 13:59:50 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=111.118.248.106 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x20 TTL=42 ID=23351 DF PROTO=TCP SPT=1607 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:03:14 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=83.25.32.57 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x40 TTL=44 ID=1190 DF PROTO=TCP SPT=55850 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:11:44 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=98.66.176.194 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=47 ID=57504 DF PROTO=TCP SPT=4035 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:12:02 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=218.77.79.34 DST=149.165.235.1 LEN=40 TOS=0x00 PREC=0x00 TTL=241 ID=54321 PROTO=TCP SPT=44888 DPT=21 WINDOW=65535 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:15:15 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=180.153.113.141 DST=149.165.235.1 LEN=40 TOS=0x00 PREC=0x00 TTL=50 ID=0 DF PROTO=TCP SPT=22204 DPT=1723 WINDOW=8192 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:21:12 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=186.235.165.100 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x20 TTL=48 ID=34371 DF PROTO=TCP SPT=43697 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:27:29 dc2oss01 kernel: : UDP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=89.248.168.200 DST=149.165.235.1 LEN=64 TOS=0x08 PREC=0x20 TTL=49 ID=0 DF PROTO=UDP SPT=39940 DPT=53 LEN=44 ./dc2oss01/logs/kern:Mar 3 14:28:30 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=179.154.69.74 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=51244 DF PROTO=TCP SPT=47306 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:30:02 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=200.74.32.12 DST=149.165.235.1 LEN=52 TOS=0x00 PREC=0x00 TTL=54 ID=14765 DF PROTO=TCP SPT=57667 DPT=5000 WINDOW=14600 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:31:31 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=187.20.235.175 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=63204 DF PROTO=TCP SPT=53089 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:31:50 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=177.189.210.209 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x40 TTL=46 ID=2781 DF PROTO=TCP SPT=51410 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:33:31 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=181.164.12.48 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=39628 DF PROTO=TCP SPT=44377 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:35:39 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=201.9.250.205 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=9181 DF PROTO=TCP SPT=57884 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:35:59 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=187.65.18.238 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=21019 DF PROTO=TCP SPT=41771 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:38:21 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=177.19.218.12 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=62 DF PROTO=TCP SPT=37152 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:38:28 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=190.246.171.188 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=4731 DF PROTO=TCP SPT=35509 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:40:58 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=174.17.95.187 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=55 ID=60736 DF PROTO=TCP SPT=34693 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:42:04 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=223.4.212.106 DST=149.165.235.1 LEN=40 TOS=0x00 PREC=0x00 TTL=104 ID=256 PROTO=TCP SPT=6000 DPT=3306 WINDOW=16384 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:44:48 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=201.83.164.27 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=52724 DF PROTO=TCP SPT=40740 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:46:18 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=181.95.138.40 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=53 ID=57398 DF PROTO=TCP SPT=58840 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:50:34 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=115.135.192.57 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=18982 DF PROTO=TCP SPT=1973 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:51:13 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=179.25.178.217 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=4288 DF PROTO=TCP SPT=4202 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:53:17 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=84.95.88.120 DST=149.165.235.1 LEN=60 TOS=0x1C PREC=0x00 TTL=52 ID=63660 DF PROTO=TCP SPT=43159 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:53:32 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=5.66.70.185 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=62593 DF PROTO=TCP SPT=48799 DPT=5000 WINDOW=14600 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:53:48 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=201.190.0.236 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x40 TTL=47 ID=2317 DF PROTO=TCP SPT=51936 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:54:06 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=60.189.156.115 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=65001 DF PROTO=TCP SPT=29430 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:54:16 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=95.92.28.133 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=42763 DF PROTO=TCP SPT=60285 DPT=5000 WINDOW=4380 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:54:24 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=149.156.163.225 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=56636 DF PROTO=TCP SPT=57602 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:54:46 dc2oss01 kernel: : ib_send_cm_mra: cm_id_priv->id.state: 0x6 ./dc2oss01/logs/kern:Mar 3 14:54:46 dc2oss01 kernel: : Lustre: 20377:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0000: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2oss01/logs/kern:Mar 3 14:54:46 dc2oss01 kernel: : Lustre: 20377:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 6 previous similar messages ./dc2oss01/logs/kern:Mar 3 14:55:46 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=187.126.125.103 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=6947 DF PROTO=TCP SPT=44854 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:56:39 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=201.227.89.66 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=59592 DF PROTO=TCP SPT=3712 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:57:05 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=201.143.177.64 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=40965 DF PROTO=TCP SPT=40622 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:57:17 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=123.142.77.66 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=63497 DF PROTO=TCP SPT=43629 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:57:30 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=88.102.151.202 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=28175 DF PROTO=TCP SPT=49718 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:57:30 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=90.179.24.32 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=2276 DF PROTO=TCP SPT=37292 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:57:30 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=88.101.77.161 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=43538 DF PROTO=TCP SPT=42447 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:57:31 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=78.173.13.254 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x40 TTL=42 ID=24197 DF PROTO=TCP SPT=60099 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:57:34 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=182.74.78.170 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=41511 DF PROTO=TCP SPT=41738 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:57:35 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=121.96.83.91 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=18549 DF PROTO=TCP SPT=1830 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:57:36 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=82.154.62.224 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=51668 DF PROTO=TCP SPT=3268 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:57:41 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=74.100.132.100 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x40 TTL=48 ID=25292 DF PROTO=TCP SPT=1947 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:58:19 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=190.147.105.167 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=4670 DF PROTO=TCP SPT=4613 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:58:24 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=82.59.85.207 DST=149.165.235.1 LEN=60 TOS=0x08 PREC=0x40 TTL=42 ID=5665 DF PROTO=TCP SPT=4163 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:58:35 dc2oss01 kernel: : UDP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=98.143.175.92 DST=149.165.235.1 LEN=442 TOS=0x00 PREC=0x00 TTL=56 ID=0 DF PROTO=UDP SPT=5064 DPT=5061 LEN=422 ./dc2oss01/logs/kern:Mar 3 14:59:02 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=218.16.96.2 DST=149.165.235.1 LEN=48 TOS=0x00 PREC=0x00 TTL=113 ID=30102 DF PROTO=TCP SPT=46699 DPT=3389 WINDOW=65535 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:59:05 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=218.16.96.2 DST=149.165.235.1 LEN=48 TOS=0x00 PREC=0x00 TTL=113 ID=30189 DF PROTO=TCP SPT=46699 DPT=3389 WINDOW=65535 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:59:06 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=85.245.233.179 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=46 ID=54279 DF PROTO=TCP SPT=4021 DPT=5000 WINDOW=5808 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:59:11 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=218.16.96.2 DST=149.165.235.1 LEN=48 TOS=0x00 PREC=0x00 TTL=113 ID=30419 DF PROTO=TCP SPT=46699 DPT=3389 WINDOW=65535 RES=0x00 SYN URGP=0 ./dc2oss01/logs/kern:Mar 3 14:59:13 dc2oss01 kernel: : TCP Dropped: IN=eth2 OUT= MAC=d4:ae:52:b5:99:40:00:24:38:a8:dd:00:08:00 SRC=177.141.36.226 DST=149.165.235.1 LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=8185 DF PROTO=TCP SPT=61535 DPT=5000 WINDOW=5840 RES=0x00 SYN URGP=0 ./dc2oss02/logs/kern:Mar 3 11:53:39 dc2oss02 kernel: : Lustre: 14596:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000b: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss02/logs/kern:Mar 3 11:53:39 dc2oss02 kernel: : Lustre: 14596:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 6 previous similar messages ./dc2oss02/logs/kern:Mar 3 12:33:46 dc2oss02 kernel: : Lustre: 14566:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1393868019/real 1393868020] req@ffff8817c77f8c00 x1456766148346457/t0(0) o104->dc2-OST0011@149.165.226.202@tcp:15/16 lens 296/192 e 0 to 1 dl 1393868026 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 ./dc2oss02/logs/kern:Mar 3 12:33:46 dc2oss02 kernel: : LustreError: 138-a: dc2-OST0011: A client on nid 149.165.226.202@tcp was evicted due to a lock blocking callback time out: rc -107 ./dc2oss02/logs/kern:Mar 3 12:33:46 dc2oss02 kernel: : LustreError: 14511:0:(client.c:1045:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880bd9c23400 x1456766148347409/t0(0) o105->dc2-OST0011@149.165.226.202@tcp:15/16 lens 344/192 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 ./dc2oss02/logs/kern:Mar 3 12:33:46 dc2oss02 kernel: : LustreError: 14511:0:(ldlm_lockd.c:656:ldlm_handle_ast_error()) ### client (nid 149.165.226.202@tcp) returned 0 from completion AST ns: filter-dc2-OST0011_UUID lock: ffff880e83b5db40/0xe73a71c291c8e1cb lrc: 3/0,0 mode: PW/PW res: 22017784/0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x0 remote: 0xae970e13c5cd39de expref: 33 pid: 14566 timeout 0 ./dc2oss02/logs/kern:Mar 3 12:33:49 dc2oss02 kernel: : Lustre: 15460:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0011: connection from 16a30cbc-2b02-8ef3-3d33-dbb18f4b0ff3@149.165.226.202@tcp t0 exp (null) cur 1393868029 last 0 ./dc2oss02/logs/kern:Mar 3 12:33:49 dc2oss02 kernel: : Lustre: 15460:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 9 previous similar messages ./dc2oss02/logs/kern:Mar 3 12:33:57 dc2oss02 kernel: : Lustre: dc2-OST0011: Client e6cd2599-383c-225f-41ef-71626f4873b7 (at 149.165.229.17@tcp) reconnecting ./dc2oss02/logs/kern:Mar 3 12:33:57 dc2oss02 kernel: : Lustre: 15495:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0011: connection from e6cd2599-383c-225f-41ef-71626f4873b7@149.165.229.17@tcp t0 exp ffff8817cb512c00 cur 1393868037 last 1393868037 ./dc2oss02/logs/kern:Mar 3 12:33:57 dc2oss02 kernel: : Lustre: dc2-OST0012: Client e6cd2599-383c-225f-41ef-71626f4873b7 (at 149.165.229.17@tcp) reconnecting ./dc2oss02/logs/kern:Mar 3 12:38:31 dc2oss02 kernel: : Lustre: dc2-OST000b: Client 8f55aefa-1141-e9ee-6e8e-c7ef84ac0a50 (at 149.165.229.194@tcp) reconnecting ./dc2oss02/logs/kern:Mar 3 12:38:31 dc2oss02 kernel: : Lustre: 15569:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000b: connection from 8f55aefa-1141-e9ee-6e8e-c7ef84ac0a50@149.165.229.194@tcp t0 exp ffff8817c933c400 cur 1393868311 last 1393868311 ./dc2oss02/logs/kern:Mar 3 12:38:31 dc2oss02 kernel: : Lustre: 15569:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss02/logs/kern:Mar 3 12:38:42 dc2oss02 kernel: : Lustre: dc2-OST000d: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss02/logs/kern:Mar 3 12:38:42 dc2oss02 kernel: : Lustre: Skipped 8 previous similar messages ./dc2oss02/logs/kern:Mar 3 12:38:42 dc2oss02 kernel: : Lustre: dc2-OST000d: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) refused reconnection, still busy with 5 active RPCs ./dc2oss02/logs/kern:Mar 3 12:38:43 dc2oss02 kernel: : LustreError: 14745:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880a43a2a000 x1459136108324229/t0(0) o3->1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp:0/0 lens 448/400 e 0 to 0 dl 1393868334 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss02/logs/kern:Mar 3 12:38:43 dc2oss02 kernel: : Lustre: dc2-OST000d: Bulk IO read error with 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp), client will retry: rc -110 ./dc2oss02/logs/kern:Mar 3 12:38:43 dc2oss02 kernel: : LustreError: 14747:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8808b7b98800 x1459136108324579/t0(0) o3->1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp:0/0 lens 448/400 e 0 to 0 dl 1393868357 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss02/logs/kern:Mar 3 12:38:43 dc2oss02 kernel: : LustreError: 14747:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 3 previous similar messages ./dc2oss02/logs/kern:Mar 3 12:38:43 dc2oss02 kernel: : Lustre: dc2-OST000d: Bulk IO read error with 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp), client will retry: rc -110 ./dc2oss02/logs/kern:Mar 3 12:38:43 dc2oss02 kernel: : Lustre: Skipped 3 previous similar messages ./dc2oss02/logs/kern:Mar 3 12:39:08 dc2oss02 kernel: : Lustre: dc2-OST000d: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss02/logs/kern:Mar 3 12:39:08 dc2oss02 kernel: : Lustre: 15336:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000d: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8817c5be8400 cur 1393868348 last 1393868348 ./dc2oss02/logs/kern:Mar 3 12:39:08 dc2oss02 kernel: : Lustre: 15336:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 9 previous similar messages ./dc2oss02/logs/kern:Mar 3 12:39:30 dc2oss02 kernel: : Lustre: dc2-OST000b: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss02/logs/kern:Mar 3 12:39:30 dc2oss02 kernel: : Lustre: 15237:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000b: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8805f85f1800 cur 1393868370 last 1393868369 ./dc2oss02/logs/kern:Mar 3 12:41:08 dc2oss02 kernel: : Lustre: dc2-OST0010: Client 1c39cdb9-ff82-76d3-8850-5d884d059792 (at 149.165.229.2@tcp) reconnecting ./dc2oss02/logs/kern:Mar 3 12:41:08 dc2oss02 kernel: : Lustre: Skipped 8 previous similar messages ./dc2oss02/logs/kern:Mar 3 12:41:08 dc2oss02 kernel: : Lustre: 15280:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0010: connection from 1c39cdb9-ff82-76d3-8850-5d884d059792@149.165.229.2@tcp t0 exp ffff880c08fc7800 cur 1393868468 last 1393868468 ./dc2oss02/logs/kern:Mar 3 12:41:08 dc2oss02 kernel: : Lustre: 15280:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 8 previous similar messages ./dc2oss02/logs/kern:Mar 3 12:54:47 dc2oss02 kernel: : Lustre: dc2-OST000c: Client 3792c245-21cd-8eb3-61bd-37df8472501c (at 149.165.229.22@tcp) reconnecting ./dc2oss02/logs/kern:Mar 3 12:54:47 dc2oss02 kernel: : Lustre: 14566:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000c: connection from 3792c245-21cd-8eb3-61bd-37df8472501c@149.165.229.22@tcp t0 exp ffff8808b63d3400 cur 1393869287 last 1393869287 ./dc2oss02/logs/kern:Mar 3 13:00:17 dc2oss02 kernel: : Lustre: dc2-OST000f: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) reconnecting ./dc2oss02/logs/kern:Mar 3 13:00:17 dc2oss02 kernel: : Lustre: dc2-OST0012: Client fb5d22a7-b346-9eb5-e313-762b97391b1e (at 149.165.235.18@tcp) reconnecting ./dc2oss02/logs/kern:Mar 3 13:00:17 dc2oss02 kernel: : Lustre: 15614:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000d: connection from fb5d22a7-b346-9eb5-e313-762b97391b1e@149.165.235.18@tcp t0 exp ffff880bf9382400 cur 1393869617 last 1393869617 ./dc2oss02/logs/kern:Mar 3 13:00:17 dc2oss02 kernel: : Lustre: Skipped 3 previous similar messages ./dc2oss02/logs/kern:Mar 3 13:00:17 dc2oss02 kernel: : Lustre: 15614:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss02/logs/kern:Mar 3 13:15:42 dc2oss02 kernel: : Lustre: dc2-OST000c: haven't heard from client 7097ef4b-b519-de72-4f21-b6c880acdc47 (at 149.165.230.179@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88036838f400, cur 1393870542 expire 1393870392 last 1393870315 ./dc2oss02/logs/kern:Mar 3 13:15:42 dc2oss02 kernel: : Lustre: Skipped 9 previous similar messages ./dc2oss02/logs/kern:Mar 3 13:36:04 dc2oss02 kernel: : Lustre: 14608:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000b: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss02/logs/kern:Mar 3 13:36:04 dc2oss02 kernel: : Lustre: 14608:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 6 previous similar messages ./dc2oss02/logs/kern:Mar 3 14:54:46 dc2oss02 kernel: : ib_send_cm_mra: cm_id_priv->id.state: 0x6 ./dc2oss02/logs/kern:Mar 3 14:54:46 dc2oss02 kernel: : Lustre: 14547:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000c: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2oss02/logs/kern:Mar 3 14:54:46 dc2oss02 kernel: : Lustre: 15332:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST000b: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2oss02/logs/kern:Mar 3 14:54:46 dc2oss02 kernel: : Lustre: 15332:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 8 previous similar messages ./dc2oss03/logs/kern:Mar 3 11:53:39 dc2oss03 kernel: : Lustre: 15294:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0015: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss03/logs/kern:Mar 3 11:53:39 dc2oss03 kernel: : Lustre: 15294:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 10 previous similar messages ./dc2oss03/logs/kern:Mar 3 12:38:31 dc2oss03 kernel: : Lustre: dc2-OST0018: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss03/logs/kern:Mar 3 12:38:31 dc2oss03 kernel: : Lustre: 15375:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0018: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff881776589000 cur 1393868311 last 1393868311 ./dc2oss03/logs/kern:Mar 3 12:38:31 dc2oss03 kernel: : Lustre: 15375:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 10 previous similar messages ./dc2oss03/logs/kern:Mar 3 12:38:52 dc2oss03 kernel: : Lustre: dc2-OST0016: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss03/logs/kern:Mar 3 12:38:52 dc2oss03 kernel: : Lustre: Skipped 2 previous similar messages ./dc2oss03/logs/kern:Mar 3 12:38:52 dc2oss03 kernel: : Lustre: 14564:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0016: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff881014c82800 cur 1393868332 last 1393868332 ./dc2oss03/logs/kern:Mar 3 12:38:52 dc2oss03 kernel: : Lustre: 14564:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss03/logs/kern:Mar 3 12:38:56 dc2oss03 kernel: : Lustre: dc2-OST001a: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss03/logs/kern:Mar 3 12:38:56 dc2oss03 kernel: : Lustre: 15301:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST001a: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8814c990d800 cur 1393868336 last 1393868336 ./dc2oss03/logs/kern:Mar 3 12:39:30 dc2oss03 kernel: : Lustre: dc2-OST0015: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss03/logs/kern:Mar 3 12:39:30 dc2oss03 kernel: : Lustre: Skipped 2 previous similar messages ./dc2oss03/logs/kern:Mar 3 12:39:30 dc2oss03 kernel: : Lustre: 15647:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0015: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff880ca6eca400 cur 1393868370 last 1393868369 ./dc2oss03/logs/kern:Mar 3 12:39:30 dc2oss03 kernel: : Lustre: 15647:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss03/logs/kern:Mar 3 13:11:53 dc2oss03 kernel: : Lustre: dc2-OST0016: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss03/logs/kern:Mar 3 13:11:53 dc2oss03 kernel: : Lustre: Skipped 10 previous similar messages ./dc2oss03/logs/kern:Mar 3 13:11:53 dc2oss03 kernel: : Lustre: dc2-OST0016: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) refused reconnection, still busy with 6 active RPCs ./dc2oss03/logs/kern:Mar 3 13:11:53 dc2oss03 kernel: : LustreError: 26233:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880be3b4e000 x1459136108395187/t0(0) o3->1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp:0/0 lens 448/400 e 1 to 0 dl 1393870342 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss03/logs/kern:Mar 3 13:11:53 dc2oss03 kernel: : LustreError: 26233:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages ./dc2oss03/logs/kern:Mar 3 13:11:53 dc2oss03 kernel: : Lustre: dc2-OST0016: Bulk IO read error with 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp), client will retry: rc -110 ./dc2oss03/logs/kern:Mar 3 13:11:53 dc2oss03 kernel: : Lustre: Skipped 2 previous similar messages ./dc2oss03/logs/kern:Mar 3 13:11:53 dc2oss03 kernel: : LustreError: 5096:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880be6880850 x1459136108395179/t0(0) o3->1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp:0/0 lens 448/400 e 1 to 0 dl 1393870340 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss03/logs/kern:Mar 3 13:11:53 dc2oss03 kernel: : Lustre: dc2-OST0016: Bulk IO read error with 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp), client will retry: rc -110 ./dc2oss03/logs/kern:Mar 3 13:11:53 dc2oss03 kernel: : LustreError: 14643:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd4aa4400 x1459136108395181/t0(0) o3->1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp:0/0 lens 448/400 e 1 to 0 dl 1393870340 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss03/logs/kern:Mar 3 13:11:53 dc2oss03 kernel: : LustreError: 14643:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages ./dc2oss03/logs/kern:Mar 3 13:11:53 dc2oss03 kernel: : Lustre: dc2-OST0016: Bulk IO read error with 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp), client will retry: rc -110 ./dc2oss03/logs/kern:Mar 3 13:11:53 dc2oss03 kernel: : Lustre: Skipped 2 previous similar messages ./dc2oss03/logs/kern:Mar 3 13:12:01 dc2oss03 kernel: : Lustre: dc2-OST0016: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss03/logs/kern:Mar 3 13:12:01 dc2oss03 kernel: : Lustre: 15403:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0016: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff881014c82800 cur 1393870321 last 1393870321 ./dc2oss03/logs/kern:Mar 3 13:12:01 dc2oss03 kernel: : Lustre: 15403:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 10 previous similar messages ./dc2oss03/logs/kern:Mar 3 13:15:42 dc2oss03 kernel: : Lustre: dc2-OST0016: haven't heard from client 7097ef4b-b519-de72-4f21-b6c880acdc47 (at 149.165.230.179@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817bbe8fc00, cur 1393870542 expire 1393870392 last 1393870315 ./dc2oss03/logs/kern:Mar 3 13:15:42 dc2oss03 kernel: : Lustre: Skipped 10 previous similar messages ./dc2oss03/logs/kern:Mar 3 13:36:04 dc2oss03 kernel: : Lustre: 15306:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0016: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss03/logs/kern:Mar 3 13:36:04 dc2oss03 kernel: : Lustre: 14570:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0018: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss03/logs/kern:Mar 3 13:36:04 dc2oss03 kernel: : Lustre: 14570:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss03/logs/kern:Mar 3 13:36:04 dc2oss03 kernel: : Lustre: 15306:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss03/logs/kern:Mar 3 14:54:46 dc2oss03 kernel: : ib_send_cm_mra: cm_id_priv->id.state: 0x6 ./dc2oss03/logs/kern:Mar 3 14:54:46 dc2oss03 kernel: : Lustre: 15418:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0015: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2oss03/logs/kern:Mar 3 14:54:46 dc2oss03 kernel: : Lustre: 15418:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 8 previous similar messages ./dc2oss04/logs/kern:Mar 3 11:53:39 dc2oss04 kernel: : Lustre: 15479:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0020: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss04/logs/kern:Mar 3 11:53:39 dc2oss04 kernel: : Lustre: 15479:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 9 previous similar messages ./dc2oss04/logs/kern:Mar 3 12:38:31 dc2oss04 kernel: : Lustre: dc2-OST0020: Client 6cf88260-fe65-1a9d-cec8-f066cce5bed7 (at 149.165.230.149@tcp) reconnecting ./dc2oss04/logs/kern:Mar 3 12:38:31 dc2oss04 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss04/logs/kern:Mar 3 12:38:31 dc2oss04 kernel: : Lustre: 15381:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0020: connection from 6cf88260-fe65-1a9d-cec8-f066cce5bed7@149.165.230.149@tcp t0 exp ffff880817689400 cur 1393868311 last 1393868311 ./dc2oss04/logs/kern:Mar 3 12:38:31 dc2oss04 kernel: : Lustre: 15381:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 9 previous similar messages ./dc2oss04/logs/kern:Mar 3 12:38:31 dc2oss04 kernel: : Lustre: 15381:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0020: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8817d192a400 cur 1393868311 last 1393868311 ./dc2oss04/logs/kern:Mar 3 12:38:31 dc2oss04 kernel: : Lustre: dc2-OST0025: Client d9951213-87b4-dc02-4ae5-70ac2d71d6b9 (at 149.165.230.171@tcp) reconnecting ./dc2oss04/logs/kern:Mar 3 12:38:31 dc2oss04 kernel: : Lustre: Skipped 21 previous similar messages ./dc2oss04/logs/kern:Mar 3 12:38:31 dc2oss04 kernel: : Lustre: dc2-OST0021: Client f3f701b7-742f-fe5b-aac4-42c3cb06fbfb (at 149.165.229.187@tcp) reconnecting ./dc2oss04/logs/kern:Mar 3 12:38:31 dc2oss04 kernel: : Lustre: Skipped 10 previous similar messages ./dc2oss04/logs/kern:Mar 3 12:38:31 dc2oss04 kernel: : Lustre: 15398:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0025: connection from f3f701b7-742f-fe5b-aac4-42c3cb06fbfb@149.165.229.187@tcp t0 exp ffff8817b0cbf400 cur 1393868311 last 1393868311 ./dc2oss04/logs/kern:Mar 3 12:38:31 dc2oss04 kernel: : Lustre: 15398:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 31 previous similar messages ./dc2oss04/logs/kern:Mar 3 12:40:15 dc2oss04 kernel: : Lustre: dc2-OST0029: Client 1c39cdb9-ff82-76d3-8850-5d884d059792 (at 149.165.229.2@tcp) reconnecting ./dc2oss04/logs/kern:Mar 3 12:40:15 dc2oss04 kernel: : Lustre: Skipped 15 previous similar messages ./dc2oss04/logs/kern:Mar 3 12:40:15 dc2oss04 kernel: : Lustre: 15416:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0029: connection from 1c39cdb9-ff82-76d3-8850-5d884d059792@149.165.229.2@tcp t0 exp ffff88174ae0a800 cur 1393868415 last 1393868415 ./dc2oss04/logs/kern:Mar 3 12:40:15 dc2oss04 kernel: : Lustre: 15416:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 16 previous similar messages ./dc2oss04/logs/kern:Mar 3 12:51:05 dc2oss04 kernel: : Lustre: dc2-OST0020: Client 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp) reconnecting ./dc2oss04/logs/kern:Mar 3 12:51:05 dc2oss04 kernel: : Lustre: dc2-OST0020: Client 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp) refused reconnection, still busy with 3 active RPCs ./dc2oss04/logs/kern:Mar 3 12:51:06 dc2oss04 kernel: : LustreError: 31946:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd671fc00 x1459134275163351/t0(0) o3->14a5b49b-639b-b462-1562-f1e899aebd9c@149.165.229.49@tcp:0/0 lens 448/400 e 1 to 0 dl 1393869104 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss04/logs/kern:Mar 3 12:51:06 dc2oss04 kernel: : LustreError: 31946:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 3 previous similar messages ./dc2oss04/logs/kern:Mar 3 12:51:06 dc2oss04 kernel: : Lustre: dc2-OST0020: Bulk IO read error with 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp), client will retry: rc -110 ./dc2oss04/logs/kern:Mar 3 12:51:06 dc2oss04 kernel: : Lustre: Skipped 3 previous similar messages ./dc2oss04/logs/kern:Mar 3 12:51:06 dc2oss04 kernel: : LustreError: 31951:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880702b07850 x1459134275163261/t0(0) o3->14a5b49b-639b-b462-1562-f1e899aebd9c@149.165.229.49@tcp:0/0 lens 448/400 e 1 to 0 dl 1393869102 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss04/logs/kern:Mar 3 12:51:06 dc2oss04 kernel: : Lustre: dc2-OST0020: Bulk IO read error with 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp), client will retry: rc -110 ./dc2oss04/logs/kern:Mar 3 12:51:07 dc2oss04 kernel: : Lustre: dc2-OST0027: Client 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp) reconnecting ./dc2oss04/logs/kern:Mar 3 12:51:07 dc2oss04 kernel: : Lustre: dc2-OST0027: Client 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp) refused reconnection, still busy with 2 active RPCs ./dc2oss04/logs/kern:Mar 3 12:51:07 dc2oss04 kernel: : LustreError: 19941:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd633a400 x1459134275163270/t0(0) o3->14a5b49b-639b-b462-1562-f1e899aebd9c@149.165.229.49@tcp:0/0 lens 448/400 e 1 to 0 dl 1393869102 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss04/logs/kern:Mar 3 12:51:07 dc2oss04 kernel: : LustreError: 19941:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message ./dc2oss04/logs/kern:Mar 3 12:51:07 dc2oss04 kernel: : Lustre: dc2-OST0027: Bulk IO read error with 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp), client will retry: rc -110 ./dc2oss04/logs/kern:Mar 3 12:51:07 dc2oss04 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss04/logs/kern:Mar 3 12:51:16 dc2oss04 kernel: : Lustre: dc2-OST0020: Client 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp) reconnecting ./dc2oss04/logs/kern:Mar 3 12:51:16 dc2oss04 kernel: : Lustre: 15398:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0020: connection from 14a5b49b-639b-b462-1562-f1e899aebd9c@149.165.229.49@tcp t0 exp ffff8817d2fa0c00 cur 1393869076 last 1393869076 ./dc2oss04/logs/kern:Mar 3 12:51:19 dc2oss04 kernel: : Lustre: dc2-OST0027: Client 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp) reconnecting ./dc2oss04/logs/kern:Mar 3 12:51:19 dc2oss04 kernel: : Lustre: 15404:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0027: connection from 14a5b49b-639b-b462-1562-f1e899aebd9c@149.165.229.49@tcp t0 exp ffff8817aec34c00 cur 1393869079 last 1393869079 ./dc2oss04/logs/kern:Mar 3 13:15:42 dc2oss04 kernel: : Lustre: dc2-OST0023: haven't heard from client 7097ef4b-b519-de72-4f21-b6c880acdc47 (at 149.165.230.179@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee9c25400, cur 1393870542 expire 1393870392 last 1393870315 ./dc2oss04/logs/kern:Mar 3 13:15:42 dc2oss04 kernel: : Lustre: Skipped 9 previous similar messages ./dc2oss04/logs/kern:Mar 3 13:36:04 dc2oss04 kernel: : Lustre: 15416:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0023: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss04/logs/kern:Mar 3 13:36:04 dc2oss04 kernel: : Lustre: 15416:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss04/logs/kern:Mar 3 14:54:47 dc2oss04 kernel: : ib_send_cm_mra: cm_id_priv->id.state: 0x6 ./dc2oss04/logs/kern:Mar 3 14:54:47 dc2oss04 kernel: : Lustre: 15436:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0022: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876487 last 0 ./dc2oss04/logs/kern:Mar 3 14:54:47 dc2oss04 kernel: : Lustre: 15383:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0023: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876487 last 0 ./dc2oss04/logs/kern:Mar 3 14:54:47 dc2oss04 kernel: : Lustre: 15388:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0020: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876487 last 0 ./dc2oss04/logs/kern:Mar 3 14:54:47 dc2oss04 kernel: : Lustre: 15383:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 8 previous similar messages ./dc2oss04/logs/kern:Mar 3 14:54:47 dc2oss04 kernel: : Lustre: 15388:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 8 previous similar messages ./dc2oss04/logs/kern:Mar 3 14:54:47 dc2oss04 kernel: : Lustre: 15388:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0028: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876487 last 0 ./dc2oss04/logs/kern:Mar 3 14:54:47 dc2oss04 kernel: : Lustre: 15388:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 5 previous similar messages ./dc2oss05/logs/kern:Mar 3 11:53:39 dc2oss05 kernel: : Lustre: 15973:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002a: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss05/logs/kern:Mar 3 11:53:39 dc2oss05 kernel: : Lustre: 15973:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 10 previous similar messages ./dc2oss05/logs/kern:Mar 3 12:33:57 dc2oss05 kernel: : Lustre: dc2-OST002c: Client 4017c176-bbe0-d45e-f18e-9f8e9ee2f246 (at 149.165.229.215@tcp) reconnecting ./dc2oss05/logs/kern:Mar 3 12:33:57 dc2oss05 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss05/logs/kern:Mar 3 12:33:57 dc2oss05 kernel: : Lustre: 15862:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002c: connection from 4017c176-bbe0-d45e-f18e-9f8e9ee2f246@149.165.229.215@tcp t0 exp ffff880be0ac8800 cur 1393868037 last 1393868037 ./dc2oss05/logs/kern:Mar 3 12:33:57 dc2oss05 kernel: : Lustre: 15862:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 10 previous similar messages ./dc2oss05/logs/kern:Mar 3 12:33:57 dc2oss05 kernel: : Lustre: dc2-OST0033: Client 1951e547-ef58-f071-b440-91732bd33942 (at 149.165.229.191@tcp) reconnecting ./dc2oss05/logs/kern:Mar 3 12:33:57 dc2oss05 kernel: : Lustre: 16132:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0033: connection from 1951e547-ef58-f071-b440-91732bd33942@149.165.229.191@tcp t0 exp ffff880bc1ceac00 cur 1393868037 last 1393868037 ./dc2oss05/logs/kern:Mar 3 12:38:31 dc2oss05 kernel: : Lustre: dc2-OST002b: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss05/logs/kern:Mar 3 12:38:31 dc2oss05 kernel: : Lustre: 15849:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002d: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8817af0d2400 cur 1393868311 last 1393868311 ./dc2oss05/logs/kern:Mar 3 12:38:31 dc2oss05 kernel: : Lustre: 15849:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss05/logs/kern:Mar 3 12:38:31 dc2oss05 kernel: : Lustre: Skipped 3 previous similar messages ./dc2oss05/logs/kern:Mar 3 12:38:33 dc2oss05 kernel: : Lustre: dc2-OST002b: Client f3f701b7-742f-fe5b-aac4-42c3cb06fbfb (at 149.165.229.187@tcp) reconnecting ./dc2oss05/logs/kern:Mar 3 12:38:33 dc2oss05 kernel: : Lustre: dc2-OST0031: Client f3f701b7-742f-fe5b-aac4-42c3cb06fbfb (at 149.165.229.187@tcp) reconnecting ./dc2oss05/logs/kern:Mar 3 12:38:33 dc2oss05 kernel: : Lustre: Skipped 34 previous similar messages ./dc2oss05/logs/kern:Mar 3 12:38:33 dc2oss05 kernel: : Lustre: Skipped 34 previous similar messages ./dc2oss05/logs/kern:Mar 3 12:38:33 dc2oss05 kernel: : Lustre: 14512:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002b: connection from f3f701b7-742f-fe5b-aac4-42c3cb06fbfb@149.165.229.187@tcp t0 exp ffff88180238d400 cur 1393868313 last 1393868313 ./dc2oss05/logs/kern:Mar 3 12:38:33 dc2oss05 kernel: : Lustre: 14512:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 36 previous similar messages ./dc2oss05/logs/kern:Mar 3 12:38:39 dc2oss05 kernel: : LustreError: 137-5: UUID 'dc2-OST003a_UUID' is not available for connect (no target) ./dc2oss05/logs/kern:Mar 3 12:52:29 dc2oss05 kernel: : Lustre: dc2-OST002a: Client fe283d84-ddff-5e8e-169f-0dee01362a65 (at 149.165.229.106@tcp) reconnecting ./dc2oss05/logs/kern:Mar 3 12:52:29 dc2oss05 kernel: : Lustre: 15933:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002a: connection from fe283d84-ddff-5e8e-169f-0dee01362a65@149.165.229.106@tcp t0 exp ffff880da08eac00 cur 1393869149 last 1393869149 ./dc2oss05/logs/kern:Mar 3 12:52:30 dc2oss05 kernel: : Lustre: dc2-OST002b: Client fe283d84-ddff-5e8e-169f-0dee01362a65 (at 149.165.229.106@tcp) reconnecting ./dc2oss05/logs/kern:Mar 3 12:52:30 dc2oss05 kernel: : Lustre: 15852:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002b: connection from fe283d84-ddff-5e8e-169f-0dee01362a65@149.165.229.106@tcp t0 exp ffff880a03984c00 cur 1393869150 last 1393869150 ./dc2oss05/logs/kern:Mar 3 12:52:31 dc2oss05 kernel: : Lustre: 14601:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002c: connection from 87ae1001-0cff-8218-9e34-50609682b7a6@149.165.230.143@tcp t0 exp ffff8817d4db1000 cur 1393869151 last 1393869151 ./dc2oss05/logs/kern:Mar 3 12:52:31 dc2oss05 kernel: : Lustre: 14601:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss05/logs/kern:Mar 3 12:54:47 dc2oss05 kernel: : Lustre: dc2-OST002c: Client 3792c245-21cd-8eb3-61bd-37df8472501c (at 149.165.229.22@tcp) reconnecting ./dc2oss05/logs/kern:Mar 3 12:54:47 dc2oss05 kernel: : Lustre: Skipped 5 previous similar messages ./dc2oss05/logs/kern:Mar 3 12:54:47 dc2oss05 kernel: : Lustre: 16048:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002c: connection from 3792c245-21cd-8eb3-61bd-37df8472501c@149.165.229.22@tcp t0 exp ffff8817d37e7800 cur 1393869287 last 1393869287 ./dc2oss05/logs/kern:Mar 3 12:54:47 dc2oss05 kernel: : Lustre: 16048:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss05/logs/kern:Mar 3 12:54:50 dc2oss05 kernel: : Lustre: dc2-OST002a: Client 3792c245-21cd-8eb3-61bd-37df8472501c (at 149.165.229.22@tcp) reconnecting ./dc2oss05/logs/kern:Mar 3 12:54:50 dc2oss05 kernel: : Lustre: Skipped 4 previous similar messages ./dc2oss05/logs/kern:Mar 3 12:54:50 dc2oss05 kernel: : Lustre: 16010:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002a: connection from 3792c245-21cd-8eb3-61bd-37df8472501c@149.165.229.22@tcp t0 exp ffff880cec5bc400 cur 1393869290 last 1393869290 ./dc2oss05/logs/kern:Mar 3 12:54:50 dc2oss05 kernel: : Lustre: 16010:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 4 previous similar messages ./dc2oss05/logs/kern:Mar 3 13:15:42 dc2oss05 kernel: : Lustre: dc2-OST002a: haven't heard from client 7097ef4b-b519-de72-4f21-b6c880acdc47 (at 149.165.230.179@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88009239e800, cur 1393870542 expire 1393870392 last 1393870315 ./dc2oss05/logs/kern:Mar 3 13:15:42 dc2oss05 kernel: : Lustre: Skipped 10 previous similar messages ./dc2oss05/logs/kern:Mar 3 13:36:04 dc2oss05 kernel: : Lustre: 16023:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002b: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss05/logs/kern:Mar 3 13:36:04 dc2oss05 kernel: : Lustre: 15895:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002e: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss05/logs/kern:Mar 3 13:36:04 dc2oss05 kernel: : Lustre: 15895:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss05/logs/kern:Mar 3 13:36:04 dc2oss05 kernel: : Lustre: 16023:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss05/logs/kern:Mar 3 14:54:46 dc2oss05 kernel: : ib_send_cm_mra: cm_id_priv->id.state: 0x6 ./dc2oss05/logs/kern:Mar 3 14:54:46 dc2oss05 kernel: : Lustre: 15814:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST002a: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2oss05/logs/kern:Mar 3 14:54:46 dc2oss05 kernel: : Lustre: 15814:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 7 previous similar messages ./dc2oss06/logs/kern:Mar 3 11:53:39 dc2oss06 kernel: : Lustre: 14430:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0036: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss06/logs/kern:Mar 3 11:53:39 dc2oss06 kernel: : Lustre: 15830:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003a: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss06/logs/kern:Mar 3 11:53:39 dc2oss06 kernel: : Lustre: 14398:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003d: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss06/logs/kern:Mar 3 11:53:39 dc2oss06 kernel: : Lustre: 14398:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 6 previous similar messages ./dc2oss06/logs/kern:Mar 3 12:38:31 dc2oss06 kernel: : Lustre: dc2-OST0036: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss06/logs/kern:Mar 3 12:38:31 dc2oss06 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss06/logs/kern:Mar 3 12:38:31 dc2oss06 kernel: : Lustre: 15963:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0036: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8817c8000c00 cur 1393868311 last 1393868311 ./dc2oss06/logs/kern:Mar 3 12:38:31 dc2oss06 kernel: : Lustre: 15963:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss06/logs/kern:Mar 3 12:38:46 dc2oss06 kernel: : Lustre: dc2-OST003a: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss06/logs/kern:Mar 3 12:38:46 dc2oss06 kernel: : Lustre: Skipped 4 previous similar messages ./dc2oss06/logs/kern:Mar 3 12:38:46 dc2oss06 kernel: : Lustre: 15910:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003a: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8814b3694800 cur 1393868326 last 1393868326 ./dc2oss06/logs/kern:Mar 3 12:38:46 dc2oss06 kernel: : Lustre: 15910:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 4 previous similar messages ./dc2oss06/logs/kern:Mar 3 12:38:56 dc2oss06 kernel: : Lustre: dc2-OST0035: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss06/logs/kern:Mar 3 12:38:56 dc2oss06 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss06/logs/kern:Mar 3 12:38:56 dc2oss06 kernel: : Lustre: 15847:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0038: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff88135e59e000 cur 1393868336 last 1393868336 ./dc2oss06/logs/kern:Mar 3 12:51:01 dc2oss06 kernel: : Lustre: dc2-OST0038: Client 20fb5f9f-5fd2-532b-4e29-fbc73fc84137 (at 149.165.229.195@tcp) reconnecting ./dc2oss06/logs/kern:Mar 3 12:51:01 dc2oss06 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss06/logs/kern:Mar 3 12:51:01 dc2oss06 kernel: : Lustre: 15847:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0038: connection from 20fb5f9f-5fd2-532b-4e29-fbc73fc84137@149.165.229.195@tcp t0 exp ffff8817f8fdec00 cur 1393869061 last 1393869061 ./dc2oss06/logs/kern:Mar 3 12:51:01 dc2oss06 kernel: : Lustre: 15847:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss06/logs/kern:Mar 3 12:51:02 dc2oss06 kernel: : Lustre: dc2-OST003b: Client 20fb5f9f-5fd2-532b-4e29-fbc73fc84137 (at 149.165.229.195@tcp) reconnecting ./dc2oss06/logs/kern:Mar 3 12:51:02 dc2oss06 kernel: : Lustre: 14482:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003b: connection from 20fb5f9f-5fd2-532b-4e29-fbc73fc84137@149.165.229.195@tcp t0 exp ffff8817cc0c5400 cur 1393869062 last 1393869062 ./dc2oss06/logs/kern:Mar 3 12:54:45 dc2oss06 kernel: : Lustre: dc2-OST0038: Client 99b1f51a-1bfd-245b-75c9-cb96a7365778 (at 149.165.229.161@tcp) reconnecting ./dc2oss06/logs/kern:Mar 3 12:54:45 dc2oss06 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss06/logs/kern:Mar 3 12:54:45 dc2oss06 kernel: : Lustre: 16006:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0038: connection from 99b1f51a-1bfd-245b-75c9-cb96a7365778@149.165.229.161@tcp t0 exp ffff8817caaab000 cur 1393869285 last 1393869285 ./dc2oss06/logs/kern:Mar 3 12:54:45 dc2oss06 kernel: : Lustre: 16006:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss06/logs/kern:Mar 3 13:15:42 dc2oss06 kernel: : Lustre: dc2-OST0038: haven't heard from client 7097ef4b-b519-de72-4f21-b6c880acdc47 (at 149.165.230.179@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817cadc0800, cur 1393870542 expire 1393870392 last 1393870315 ./dc2oss06/logs/kern:Mar 3 13:15:42 dc2oss06 kernel: : Lustre: Skipped 9 previous similar messages ./dc2oss06/logs/kern:Mar 3 13:36:04 dc2oss06 kernel: : Lustre: 15963:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0037: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss06/logs/kern:Mar 3 13:36:04 dc2oss06 kernel: : Lustre: 15829:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003a: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss06/logs/kern:Mar 3 13:36:04 dc2oss06 kernel: : Lustre: 15829:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss06/logs/kern:Mar 3 13:36:04 dc2oss06 kernel: : Lustre: 15963:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss06/logs/kern:Mar 3 14:54:46 dc2oss06 kernel: : ib_send_cm_mra: cm_id_priv->id.state: 0x6 ./dc2oss06/logs/kern:Mar 3 14:54:46 dc2oss06 kernel: : Lustre: 15963:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0035: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2oss06/logs/kern:Mar 3 14:54:46 dc2oss06 kernel: : Lustre: 15963:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 7 previous similar messages ./dc2oss07/logs/kern:Mar 3 11:53:39 dc2oss07 kernel: : Lustre: 15819:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0042: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss07/logs/kern:Mar 3 11:53:39 dc2oss07 kernel: : Lustre: 15943:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0044: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss07/logs/kern:Mar 3 11:53:39 dc2oss07 kernel: : Lustre: 15943:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 11 previous similar messages ./dc2oss07/logs/kern:Mar 3 11:53:39 dc2oss07 kernel: : Lustre: 15920:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0040: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss07/logs/kern:Mar 3 11:53:39 dc2oss07 kernel: : Lustre: 15920:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss07/logs/kern:Mar 3 12:38:31 dc2oss07 kernel: : Lustre: dc2-OST0041: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss07/logs/kern:Mar 3 12:38:31 dc2oss07 kernel: : Lustre: 15940:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0041: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff880ccc6d1c00 cur 1393868311 last 1393868311 ./dc2oss07/logs/kern:Mar 3 12:38:31 dc2oss07 kernel: : Lustre: 15940:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 6 previous similar messages ./dc2oss07/logs/kern:Mar 3 12:38:39 dc2oss07 kernel: : LustreError: 137-5: UUID 'dc2-OST0051_UUID' is not available for connect (no target) ./dc2oss07/logs/kern:Mar 3 12:38:56 dc2oss07 kernel: : Lustre: dc2-OST0041: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss07/logs/kern:Mar 3 12:38:56 dc2oss07 kernel: : Lustre: Skipped 6 previous similar messages ./dc2oss07/logs/kern:Mar 3 12:38:56 dc2oss07 kernel: : Lustre: 14393:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0044: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff880bcbeb4000 cur 1393868336 last 1393868336 ./dc2oss07/logs/kern:Mar 3 12:38:56 dc2oss07 kernel: : Lustre: 14393:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 6 previous similar messages ./dc2oss07/logs/kern:Mar 3 12:39:30 dc2oss07 kernel: : Lustre: dc2-OST003f: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss07/logs/kern:Mar 3 12:39:30 dc2oss07 kernel: : Lustre: Skipped 2 previous similar messages ./dc2oss07/logs/kern:Mar 3 12:39:30 dc2oss07 kernel: : Lustre: 15963:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0041: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff880ccc6d1c00 cur 1393868370 last 1393868370 ./dc2oss07/logs/kern:Mar 3 12:39:30 dc2oss07 kernel: : Lustre: 15963:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 3 previous similar messages ./dc2oss07/logs/kern:Mar 3 12:54:47 dc2oss07 kernel: : Lustre: dc2-OST0047: Client 3792c245-21cd-8eb3-61bd-37df8472501c (at 149.165.229.22@tcp) reconnecting ./dc2oss07/logs/kern:Mar 3 12:54:47 dc2oss07 kernel: : Lustre: Skipped 8 previous similar messages ./dc2oss07/logs/kern:Mar 3 12:54:47 dc2oss07 kernel: : Lustre: 14418:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0047: connection from 3792c245-21cd-8eb3-61bd-37df8472501c@149.165.229.22@tcp t0 exp ffff880347471400 cur 1393869287 last 1393869287 ./dc2oss07/logs/kern:Mar 3 12:54:47 dc2oss07 kernel: : Lustre: 14418:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 8 previous similar messages ./dc2oss07/logs/kern:Mar 3 13:02:35 dc2oss07 kernel: : Lustre: 29492:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1393869748/real 1393869748] req@ffff880bc5887c00 x1456766149260668/t0(0) o105->dc2-OST0045@149.165.229.154@tcp:15/16 lens 344/192 e 0 to 1 dl 1393869755 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 ./dc2oss07/logs/kern:Mar 3 13:02:35 dc2oss07 kernel: : LustreError: 138-a: dc2-OST0045: A client on nid 149.165.229.154@tcp was evicted due to a lock completion callback time out: rc -107 ./dc2oss07/logs/kern:Mar 3 13:02:40 dc2oss07 kernel: : Lustre: 16065:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0045: connection from 4b5d9d49-df6c-109b-69ee-e20ed06ff615@149.165.229.154@tcp t0 exp (null) cur 1393869760 last 0 ./dc2oss07/logs/kern:Mar 3 13:02:40 dc2oss07 kernel: : Lustre: 16065:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss07/logs/kern:Mar 3 13:15:42 dc2oss07 kernel: : Lustre: dc2-OST0049: haven't heard from client 7097ef4b-b519-de72-4f21-b6c880acdc47 (at 149.165.230.179@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8815ec832c00, cur 1393870542 expire 1393870392 last 1393870315 ./dc2oss07/logs/kern:Mar 3 13:15:42 dc2oss07 kernel: : Lustre: Skipped 10 previous similar messages ./dc2oss07/logs/kern:Mar 3 13:36:04 dc2oss07 kernel: : Lustre: 15943:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003f: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss07/logs/kern:Mar 3 14:54:46 dc2oss07 kernel: : ib_send_cm_mra: cm_id_priv->id.state: 0x6 ./dc2oss07/logs/kern:Mar 3 14:54:46 dc2oss07 kernel: : Lustre: 15775:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0040: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2oss07/logs/kern:Mar 3 14:54:46 dc2oss07 kernel: : Lustre: 15940:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST003f: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2oss07/logs/kern:Mar 3 14:54:46 dc2oss07 kernel: : Lustre: 15940:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 10 previous similar messages ./dc2oss08/logs/kern:Mar 3 11:53:39 dc2oss08 kernel: : Lustre: 16022:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004c: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss08/logs/kern:Mar 3 11:53:39 dc2oss08 kernel: : Lustre: 16022:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 11 previous similar messages ./dc2oss08/logs/kern:Mar 3 11:53:39 dc2oss08 kernel: : Lustre: 15855:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004d: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss08/logs/kern:Mar 3 11:53:39 dc2oss08 kernel: : Lustre: 15855:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 5 previous similar messages ./dc2oss08/logs/kern:Mar 3 12:08:36 dc2oss08 kernel: : Lustre: 15999:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1393866509/real 1393866509] req@ffff880b77238000 x1456766185711953/t0(0) o106->dc2-OST004c@149.165.229.5@tcp:15/16 lens 296/232 e 0 to 1 dl 1393866516 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 ./dc2oss08/logs/kern:Mar 3 12:38:31 dc2oss08 kernel: : Lustre: dc2-OST004e: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss08/logs/kern:Mar 3 12:38:31 dc2oss08 kernel: : Lustre: dc2-OST0051: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss08/logs/kern:Mar 3 12:38:31 dc2oss08 kernel: : Lustre: dc2-OST004b: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss08/logs/kern:Mar 3 12:38:31 dc2oss08 kernel: : Lustre: Skipped 4 previous similar messages ./dc2oss08/logs/kern:Mar 3 12:38:31 dc2oss08 kernel: : Lustre: Skipped 4 previous similar messages ./dc2oss08/logs/kern:Mar 3 12:38:31 dc2oss08 kernel: : Lustre: 14516:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0051: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff881131fd7c00 cur 1393868311 last 1393868311 ./dc2oss08/logs/kern:Mar 3 12:38:31 dc2oss08 kernel: : Lustre: 14516:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 3 previous similar messages ./dc2oss08/logs/kern:Mar 3 12:38:46 dc2oss08 kernel: : Lustre: dc2-OST0051: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss08/logs/kern:Mar 3 12:38:46 dc2oss08 kernel: : Lustre: 16005:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0051: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff881131fd7c00 cur 1393868326 last 1393868326 ./dc2oss08/logs/kern:Mar 3 12:38:56 dc2oss08 kernel: : Lustre: dc2-OST004a: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss08/logs/kern:Mar 3 12:38:56 dc2oss08 kernel: : Lustre: dc2-OST004e: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss08/logs/kern:Mar 3 12:38:56 dc2oss08 kernel: : Lustre: 15992:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004e: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8817d5156c00 cur 1393868336 last 1393868336 ./dc2oss08/logs/kern:Mar 3 12:38:56 dc2oss08 kernel: : Lustre: 15992:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss08/logs/kern:Mar 3 12:54:46 dc2oss08 kernel: : Lustre: dc2-OST0052: Client d9951213-87b4-dc02-4ae5-70ac2d71d6b9 (at 149.165.230.171@tcp) reconnecting ./dc2oss08/logs/kern:Mar 3 12:54:46 dc2oss08 kernel: : Lustre: Skipped 2 previous similar messages ./dc2oss08/logs/kern:Mar 3 12:54:46 dc2oss08 kernel: : Lustre: 15864:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0051: connection from d9951213-87b4-dc02-4ae5-70ac2d71d6b9@149.165.230.171@tcp t0 exp ffff880bd5c08000 cur 1393869286 last 1393869286 ./dc2oss08/logs/kern:Mar 3 12:54:46 dc2oss08 kernel: : Lustre: 15864:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss08/logs/kern:Mar 3 12:54:46 dc2oss08 kernel: : Lustre: 15864:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0052: connection from 0d91a3ec-b287-fd61-51b6-e01c88e51e08@149.165.230.153@tcp t0 exp ffff881806bfa800 cur 1393869286 last 1393869286 ./dc2oss08/logs/kern:Mar 3 12:54:46 dc2oss08 kernel: : Lustre: 15864:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss08/logs/kern:Mar 3 12:54:46 dc2oss08 kernel: : Lustre: dc2-OST004b: Client 121eb509-1975-1ed3-c589-fa858fd094d4 (at 149.165.229.188@tcp) reconnecting ./dc2oss08/logs/kern:Mar 3 12:54:46 dc2oss08 kernel: : Lustre: Skipped 6 previous similar messages ./dc2oss08/logs/kern:Mar 3 12:54:47 dc2oss08 kernel: : Lustre: 16027:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004d: connection from 121eb509-1975-1ed3-c589-fa858fd094d4@149.165.229.188@tcp t0 exp ffff880f6b569c00 cur 1393869287 last 1393869287 ./dc2oss08/logs/kern:Mar 3 12:54:47 dc2oss08 kernel: : Lustre: 16027:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 10 previous similar messages ./dc2oss08/logs/kern:Mar 3 12:54:47 dc2oss08 kernel: : Lustre: dc2-OST004c: Client 3792c245-21cd-8eb3-61bd-37df8472501c (at 149.165.229.22@tcp) reconnecting ./dc2oss08/logs/kern:Mar 3 12:54:47 dc2oss08 kernel: : Lustre: dc2-OST0052: Client 3792c245-21cd-8eb3-61bd-37df8472501c (at 149.165.229.22@tcp) reconnecting ./dc2oss08/logs/kern:Mar 3 12:54:47 dc2oss08 kernel: : Lustre: Skipped 7 previous similar messages ./dc2oss08/logs/kern:Mar 3 12:54:48 dc2oss08 kernel: : Lustre: 15999:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0050: connection from 121eb509-1975-1ed3-c589-fa858fd094d4@149.165.229.188@tcp t0 exp ffff881308cbb800 cur 1393869288 last 1393869288 ./dc2oss08/logs/kern:Mar 3 12:54:48 dc2oss08 kernel: : Lustre: 15999:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 18 previous similar messages ./dc2oss08/logs/kern:Mar 3 12:54:51 dc2oss08 kernel: : Lustre: dc2-OST004b: Client 4017c176-bbe0-d45e-f18e-9f8e9ee2f246 (at 149.165.229.215@tcp) reconnecting ./dc2oss08/logs/kern:Mar 3 12:54:51 dc2oss08 kernel: : Lustre: Skipped 24 previous similar messages ./dc2oss08/logs/kern:Mar 3 12:54:51 dc2oss08 kernel: : Lustre: 15992:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004b: connection from 4017c176-bbe0-d45e-f18e-9f8e9ee2f246@149.165.229.215@tcp t0 exp ffff8817d0c6cc00 cur 1393869291 last 1393869291 ./dc2oss08/logs/kern:Mar 3 12:54:51 dc2oss08 kernel: : Lustre: 15992:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 8 previous similar messages ./dc2oss08/logs/kern:Mar 3 13:15:42 dc2oss08 kernel: : Lustre: dc2-OST004f: haven't heard from client 7097ef4b-b519-de72-4f21-b6c880acdc47 (at 149.165.230.179@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817d7c76400, cur 1393870542 expire 1393870392 last 1393870315 ./dc2oss08/logs/kern:Mar 3 13:15:42 dc2oss08 kernel: : Lustre: Skipped 9 previous similar messages ./dc2oss08/logs/kern:Mar 3 13:36:04 dc2oss08 kernel: : Lustre: 16002:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004b: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss08/logs/kern:Mar 3 13:36:04 dc2oss08 kernel: : Lustre: 16002:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 5 previous similar messages ./dc2oss08/logs/kern:Mar 3 14:54:46 dc2oss08 kernel: : ib_send_cm_mra: cm_id_priv->id.state: 0x6 ./dc2oss08/logs/kern:Mar 3 14:54:46 dc2oss08 kernel: : Lustre: 15961:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST004a: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2oss08/logs/kern:Mar 3 14:54:46 dc2oss08 kernel: : Lustre: 15961:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 8 previous similar messages ./dc2oss09/logs/kern:Mar 3 11:53:39 dc2oss09 kernel: : Lustre: 16127:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0055: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss09/logs/kern:Mar 3 11:53:39 dc2oss09 kernel: : Lustre: 16127:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 11 previous similar messages ./dc2oss09/logs/kern:Mar 3 12:38:31 dc2oss09 kernel: : Lustre: dc2-OST0055: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss09/logs/kern:Mar 3 12:38:31 dc2oss09 kernel: : Lustre: dc2-OST0054: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss09/logs/kern:Mar 3 12:38:31 dc2oss09 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss09/logs/kern:Mar 3 12:38:31 dc2oss09 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss09/logs/kern:Mar 3 12:38:31 dc2oss09 kernel: : Lustre: 16444:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0055: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff880846a73800 cur 1393868311 last 1393868311 ./dc2oss09/logs/kern:Mar 3 12:38:31 dc2oss09 kernel: : Lustre: 16444:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 10 previous similar messages ./dc2oss09/logs/kern:Mar 3 12:38:56 dc2oss09 kernel: : Lustre: dc2-OST0056: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss09/logs/kern:Mar 3 12:38:56 dc2oss09 kernel: : Lustre: Skipped 4 previous similar messages ./dc2oss09/logs/kern:Mar 3 12:38:56 dc2oss09 kernel: : Lustre: 16115:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0056: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff881657c5c000 cur 1393868336 last 1393868336 ./dc2oss09/logs/kern:Mar 3 12:38:56 dc2oss09 kernel: : Lustre: 16169:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0057: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8817c605dc00 cur 1393868336 last 1393868336 ./dc2oss09/logs/kern:Mar 3 12:38:56 dc2oss09 kernel: : Lustre: 16169:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 5 previous similar messages ./dc2oss09/logs/kern:Mar 3 12:38:56 dc2oss09 kernel: : Lustre: 16115:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 5 previous similar messages ./dc2oss09/logs/kern:Mar 3 12:54:47 dc2oss09 kernel: : Lustre: dc2-OST0054: Client 3792c245-21cd-8eb3-61bd-37df8472501c (at 149.165.229.22@tcp) reconnecting ./dc2oss09/logs/kern:Mar 3 12:54:47 dc2oss09 kernel: : Lustre: 14539:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005a: connection from 3792c245-21cd-8eb3-61bd-37df8472501c@149.165.229.22@tcp t0 exp ffff8812db724000 cur 1393869287 last 1393869287 ./dc2oss09/logs/kern:Mar 3 12:54:47 dc2oss09 kernel: : Lustre: 14539:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss09/logs/kern:Mar 3 12:54:47 dc2oss09 kernel: : Lustre: Skipped 9 previous similar messages ./dc2oss09/logs/kern:Mar 3 13:15:42 dc2oss09 kernel: : Lustre: dc2-OST0054: haven't heard from client 7097ef4b-b519-de72-4f21-b6c880acdc47 (at 149.165.230.179@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817c6d22c00, cur 1393870542 expire 1393870392 last 1393870315 ./dc2oss09/logs/kern:Mar 3 13:15:42 dc2oss09 kernel: : Lustre: Skipped 10 previous similar messages ./dc2oss09/logs/kern:Mar 3 13:36:04 dc2oss09 kernel: : Lustre: 16364:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0055: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss09/logs/kern:Mar 3 13:36:04 dc2oss09 kernel: : Lustre: 16364:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 5 previous similar messages ./dc2oss09/logs/kern:Mar 3 14:54:46 dc2oss09 kernel: : ib_send_cm_mra: cm_id_priv->id.state: 0x6 ./dc2oss09/logs/kern:Mar 3 14:54:46 dc2oss09 kernel: : Lustre: 14563:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0056: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2oss09/logs/kern:Mar 3 14:54:46 dc2oss09 kernel: : Lustre: 14563:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 11 previous similar messages ./dc2oss10/logs/kern:Mar 3 11:53:39 dc2oss10 kernel: : Lustre: 16289:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0061: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss10/logs/kern:Mar 3 11:53:39 dc2oss10 kernel: : Lustre: 16289:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 9 previous similar messages ./dc2oss10/logs/kern:Mar 3 11:53:39 dc2oss10 kernel: : Lustre: 15879:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005f: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss10/logs/kern:Mar 3 11:53:39 dc2oss10 kernel: : Lustre: 15879:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 6 previous similar messages ./dc2oss10/logs/kern:Mar 3 12:10:44 dc2oss10 kernel: : Lustre: dc2-OST0060: Client d0adfdce-837b-2e4e-6d85-5e798bdbf5f5 (at 149.165.229.24@tcp) reconnecting ./dc2oss10/logs/kern:Mar 3 12:10:44 dc2oss10 kernel: : Lustre: dc2-OST0060: Client d0adfdce-837b-2e4e-6d85-5e798bdbf5f5 (at 149.165.229.24@tcp) refused reconnection, still busy with 1 active RPCs ./dc2oss10/logs/kern:Mar 3 12:10:45 dc2oss10 kernel: : LustreError: 8911:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bfc14d000 x1459136123667494/t0(0) o3->d0adfdce-837b-2e4e-6d85-5e798bdbf5f5@149.165.229.24@tcp:0/0 lens 448/400 e 1 to 0 dl 1393866681 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss10/logs/kern:Mar 3 12:10:45 dc2oss10 kernel: : LustreError: 8911:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages ./dc2oss10/logs/kern:Mar 3 12:10:45 dc2oss10 kernel: : Lustre: dc2-OST0060: Bulk IO read error with d0adfdce-837b-2e4e-6d85-5e798bdbf5f5 (at 149.165.229.24@tcp), client will retry: rc -110 ./dc2oss10/logs/kern:Mar 3 12:10:45 dc2oss10 kernel: : Lustre: Skipped 2 previous similar messages ./dc2oss10/logs/kern:Mar 3 12:10:54 dc2oss10 kernel: : Lustre: dc2-OST0060: Client d0adfdce-837b-2e4e-6d85-5e798bdbf5f5 (at 149.165.229.24@tcp) reconnecting ./dc2oss10/logs/kern:Mar 3 12:10:54 dc2oss10 kernel: : Lustre: 16462:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0060: connection from d0adfdce-837b-2e4e-6d85-5e798bdbf5f5@149.165.229.24@tcp t0 exp ffff8817d278fc00 cur 1393866654 last 1393866654 ./dc2oss10/logs/kern:Mar 3 12:10:54 dc2oss10 kernel: : Lustre: 16462:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss10/logs/kern:Mar 3 12:38:18 dc2oss10 kernel: : Lustre: dc2-OST0065: Client 461777b5-a835-7c77-379b-1f12d2aa7ca3 (at 149.165.226.209@tcp) reconnecting ./dc2oss10/logs/kern:Mar 3 12:38:18 dc2oss10 kernel: : Lustre: 16357:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0065: connection from 461777b5-a835-7c77-379b-1f12d2aa7ca3@149.165.226.209@tcp t0 exp ffff8812e778cc00 cur 1393868298 last 1393868298 ./dc2oss10/logs/kern:Mar 3 12:38:23 dc2oss10 kernel: : Lustre: dc2-OST005f: Client a5f2f44d-2068-b3cd-c596-c8a17da315ad (at 149.165.229.45@tcp) reconnecting ./dc2oss10/logs/kern:Mar 3 12:38:23 dc2oss10 kernel: : Lustre: 16347:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005f: connection from a5f2f44d-2068-b3cd-c596-c8a17da315ad@149.165.229.45@tcp t0 exp ffff880bc1b21800 cur 1393868303 last 1393868303 ./dc2oss10/logs/kern:Mar 3 12:38:24 dc2oss10 kernel: : Lustre: dc2-OST0067: Client a5f2f44d-2068-b3cd-c596-c8a17da315ad (at 149.165.229.45@tcp) reconnecting ./dc2oss10/logs/kern:Mar 3 12:38:24 dc2oss10 kernel: : Lustre: 16357:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0067: connection from a5f2f44d-2068-b3cd-c596-c8a17da315ad@149.165.229.45@tcp t0 exp ffff8817d6a30800 cur 1393868304 last 1393868304 ./dc2oss10/logs/kern:Mar 3 12:38:26 dc2oss10 kernel: : Lustre: dc2-OST005f: Client fc059ef3-dd8f-085a-be7d-8f93663a6eec (at 149.165.229.111@tcp) reconnecting ./dc2oss10/logs/kern:Mar 3 12:38:26 dc2oss10 kernel: : Lustre: 16332:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005f: connection from fc059ef3-dd8f-085a-be7d-8f93663a6eec@149.165.229.111@tcp t0 exp ffff8817d5d78c00 cur 1393868306 last 1393868306 ./dc2oss10/logs/kern:Mar 3 12:38:39 dc2oss10 kernel: : Lustre: dc2-OST0062: Client a5f2f44d-2068-b3cd-c596-c8a17da315ad (at 149.165.229.45@tcp) reconnecting ./dc2oss10/logs/kern:Mar 3 12:38:39 dc2oss10 kernel: : Lustre: Skipped 9 previous similar messages ./dc2oss10/logs/kern:Mar 3 12:38:39 dc2oss10 kernel: : Lustre: 16288:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0062: connection from a5f2f44d-2068-b3cd-c596-c8a17da315ad@149.165.229.45@tcp t0 exp ffff8817cfbb8800 cur 1393868319 last 1393868319 ./dc2oss10/logs/kern:Mar 3 12:38:39 dc2oss10 kernel: : Lustre: 16288:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 9 previous similar messages ./dc2oss10/logs/kern:Mar 3 12:38:49 dc2oss10 kernel: : Lustre: dc2-OST0061: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss10/logs/kern:Mar 3 12:38:49 dc2oss10 kernel: : Lustre: Skipped 3 previous similar messages ./dc2oss10/logs/kern:Mar 3 12:38:49 dc2oss10 kernel: : Lustre: dc2-OST0061: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) refused reconnection, still busy with 6 active RPCs ./dc2oss10/logs/kern:Mar 3 12:38:49 dc2oss10 kernel: : LustreError: 16013:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8817fe832050 x1459136108324260/t0(0) o3->1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp:0/0 lens 448/400 e 1 to 0 dl 1393868402 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss10/logs/kern:Mar 3 12:38:49 dc2oss10 kernel: : Lustre: dc2-OST0061: Bulk IO read error with 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp), client will retry: rc -110 ./dc2oss10/logs/kern:Mar 3 12:38:50 dc2oss10 kernel: : LustreError: 1149:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bd01a4400 x1459136108324258/t0(0) o3->1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp:0/0 lens 448/400 e 1 to 0 dl 1393868402 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss10/logs/kern:Mar 3 12:38:50 dc2oss10 kernel: : Lustre: dc2-OST0061: Bulk IO read error with 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp), client will retry: rc -110 ./dc2oss10/logs/kern:Mar 3 12:38:58 dc2oss10 kernel: : Lustre: dc2-OST0061: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss10/logs/kern:Mar 3 12:38:58 dc2oss10 kernel: : Lustre: 16559:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0061: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff880bd6175000 cur 1393868338 last 1393868338 ./dc2oss10/logs/kern:Mar 3 12:38:58 dc2oss10 kernel: : Lustre: 16559:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 3 previous similar messages ./dc2oss10/logs/kern:Mar 3 12:39:31 dc2oss10 kernel: : Lustre: dc2-OST0060: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss10/logs/kern:Mar 3 12:39:31 dc2oss10 kernel: : Lustre: 16622:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0060: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8817c7a56000 cur 1393868371 last 1393868371 ./dc2oss10/logs/kern:Mar 3 12:39:31 dc2oss10 kernel: : Lustre: dc2-OST0061: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) refused reconnection, still busy with 4 active RPCs ./dc2oss10/logs/kern:Mar 3 12:39:31 dc2oss10 kernel: : LustreError: 15961:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8817fea3b050 x1459136108325031/t0(0) o3->1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp:0/0 lens 448/400 e 0 to 0 dl 1393868446 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss10/logs/kern:Mar 3 12:39:31 dc2oss10 kernel: : LustreError: 15961:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 4 previous similar messages ./dc2oss10/logs/kern:Mar 3 12:39:31 dc2oss10 kernel: : Lustre: dc2-OST0061: Bulk IO read error with 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp), client will retry: rc -110 ./dc2oss10/logs/kern:Mar 3 12:39:31 dc2oss10 kernel: : Lustre: Skipped 4 previous similar messages ./dc2oss10/logs/kern:Mar 3 12:40:00 dc2oss10 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 111s: evicting client at 149.165.226.209@tcp ns: filter-dc2-OST0065_UUID lock: ffff8812e3ff4480/0xef62ad91ab28b15e lrc: 3/0,0 mode: PR/PR res: 22934553/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 remote: 0x2f71511e6ffa551 expref: 761 pid: 16498 timeout 8883632107 ./dc2oss10/logs/kern:Mar 3 12:40:00 dc2oss10 kernel: : LustreError: 15761:0:(client.c:1045:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880bded33000 x1456766113065938/t0(0) o105->dc2-OST0065@149.165.226.209@tcp:15/16 lens 344/192 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1 ./dc2oss10/logs/kern:Mar 3 12:40:00 dc2oss10 kernel: : LustreError: 15761:0:(ldlm_lockd.c:656:ldlm_handle_ast_error()) ### client (nid 149.165.226.209@tcp) returned 0 from completion AST ns: filter-dc2-OST0065_UUID lock: ffff88101f365900/0xef62ad91ab28b563 lrc: 3/0,0 mode: PW/PW res: 22934553/0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->8191) flags: 0x0 remote: 0x2f71511e6ffa566 expref: 714 pid: 16363 timeout 0 ./dc2oss10/logs/kern:Mar 3 12:40:12 dc2oss10 kernel: : Lustre: 16529:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0065: connection from 461777b5-a835-7c77-379b-1f12d2aa7ca3@149.165.226.209@tcp t0 exp (null) cur 1393868412 last 0 ./dc2oss10/logs/kern:Mar 3 12:40:12 dc2oss10 kernel: : Lustre: 16529:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 4 previous similar messages ./dc2oss10/logs/kern:Mar 3 12:51:04 dc2oss10 kernel: : Lustre: dc2-OST0064: Client 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp) reconnecting ./dc2oss10/logs/kern:Mar 3 12:51:04 dc2oss10 kernel: : Lustre: Skipped 5 previous similar messages ./dc2oss10/logs/kern:Mar 3 12:51:04 dc2oss10 kernel: : Lustre: dc2-OST0064: Client 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp) refused reconnection, still busy with 1 active RPCs ./dc2oss10/logs/kern:Mar 3 12:51:05 dc2oss10 kernel: : LustreError: 8663:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880458353c00 x1459134275163263/t0(0) o3->14a5b49b-639b-b462-1562-f1e899aebd9c@149.165.229.49@tcp:0/0 lens 448/400 e 1 to 0 dl 1393869102 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss10/logs/kern:Mar 3 12:51:05 dc2oss10 kernel: : LustreError: 8663:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 3 previous similar messages ./dc2oss10/logs/kern:Mar 3 12:51:05 dc2oss10 kernel: : Lustre: dc2-OST0064: Bulk IO read error with 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp), client will retry: rc -110 ./dc2oss10/logs/kern:Mar 3 12:51:05 dc2oss10 kernel: : Lustre: Skipped 3 previous similar messages ./dc2oss10/logs/kern:Mar 3 12:51:14 dc2oss10 kernel: : Lustre: dc2-OST0064: Client 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp) reconnecting ./dc2oss10/logs/kern:Mar 3 12:51:14 dc2oss10 kernel: : Lustre: 16518:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0064: connection from 14a5b49b-639b-b462-1562-f1e899aebd9c@149.165.229.49@tcp t0 exp ffff8817d64ec800 cur 1393869074 last 1393869074 ./dc2oss10/logs/kern:Mar 3 12:54:47 dc2oss10 kernel: : Lustre: dc2-OST0065: Client 3792c245-21cd-8eb3-61bd-37df8472501c (at 149.165.229.22@tcp) reconnecting ./dc2oss10/logs/kern:Mar 3 12:54:47 dc2oss10 kernel: : Lustre: 16462:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0061: connection from 3792c245-21cd-8eb3-61bd-37df8472501c@149.165.229.22@tcp t0 exp ffff8817d1df3800 cur 1393869287 last 1393869287 ./dc2oss10/logs/kern:Mar 3 12:54:47 dc2oss10 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss10/logs/kern:Mar 3 13:15:42 dc2oss10 kernel: : Lustre: dc2-OST0068: haven't heard from client 7097ef4b-b519-de72-4f21-b6c880acdc47 (at 149.165.230.179@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ac83bc000, cur 1393870542 expire 1393870392 last 1393870315 ./dc2oss10/logs/kern:Mar 3 13:15:42 dc2oss10 kernel: : Lustre: Skipped 9 previous similar messages ./dc2oss10/logs/kern:Mar 3 13:36:05 dc2oss10 kernel: : Lustre: 16506:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0061: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871765 last 0 ./dc2oss10/logs/kern:Mar 3 13:36:05 dc2oss10 kernel: : Lustre: 16288:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0063: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871765 last 0 ./dc2oss10/logs/kern:Mar 3 13:36:05 dc2oss10 kernel: : Lustre: 16462:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005f: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871765 last 0 ./dc2oss10/logs/kern:Mar 3 13:36:05 dc2oss10 kernel: : Lustre: 16288:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss10/logs/kern:Mar 3 13:36:05 dc2oss10 kernel: : Lustre: 16462:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss10/logs/kern:Mar 3 14:54:47 dc2oss10 kernel: : ib_send_cm_mra: cm_id_priv->id.state: 0x6 ./dc2oss10/logs/kern:Mar 3 14:54:47 dc2oss10 kernel: : Lustre: 16379:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST005f: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876487 last 0 ./dc2oss10/logs/kern:Mar 3 14:54:47 dc2oss10 kernel: : Lustre: 16379:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 6 previous similar messages ./dc2oss11/logs/kern:Mar 3 11:53:39 dc2oss11 kernel: : Lustre: 16171:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0069: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss11/logs/kern:Mar 3 11:53:39 dc2oss11 kernel: : Lustre: 16171:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 10 previous similar messages ./dc2oss11/logs/kern:Mar 3 11:53:39 dc2oss11 kernel: : Lustre: 14605:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006c: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss11/logs/kern:Mar 3 11:53:39 dc2oss11 kernel: : Lustre: 14605:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 7 previous similar messages ./dc2oss11/logs/kern:Mar 3 12:10:44 dc2oss11 kernel: : Lustre: dc2-OST006d: Client 461777b5-a835-7c77-379b-1f12d2aa7ca3 (at 149.165.226.209@tcp) reconnecting ./dc2oss11/logs/kern:Mar 3 12:10:44 dc2oss11 kernel: : Lustre: Skipped 2 previous similar messages ./dc2oss11/logs/kern:Mar 3 12:10:44 dc2oss11 kernel: : Lustre: 15536:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006d: connection from 461777b5-a835-7c77-379b-1f12d2aa7ca3@149.165.226.209@tcp t0 exp ffff881306c0d400 cur 1393866644 last 1393866644 ./dc2oss11/logs/kern:Mar 3 12:10:44 dc2oss11 kernel: : Lustre: 15536:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss11/logs/kern:Mar 3 12:23:22 dc2oss11 kernel: : Lustre: dc2-OST0071: Client c1831dad-e9f7-34e7-e63b-680077105dc6 (at 149.165.229.3@tcp) reconnecting ./dc2oss11/logs/kern:Mar 3 12:23:22 dc2oss11 kernel: : Lustre: 14493:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0071: connection from c1831dad-e9f7-34e7-e63b-680077105dc6@149.165.229.3@tcp t164994809 exp ffff881798b2e000 cur 1393867402 last 1393867402 ./dc2oss11/logs/kern:Mar 3 12:25:02 dc2oss11 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 108s: evicting client at 149.165.229.3@tcp ns: filter-dc2-OST0071_UUID lock: ffff8805f310e6c0/0x309c239b379ae703 lrc: 3/0,0 mode: PW/PW res: 22258463/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x20 remote: 0xb12b585f26a87e4c expref: 804 pid: 16171 timeout 8882735678 ./dc2oss11/logs/kern:Mar 3 12:25:03 dc2oss11 kernel: : Lustre: 16238:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0071: connection from c1831dad-e9f7-34e7-e63b-680077105dc6@149.165.229.3@tcp t164994809 exp (null) cur 1393867503 last 0 ./dc2oss11/logs/kern:Mar 3 12:38:31 dc2oss11 kernel: : Lustre: dc2-OST0069: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss11/logs/kern:Mar 3 12:38:31 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss11/logs/kern:Mar 3 12:38:31 dc2oss11 kernel: : Lustre: 14552:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0070: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8817c608c000 cur 1393868311 last 1393868311 ./dc2oss11/logs/kern:Mar 3 12:38:31 dc2oss11 kernel: : Lustre: 14552:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss11/logs/kern:Mar 3 12:38:39 dc2oss11 kernel: : LustreError: 137-5: UUID 'dc2-OST0075_UUID' is not available for connect (no target) ./dc2oss11/logs/kern:Mar 3 12:39:00 dc2oss11 kernel: : Lustre: dc2-OST006d: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss11/logs/kern:Mar 3 12:39:00 dc2oss11 kernel: : Lustre: Skipped 2 previous similar messages ./dc2oss11/logs/kern:Mar 3 12:39:00 dc2oss11 kernel: : Lustre: dc2-OST006d: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) refused reconnection, still busy with 4 active RPCs ./dc2oss11/logs/kern:Mar 3 12:39:00 dc2oss11 kernel: : LustreError: 25458:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bc65f8c00 x1459136108324297/t0(0) o3->1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp:0/0 lens 448/400 e 1 to 0 dl 1393868412 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss11/logs/kern:Mar 3 12:39:00 dc2oss11 kernel: : LustreError: 25458:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 2 previous similar messages ./dc2oss11/logs/kern:Mar 3 12:39:00 dc2oss11 kernel: : Lustre: dc2-OST006d: Bulk IO read error with 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp), client will retry: rc -110 ./dc2oss11/logs/kern:Mar 3 12:39:00 dc2oss11 kernel: : Lustre: Skipped 2 previous similar messages ./dc2oss11/logs/kern:Mar 3 12:39:00 dc2oss11 kernel: : LustreError: 25449:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88097410ec00 x1459136108324322/t0(0) o3->1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp:0/0 lens 448/400 e 1 to 0 dl 1393868419 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss11/logs/kern:Mar 3 12:39:00 dc2oss11 kernel: : LustreError: 25449:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message ./dc2oss11/logs/kern:Mar 3 12:39:00 dc2oss11 kernel: : Lustre: dc2-OST006d: Bulk IO read error with 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp), client will retry: rc -110 ./dc2oss11/logs/kern:Mar 3 12:39:00 dc2oss11 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss11/logs/kern:Mar 3 12:39:11 dc2oss11 kernel: : Lustre: dc2-OST006d: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss11/logs/kern:Mar 3 12:39:11 dc2oss11 kernel: : Lustre: 15363:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006d: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff881801f9c800 cur 1393868351 last 1393868351 ./dc2oss11/logs/kern:Mar 3 12:39:11 dc2oss11 kernel: : Lustre: 15363:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss11/logs/kern:Mar 3 12:54:47 dc2oss11 kernel: : Lustre: dc2-OST006d: Client 3792c245-21cd-8eb3-61bd-37df8472501c (at 149.165.229.22@tcp) reconnecting ./dc2oss11/logs/kern:Mar 3 12:54:47 dc2oss11 kernel: : Lustre: 15359:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST006d: connection from 3792c245-21cd-8eb3-61bd-37df8472501c@149.165.229.22@tcp t0 exp ffff880b02d27400 cur 1393869287 last 1393869287 ./dc2oss11/logs/kern:Mar 3 13:15:42 dc2oss11 kernel: : Lustre: dc2-OST0072: haven't heard from client 7097ef4b-b519-de72-4f21-b6c880acdc47 (at 149.165.230.179@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881299844400, cur 1393870542 expire 1393870392 last 1393870315 ./dc2oss11/logs/kern:Mar 3 13:15:42 dc2oss11 kernel: : Lustre: Skipped 10 previous similar messages ./dc2oss11/logs/kern:Mar 3 13:36:04 dc2oss11 kernel: : Lustre: 14576:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0069: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss11/logs/kern:Mar 3 13:36:04 dc2oss11 kernel: : Lustre: 14576:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 4 previous similar messages ./dc2oss11/logs/kern:Mar 3 14:54:46 dc2oss11 kernel: : ib_send_cm_mra: cm_id_priv->id.state: 0x6 ./dc2oss11/logs/kern:Mar 3 14:54:46 dc2oss11 kernel: : Lustre: 15486:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0069: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2oss11/logs/kern:Mar 3 14:54:46 dc2oss11 kernel: : Lustre: 15486:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 10 previous similar messages ./dc2oss12/logs/kern:Mar 3 11:53:39 dc2oss12 kernel: : Lustre: 14536:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0075: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss12/logs/kern:Mar 3 11:53:39 dc2oss12 kernel: : Lustre: 15336:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0076: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss12/logs/kern:Mar 3 11:53:39 dc2oss12 kernel: : Lustre: 14517:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0074: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss12/logs/kern:Mar 3 11:53:39 dc2oss12 kernel: : Lustre: 15336:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 7 previous similar messages ./dc2oss12/logs/kern:Mar 3 11:53:39 dc2oss12 kernel: : Lustre: 14517:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 7 previous similar messages ./dc2oss12/logs/kern:Mar 3 11:53:39 dc2oss12 kernel: : Lustre: 14536:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007b: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss12/logs/kern:Mar 3 11:53:39 dc2oss12 kernel: : Lustre: 14536:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 4 previous similar messages ./dc2oss12/logs/kern:Mar 3 12:33:57 dc2oss12 kernel: : Lustre: dc2-OST0079: Client 20fb5f9f-5fd2-532b-4e29-fbc73fc84137 (at 149.165.229.195@tcp) reconnecting ./dc2oss12/logs/kern:Mar 3 12:33:57 dc2oss12 kernel: : Lustre: 14512:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0076: connection from 20fb5f9f-5fd2-532b-4e29-fbc73fc84137@149.165.229.195@tcp t0 exp ffff8817d2e9c800 cur 1393868037 last 1393868037 ./dc2oss12/logs/kern:Mar 3 12:33:57 dc2oss12 kernel: : Lustre: Skipped 2 previous similar messages ./dc2oss12/logs/kern:Mar 3 12:33:57 dc2oss12 kernel: : Lustre: 14512:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss12/logs/kern:Mar 3 12:38:31 dc2oss12 kernel: : Lustre: dc2-OST0075: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss12/logs/kern:Mar 3 12:38:31 dc2oss12 kernel: : Lustre: dc2-OST0079: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss12/logs/kern:Mar 3 12:38:31 dc2oss12 kernel: : Lustre: 14536:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0079: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff880bcf9c9800 cur 1393868311 last 1393868311 ./dc2oss12/logs/kern:Mar 3 12:38:31 dc2oss12 kernel: : Lustre: 14536:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss12/logs/kern:Mar 3 12:38:46 dc2oss12 kernel: : Lustre: dc2-OST0075: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss12/logs/kern:Mar 3 12:38:46 dc2oss12 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss12/logs/kern:Mar 3 12:38:46 dc2oss12 kernel: : Lustre: 16034:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0075: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8817d288dc00 cur 1393868326 last 1393868326 ./dc2oss12/logs/kern:Mar 3 12:38:46 dc2oss12 kernel: : Lustre: 16034:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss12/logs/kern:Mar 3 12:38:56 dc2oss12 kernel: : Lustre: dc2-OST007d: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss12/logs/kern:Mar 3 12:38:56 dc2oss12 kernel: : Lustre: 16043:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007d: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff880bd0706400 cur 1393868336 last 1393868336 ./dc2oss12/logs/kern:Mar 3 12:39:30 dc2oss12 kernel: : Lustre: dc2-OST0074: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss12/logs/kern:Mar 3 12:39:30 dc2oss12 kernel: : Lustre: Skipped 2 previous similar messages ./dc2oss12/logs/kern:Mar 3 12:39:30 dc2oss12 kernel: : Lustre: 14453:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0079: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff880bcf9c9800 cur 1393868370 last 1393868370 ./dc2oss12/logs/kern:Mar 3 12:39:30 dc2oss12 kernel: : Lustre: 14453:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss12/logs/kern:Mar 3 12:51:04 dc2oss12 kernel: : Lustre: dc2-OST0074: Client 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp) reconnecting ./dc2oss12/logs/kern:Mar 3 12:51:04 dc2oss12 kernel: : Lustre: Skipped 2 previous similar messages ./dc2oss12/logs/kern:Mar 3 12:51:04 dc2oss12 kernel: : Lustre: dc2-OST0074: Client 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp) refused reconnection, still busy with 2 active RPCs ./dc2oss12/logs/kern:Mar 3 12:51:05 dc2oss12 kernel: : LustreError: 32734:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8811a84bb850 x1459134275163255/t0(0) o3->14a5b49b-639b-b462-1562-f1e899aebd9c@149.165.229.49@tcp:0/0 lens 448/400 e 1 to 0 dl 1393869100 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss12/logs/kern:Mar 3 12:51:05 dc2oss12 kernel: : Lustre: dc2-OST0074: Bulk IO read error with 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp), client will retry: rc -110 ./dc2oss12/logs/kern:Mar 3 12:51:05 dc2oss12 kernel: : LustreError: 9267:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880c0010a050 x1459134275163266/t0(0) o3->14a5b49b-639b-b462-1562-f1e899aebd9c@149.165.229.49@tcp:0/0 lens 448/400 e 1 to 0 dl 1393869100 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss12/logs/kern:Mar 3 12:51:05 dc2oss12 kernel: : Lustre: dc2-OST0074: Bulk IO read error with 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp), client will retry: rc -110 ./dc2oss12/logs/kern:Mar 3 12:51:15 dc2oss12 kernel: : Lustre: dc2-OST0074: Client 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp) reconnecting ./dc2oss12/logs/kern:Mar 3 12:51:15 dc2oss12 kernel: : Lustre: 16001:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0074: connection from 14a5b49b-639b-b462-1562-f1e899aebd9c@149.165.229.49@tcp t0 exp ffff880bc0fe0c00 cur 1393869075 last 1393869075 ./dc2oss12/logs/kern:Mar 3 12:51:15 dc2oss12 kernel: : Lustre: 16001:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 3 previous similar messages ./dc2oss12/logs/kern:Mar 3 12:54:47 dc2oss12 kernel: : Lustre: dc2-OST0074: Client 3792c245-21cd-8eb3-61bd-37df8472501c (at 149.165.229.22@tcp) reconnecting ./dc2oss12/logs/kern:Mar 3 12:54:47 dc2oss12 kernel: : Lustre: 14469:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007c: connection from 3792c245-21cd-8eb3-61bd-37df8472501c@149.165.229.22@tcp t0 exp ffff880fadf8a800 cur 1393869287 last 1393869287 ./dc2oss12/logs/kern:Mar 3 12:54:47 dc2oss12 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss12/logs/kern:Mar 3 12:56:09 dc2oss12 kernel: : Lustre: dc2-OST0078: Client c1831dad-e9f7-34e7-e63b-680077105dc6 (at 149.165.229.3@tcp) reconnecting ./dc2oss12/logs/kern:Mar 3 12:56:09 dc2oss12 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss12/logs/kern:Mar 3 12:56:09 dc2oss12 kernel: : Lustre: 15389:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0078: connection from c1831dad-e9f7-34e7-e63b-680077105dc6@149.165.229.3@tcp t0 exp ffff8817d279cc00 cur 1393869369 last 1393869369 ./dc2oss12/logs/kern:Mar 3 12:56:09 dc2oss12 kernel: : Lustre: 15389:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss12/logs/kern:Mar 3 13:15:41 dc2oss12 kernel: : Lustre: dc2-OST007b: haven't heard from client 7097ef4b-b519-de72-4f21-b6c880acdc47 (at 149.165.230.179@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817c0736800, cur 1393870541 expire 1393870391 last 1393870314 ./dc2oss12/logs/kern:Mar 3 13:15:41 dc2oss12 kernel: : Lustre: Skipped 9 previous similar messages ./dc2oss12/logs/kern:Mar 3 13:36:04 dc2oss12 kernel: : Lustre: 15307:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0075: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss12/logs/kern:Mar 3 13:36:04 dc2oss12 kernel: : Lustre: 15307:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss12/logs/kern:Mar 3 14:35:27 dc2oss12 kernel: : Lustre: dc2-OST007c: Client d67cf4e1-6b8e-d7a0-a53b-a82a2a7b2cc0 (at 149.165.229.5@tcp) reconnecting ./dc2oss12/logs/kern:Mar 3 14:35:27 dc2oss12 kernel: : Lustre: 16150:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007c: connection from d67cf4e1-6b8e-d7a0-a53b-a82a2a7b2cc0@149.165.229.5@tcp t0 exp ffff8817cfc40400 cur 1393875327 last 1393875327 ./dc2oss12/logs/kern:Mar 3 14:35:27 dc2oss12 kernel: : Lustre: 16150:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 7 previous similar messages ./dc2oss12/logs/kern:Mar 3 14:54:46 dc2oss12 kernel: : ib_send_cm_mra: cm_id_priv->id.state: 0x6 ./dc2oss12/logs/kern:Mar 3 14:54:46 dc2oss12 kernel: : Lustre: 14453:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0074: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2oss13/logs/kern:Mar 3 11:53:39 dc2oss13 kernel: : Lustre: 15825:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0082: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss13/logs/kern:Mar 3 11:53:39 dc2oss13 kernel: : Lustre: 15825:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 11 previous similar messages ./dc2oss13/logs/kern:Mar 3 11:53:39 dc2oss13 kernel: : Lustre: 15825:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007f: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss13/logs/kern:Mar 3 11:53:39 dc2oss13 kernel: : Lustre: 15825:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 4 previous similar messages ./dc2oss13/logs/kern:Mar 3 12:05:38 dc2oss13 kernel: : Lustre: dc2-OST0081: Client f0ce86c8-d1d3-0c34-94f7-9f786292e7d1 (at 149.165.230.208@tcp) reconnecting ./dc2oss13/logs/kern:Mar 3 12:05:38 dc2oss13 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss13/logs/kern:Mar 3 12:05:38 dc2oss13 kernel: : Lustre: 14447:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0081: connection from f0ce86c8-d1d3-0c34-94f7-9f786292e7d1@149.165.230.208@tcp t0 exp ffff88100c98a800 cur 1393866338 last 1393866338 ./dc2oss13/logs/kern:Mar 3 12:05:38 dc2oss13 kernel: : Lustre: 14447:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 4 previous similar messages ./dc2oss13/logs/kern:Mar 3 12:05:38 dc2oss13 kernel: : Lustre: dc2-OST0082: Client f0ce86c8-d1d3-0c34-94f7-9f786292e7d1 (at 149.165.230.208@tcp) reconnecting ./dc2oss13/logs/kern:Mar 3 12:05:38 dc2oss13 kernel: : Lustre: 14469:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0082: connection from f0ce86c8-d1d3-0c34-94f7-9f786292e7d1@149.165.230.208@tcp t0 exp ffff88104b6fb800 cur 1393866338 last 1393866338 ./dc2oss13/logs/kern:Mar 3 12:32:25 dc2oss13 kernel: : Lustre: dc2-OST007e: Client f0ce86c8-d1d3-0c34-94f7-9f786292e7d1 (at 149.165.230.208@tcp) reconnecting ./dc2oss13/logs/kern:Mar 3 12:32:25 dc2oss13 kernel: : Lustre: 14469:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007e: connection from f0ce86c8-d1d3-0c34-94f7-9f786292e7d1@149.165.230.208@tcp t0 exp ffff880d70612c00 cur 1393867945 last 1393867945 ./dc2oss13/logs/kern:Mar 3 12:32:25 dc2oss13 kernel: : Lustre: dc2-OST0081: Client f0ce86c8-d1d3-0c34-94f7-9f786292e7d1 (at 149.165.230.208@tcp) reconnecting ./dc2oss13/logs/kern:Mar 3 12:32:25 dc2oss13 kernel: : Lustre: 16001:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0081: connection from f0ce86c8-d1d3-0c34-94f7-9f786292e7d1@149.165.230.208@tcp t0 exp ffff88100c98a800 cur 1393867945 last 1393867945 ./dc2oss13/logs/kern:Mar 3 12:34:30 dc2oss13 kernel: : Lustre: dc2-OST007f: Client c1831dad-e9f7-34e7-e63b-680077105dc6 (at 149.165.229.3@tcp) reconnecting ./dc2oss13/logs/kern:Mar 3 12:34:30 dc2oss13 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss13/logs/kern:Mar 3 12:34:30 dc2oss13 kernel: : Lustre: 14489:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007f: connection from c1831dad-e9f7-34e7-e63b-680077105dc6@149.165.229.3@tcp t0 exp ffff8817c722bc00 cur 1393868070 last 1393868070 ./dc2oss13/logs/kern:Mar 3 12:34:30 dc2oss13 kernel: : Lustre: 14489:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss13/logs/kern:Mar 3 12:36:10 dc2oss13 kernel: : LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 108s: evicting client at 149.165.229.3@tcp ns: filter-dc2-OST007f_UUID lock: ffff8817ea332900/0x5185b69bb11de900 lrc: 3/0,0 mode: PW/PW res: 22603950/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x20 remote: 0xb12b585f26ad0abb expref: 805 pid: 14489 timeout 8883386421 ./dc2oss13/logs/kern:Mar 3 12:36:18 dc2oss13 kernel: : Lustre: 15863:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007f: connection from c1831dad-e9f7-34e7-e63b-680077105dc6@149.165.229.3@tcp t0 exp (null) cur 1393868178 last 0 ./dc2oss13/logs/kern:Mar 3 12:38:31 dc2oss13 kernel: : Lustre: dc2-OST0085: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss13/logs/kern:Mar 3 12:38:31 dc2oss13 kernel: : Lustre: dc2-OST0085: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) refused reconnection, still busy with 6 active RPCs ./dc2oss13/logs/kern:Mar 3 12:38:31 dc2oss13 kernel: : LustreError: 6399:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8803f19a0850 x1459136108324239/t0(0) o3->1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp:0/0 lens 448/400 e 1 to 0 dl 1393868363 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss13/logs/kern:Mar 3 12:38:31 dc2oss13 kernel: : LustreError: 6399:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message ./dc2oss13/logs/kern:Mar 3 12:38:31 dc2oss13 kernel: : Lustre: dc2-OST0085: Bulk IO read error with 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp), client will retry: rc -110 ./dc2oss13/logs/kern:Mar 3 12:38:31 dc2oss13 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss13/logs/kern:Mar 3 12:38:31 dc2oss13 kernel: : LustreError: 6471:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff8803b6627050 x1459136108324241/t0(0) o3->1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp:0/0 lens 448/400 e 1 to 0 dl 1393868363 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss13/logs/kern:Mar 3 12:38:31 dc2oss13 kernel: : Lustre: dc2-OST0085: Bulk IO read error with 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp), client will retry: rc -110 ./dc2oss13/logs/kern:Mar 3 12:38:32 dc2oss13 kernel: : LustreError: 24884:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bc4086400 x1459136108324233/t0(0) o3->1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp:0/0 lens 448/400 e 1 to 0 dl 1393868363 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss13/logs/kern:Mar 3 12:38:32 dc2oss13 kernel: : LustreError: 24884:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 3 previous similar messages ./dc2oss13/logs/kern:Mar 3 12:38:32 dc2oss13 kernel: : Lustre: dc2-OST0085: Bulk IO read error with 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp), client will retry: rc -110 ./dc2oss13/logs/kern:Mar 3 12:38:32 dc2oss13 kernel: : Lustre: Skipped 3 previous similar messages ./dc2oss13/logs/kern:Mar 3 12:38:42 dc2oss13 kernel: : Lustre: dc2-OST0085: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss13/logs/kern:Mar 3 12:38:42 dc2oss13 kernel: : Lustre: 16116:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0085: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff880d3c164400 cur 1393868322 last 1393868322 ./dc2oss13/logs/kern:Mar 3 12:38:43 dc2oss13 kernel: : LustreError: 137-5: UUID 'dc2-OST0090_UUID' is not available for connect (no target) ./dc2oss13/logs/kern:Mar 3 12:39:30 dc2oss13 kernel: : Lustre: dc2-OST0085: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss13/logs/kern:Mar 3 12:39:30 dc2oss13 kernel: : Lustre: dc2-OST0085: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) refused reconnection, still busy with 1 active RPCs ./dc2oss13/logs/kern:Mar 3 12:39:30 dc2oss13 kernel: : LustreError: 14626:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bfa1bd850 x1459136108324941/t0(0) o3->1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp:0/0 lens 448/400 e 0 to 0 dl 1393868403 ref 1 fl Interpret:/2/0 rc 0/0 ./dc2oss13/logs/kern:Mar 3 12:39:30 dc2oss13 kernel: : Lustre: dc2-OST0085: Bulk IO read error with 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp), client will retry: rc -110 ./dc2oss13/logs/kern:Mar 3 12:39:30 dc2oss13 kernel: : Lustre: 15912:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0080: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8817c722b400 cur 1393868370 last 1393868369 ./dc2oss13/logs/kern:Mar 3 12:39:40 dc2oss13 kernel: : Lustre: dc2-OST0085: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss13/logs/kern:Mar 3 12:39:40 dc2oss13 kernel: : Lustre: Skipped 4 previous similar messages ./dc2oss13/logs/kern:Mar 3 12:39:40 dc2oss13 kernel: : Lustre: 15879:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0085: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff880d3c164400 cur 1393868380 last 1393868380 ./dc2oss13/logs/kern:Mar 3 12:39:40 dc2oss13 kernel: : Lustre: 15879:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 3 previous similar messages ./dc2oss13/logs/kern:Mar 3 12:50:52 dc2oss13 kernel: : Lustre: dc2-OST0080: Client 1c39cdb9-ff82-76d3-8850-5d884d059792 (at 149.165.229.2@tcp) reconnecting ./dc2oss13/logs/kern:Mar 3 12:50:52 dc2oss13 kernel: : Lustre: 15808:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0080: connection from 1c39cdb9-ff82-76d3-8850-5d884d059792@149.165.229.2@tcp t0 exp ffff8817cc577400 cur 1393869052 last 1393869052 ./dc2oss13/logs/kern:Mar 3 12:51:00 dc2oss13 kernel: : Lustre: dc2-OST007e: Client bbd48cd8-fc59-b191-1012-74b80818fb73 (at 149.165.229.162@tcp) reconnecting ./dc2oss13/logs/kern:Mar 3 12:51:00 dc2oss13 kernel: : Lustre: 16003:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007e: connection from bbd48cd8-fc59-b191-1012-74b80818fb73@149.165.229.162@tcp t0 exp ffff8817cad64400 cur 1393869060 last 1393869060 ./dc2oss13/logs/kern:Mar 3 12:51:04 dc2oss13 kernel: : Lustre: dc2-OST007e: Client 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp) refused reconnection, still busy with 4 active RPCs ./dc2oss13/logs/kern:Mar 3 12:51:04 dc2oss13 kernel: : LustreError: 6418:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bce0fb400 x1459134275163355/t0(0) o3->14a5b49b-639b-b462-1562-f1e899aebd9c@149.165.229.49@tcp:0/0 lens 448/400 e 1 to 0 dl 1393869102 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss13/logs/kern:Mar 3 12:51:04 dc2oss13 kernel: : Lustre: dc2-OST007e: Bulk IO read error with 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp), client will retry: rc -110 ./dc2oss13/logs/kern:Mar 3 12:51:05 dc2oss13 kernel: : LustreError: 6426:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880085f07000 x1459134275163241/t0(0) o3->14a5b49b-639b-b462-1562-f1e899aebd9c@149.165.229.49@tcp:0/0 lens 448/400 e 1 to 0 dl 1393869100 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss13/logs/kern:Mar 3 12:51:05 dc2oss13 kernel: : LustreError: 6426:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message ./dc2oss13/logs/kern:Mar 3 12:51:05 dc2oss13 kernel: : Lustre: dc2-OST007e: Bulk IO read error with 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp), client will retry: rc -110 ./dc2oss13/logs/kern:Mar 3 12:51:05 dc2oss13 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss13/logs/kern:Mar 3 12:51:15 dc2oss13 kernel: : Lustre: dc2-OST007e: Client 14a5b49b-639b-b462-1562-f1e899aebd9c (at 149.165.229.49@tcp) reconnecting ./dc2oss13/logs/kern:Mar 3 12:51:15 dc2oss13 kernel: : Lustre: Skipped 5 previous similar messages ./dc2oss13/logs/kern:Mar 3 12:51:15 dc2oss13 kernel: : Lustre: 14474:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007e: connection from 14a5b49b-639b-b462-1562-f1e899aebd9c@149.165.229.49@tcp t0 exp ffff8817cb920400 cur 1393869075 last 1393869075 ./dc2oss13/logs/kern:Mar 3 12:51:15 dc2oss13 kernel: : Lustre: 14474:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 4 previous similar messages ./dc2oss13/logs/kern:Mar 3 13:15:42 dc2oss13 kernel: : Lustre: dc2-OST007e: haven't heard from client 7097ef4b-b519-de72-4f21-b6c880acdc47 (at 149.165.230.179@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817c97a6400, cur 1393870542 expire 1393870392 last 1393870315 ./dc2oss13/logs/kern:Mar 3 13:15:42 dc2oss13 kernel: : Lustre: Skipped 10 previous similar messages ./dc2oss13/logs/kern:Mar 3 13:36:04 dc2oss13 kernel: : Lustre: 15944:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007e: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss13/logs/kern:Mar 3 13:36:04 dc2oss13 kernel: : Lustre: 15912:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007f: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss13/logs/kern:Mar 3 14:54:46 dc2oss13 kernel: : ib_send_cm_mra: cm_id_priv->id.state: 0x6 ./dc2oss13/logs/kern:Mar 3 14:54:46 dc2oss13 kernel: : Lustre: 15863:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST007e: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2oss13/logs/kern:Mar 3 14:54:46 dc2oss13 kernel: : Lustre: 15863:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 12 previous similar messages ./dc2oss14/logs/kern:Mar 3 11:53:39 dc2oss14 kernel: : Lustre: 14491:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0089: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss14/logs/kern:Mar 3 11:53:39 dc2oss14 kernel: : Lustre: 15470:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008a: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss14/logs/kern:Mar 3 11:53:39 dc2oss14 kernel: : Lustre: 15470:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 9 previous similar messages ./dc2oss14/logs/kern:Mar 3 11:53:39 dc2oss14 kernel: : Lustre: 15785:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008c: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss14/logs/kern:Mar 3 11:53:39 dc2oss14 kernel: : Lustre: 15785:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss14/logs/kern:Mar 3 12:38:31 dc2oss14 kernel: : Lustre: dc2-OST008a: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss14/logs/kern:Mar 3 12:38:31 dc2oss14 kernel: : Lustre: 15537:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008a: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff880e64604400 cur 1393868311 last 1393868311 ./dc2oss14/logs/kern:Mar 3 12:38:31 dc2oss14 kernel: : Lustre: 15537:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 6 previous similar messages ./dc2oss14/logs/kern:Mar 3 12:38:35 dc2oss14 kernel: : Lustre: dc2-OST008d: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss14/logs/kern:Mar 3 12:38:35 dc2oss14 kernel: : Lustre: 15599:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0090: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8810089fe000 cur 1393868315 last 1393868315 ./dc2oss14/logs/kern:Mar 3 12:38:35 dc2oss14 kernel: : Lustre: 15599:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss14/logs/kern:Mar 3 12:38:35 dc2oss14 kernel: : Lustre: Skipped 2 previous similar messages ./dc2oss14/logs/kern:Mar 3 12:38:52 dc2oss14 kernel: : Lustre: dc2-OST0090: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss14/logs/kern:Mar 3 12:38:52 dc2oss14 kernel: : Lustre: 14477:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0090: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8810089fe000 cur 1393868332 last 1393868332 ./dc2oss14/logs/kern:Mar 3 12:38:52 dc2oss14 kernel: : Lustre: 14477:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss14/logs/kern:Mar 3 12:38:56 dc2oss14 kernel: : Lustre: dc2-OST008c: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss14/logs/kern:Mar 3 12:38:56 dc2oss14 kernel: : Lustre: 15470:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008c: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff880439d50800 cur 1393868336 last 1393868336 ./dc2oss14/logs/kern:Mar 3 12:38:56 dc2oss14 kernel: : Lustre: 15470:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss14/logs/kern:Mar 3 12:52:29 dc2oss14 kernel: : Lustre: dc2-OST0089: Client 3f75637b-a005-8cfb-23bb-0365e0b2877d (at 149.165.229.98@tcp) reconnecting ./dc2oss14/logs/kern:Mar 3 12:52:29 dc2oss14 kernel: : Lustre: Skipped 2 previous similar messages ./dc2oss14/logs/kern:Mar 3 12:52:29 dc2oss14 kernel: : Lustre: 15539:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0089: connection from 3f75637b-a005-8cfb-23bb-0365e0b2877d@149.165.229.98@tcp t0 exp ffff8817e19e1800 cur 1393869149 last 1393869149 ./dc2oss14/logs/kern:Mar 3 12:52:29 dc2oss14 kernel: : Lustre: 15539:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1 previous similar message ./dc2oss14/logs/kern:Mar 3 12:52:29 dc2oss14 kernel: : Lustre: dc2-OST008a: Client 3f75637b-a005-8cfb-23bb-0365e0b2877d (at 149.165.229.98@tcp) reconnecting ./dc2oss14/logs/kern:Mar 3 12:52:29 dc2oss14 kernel: : Lustre: 15659:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008a: connection from 3f75637b-a005-8cfb-23bb-0365e0b2877d@149.165.229.98@tcp t0 exp ffff8817dcf3c800 cur 1393869149 last 1393869149 ./dc2oss14/logs/kern:Mar 3 12:52:30 dc2oss14 kernel: : Lustre: dc2-OST008b: Client 74381b2c-e0d5-dde0-7d0c-1d9e6ce37f7f (at 149.165.229.93@tcp) reconnecting ./dc2oss14/logs/kern:Mar 3 12:52:30 dc2oss14 kernel: : Lustre: Skipped 10 previous similar messages ./dc2oss14/logs/kern:Mar 3 12:52:30 dc2oss14 kernel: : Lustre: 15672:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008b: connection from 74381b2c-e0d5-dde0-7d0c-1d9e6ce37f7f@149.165.229.93@tcp t0 exp ffff880bcca77000 cur 1393869150 last 1393869150 ./dc2oss14/logs/kern:Mar 3 12:52:30 dc2oss14 kernel: : Lustre: 15672:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 10 previous similar messages ./dc2oss14/logs/kern:Mar 3 12:52:31 dc2oss14 kernel: : Lustre: dc2-OST0089: Client 1fdf0f39-4605-afa9-0db1-9c1f651ea5a3 (at 149.165.229.100@tcp) reconnecting ./dc2oss14/logs/kern:Mar 3 12:52:31 dc2oss14 kernel: : Lustre: Skipped 6 previous similar messages ./dc2oss14/logs/kern:Mar 3 12:52:31 dc2oss14 kernel: : Lustre: 15718:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0089: connection from 1fdf0f39-4605-afa9-0db1-9c1f651ea5a3@149.165.229.100@tcp t0 exp ffff8817da1e0000 cur 1393869151 last 1393869151 ./dc2oss14/logs/kern:Mar 3 12:52:31 dc2oss14 kernel: : Lustre: 15718:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 6 previous similar messages ./dc2oss14/logs/kern:Mar 3 13:15:42 dc2oss14 kernel: : Lustre: dc2-OST008c: haven't heard from client 7097ef4b-b519-de72-4f21-b6c880acdc47 (at 149.165.230.179@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817aea07000, cur 1393870542 expire 1393870392 last 1393870315 ./dc2oss14/logs/kern:Mar 3 13:15:42 dc2oss14 kernel: : Lustre: Skipped 9 previous similar messages ./dc2oss14/logs/kern:Mar 3 13:36:04 dc2oss14 kernel: : Lustre: 14592:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST008b: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss14/logs/kern:Mar 3 13:36:04 dc2oss14 kernel: : Lustre: 14592:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 6 previous similar messages ./dc2oss14/logs/kern:Mar 3 14:54:46 dc2oss14 kernel: : ib_send_cm_mra: cm_id_priv->id.state: 0x6 ./dc2oss14/logs/kern:Mar 3 14:54:46 dc2oss14 kernel: : Lustre: 15834:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0089: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2oss14/logs/kern:Mar 3 14:54:46 dc2oss14 kernel: : Lustre: 15834:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 9 previous similar messages ./dc2oss15/logs/kern:Mar 3 11:53:39 dc2oss15 kernel: : Lustre: 15697:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0094: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss15/logs/kern:Mar 3 11:53:39 dc2oss15 kernel: : Lustre: 15697:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 12 previous similar messages ./dc2oss15/logs/kern:Mar 3 12:38:22 dc2oss15 kernel: : Lustre: dc2-OST0094: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss15/logs/kern:Mar 3 12:38:22 dc2oss15 kernel: : Lustre: dc2-OST0094: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) refused reconnection, still busy with 4 active RPCs ./dc2oss15/logs/kern:Mar 3 12:38:23 dc2oss15 kernel: : LustreError: 8414:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bbd271000 x1459136108324281/t0(0) o3->1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp:0/0 lens 448/400 e 1 to 0 dl 1393868342 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss15/logs/kern:Mar 3 12:38:23 dc2oss15 kernel: : Lustre: dc2-OST0094: Bulk IO read error with 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp), client will retry: rc -110 ./dc2oss15/logs/kern:Mar 3 12:38:23 dc2oss15 kernel: : LustreError: 8591:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880bcd9ee000 x1459136108324252/t0(0) o3->1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp:0/0 lens 448/400 e 1 to 0 dl 1393868338 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss15/logs/kern:Mar 3 12:38:23 dc2oss15 kernel: : LustreError: 8591:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message ./dc2oss15/logs/kern:Mar 3 12:38:23 dc2oss15 kernel: : Lustre: dc2-OST0094: Bulk IO read error with 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp), client will retry: rc -110 ./dc2oss15/logs/kern:Mar 3 12:38:23 dc2oss15 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss15/logs/kern:Mar 3 12:38:32 dc2oss15 kernel: : Lustre: dc2-OST0094: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss15/logs/kern:Mar 3 12:38:32 dc2oss15 kernel: : Lustre: 14471:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0094: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8817cd757c00 cur 1393868312 last 1393868312 ./dc2oss15/logs/kern:Mar 3 12:38:32 dc2oss15 kernel: : Lustre: 14471:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 8 previous similar messages ./dc2oss15/logs/kern:Mar 3 12:39:30 dc2oss15 kernel: : Lustre: dc2-OST0095: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss15/logs/kern:Mar 3 12:39:30 dc2oss15 kernel: : Lustre: 15830:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0095: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8817cc6fe800 cur 1393868370 last 1393868369 ./dc2oss15/logs/kern:Mar 3 12:41:03 dc2oss15 kernel: : Lustre: dc2-OST0094: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss15/logs/kern:Mar 3 12:41:03 dc2oss15 kernel: : Lustre: Skipped 2 previous similar messages ./dc2oss15/logs/kern:Mar 3 12:41:03 dc2oss15 kernel: : Lustre: dc2-OST0094: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) refused reconnection, still busy with 3 active RPCs ./dc2oss15/logs/kern:Mar 3 12:41:03 dc2oss15 kernel: : LustreError: 29037:0:(ldlm_lib.c:2674:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff880be8f43800 x1459136108324932/t0(0) o3->1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp:0/0 lens 448/400 e 1 to 0 dl 1393868473 ref 1 fl Interpret:/0/0 rc 0/0 ./dc2oss15/logs/kern:Mar 3 12:41:03 dc2oss15 kernel: : LustreError: 29037:0:(ldlm_lib.c:2674:target_bulk_io()) Skipped 1 previous similar message ./dc2oss15/logs/kern:Mar 3 12:41:03 dc2oss15 kernel: : Lustre: dc2-OST0094: Bulk IO read error with 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp), client will retry: rc -110 ./dc2oss15/logs/kern:Mar 3 12:41:03 dc2oss15 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss15/logs/kern:Mar 3 12:41:12 dc2oss15 kernel: : Lustre: dc2-OST0094: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss15/logs/kern:Mar 3 12:41:12 dc2oss15 kernel: : Lustre: 15727:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0094: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8817cd757c00 cur 1393868472 last 1393868472 ./dc2oss15/logs/kern:Mar 3 12:41:12 dc2oss15 kernel: : Lustre: 15727:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss15/logs/kern:Mar 3 13:15:42 dc2oss15 kernel: : Lustre: dc2-OST0097: haven't heard from client 7097ef4b-b519-de72-4f21-b6c880acdc47 (at 149.165.230.179@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880240d5d800, cur 1393870542 expire 1393870392 last 1393870315 ./dc2oss15/logs/kern:Mar 3 13:15:42 dc2oss15 kernel: : Lustre: Skipped 10 previous similar messages ./dc2oss15/logs/kern:Mar 3 13:36:04 dc2oss15 kernel: : Lustre: 15849:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0093: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss15/logs/kern:Mar 3 14:54:46 dc2oss15 kernel: : ib_send_cm_mra: cm_id_priv->id.state: 0x6 ./dc2oss15/logs/kern:Mar 3 14:54:46 dc2oss15 kernel: : Lustre: 15738:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0093: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2oss15/logs/kern:Mar 3 14:54:46 dc2oss15 kernel: : Lustre: 15650:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST0094: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2oss15/logs/kern:Mar 3 14:54:46 dc2oss15 kernel: : Lustre: 15650:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 12 previous similar messages ./dc2oss16/logs/kern:Mar 3 11:53:39 dc2oss16 kernel: : Lustre: 17319:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a0: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss16/logs/kern:Mar 3 11:53:39 dc2oss16 kernel: : Lustre: 17319:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 9 previous similar messages ./dc2oss16/logs/kern:Mar 3 11:53:39 dc2oss16 kernel: : Lustre: 17158:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a4: connection from a69ef170-baaf-a6ea-059d-2e445759c567@149.165.235.42@tcp t0 exp (null) cur 1393865619 last 0 ./dc2oss16/logs/kern:Mar 3 11:53:39 dc2oss16 kernel: : Lustre: 17158:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 5 previous similar messages ./dc2oss16/logs/kern:Mar 3 12:38:31 dc2oss16 kernel: : Lustre: dc2-OST009e: Client 6cf88260-fe65-1a9d-cec8-f066cce5bed7 (at 149.165.230.149@tcp) reconnecting ./dc2oss16/logs/kern:Mar 3 12:38:31 dc2oss16 kernel: : Lustre: Skipped 1 previous similar message ./dc2oss16/logs/kern:Mar 3 12:38:31 dc2oss16 kernel: : Lustre: 15806:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009e: connection from 6cf88260-fe65-1a9d-cec8-f066cce5bed7@149.165.230.149@tcp t0 exp ffff880bd045a800 cur 1393868311 last 1393868311 ./dc2oss16/logs/kern:Mar 3 12:38:31 dc2oss16 kernel: : Lustre: 15806:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 3 previous similar messages ./dc2oss16/logs/kern:Mar 3 12:38:31 dc2oss16 kernel: : Lustre: dc2-OST00a4: Client f078186f-799e-d74b-332c-ae05343757bf (at 149.165.230.170@tcp) reconnecting ./dc2oss16/logs/kern:Mar 3 12:38:31 dc2oss16 kernel: : Lustre: Skipped 9 previous similar messages ./dc2oss16/logs/kern:Mar 3 12:38:31 dc2oss16 kernel: : Lustre: 17271:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a4: connection from f078186f-799e-d74b-332c-ae05343757bf@149.165.230.170@tcp t0 exp ffff8817d7da0c00 cur 1393868311 last 1393868311 ./dc2oss16/logs/kern:Mar 3 12:38:31 dc2oss16 kernel: : Lustre: 17271:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 9 previous similar messages ./dc2oss16/logs/kern:Mar 3 12:38:34 dc2oss16 kernel: : Lustre: dc2-OST009e: Client 45bec439-e3c6-b444-ca4e-c5c2a83bd283 (at 149.165.229.27@tcp) reconnecting ./dc2oss16/logs/kern:Mar 3 12:38:34 dc2oss16 kernel: : Lustre: Skipped 11 previous similar messages ./dc2oss16/logs/kern:Mar 3 12:38:34 dc2oss16 kernel: : Lustre: 17106:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009e: connection from 45bec439-e3c6-b444-ca4e-c5c2a83bd283@149.165.229.27@tcp t0 exp ffff8814e2f79c00 cur 1393868314 last 1393868314 ./dc2oss16/logs/kern:Mar 3 12:38:34 dc2oss16 kernel: : Lustre: 17106:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 11 previous similar messages ./dc2oss16/logs/kern:Mar 3 12:39:30 dc2oss16 kernel: : Lustre: dc2-OST009e: Client 1c13518f-39c8-5628-7d9a-b34ab0b0af7d (at 149.165.229.20@tcp) reconnecting ./dc2oss16/logs/kern:Mar 3 12:39:30 dc2oss16 kernel: : Lustre: 17381:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009f: connection from 1c13518f-39c8-5628-7d9a-b34ab0b0af7d@149.165.229.20@tcp t0 exp ffff8817d8bcf400 cur 1393868370 last 1393868369 ./dc2oss16/logs/kern:Mar 3 12:39:30 dc2oss16 kernel: : Lustre: Skipped 10 previous similar messages ./dc2oss16/logs/kern:Mar 3 12:39:30 dc2oss16 kernel: : Lustre: 17381:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 9 previous similar messages ./dc2oss16/logs/kern:Mar 3 12:54:47 dc2oss16 kernel: : Lustre: dc2-OST00a7: Client 3792c245-21cd-8eb3-61bd-37df8472501c (at 149.165.229.22@tcp) reconnecting ./dc2oss16/logs/kern:Mar 3 12:54:47 dc2oss16 kernel: : Lustre: Skipped 8 previous similar messages ./dc2oss16/logs/kern:Mar 3 12:54:47 dc2oss16 kernel: : Lustre: 15731:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a7: connection from 3792c245-21cd-8eb3-61bd-37df8472501c@149.165.229.22@tcp t0 exp ffff8817d350cc00 cur 1393869287 last 1393869287 ./dc2oss16/logs/kern:Mar 3 12:54:47 dc2oss16 kernel: : Lustre: 15731:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 8 previous similar messages ./dc2oss16/logs/kern:Mar 3 13:15:42 dc2oss16 kernel: : Lustre: dc2-OST00a4: haven't heard from client 7097ef4b-b519-de72-4f21-b6c880acdc47 (at 149.165.230.179@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817d3e20800, cur 1393870542 expire 1393870392 last 1393870315 ./dc2oss16/logs/kern:Mar 3 13:15:42 dc2oss16 kernel: : Lustre: Skipped 9 previous similar messages ./dc2oss16/logs/kern:Mar 3 13:36:04 dc2oss16 kernel: : Lustre: 17374:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST00a1: connection from 21750fe4-5671-53c2-ff4a-3bd0875d2f03@149.165.230.179@tcp t0 exp (null) cur 1393871764 last 0 ./dc2oss16/logs/kern:Mar 3 13:36:04 dc2oss16 kernel: : Lustre: 17374:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 2 previous similar messages ./dc2oss16/logs/kern:Mar 3 14:54:46 dc2oss16 kernel: : ib_send_cm_mra: cm_id_priv->id.state: 0x6 ./dc2oss16/logs/kern:Mar 3 14:54:46 dc2oss16 kernel: : Lustre: 17307:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009f: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2oss16/logs/kern:Mar 3 14:54:46 dc2oss16 kernel: : Lustre: 17319:0:(ldlm_lib.c:952:target_handle_connect()) dc2-OST009e: connection from ec1052a3-2f6f-1062-5d4f-41704ccd2065@10.10.1.10@o2ib t0 exp (null) cur 1393876486 last 0 ./dc2oss16/logs/kern:Mar 3 14:54:46 dc2oss16 kernel: : Lustre: 17319:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 10 previous similar messages ./dc2xfer02/logs/kern:Mar 3 13:12:35 dc2xfer02 kernel: : Lustre: There was an unexpected network error while writing to 149.165.230.179: -110. ./dc2xfer03/logs/kern:Mar 3 13:12:35 dc2xfer03 kernel: : Lustre: There was an unexpected network error while writing to 149.165.230.179: -110. ./dc2xfer04/logs/kern:Mar 3 13:12:35 dc2xfer04 kernel: : Lustre: There was an unexpected network error while writing to 149.165.230.179: -110.