Apr 8 03:31:15 ts-xxxxxxxx-03 kernel: imklog 4.6.2, log source = /proc/kmsg started. Apr 8 03:31:15 ts-xxxxxxxx-03 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="3844" x-info="http://www.rsyslog.com"] (re)start Apr 9 00:45:12 ts-xxxxxxxx-03 ts-log-date: Timestamp 1333932312 Apr 9 00:46:30 ts-xxxxxxxx-03 ts-log-date: Timestamp 1333932390 Apr 9 21:54:44 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0002: haven't heard from client c3cde13e-e40b-6899-6368-2bb2ff23fd40 (at 192.168.185.12@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88036b255c00, cur 1334008484 expire 1334008334 last 1334008257 Apr 9 21:54:44 ts-xxxxxxxx-03 kernel: Lustre: Skipped 5 previous similar messages Apr 9 21:54:45 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0004: haven't heard from client c3cde13e-e40b-6899-6368-2bb2ff23fd40 (at 192.168.185.12@tcp) in 228 seconds. I think it's dead, and I am evicting it. exp ffff88030a14bc00, cur 1334008485 expire 1334008335 last 1334008257 Apr 9 21:54:45 ts-xxxxxxxx-03 kernel: Lustre: Skipped 1 previous similar message Apr 9 21:54:46 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0003: haven't heard from client c3cde13e-e40b-6899-6368-2bb2ff23fd40 (at 192.168.185.12@tcp) in 229 seconds. I think it's dead, and I am evicting it. exp ffff8803b6d2f400, cur 1334008486 expire 1334008336 last 1334008257 Apr 9 21:54:46 ts-xxxxxxxx-03 kernel: Lustre: Skipped 2 previous similar messages Apr 9 22:28:56 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0000: haven't heard from client fcd3ac04-b93c-21d5-1bcf-7f0726f17da9 (at 192.168.185.3@tcp) in 229 seconds. I think it's dead, and I am evicting it. exp ffff8803fdb0e000, cur 1334010536 expire 1334010386 last 1334010307 Apr 9 22:28:56 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0001: haven't heard from client fcd3ac04-b93c-21d5-1bcf-7f0726f17da9 (at 192.168.185.3@tcp) in 229 seconds. I think it's dead, and I am evicting it. exp ffff8803fdb13400, cur 1334010536 expire 1334010386 last 1334010307 Apr 9 22:29:19 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0003: haven't heard from client fcd3ac04-b93c-21d5-1bcf-7f0726f17da9 (at 192.168.185.3@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880352c16400, cur 1334010559 expire 1334010409 last 1334010332 Apr 9 22:29:21 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0005: haven't heard from client fcd3ac04-b93c-21d5-1bcf-7f0726f17da9 (at 192.168.185.3@tcp) in 229 seconds. I think it's dead, and I am evicting it. exp ffff8804228ac000, cur 1334010561 expire 1334010411 last 1334010332 Apr 9 22:29:21 ts-xxxxxxxx-03 kernel: Lustre: Skipped 1 previous similar message Apr 10 00:45:12 ts-xxxxxxxx-03 ts-log-date: Timestamp 1334018712 Apr 10 00:46:31 ts-xxxxxxxx-03 ts-log-date: Timestamp 1334018791 Apr 10 04:03:35 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0002: haven't heard from client debe3c76-c7c2-f83a-f7b0-8049110a3e27 (at 192.168.185.11@tcp) in 233 seconds. I think it's dead, and I am evicting it. exp ffff880111476000, cur 1334030615 expire 1334030465 last 1334030382 Apr 10 04:03:35 ts-xxxxxxxx-03 kernel: Lustre: Skipped 1 previous similar message Apr 10 04:03:36 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0005: haven't heard from client debe3c76-c7c2-f83a-f7b0-8049110a3e27 (at 192.168.185.11@tcp) in 234 seconds. I think it's dead, and I am evicting it. exp ffff8803e8094800, cur 1334030616 expire 1334030466 last 1334030382 Apr 10 04:03:38 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0000: haven't heard from client debe3c76-c7c2-f83a-f7b0-8049110a3e27 (at 192.168.185.11@tcp) in 236 seconds. I think it's dead, and I am evicting it. exp ffff8803fa19c800, cur 1334030618 expire 1334030468 last 1334030382 Apr 10 04:03:38 ts-xxxxxxxx-03 kernel: Lustre: Skipped 1 previous similar message Apr 10 04:03:49 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0003: haven't heard from client debe3c76-c7c2-f83a-f7b0-8049110a3e27 (at 192.168.185.11@tcp) in 247 seconds. I think it's dead, and I am evicting it. exp ffff8803fdb70c00, cur 1334030629 expire 1334030479 last 1334030382 Apr 10 11:55:10 ts-xxxxxxxx-03 tslagentd: tslagent: Running: /root/bin/ts-gather-support-material 0 1873710108 4096 support-material-ts-xxxxxxxx-03-2012.03.10_11.55.10 > /tmp/tmp.gather-support.log 2>&1 & Apr 10 11:55:13 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030100): originator(IOP), code(0x03), sub_code(0x0100) Apr 10 11:55:13 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030100): originator(IOP), code(0x03), sub_code(0x0100) Apr 10 11:55:13 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) Apr 10 11:55:13 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) Apr 10 11:55:13 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) Apr 10 11:55:13 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) Apr 10 11:55:13 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030107): originator(IOP), code(0x03), sub_code(0x0107) Apr 10 11:55:13 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030107): originator(IOP), code(0x03), sub_code(0x0107) Apr 10 11:55:13 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x3003010b): originator(IOP), code(0x03), sub_code(0x010b) Apr 10 11:55:13 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030110): originator(IOP), code(0x03), sub_code(0x0110) Apr 10 11:55:13 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030110): originator(IOP), code(0x03), sub_code(0x0110) Apr 10 11:55:15 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030100): originator(IOP), code(0x03), sub_code(0x0100) Apr 10 11:55:15 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030100): originator(IOP), code(0x03), sub_code(0x0100) Apr 10 11:55:16 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) Apr 10 11:55:16 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) Apr 10 11:55:16 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) Apr 10 11:55:16 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) Apr 10 11:55:16 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030107): originator(IOP), code(0x03), sub_code(0x0107) Apr 10 11:55:16 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030107): originator(IOP), code(0x03), sub_code(0x0107) Apr 10 11:55:16 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x3003010b): originator(IOP), code(0x03), sub_code(0x010b) Apr 10 11:55:16 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030110): originator(IOP), code(0x03), sub_code(0x0110) Apr 10 11:55:16 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030110): originator(IOP), code(0x03), sub_code(0x0110) Apr 10 11:55:19 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030100): originator(IOP), code(0x03), sub_code(0x0100) Apr 10 11:55:19 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030100): originator(IOP), code(0x03), sub_code(0x0100) Apr 10 11:55:19 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) Apr 10 11:55:19 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) Apr 10 11:55:19 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) Apr 10 11:55:19 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) Apr 10 11:55:19 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030107): originator(IOP), code(0x03), sub_code(0x0107) Apr 10 11:55:19 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030107): originator(IOP), code(0x03), sub_code(0x0107) Apr 10 11:55:19 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x3003010b): originator(IOP), code(0x03), sub_code(0x010b) Apr 10 11:55:19 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030110): originator(IOP), code(0x03), sub_code(0x0110) Apr 10 11:55:19 ts-xxxxxxxx-03 kernel: mpt2sas0: log_info(0x30030110): originator(IOP), code(0x03), sub_code(0x0110) Apr 10 11:55:20 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030100): originator(IOP), code(0x03), sub_code(0x0100) Apr 10 11:55:20 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030100): originator(IOP), code(0x03), sub_code(0x0100) Apr 10 11:55:20 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) Apr 10 11:55:20 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) Apr 10 11:55:20 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) Apr 10 11:55:20 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) Apr 10 11:55:20 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030107): originator(IOP), code(0x03), sub_code(0x0107) Apr 10 11:55:20 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030107): originator(IOP), code(0x03), sub_code(0x0107) Apr 10 11:55:20 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x3003010b): originator(IOP), code(0x03), sub_code(0x010b) Apr 10 11:55:20 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030110): originator(IOP), code(0x03), sub_code(0x0110) Apr 10 11:55:20 ts-xxxxxxxx-03 kernel: mpt2sas1: log_info(0x30030110): originator(IOP), code(0x03), sub_code(0x0110) Apr 10 11:55:21 ts-xxxxxxxx-03 kernel: MegaCli64[20062] general protection ip:35f8665da4 sp:7fff44eaa900 error:0 in libc-2.12.so[35f8600000+175000] Apr 10 12:09:39 ts-xxxxxxxx-03 kernel: Lustre: 6282:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from e1e9d33a-99de-958d-3e58-f0e99fb5ec56@172.30.246.117@tcp t0 exp (null) cur 1334059779 last 0 Apr 10 12:09:39 ts-xxxxxxxx-03 kernel: Lustre: 6272:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0002->NET_0x20000ac1ef675_UUID netid 20000: select flavor null Apr 10 12:09:39 ts-xxxxxxxx-03 kernel: Lustre: 6272:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 4 previous similar messages Apr 10 12:09:39 ts-xxxxxxxx-03 kernel: Lustre: 6282:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 9 previous similar messages Apr 10 12:10:45 ts-xxxxxxxx-03 kernel: Lustre: 6285:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from ab19e8b1-01fa-ae26-a71b-b5b39e1877e4@192.168.185.3@tcp t0 exp (null) cur 1334059845 last 0 Apr 10 12:10:45 ts-xxxxxxxx-03 kernel: Lustre: 6243:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001->NET_0x20000c0a8b903_UUID netid 20000: select flavor null Apr 10 12:10:45 ts-xxxxxxxx-03 kernel: Lustre: 6243:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages Apr 10 12:10:45 ts-xxxxxxxx-03 kernel: Lustre: 6285:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 10 12:11:26 ts-xxxxxxxx-03 kernel: Lustre: 6266:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0001: connection from 92fec929-7f5a-d025-f46b-59d45056afd8@172.30.246.126@tcp t0 exp (null) cur 1334059886 last 0 Apr 10 12:11:26 ts-xxxxxxxx-03 kernel: Lustre: 6252:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0000->NET_0x20000ac1ef67e_UUID netid 20000: select flavor null Apr 10 12:11:26 ts-xxxxxxxx-03 kernel: Lustre: 6252:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages Apr 10 12:11:26 ts-xxxxxxxx-03 kernel: Lustre: 6266:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 10 15:14:05 ts-xxxxxxxx-03 kernel: Lustre: 6267:0:(ldlm_lib.c:609:target_handle_reconnect()) xxxxxx-OST0001: cc4b75a4-8d81-475b-4e2b-1b8e52e7eef1 reconnecting Apr 10 15:14:05 ts-xxxxxxxx-03 kernel: Lustre: 6267:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0001: connection from cc4b75a4-8d81-475b-4e2b-1b8e52e7eef1@192.168.185.32@tcp t0 exp ffff880398d5b400 cur 1334070845 last 1334070845 Apr 10 15:14:05 ts-xxxxxxxx-03 kernel: Lustre: 6267:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001-> netid 20000: select flavor null Apr 10 15:14:05 ts-xxxxxxxx-03 kernel: Lustre: 6267:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages Apr 10 15:17:57 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0005: haven't heard from client cc4b75a4-8d81-475b-4e2b-1b8e52e7eef1 (at 192.168.185.32@tcp) in 239 seconds. I think it's dead, and I am evicting it. exp ffff88014cdcac00, cur 1334071077 expire 1334070927 last 1334070838 Apr 10 15:17:57 ts-xxxxxxxx-03 kernel: Lustre: Skipped 1 previous similar message Apr 10 15:17:59 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0001: haven't heard from client cc4b75a4-8d81-475b-4e2b-1b8e52e7eef1 (at 192.168.185.32@tcp) in 234 seconds. I think it's dead, and I am evicting it. exp ffff880398d5b400, cur 1334071079 expire 1334070929 last 1334070845 Apr 10 15:17:59 ts-xxxxxxxx-03 kernel: Lustre: Skipped 4 previous similar messages Apr 10 15:27:57 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0005: haven't heard from client 9854cd6a-380a-716a-2cdd-41392129f226 (at 192.168.185.5@tcp) in 239 seconds. I think it's dead, and I am evicting it. exp ffff880361725800, cur 1334071677 expire 1334071527 last 1334071438 Apr 10 15:32:37 ts-xxxxxxxx-03 kernel: Lustre: 6268:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0001: connection from 2ff100a2-88a7-7414-c940-c68827998b6f@192.168.185.5@tcp t0 exp (null) cur 1334071957 last 0 Apr 10 15:32:37 ts-xxxxxxxx-03 kernel: Lustre: 6259:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from 2ff100a2-88a7-7414-c940-c68827998b6f@192.168.185.5@tcp t0 exp (null) cur 1334071957 last 0 Apr 10 15:32:37 ts-xxxxxxxx-03 kernel: Lustre: 6259:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0000-> netid 20000: select flavor null Apr 10 15:32:37 ts-xxxxxxxx-03 kernel: Lustre: 6268:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 4 previous similar messages Apr 10 15:34:55 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0003: haven't heard from client f1751802-9f84-105a-f622-92396581c95c (at 192.168.185.7@tcp) in 238 seconds. I think it's dead, and I am evicting it. exp ffff8801c11a9800, cur 1334072095 expire 1334071945 last 1334071857 Apr 10 15:34:55 ts-xxxxxxxx-03 kernel: Lustre: Skipped 5 previous similar messages Apr 10 15:34:56 ts-xxxxxxxx-03 kernel: Lustre: 6228:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from 78f935a8-9275-6238-ca51-6ea3df120326@192.168.185.7@tcp t0 exp (null) cur 1334072096 last 0 Apr 10 15:34:56 ts-xxxxxxxx-03 kernel: Lustre: 6232:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001->NET_0x20000c0a8b907_UUID netid 20000: select flavor null Apr 10 15:34:56 ts-xxxxxxxx-03 kernel: Lustre: 6232:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages Apr 10 15:34:56 ts-xxxxxxxx-03 kernel: Lustre: 6228:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 10 15:35:24 ts-xxxxxxxx-03 kernel: Lustre: 6259:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from 2c329d2b-8b91-7ff6-48bf-1a4a3c3e8a88@192.168.185.6@tcp t0 exp (null) cur 1334072124 last 0 Apr 10 15:35:24 ts-xxxxxxxx-03 kernel: Lustre: 6232:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0001: connection from 2c329d2b-8b91-7ff6-48bf-1a4a3c3e8a88@192.168.185.6@tcp t0 exp (null) cur 1334072124 last 0 Apr 10 15:35:24 ts-xxxxxxxx-03 kernel: Lustre: 6232:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 1 previous similar message Apr 10 15:35:24 ts-xxxxxxxx-03 kernel: Lustre: 6267:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0002-> netid 20000: select flavor null Apr 10 15:35:24 ts-xxxxxxxx-03 kernel: Lustre: 6267:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages Apr 10 15:35:24 ts-xxxxxxxx-03 kernel: Lustre: 6259:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 3 previous similar messages Apr 10 17:57:43 ts-xxxxxxxx-03 ntpd[4028]: synchronized to 192.168.4.1, stratum 2 Apr 10 20:00:08 ts-xxxxxxxx-03 kernel: Lustre: 6266:0:(ldlm_lib.c:609:target_handle_reconnect()) xxxxxx-OST0002: ab19e8b1-01fa-ae26-a71b-b5b39e1877e4 reconnecting Apr 10 20:00:08 ts-xxxxxxxx-03 kernel: Lustre: 6249:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0004: connection from ab19e8b1-01fa-ae26-a71b-b5b39e1877e4@192.168.185.3@tcp t0 exp ffff8801c97ec800 cur 1334088008 last 1334088008 Apr 10 20:00:08 ts-xxxxxxxx-03 kernel: Lustre: 6249:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0004->NET_0x20000c0a8b903_UUID netid 20000: select flavor null Apr 10 20:00:08 ts-xxxxxxxx-03 kernel: Lustre: 6249:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages Apr 10 20:00:08 ts-xxxxxxxx-03 kernel: LustreError: 16470:0:(service.c:844:ptlrpc_check_req()) @@@ DROPPING req from old connection 1 < 2 req@ffff88033aa45c00 x1398863127658850/t0(0) o-1->ab19e8b1-01fa-ae26-a71b-b5b39e1877e4@NET_0x20000c0a8b903_UUID:0/0 lens 456/0 e 0 to 0 dl 0 ref 2 fl New:/ffffffff/ffffffff rc 0/-1 Apr 10 20:00:08 ts-xxxxxxxx-03 kernel: LustreError: 137-5: UUID 'xxxxxx-OST0008_UUID' is not available for connect (no target) Apr 10 20:00:08 ts-xxxxxxxx-03 kernel: LustreError: Skipped 32 previous similar messages Apr 10 20:00:08 ts-xxxxxxxx-03 kernel: LustreError: 6272:0:(ldlm_lib.c:2129:target_send_reply_msg()) @@@ processing error (-19) req@ffff88041bc02850 x1398863127658871/t0(0) o-1->@:0/0 lens 368/0 e 0 to 0 dl 1334088108 ref 1 fl Interpret:/ffffffff/ffffffff rc -19/-1 Apr 10 20:00:08 ts-xxxxxxxx-03 kernel: LustreError: 6272:0:(ldlm_lib.c:2129:target_send_reply_msg()) Skipped 89 previous similar messages Apr 10 20:00:08 ts-xxxxxxxx-03 kernel: Lustre: 6266:0:(ldlm_lib.c:609:target_handle_reconnect()) Skipped 5 previous similar messages Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: Lustre: Service thread pid 16358 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: Pid: 16358, comm: ll_ost_io_143 Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: Call Trace: Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ? LNetGet+0x3e9/0x830 [lnet] Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ? LNetMDBind+0x135/0x490 [lnet] Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ? lock_timer_base+0x2b/0x4f Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ? try_to_del_timer_sync+0xaa/0xb7 Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] schedule_timeout+0x1c6/0x1ee Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ? process_timeout+0x0/0x10 Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] cfs_waitq_timedwait+0x11/0x20 [libcfs] Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] target_bulk_io+0x3af/0xa40 [ptlrpc] Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ? cfs_alloc+0x30/0x60 [libcfs] Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ? default_wake_function+0x0/0x14 Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ost_brw_write+0x12af/0x2040 [ost] Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ? __alloc_pages_nodemask+0x106/0x7d6 Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ? target_bulk_timeout+0x0/0x100 [ptlrpc] Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ost_handle+0x2604/0x57e0 [ost] Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ? lustre_msg_get_opc+0x8e/0xf0 [ptlrpc] Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ptlrpc_server_handle_request+0x4ec/0xfc0 [ptlrpc] Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ? enqueue_task+0x7c/0x8b Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ? cfs_timer_arm+0xe/0x10 [libcfs] Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ? lc_watchdog_touch+0x70/0x150 [libcfs] Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ? lc_watchdog_disable+0x47/0x120 [libcfs] Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ? ptlrpc_wait_event+0xa4/0x2d0 [ptlrpc] Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ? __wake_up+0x48/0x55 Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ptlrpc_main+0x53f/0x1670 [ptlrpc] Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] child_rip+0xa/0x20 Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ? ptlrpc_main+0x0/0x1670 [ptlrpc] Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: [] ? child_rip+0x0/0x20 Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: Apr 10 20:02:37 ts-xxxxxxxx-03 kernel: LustreError: dumping log to /tmp/lustre-log.1334088157.16358 Apr 10 20:02:39 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0004: haven't heard from client 92fec929-7f5a-d025-f46b-59d45056afd8 (at 172.30.246.126@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880423c78400, cur 1334088159 expire 1334088009 last 1334087932 Apr 10 20:02:39 ts-xxxxxxxx-03 kernel: Lustre: Skipped 11 previous similar messages Apr 10 20:02:44 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0003: haven't heard from client 92fec929-7f5a-d025-f46b-59d45056afd8 (at 172.30.246.126@tcp) in 228 seconds. I think it's dead, and I am evicting it. exp ffff8803b414f800, cur 1334088164 expire 1334088014 last 1334087936 Apr 10 20:02:44 ts-xxxxxxxx-03 kernel: Lustre: Skipped 1 previous similar message Apr 10 20:02:46 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0002: haven't heard from client 92fec929-7f5a-d025-f46b-59d45056afd8 (at 172.30.246.126@tcp) in 230 seconds. I think it's dead, and I am evicting it. exp ffff8802ea562000, cur 1334088166 expire 1334088016 last 1334087936 Apr 10 20:02:46 ts-xxxxxxxx-03 kernel: Lustre: Skipped 1 previous similar message Apr 10 20:02:51 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0005: haven't heard from client 92fec929-7f5a-d025-f46b-59d45056afd8 (at 172.30.246.126@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803907e9000, cur 1334088171 expire 1334088021 last 1334087944 Apr 10 20:02:51 ts-xxxxxxxx-03 kernel: Lustre: Skipped 1 previous similar message Apr 10 20:02:59 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0001: haven't heard from client 92fec929-7f5a-d025-f46b-59d45056afd8 (at 172.30.246.126@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880360bca800, cur 1334088179 expire 1334088029 last 1334087952 Apr 10 20:02:59 ts-xxxxxxxx-03 kernel: Lustre: Skipped 1 previous similar message Apr 10 20:03:05 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0000: haven't heard from client 92fec929-7f5a-d025-f46b-59d45056afd8 (at 172.30.246.126@tcp) in 228 seconds. I think it's dead, and I am evicting it. exp ffff88014642e800, cur 1334088185 expire 1334088035 last 1334087957 Apr 10 20:03:05 ts-xxxxxxxx-03 kernel: Lustre: Skipped 1 previous similar message Apr 10 20:03:05 ts-xxxxxxxx-03 kernel: LustreError: 6426:0:(filter.c:2928:filter_grant_sanity_check()) filter_disconnect: tot_granted 4938354688 != fo_tot_granted 4938465280 Apr 10 20:03:05 ts-xxxxxxxx-03 kernel: LustreError: 6426:0:(filter.c:2931:filter_grant_sanity_check()) filter_disconnect: tot_pending 0 != fo_tot_pending 110592 Apr 10 20:03:05 ts-xxxxxxxx-03 kernel: LustreError: 7909:0:(filter.c:2928:filter_grant_sanity_check()) filter_destroy_export: tot_granted 4936257536 != fo_tot_granted 4936368128 Apr 10 20:03:05 ts-xxxxxxxx-03 kernel: LustreError: 7909:0:(filter.c:2931:filter_grant_sanity_check()) filter_destroy_export: tot_pending 0 != fo_tot_pending 110592 Apr 10 20:03:05 ts-xxxxxxxx-03 kernel: LustreError: 16358:0:(ldlm_lib.c:2547:target_bulk_io()) @@@ Eviction on bulk GET req@ffff88022c920000 x1398863170650498/t0(0) o-1->92fec929-7f5a-d025-f46b-59d45056afd8@NET_0x20000ac1ef67e_UUID:0/0 lens 456/416 e 4 to 0 dl 1334088353 ref 1 fl Interpret:/ffffffff/ffffffff rc 0/-1 Apr 10 20:03:05 ts-xxxxxxxx-03 kernel: Lustre: 16358:0:(ost_handler.c:1242:ost_brw_write()) xxxxxx-OST0000: ignoring bulk IO comm error with 92fec929-7f5a-d025-f46b-59d45056afd8@NET_0x20000ac1ef67e_UUID id 12345-172.30.246.126@tcp - client will retry Apr 10 20:03:05 ts-xxxxxxxx-03 kernel: Lustre: Service thread pid 16358 completed after 228.00s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Apr 11 00:45:12 ts-xxxxxxxx-03 ts-log-date: Timestamp 1334105112 Apr 11 00:46:30 ts-xxxxxxxx-03 ts-log-date: Timestamp 1334105190 Apr 11 05:22:07 ts-xxxxxxxx-03 ntpd[4028]: synchronized to 192.168.3.1, stratum 2 Apr 11 14:24:05 ts-xxxxxxxx-03 kernel: Lustre: 6252:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from 916159e5-bb29-18c2-c149-182aa58e0523@192.168.185.3@tcp t0 exp (null) cur 1334154245 last 0 Apr 11 14:24:05 ts-xxxxxxxx-03 kernel: Lustre: 6259:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001->NET_0x20000c0a8b903_UUID netid 20000: select flavor null Apr 11 14:24:05 ts-xxxxxxxx-03 kernel: Lustre: 6259:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages Apr 11 14:24:05 ts-xxxxxxxx-03 kernel: Lustre: 6252:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 10 previous similar messages Apr 11 14:24:06 ts-xxxxxxxx-03 kernel: Lustre: 6252:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0000->NET_0x20000c0a8b903_UUID netid 20000: select flavor null Apr 11 14:24:06 ts-xxxxxxxx-03 kernel: Lustre: 6252:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 3 previous similar messages Apr 11 15:18:00 ts-xxxxxxxx-03 ntpd[4028]: synchronized to 192.168.4.1, stratum 2 Apr 11 16:45:10 ts-xxxxxxxx-03 ntpd[4028]: synchronized to 192.168.3.1, stratum 2 Apr 11 18:07:58 ts-xxxxxxxx-03 kernel: Lustre: 6264:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from a182ffea-d75b-0158-5553-f9a55337da9b@192.168.185.11@tcp t0 exp (null) cur 1334167678 last 0 Apr 11 18:07:58 ts-xxxxxxxx-03 kernel: Lustre: 6243:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001-> netid 20000: select flavor null Apr 11 18:07:58 ts-xxxxxxxx-03 kernel: Lustre: 6293:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0002-> netid 20000: select flavor null Apr 11 18:07:58 ts-xxxxxxxx-03 kernel: Lustre: 6264:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 12 00:45:14 ts-xxxxxxxx-03 ts-log-date: Timestamp 1334191514 Apr 12 00:46:30 ts-xxxxxxxx-03 ts-log-date: Timestamp 1334191590 Apr 12 19:35:00 ts-xxxxxxxx-03 kernel: Lustre: 6269:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from c890b421-1209-2800-f43a-51f159be2343@192.168.117.127@tcp t0 exp (null) cur 1334259300 last 0 Apr 12 19:35:00 ts-xxxxxxxx-03 kernel: Lustre: 6285:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0002-> netid 20000: select flavor null Apr 12 19:35:00 ts-xxxxxxxx-03 kernel: Lustre: 6250:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001-> netid 20000: select flavor null Apr 12 19:35:00 ts-xxxxxxxx-03 kernel: Lustre: 6285:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 4 previous similar messages Apr 12 19:35:00 ts-xxxxxxxx-03 kernel: Lustre: 6250:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 4 previous similar messages Apr 12 19:35:00 ts-xxxxxxxx-03 kernel: Lustre: 6269:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 13 00:45:14 ts-xxxxxxxx-03 ts-log-date: Timestamp 1334277914 Apr 13 00:46:32 ts-xxxxxxxx-03 ts-log-date: Timestamp 1334277992 Apr 13 00:54:14 ts-xxxxxxxx-03 ntpd[4028]: synchronized to 192.168.4.1, stratum 2 Apr 13 02:00:58 ts-xxxxxxxx-03 ntpd[4028]: synchronized to 192.168.3.1, stratum 2 Apr 13 08:20:01 ts-xxxxxxxx-03 auditd[3829]: Audit daemon rotating log files Apr 13 10:48:48 ts-xxxxxxxx-03 kernel: Lustre: 6281:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from 35ac6bf2-d849-689e-6917-6a101cb711a1@172.30.246.117@tcp t0 exp (null) cur 1334314128 last 0 Apr 13 10:48:48 ts-xxxxxxxx-03 kernel: Lustre: 6243:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001->NET_0x20000ac1ef675_UUID netid 20000: select flavor null Apr 13 10:48:48 ts-xxxxxxxx-03 kernel: Lustre: 6243:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages Apr 13 10:48:48 ts-xxxxxxxx-03 kernel: Lustre: 6281:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 13 10:48:48 ts-xxxxxxxx-03 kernel: Lustre: 6281:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0000->NET_0x20000ac1ef675_UUID netid 20000: select flavor null Apr 13 10:48:48 ts-xxxxxxxx-03 kernel: Lustre: 6281:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 2 previous similar messages Apr 13 10:48:49 ts-xxxxxxxx-03 kernel: Lustre: 6266:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from 25b812b5-346e-21b9-a36f-97897ad89137@172.30.246.113@tcp t0 exp (null) cur 1334314129 last 0 Apr 13 10:48:49 ts-xxxxxxxx-03 kernel: Lustre: 6272:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001->NET_0x20000ac1ef671_UUID netid 20000: select flavor null Apr 13 10:48:49 ts-xxxxxxxx-03 kernel: Lustre: 6266:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 13 10:48:50 ts-xxxxxxxx-03 kernel: Lustre: 6295:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from db30da88-4bea-3134-33f9-4de3b97272e1@192.168.185.30@tcp t0 exp (null) cur 1334314130 last 0 Apr 13 10:48:50 ts-xxxxxxxx-03 kernel: Lustre: 6295:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 13 10:48:51 ts-xxxxxxxx-03 kernel: Lustre: 6264:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from 0ab9bd56-e079-ff8d-a5b5-fc70a6fd625a@172.30.246.127@tcp t0 exp (null) cur 1334314131 last 0 Apr 13 10:48:51 ts-xxxxxxxx-03 kernel: Lustre: 6281:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001-> netid 20000: select flavor null Apr 13 10:48:51 ts-xxxxxxxx-03 kernel: Lustre: 6281:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 11 previous similar messages Apr 13 10:48:51 ts-xxxxxxxx-03 kernel: Lustre: 6264:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 13 10:48:53 ts-xxxxxxxx-03 kernel: Lustre: 6293:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from 6313cb21-cbdf-aa2e-d301-115b63c06686@192.168.185.6@tcp t0 exp (null) cur 1334314133 last 0 Apr 13 10:48:53 ts-xxxxxxxx-03 kernel: Lustre: 6286:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001-> netid 20000: select flavor null Apr 13 10:48:53 ts-xxxxxxxx-03 kernel: Lustre: 6286:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 4 previous similar messages Apr 13 10:48:53 ts-xxxxxxxx-03 kernel: Lustre: 6293:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 13 10:51:48 ts-xxxxxxxx-03 kernel: Lustre: 6250:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from e018da1e-292c-6de1-0800-f3509dbc6b14@172.30.246.122@tcp t0 exp (null) cur 1334314308 last 0 Apr 13 10:51:48 ts-xxxxxxxx-03 kernel: Lustre: 6266:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001->NET_0x20000ac1ef67a_UUID netid 20000: select flavor null Apr 13 10:51:48 ts-xxxxxxxx-03 kernel: Lustre: 6266:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages Apr 13 10:51:48 ts-xxxxxxxx-03 kernel: Lustre: 6250:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 13 11:53:20 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0004: haven't heard from client 18ddcd91-6503-4467-a7b1-515a4c979c1e (at 192.168.185.2@tcp) in 243 seconds. I think it's dead, and I am evicting it. exp ffff880396e33000, cur 1334318000 expire 1334317850 last 1334317757 Apr 13 11:53:20 ts-xxxxxxxx-03 kernel: Lustre: Skipped 1 previous similar message Apr 13 11:54:57 ts-xxxxxxxx-03 kernel: Lustre: 6250:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from 54acd2c6-aae8-b7aa-a081-5edf782809bc@192.168.185.2@tcp t0 exp (null) cur 1334318097 last 0 Apr 13 11:54:57 ts-xxxxxxxx-03 kernel: Lustre: 6240:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001->NET_0x20000c0a8b902_UUID netid 20000: select flavor null Apr 13 11:54:57 ts-xxxxxxxx-03 kernel: Lustre: 6240:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 17 previous similar messages Apr 13 11:54:57 ts-xxxxxxxx-03 kernel: Lustre: 6250:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 17 previous similar messages Apr 13 11:56:30 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0004: haven't heard from client 39bb32ba-852c-eedd-1ea2-7f817a625dd6 (at 192.168.185.29@tcp) in 233 seconds. I think it's dead, and I am evicting it. exp ffff8803e8496400, cur 1334318190 expire 1334318040 last 1334317957 Apr 13 11:56:30 ts-xxxxxxxx-03 kernel: Lustre: Skipped 5 previous similar messages Apr 13 11:56:40 ts-xxxxxxxx-03 kernel: Lustre: 6264:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from 3cd34c45-ed41-c25b-6e57-9805706e69da@172.30.246.121@tcp t0 exp (null) cur 1334318200 last 0 Apr 13 11:56:40 ts-xxxxxxxx-03 kernel: Lustre: 6268:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001->NET_0x20000ac1ef679_UUID netid 20000: select flavor null Apr 13 11:56:40 ts-xxxxxxxx-03 kernel: Lustre: 6268:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages Apr 13 11:56:40 ts-xxxxxxxx-03 kernel: Lustre: 6264:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 13 12:24:10 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0004: haven't heard from client 79324aa4-1a2a-89aa-7eec-cacd4bb38d24 (at 192.168.185.8@tcp) in 234 seconds. I think it's dead, and I am evicting it. exp ffff8803bf3d1000, cur 1334319850 expire 1334319700 last 1334319616 Apr 13 12:24:10 ts-xxxxxxxx-03 kernel: Lustre: Skipped 5 previous similar messages Apr 13 12:24:10 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0001: haven't heard from client 79324aa4-1a2a-89aa-7eec-cacd4bb38d24 (at 192.168.185.8@tcp) in 233 seconds. I think it's dead, and I am evicting it. exp ffff8803fdfac000, cur 1334319850 expire 1334319700 last 1334319617 Apr 13 12:24:10 ts-xxxxxxxx-03 kernel: Lustre: Skipped 24 previous similar messages Apr 13 12:24:28 ts-xxxxxxxx-03 kernel: Lustre: 6281:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from c0588737-d7f6-ccb6-9a7f-8c80268cf61b@192.168.185.8@tcp t0 exp (null) cur 1334319868 last 0 Apr 13 12:24:28 ts-xxxxxxxx-03 kernel: Lustre: 6286:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001-> netid 20000: select flavor null Apr 13 12:24:28 ts-xxxxxxxx-03 kernel: Lustre: 6286:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 4 previous similar messages Apr 13 12:24:28 ts-xxxxxxxx-03 kernel: Lustre: 6281:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 13 12:24:35 ts-xxxxxxxx-03 kernel: Lustre: 6281:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from 7c0d4e2e-e0e7-57e4-00af-95df126928fd@192.168.185.1@tcp t0 exp (null) cur 1334319874 last 0 Apr 13 12:24:35 ts-xxxxxxxx-03 kernel: Lustre: 6295:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0003-> netid 20000: select flavor null Apr 13 12:24:35 ts-xxxxxxxx-03 kernel: Lustre: 6295:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages Apr 13 12:24:35 ts-xxxxxxxx-03 kernel: Lustre: 6281:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 13 12:24:35 ts-xxxxxxxx-03 kernel: Lustre: 6240:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from cdcf0488-8f85-19b1-f706-cf322641d695@172.30.246.124@tcp t0 exp (null) cur 1334319875 last 0 Apr 13 12:24:35 ts-xxxxxxxx-03 kernel: Lustre: 6286:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001-> netid 20000: select flavor null Apr 13 12:24:35 ts-xxxxxxxx-03 kernel: Lustre: 6286:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 3 previous similar messages Apr 13 12:24:35 ts-xxxxxxxx-03 kernel: Lustre: 6240:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 13 12:24:38 ts-xxxxxxxx-03 kernel: Lustre: 6250:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from 70548a54-d26e-1eb9-59f5-2a54bad23c30@192.168.185.11@tcp t0 exp (null) cur 1334319878 last 0 Apr 13 12:24:38 ts-xxxxxxxx-03 kernel: Lustre: 6285:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001-> netid 20000: select flavor null Apr 13 12:24:38 ts-xxxxxxxx-03 kernel: Lustre: 6285:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages Apr 13 12:24:38 ts-xxxxxxxx-03 kernel: Lustre: 6250:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 13 12:25:39 ts-xxxxxxxx-03 kernel: Lustre: 6250:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from b14e3a89-81ae-c6c2-f34b-02fca864e126@192.168.185.9@tcp t0 exp (null) cur 1334319939 last 0 Apr 13 12:25:39 ts-xxxxxxxx-03 kernel: Lustre: 6228:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001->NET_0x20000c0a8b909_UUID netid 20000: select flavor null Apr 13 12:25:39 ts-xxxxxxxx-03 kernel: Lustre: 6228:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 4 previous similar messages Apr 13 12:25:39 ts-xxxxxxxx-03 kernel: Lustre: 6250:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 13 14:01:32 ts-xxxxxxxx-03 kernel: Lustre: 6259:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from 7aa82aea-b7c5-3cf5-93c0-fc148cf532a3@192.168.185.32@tcp t0 exp (null) cur 1334325692 last 0 Apr 13 14:01:32 ts-xxxxxxxx-03 kernel: Lustre: 6236:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001-> netid 20000: select flavor null Apr 13 14:01:32 ts-xxxxxxxx-03 kernel: Lustre: 6236:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 6 previous similar messages Apr 13 14:01:32 ts-xxxxxxxx-03 kernel: Lustre: 6259:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 83 previous similar messages Apr 13 18:01:14 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0004: haven't heard from client ede7bd91-1ecb-5a13-865f-08152176987b (at 192.168.185.4@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880419298000, cur 1334340074 expire 1334339924 last 1334339847 Apr 13 18:01:14 ts-xxxxxxxx-03 kernel: Lustre: Skipped 4 previous similar messages Apr 13 18:01:14 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0000: haven't heard from client ede7bd91-1ecb-5a13-865f-08152176987b (at 192.168.185.4@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88041bd7a800, cur 1334340074 expire 1334339924 last 1334339847 Apr 13 18:01:15 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0002: haven't heard from client ede7bd91-1ecb-5a13-865f-08152176987b (at 192.168.185.4@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88036ae58000, cur 1334340075 expire 1334339925 last 1334339848 Apr 13 18:01:15 ts-xxxxxxxx-03 kernel: Lustre: Skipped 2 previous similar messages Apr 13 18:24:16 ts-xxxxxxxx-03 kernel: Lustre: 6264:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from 9dd74c95-8ebd-75dd-5a1d-d62736f60852@172.30.246.124@tcp t0 exp (null) cur 1334341456 last 0 Apr 13 18:24:16 ts-xxxxxxxx-03 kernel: Lustre: 6257:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0002-> netid 20000: select flavor null Apr 13 18:24:16 ts-xxxxxxxx-03 kernel: Lustre: 6295:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001-> netid 20000: select flavor null Apr 13 18:24:16 ts-xxxxxxxx-03 kernel: Lustre: 6257:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 94 previous similar messages Apr 13 18:24:16 ts-xxxxxxxx-03 kernel: Lustre: 6295:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 94 previous similar messages Apr 13 18:24:16 ts-xxxxxxxx-03 kernel: Lustre: 6264:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 17 previous similar messages Apr 13 19:30:06 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0003: haven't heard from client 3fe301f1-f205-3a6a-1e4c-c626adcc26bb (at 192.168.185.29@tcp) in 231 seconds. I think it's dead, and I am evicting it. exp ffff88041615bc00, cur 1334345406 expire 1334345256 last 1334345175 Apr 13 19:30:06 ts-xxxxxxxx-03 kernel: Lustre: Skipped 1 previous similar message Apr 13 19:30:06 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0000: haven't heard from client 3fe301f1-f205-3a6a-1e4c-c626adcc26bb (at 192.168.185.29@tcp) in 231 seconds. I think it's dead, and I am evicting it. exp ffff88036160b800, cur 1334345406 expire 1334345256 last 1334345175 Apr 13 19:30:11 ts-xxxxxxxx-03 kernel: Lustre: xxxxxx-OST0005: haven't heard from client 3fe301f1-f205-3a6a-1e4c-c626adcc26bb (at 192.168.185.29@tcp) in 232 seconds. I think it's dead, and I am evicting it. exp ffff880361a9f800, cur 1334345411 expire 1334345261 last 1334345179 Apr 13 19:30:40 ts-xxxxxxxx-03 kernel: Lustre: 6249:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from 26c1c6e8-f333-8062-fc5e-3609e6504a1a@192.168.185.30@tcp t0 exp (null) cur 1334345440 last 0 Apr 13 19:30:40 ts-xxxxxxxx-03 kernel: Lustre: 6288:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001->NET_0x20000c0a8b91e_UUID netid 20000: select flavor null Apr 13 19:30:40 ts-xxxxxxxx-03 kernel: Lustre: 6252:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0003->NET_0x20000c0a8b91e_UUID netid 20000: select flavor null Apr 13 19:30:40 ts-xxxxxxxx-03 kernel: Lustre: 6288:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages Apr 13 19:30:40 ts-xxxxxxxx-03 kernel: Lustre: 6252:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages Apr 13 19:30:40 ts-xxxxxxxx-03 kernel: Lustre: 6249:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 13 19:36:40 ts-xxxxxxxx-03 kernel: Lustre: 6268:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from c0280a79-3da2-1372-b200-401132da7eda@172.30.246.121@tcp t0 exp (null) cur 1334345800 last 0 Apr 13 19:36:40 ts-xxxxxxxx-03 kernel: Lustre: 6281:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001->NET_0x20000ac1ef679_UUID netid 20000: select flavor null Apr 13 19:36:40 ts-xxxxxxxx-03 kernel: Lustre: 6239:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0002->NET_0x20000ac1ef679_UUID netid 20000: select flavor null Apr 13 19:36:40 ts-xxxxxxxx-03 kernel: Lustre: 6281:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 3 previous similar messages Apr 13 19:36:40 ts-xxxxxxxx-03 kernel: Lustre: 6239:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 3 previous similar messages Apr 13 19:36:40 ts-xxxxxxxx-03 kernel: Lustre: 6268:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 13 19:45:07 ts-xxxxxxxx-03 kernel: Lustre: 6287:0:(ldlm_lib.c:877:target_handle_connect()) xxxxxx-OST0000: connection from 0f85e1f5-64fa-b5ac-1d0c-4da05b799959@192.168.185.32@tcp t0 exp (null) cur 1334346307 last 0 Apr 13 19:45:07 ts-xxxxxxxx-03 kernel: Lustre: 6269:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import xxxxxx-OST0001-> netid 20000: select flavor null Apr 13 19:45:07 ts-xxxxxxxx-03 kernel: Lustre: 6269:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages Apr 13 19:45:07 ts-xxxxxxxx-03 kernel: Lustre: 6287:0:(ldlm_lib.c:877:target_handle_connect()) Skipped 5 previous similar messages Apr 14 00:45:15 ts-xxxxxxxx-03 ts-log-date: Timestamp 1334364315 Apr 14 00:46:36 ts-xxxxxxxx-03 ts-log-date: Timestamp 1334364395 Apr 15 00:45:15 ts-xxxxxxxx-03 ts-log-date: Timestamp 1334450715 Apr 15 00:46:37 ts-xxxxxxxx-03 ts-log-date: Timestamp 1334450797 Apr 15 04:37:16 ts-xxxxxxxx-03 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="3844" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'restart'. Apr 15 04:37:16 ts-xxxxxxxx-03 kernel: Kernel logging (proc) stopped.