Jun 19 00:00:01 weisshorn10 CROND[119740]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:00:01 weisshorn12 CROND[119612]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:00:01 weisshorn03 CROND[120362]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:00:01 weisshorn11 CROND[120224]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:00:01 weisshorn09 CROND[119401]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:00:01 weisshorn14 CROND[48276]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:00:01 weisshorn05 CROND[119716]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:00:01 weisshorn13 CROND[123030]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:00:01 weisshorn06 CROND[120563]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:00:01 weisshorn07 CROND[119828]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:00:01 weisshorn02 CROND[12819]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:00:01 weisshorn01 CROND[46161]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:00:01 weisshorn08 CROND[118476]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:00:01 weisshorn04 CROND[119180]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:00:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:00:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:00:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:00:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:01:01 weisshorn01 CROND[46170]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 00:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4617 starting 0anacron
Jun 19 00:01:01 weisshorn08 CROND[118481]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 00:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1184 starting 0anacron
Jun 19 00:01:01 weisshorn01 anacron[46181]: Anacron started on 2012-06-19
Jun 19 00:01:01 weisshorn04 CROND[119185]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 00:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4618 finished 0anacron
Jun 19 00:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1191 starting 0anacron
Jun 19 00:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4617 starting mcelog.cron
Jun 19 00:01:01 weisshorn01 anacron[46181]: Jobs will be executed sequentially
Jun 19 00:01:01 weisshorn01 anacron[46181]: Normal exit (0 jobs run)
Jun 19 00:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4619 finished mcelog.cron
Jun 19 00:01:01 weisshorn04 anacron[119196]: Anacron started on 2012-06-19
Jun 19 00:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1191 finished 0anacron
Jun 19 00:01:01 weisshorn08 anacron[118492]: Anacron started on 2012-06-19
Jun 19 00:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1191 starting mcelog.cron
Jun 19 00:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1184 finished 0anacron
Jun 19 00:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1184 starting mcelog.cron
Jun 19 00:01:01 weisshorn04 anacron[119196]: Jobs will be executed sequentially
Jun 19 00:01:01 weisshorn04 anacron[119196]: Normal exit (0 jobs run)
Jun 19 00:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1192 finished mcelog.cron
Jun 19 00:01:01 weisshorn08 anacron[118492]: Jobs will be executed sequentially
Jun 19 00:01:01 weisshorn08 anacron[118492]: Normal exit (0 jobs run)
Jun 19 00:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1185 finished mcelog.cron
Jun 19 00:01:01 weisshorn10 CROND[119745]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 00:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1197 starting 0anacron
Jun 19 00:01:01 weisshorn10 anacron[119757]: Anacron started on 2012-06-19
Jun 19 00:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1197 finished 0anacron
Jun 19 00:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1197 starting mcelog.cron
Jun 19 00:01:01 weisshorn12 CROND[119617]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 00:01:01 weisshorn10 anacron[119757]: Jobs will be executed sequentially
Jun 19 00:01:01 weisshorn10 anacron[119757]: Normal exit (0 jobs run)
Jun 19 00:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1197 finished mcelog.cron
Jun 19 00:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1196 starting 0anacron
Jun 19 00:01:01 weisshorn12 anacron[119629]: Anacron started on 2012-06-19
Jun 19 00:01:01 weisshorn12 anacron[119629]: Jobs will be executed sequentially
Jun 19 00:01:01 weisshorn12 anacron[119629]: Normal exit (0 jobs run)
Jun 19 00:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1196 finished 0anacron
Jun 19 00:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1196 starting mcelog.cron
Jun 19 00:01:01 weisshorn03 CROND[120367]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 00:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1196 finished mcelog.cron
Jun 19 00:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1203 starting 0anacron
Jun 19 00:01:01 weisshorn03 anacron[120379]: Anacron started on 2012-06-19
Jun 19 00:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1203 finished 0anacron
Jun 19 00:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1203 starting mcelog.cron
Jun 19 00:01:01 weisshorn11 CROND[120229]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 00:01:01 weisshorn03 anacron[120379]: Jobs will be executed sequentially
Jun 19 00:01:01 weisshorn03 anacron[120379]: Normal exit (0 jobs run)
Jun 19 00:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1202 starting 0anacron
Jun 19 00:01:01 weisshorn09 CROND[119406]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 00:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1194 starting 0anacron
Jun 19 00:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1203 finished mcelog.cron
Jun 19 00:01:01 weisshorn09 anacron[119418]: Anacron started on 2012-06-19
Jun 19 00:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1194 finished 0anacron
Jun 19 00:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1194 starting mcelog.cron
Jun 19 00:01:01 weisshorn11 anacron[120241]: Anacron started on 2012-06-19
Jun 19 00:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1202 finished 0anacron
Jun 19 00:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1202 starting mcelog.cron
Jun 19 00:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1194 finished mcelog.cron
Jun 19 00:01:01 weisshorn09 anacron[119418]: Jobs will be executed sequentially
Jun 19 00:01:01 weisshorn09 anacron[119418]: Normal exit (0 jobs run)
Jun 19 00:01:01 weisshorn11 anacron[120241]: Jobs will be executed sequentially
Jun 19 00:01:01 weisshorn11 anacron[120241]: Normal exit (0 jobs run)
Jun 19 00:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1202 finished mcelog.cron
Jun 19 00:01:01 weisshorn14 CROND[48281]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 00:01:01 weisshorn05 CROND[119721]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 00:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4828 starting 0anacron
Jun 19 00:01:01 weisshorn14 anacron[48293]: Anacron started on 2012-06-19
Jun 19 00:01:01 weisshorn14 anacron[48293]: Jobs will be executed sequentially
Jun 19 00:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4829 finished 0anacron
Jun 19 00:01:01 weisshorn14 anacron[48293]: Normal exit (0 jobs run)
Jun 19 00:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4828 starting mcelog.cron
Jun 19 00:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1197 starting 0anacron
Jun 19 00:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4830 finished mcelog.cron
Jun 19 00:01:01 weisshorn06 CROND[120568]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 00:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1205 starting 0anacron
Jun 19 00:01:01 weisshorn13 CROND[123035]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 00:01:01 weisshorn05 anacron[119733]: Anacron started on 2012-06-19
Jun 19 00:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1197 finished 0anacron
Jun 19 00:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1197 starting mcelog.cron
Jun 19 00:01:01 weisshorn05 anacron[119733]: Jobs will be executed sequentially
Jun 19 00:01:01 weisshorn05 anacron[119733]: Normal exit (0 jobs run)
Jun 19 00:01:01 weisshorn06 anacron[120580]: Anacron started on 2012-06-19
Jun 19 00:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1205 finished 0anacron
Jun 19 00:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1205 starting mcelog.cron
Jun 19 00:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1197 finished mcelog.cron
Jun 19 00:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1230 starting 0anacron
Jun 19 00:01:01 weisshorn06 anacron[120580]: Jobs will be executed sequentially
Jun 19 00:01:01 weisshorn06 anacron[120580]: Normal exit (0 jobs run)
Jun 19 00:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1205 finished mcelog.cron
Jun 19 00:01:01 weisshorn13 anacron[123047]: Anacron started on 2012-06-19
Jun 19 00:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1230 finished 0anacron
Jun 19 00:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1230 starting mcelog.cron
Jun 19 00:01:01 weisshorn07 CROND[119833]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 00:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1198 starting 0anacron
Jun 19 00:01:01 weisshorn13 anacron[123047]: Jobs will be executed sequentially
Jun 19 00:01:01 weisshorn13 anacron[123047]: Normal exit (0 jobs run)
Jun 19 00:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1230 finished mcelog.cron
Jun 19 00:01:01 weisshorn07 anacron[119845]: Anacron started on 2012-06-19
Jun 19 00:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1198 finished 0anacron
Jun 19 00:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1198 starting mcelog.cron
Jun 19 00:01:01 weisshorn02 CROND[12913]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 00:01:02 weisshorn02 run-parts(/etc/cron.hourly)[1291 starting 0anacron
Jun 19 00:01:02 weisshorn07 anacron[119845]: Jobs will be executed sequentially
Jun 19 00:01:02 weisshorn07 anacron[119845]: Normal exit (0 jobs run)
Jun 19 00:01:02 weisshorn07 run-parts(/etc/cron.hourly)[1198 finished mcelog.cron
Jun 19 00:01:02 weisshorn02 anacron[12924]: Anacron started on 2012-06-19
Jun 19 00:01:02 weisshorn02 anacron[12924]: Jobs will be executed sequentially
Jun 19 00:01:02 weisshorn02 anacron[12924]: Normal exit (0 jobs run)
Jun 19 00:01:02 weisshorn02 run-parts(/etc/cron.hourly)[1292 finished 0anacron
Jun 19 00:01:02 weisshorn02 run-parts(/etc/cron.hourly)[1291 starting mcelog.cron
Jun 19 00:01:02 weisshorn02 run-parts(/etc/cron.hourly)[1293 finished mcelog.cron
Jun 19 00:01:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:01:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:01:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:01:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:02:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:02:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:02:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:02:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:03:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:03:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:03:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:03:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:04:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:04:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:04:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:04:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:05:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:05:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:05:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:05:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:06:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:06:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:06:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:06:36 weisshorn03 nslcd[3944]: [5bcf61] ldap_result() failed: Can't contact LDAP server
Jun 19 00:06:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:07:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:07:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:07:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:07:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:08:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:08:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:08:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:08:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:09:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:09:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:09:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:09:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:10:01 weisshorn07 CROND[119906]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:10:01 weisshorn02 CROND[13758]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:10:01 weisshorn01 CROND[46244]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:10:01 weisshorn01 CROND[46246]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 00:10:01 weisshorn04 CROND[119237]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:10:01 weisshorn08 CROND[118531]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:10:01 weisshorn14 sshd[48322]: Accepted publickey for root from 148.187.12.231 port 49537 ssh2
Jun 19 00:10:01 weisshorn14 sshd[48322]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 00:10:01 weisshorn04 sshd[119234]: Accepted publickey for root from 148.187.12.231 port 44170 ssh2
Jun 19 00:10:01 weisshorn03 sshd[120418]: Accepted publickey for root from 148.187.12.231 port 50414 ssh2
Jun 19 00:10:01 weisshorn09 sshd[119451]: Accepted publickey for root from 148.187.12.231 port 57859 ssh2
Jun 19 00:10:01 weisshorn03 sshd[120418]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 00:10:01 weisshorn04 sshd[119234]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 00:10:01 weisshorn09 sshd[119451]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 00:10:01 weisshorn07 sshd[119908]: Accepted publickey for root from 148.187.12.231 port 48229 ssh2
Jun 19 00:10:01 weisshorn07 sshd[119908]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 00:10:01 weisshorn13 sshd[123088]: Accepted publickey for root from 148.187.12.231 port 44586 ssh2
Jun 19 00:10:01 weisshorn13 sshd[123088]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 00:10:01 weisshorn08 sshd[118528]: Accepted publickey for root from 148.187.12.231 port 40225 ssh2
Jun 19 00:10:01 weisshorn06 sshd[120640]: Accepted publickey for root from 148.187.12.231 port 44674 ssh2
Jun 19 00:10:01 weisshorn08 sshd[118528]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 00:10:01 weisshorn06 sshd[120640]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 00:10:01 weisshorn12 sshd[119667]: Accepted publickey for root from 148.187.12.231 port 46327 ssh2
Jun 19 00:10:01 weisshorn12 sshd[119667]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 00:10:01 weisshorn14 sshd[48322]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 00:10:01 weisshorn14 sshd[48322]: pam_unix(sshd:session): session closed for user root
Jun 19 00:10:01 weisshorn05 sshd[119766]: Accepted publickey for root from 148.187.12.231 port 54619 ssh2
Jun 19 00:10:01 weisshorn05 sshd[119766]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 00:10:01 weisshorn11 sshd[120320]: Accepted publickey for root from 148.187.12.231 port 45102 ssh2
Jun 19 00:10:01 weisshorn10 sshd[119795]: Accepted publickey for root from 148.187.12.231 port 36890 ssh2
Jun 19 00:10:01 weisshorn11 sshd[120320]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 00:10:01 weisshorn09 sshd[119451]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 00:10:01 weisshorn09 sshd[119451]: pam_unix(sshd:session): session closed for user root
Jun 19 00:10:01 weisshorn03 sshd[120418]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 00:10:01 weisshorn03 sshd[120418]: pam_unix(sshd:session): session closed for user root
Jun 19 00:10:01 weisshorn04 sshd[119234]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 00:10:01 weisshorn04 sshd[119234]: pam_unix(sshd:session): session closed for user root
Jun 19 00:10:01 weisshorn10 sshd[119795]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 00:10:01 weisshorn07 sshd[119908]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 00:10:01 weisshorn07 sshd[119908]: pam_unix(sshd:session): session closed for user root
Jun 19 00:10:01 weisshorn10 CROND[119800]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:10:01 weisshorn05 sshd[119766]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 00:10:01 weisshorn05 sshd[119766]: pam_unix(sshd:session): session closed for user root
Jun 19 00:10:01 weisshorn06 sshd[120640]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 00:10:01 weisshorn06 sshd[120640]: pam_unix(sshd:session): session closed for user root
Jun 19 00:10:01 weisshorn13 sshd[123088]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 00:10:01 weisshorn13 sshd[123088]: pam_unix(sshd:session): session closed for user root
Jun 19 00:10:01 weisshorn12 sshd[119667]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 00:10:01 weisshorn12 sshd[119667]: pam_unix(sshd:session): session closed for user root
Jun 19 00:10:01 weisshorn08 sshd[118528]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 00:10:01 weisshorn08 sshd[118528]: pam_unix(sshd:session): session closed for user root
Jun 19 00:10:01 weisshorn12 CROND[119690]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:10:01 weisshorn03 CROND[120441]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:10:01 weisshorn11 sshd[120320]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 00:10:01 weisshorn11 sshd[120320]: pam_unix(sshd:session): session closed for user root
Jun 19 00:10:01 weisshorn10 sshd[119795]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 00:10:01 weisshorn10 sshd[119795]: pam_unix(sshd:session): session closed for user root
Jun 19 00:10:01 weisshorn09 CROND[119474]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:10:01 weisshorn01 postfix/pickup[45858]: AA9E192224F: uid=0 from=<root>
Jun 19 00:10:01 weisshorn01 postfix/cleanup[46277]: AA9E192224F: message-id=<20120618221001.AA9E192224F@weisshorn01.admin.cscs.ch>
Jun 19 00:10:01 weisshorn11 CROND[120343]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:10:01 weisshorn01 postfix/qmgr[2710]: AA9E192224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 00:10:01 weisshorn14 CROND[48345]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:10:01 weisshorn01 postfix/local[46279]: AA9E192224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=0.46, delays=0.41/0.01/0/0.04, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 00:10:01 weisshorn01 postfix/qmgr[2710]: AA9E192224F: removed
Jun 19 00:10:01 weisshorn05 CROND[119789]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:10:01 weisshorn06 CROND[120663]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:10:01 weisshorn13 CROND[123111]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:10:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:10:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:10:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:10:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:11:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:11:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:11:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:11:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:12:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:12:06 weisshorn02 nslcd[2099]: [d601af] ldap_result() failed: Can't contact LDAP server
Jun 19 00:12:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:12:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:12:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:13:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:13:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:13:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:13:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:14:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:14:05 weisshorn05 ntpd[4877]: synchronized to 148.187.18.120, stratum 2
Jun 19 00:14:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:14:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:14:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:15:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:15:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:15:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:15:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:16:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:16:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:16:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:16:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:17:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:17:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:17:17 weisshorn07 ntpd[4804]: synchronized to 148.187.18.120, stratum 2
Jun 19 00:17:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:17:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:18:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:18:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:18:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:18:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:19:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:19:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:19:22 weisshorn08 rhnsd[118573]: running program /usr/sbin/rhn_check
Jun 19 00:19:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:19:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:20:01 weisshorn13 CROND[123175]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:20:01 weisshorn02 nslcd[2099]: [c77402] ldap_result() failed: Can't contact LDAP server
Jun 19 00:20:01 weisshorn02 CROND[14681]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:20:01 weisshorn07 CROND[119947]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:20:01 weisshorn04 CROND[119280]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:20:01 weisshorn08 CROND[118576]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:20:01 weisshorn10 CROND[119841]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:20:01 weisshorn12 CROND[119754]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:20:01 weisshorn03 CROND[120464]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:20:01 weisshorn01 CROND[46332]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:20:01 weisshorn09 CROND[119502]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:20:01 weisshorn11 CROND[120366]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:20:01 weisshorn14 nslcd[3437]: [ffae18] ldap_result() failed: Can't contact LDAP server
Jun 19 00:20:01 weisshorn14 CROND[48421]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:20:01 weisshorn05 CROND[119818]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:20:02 weisshorn06 CROND[120686]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:20:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:20:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:20:26 weisshorn01 rhnsd[46334]: running program /usr/sbin/rhn_check
Jun 19 00:20:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:20:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:21:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:21:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:21:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:21:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:22:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:22:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:22:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:22:45 weisshorn13 ntpd[4726]: synchronized to 148.187.18.120, stratum 2
Jun 19 00:22:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:23:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:23:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:23:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:23:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:24:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:24:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:24:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:24:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:25:00 weisshorn09 nslcd[3954]: [95e50a] ldap_result() failed: Can't contact LDAP server
Jun 19 00:25:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:25:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:25:23 weisshorn08 nslcd[3946]: [24de5b] ldap_result() failed: Can't contact LDAP server
Jun 19 00:25:24 weisshorn06 nslcd[3953]: [206b8e] ldap_result() failed: Can't contact LDAP server
Jun 19 00:25:26 weisshorn04 nslcd[3885]: [95e50a] ldap_result() failed: Can't contact LDAP server
Jun 19 00:25:27 weisshorn07 nslcd[4297]: [24de5b] ldap_result() failed: Can't contact LDAP server
Jun 19 00:25:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:25:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:25:47 weisshorn12 nslcd[3954]: [24de5b] ldap_result() failed: Can't contact LDAP server
Jun 19 00:25:47 weisshorn11 nslcd[3954]: [9efb0d] ldap_result() failed: Can't contact LDAP server
Jun 19 00:25:49 weisshorn13 nslcd[4224]: [24de5b] ldap_result() failed: Can't contact LDAP server
Jun 19 00:26:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:26:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:26:26 weisshorn05 nslcd[4378]: [8efe17] ldap_result() failed: Can't contact LDAP server
Jun 19 00:26:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:26:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:27:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:27:05 weisshorn02 rhnsd[15364]: running program /usr/sbin/rhn_check
Jun 19 00:27:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:27:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:27:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:28:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:28:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:28:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:28:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:29:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:29:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:29:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:29:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:30:01 weisshorn05 CROND[119884]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:30:01 weisshorn06 CROND[120730]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:30:01 weisshorn13 CROND[123206]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:30:01 weisshorn02 CROND[15640]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:30:01 weisshorn07 CROND[119996]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:30:01 weisshorn04 CROND[119346]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:30:01 weisshorn08 CROND[118643]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:30:01 weisshorn10 CROND[119911]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:30:01 weisshorn12 CROND[119779]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:30:01 weisshorn01 CROND[46422]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:30:01 weisshorn03 CROND[120529]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:30:01 weisshorn09 CROND[119568]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:30:01 weisshorn11 CROND[120399]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:30:01 weisshorn14 CROND[48445]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:30:02 weisshorn10 nslcd[4294]: [8efe17] ldap_result() failed: Can't contact LDAP server
Jun 19 00:30:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:30:16 weisshorn05 nslcd[4378]: [36b29f] ldap_result() failed: Can't contact LDAP server
Jun 19 00:30:16 weisshorn12 nslcd[3954]: [8efe17] ldap_result() failed: Can't contact LDAP server
Jun 19 00:30:16 weisshorn04 nslcd[3885]: [5bcf61] ldap_result() failed: Can't contact LDAP server
Jun 19 00:30:16 weisshorn09 nslcd[3954]: [5bcf61] ldap_result() failed: Can't contact LDAP server
Jun 19 00:30:16 weisshorn03 nslcd[3944]: [1dd403] ldap_result() failed: Can't contact LDAP server
Jun 19 00:30:16 weisshorn13 nslcd[4224]: [8efe17] ldap_result() failed: Can't contact LDAP server
Jun 19 00:30:16 weisshorn08 nslcd[3946]: [8efe17] ldap_result() failed: Can't contact LDAP server
Jun 19 00:30:16 weisshorn07 nslcd[4297]: [8efe17] ldap_result() failed: Can't contact LDAP server
Jun 19 00:30:16 weisshorn06 nslcd[3953]: [34d3d4] ldap_result() failed: Can't contact LDAP server
Jun 19 00:30:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:30:17 weisshorn11 nslcd[3954]: [0802be] ldap_result() failed: Can't contact LDAP server
Jun 19 00:30:26 weisshorn05 rhnsd[119887]: running program /usr/sbin/rhn_check
Jun 19 00:30:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:30:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:31:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:31:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:31:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:31:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:32:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:32:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:32:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:32:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:33:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:33:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:33:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:33:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:34:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:34:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:34:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:34:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:35:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:35:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:35:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:35:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:36:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:36:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:36:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:36:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:37:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:37:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:37:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:37:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:38:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:38:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:38:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:38:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:39:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:39:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:39:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:39:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:40:01 weisshorn05 CROND[119912]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:40:01 weisshorn06 CROND[120789]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:40:01 weisshorn13 CROND[123242]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:40:01 weisshorn02 CROND[16562]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:40:01 weisshorn07 CROND[120050]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:40:01 weisshorn04 CROND[119378]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:40:01 weisshorn08 CROND[118673]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:40:01 weisshorn10 CROND[119938]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:40:01 weisshorn12 CROND[119852]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:40:01 weisshorn01 CROND[46486]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:40:01 weisshorn09 CROND[119595]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:40:01 weisshorn03 CROND[120561]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:40:02 weisshorn14 CROND[48520]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:40:02 weisshorn11 CROND[120464]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:40:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:40:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:40:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:40:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:41:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:41:16 weisshorn14 nslcd[3437]: [3e47a8] ldap_result() failed: Can't contact LDAP server
Jun 19 00:41:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:41:24 weisshorn04 ntpd[4387]: synchronized to 148.187.18.120, stratum 2
Jun 19 00:41:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:41:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:42:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:42:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:42:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:42:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:43:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:43:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:43:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:43:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:44:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:44:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:44:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:44:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:45:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:45:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:45:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:45:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:46:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:46:16 weisshorn10 nslcd[4294]: [36b29f] ldap_result() failed: Can't contact LDAP server
Jun 19 00:46:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:46:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:46:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:47:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:47:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:47:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:47:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:48:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:48:09 weisshorn03 ntpd[4444]: synchronized to 148.187.18.120, stratum 2
Jun 19 00:48:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:48:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:48:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:49:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:49:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:49:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:49:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:50:01 weisshorn03 CROND[120591]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:50:01 weisshorn14 CROND[48546]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:50:01 weisshorn11 CROND[120488]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:50:01 weisshorn06 CROND[120808]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:50:01 weisshorn05 CROND[119946]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:50:01 weisshorn02 CROND[17482]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:50:01 weisshorn13 CROND[123301]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:50:01 weisshorn04 CROND[119408]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:50:01 weisshorn07 CROND[120074]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:50:01 weisshorn08 CROND[118697]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:50:01 weisshorn10 CROND[119963]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:50:01 weisshorn12 CROND[119876]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:50:01 weisshorn09 CROND[119624]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:50:02 weisshorn01 CROND[46537]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 00:50:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:50:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:50:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:50:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:51:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:51:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:51:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:51:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:52:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:52:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:52:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:52:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:53:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:53:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:53:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:53:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:54:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:54:10 weisshorn10 ntpd[4792]: synchronized to 148.187.18.120, stratum 2
Jun 19 00:54:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:54:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:54:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:55:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:55:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:55:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:55:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:56:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:56:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:56:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:56:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:56:59 weisshorn09 rhnsd[119678]: running program /usr/sbin/rhn_check
Jun 19 00:57:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:57:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:57:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:57:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:58:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:58:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:58:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:58:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:59:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:59:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:59:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 00:59:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:00:01 weisshorn09 CROND[119691]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:00:01 weisshorn03 CROND[120656]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:00:01 weisshorn14 CROND[48570]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:00:01 weisshorn01 CROND[46632]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:00:01 weisshorn11 CROND[120525]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:00:01 weisshorn05 CROND[120006]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:00:01 weisshorn06 CROND[120851]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:00:01 weisshorn13 CROND[123330]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:00:01 weisshorn02 CROND[18440]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:00:01 weisshorn04 CROND[119468]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:00:01 weisshorn07 CROND[120117]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:00:01 weisshorn08 CROND[118764]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:00:01 weisshorn10 CROND[120034]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:00:01 weisshorn12 CROND[119905]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:00:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:00:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:00:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:00:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:01:01 weisshorn12 CROND[119910]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 01:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1199 starting 0anacron
Jun 19 01:01:01 weisshorn12 anacron[119921]: Anacron started on 2012-06-19
Jun 19 01:01:01 weisshorn12 anacron[119921]: Jobs will be executed sequentially
Jun 19 01:01:01 weisshorn12 anacron[119921]: Normal exit (0 jobs run)
Jun 19 01:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1199 finished 0anacron
Jun 19 01:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1199 starting mcelog.cron
Jun 19 01:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1199 finished mcelog.cron
Jun 19 01:01:01 weisshorn09 CROND[119696]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 01:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1196 starting 0anacron
Jun 19 01:01:01 weisshorn09 anacron[119708]: Anacron started on 2012-06-19
Jun 19 01:01:01 weisshorn09 anacron[119708]: Jobs will be executed sequentially
Jun 19 01:01:01 weisshorn09 anacron[119708]: Normal exit (0 jobs run)
Jun 19 01:01:01 weisshorn03 CROND[120661]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 01:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1197 finished 0anacron
Jun 19 01:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1196 starting mcelog.cron
Jun 19 01:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1206 starting 0anacron
Jun 19 01:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1197 finished mcelog.cron
Jun 19 01:01:01 weisshorn03 anacron[120673]: Anacron started on 2012-06-19
Jun 19 01:01:01 weisshorn03 anacron[120673]: Jobs will be executed sequentially
Jun 19 01:01:01 weisshorn03 anacron[120673]: Normal exit (0 jobs run)
Jun 19 01:01:01 weisshorn14 CROND[48575]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 01:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1206 finished 0anacron
Jun 19 01:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1206 starting mcelog.cron
Jun 19 01:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4857 starting 0anacron
Jun 19 01:01:01 weisshorn14 anacron[48587]: Anacron started on 2012-06-19
Jun 19 01:01:01 weisshorn14 anacron[48587]: Jobs will be executed sequentially
Jun 19 01:01:01 weisshorn14 anacron[48587]: Normal exit (0 jobs run)
Jun 19 01:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4858 finished 0anacron
Jun 19 01:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4857 starting mcelog.cron
Jun 19 01:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1206 finished mcelog.cron
Jun 19 01:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4859 finished mcelog.cron
Jun 19 01:01:01 weisshorn01 CROND[46635]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 01:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4663 starting 0anacron
Jun 19 01:01:01 weisshorn11 CROND[120530]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 01:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1205 starting 0anacron
Jun 19 01:01:01 weisshorn11 anacron[120542]: Anacron started on 2012-06-19
Jun 19 01:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1205 finished 0anacron
Jun 19 01:01:01 weisshorn05 CROND[120011]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 01:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1205 starting mcelog.cron
Jun 19 01:01:01 weisshorn11 anacron[120542]: Jobs will be executed sequentially
Jun 19 01:01:01 weisshorn11 anacron[120542]: Normal exit (0 jobs run)
Jun 19 01:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1205 finished mcelog.cron
Jun 19 01:01:01 weisshorn01 anacron[46647]: Anacron started on 2012-06-19
Jun 19 01:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4664 finished 0anacron
Jun 19 01:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1200 starting 0anacron
Jun 19 01:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4663 starting mcelog.cron
Jun 19 01:01:01 weisshorn06 CROND[120856]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 01:01:01 weisshorn01 anacron[46647]: Jobs will be executed sequentially
Jun 19 01:01:01 weisshorn01 anacron[46647]: Normal exit (0 jobs run)
Jun 19 01:01:01 weisshorn05 anacron[120023]: Anacron started on 2012-06-19
Jun 19 01:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1200 finished 0anacron
Jun 19 01:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1208 starting 0anacron
Jun 19 01:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1200 starting mcelog.cron
Jun 19 01:01:01 weisshorn06 anacron[120868]: Anacron started on 2012-06-19
Jun 19 01:01:01 weisshorn06 anacron[120868]: Jobs will be executed sequentially
Jun 19 01:01:01 weisshorn06 anacron[120868]: Normal exit (0 jobs run)
Jun 19 01:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4665 finished mcelog.cron
Jun 19 01:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1208 finished 0anacron
Jun 19 01:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1208 starting mcelog.cron
Jun 19 01:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1200 finished mcelog.cron
Jun 19 01:01:01 weisshorn05 anacron[120023]: Jobs will be executed sequentially
Jun 19 01:01:01 weisshorn05 anacron[120023]: Normal exit (0 jobs run)
Jun 19 01:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1208 finished mcelog.cron
Jun 19 01:01:01 weisshorn04 CROND[119473]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 01:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1194 starting 0anacron
Jun 19 01:01:01 weisshorn07 CROND[120122]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 01:01:01 weisshorn02 CROND[18534]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 01:01:01 weisshorn13 CROND[123335]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 01:01:01 weisshorn02 run-parts(/etc/cron.hourly)[1853 starting 0anacron
Jun 19 01:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1201 starting 0anacron
Jun 19 01:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1233 starting 0anacron
Jun 19 01:01:01 weisshorn02 anacron[18545]: Anacron started on 2012-06-19
Jun 19 01:01:01 weisshorn02 anacron[18545]: Jobs will be executed sequentially
Jun 19 01:01:01 weisshorn02 anacron[18545]: Normal exit (0 jobs run)
Jun 19 01:01:01 weisshorn02 run-parts(/etc/cron.hourly)[1854 finished 0anacron
Jun 19 01:01:01 weisshorn02 run-parts(/etc/cron.hourly)[1853 starting mcelog.cron
Jun 19 01:01:01 weisshorn13 anacron[123347]: Anacron started on 2012-06-19
Jun 19 01:01:01 weisshorn13 anacron[123347]: Jobs will be executed sequentially
Jun 19 01:01:01 weisshorn13 anacron[123347]: Normal exit (0 jobs run)
Jun 19 01:01:01 weisshorn04 anacron[119485]: Anacron started on 2012-06-19
Jun 19 01:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1233 finished 0anacron
Jun 19 01:01:01 weisshorn04 anacron[119485]: Jobs will be executed sequentially
Jun 19 01:01:01 weisshorn04 anacron[119485]: Normal exit (0 jobs run)
Jun 19 01:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1194 finished 0anacron
Jun 19 01:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1233 starting mcelog.cron
Jun 19 01:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1194 starting mcelog.cron
Jun 19 01:01:01 weisshorn02 run-parts(/etc/cron.hourly)[1855 finished mcelog.cron
Jun 19 01:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1233 finished mcelog.cron
Jun 19 01:01:01 weisshorn07 anacron[120134]: Anacron started on 2012-06-19
Jun 19 01:01:01 weisshorn07 anacron[120134]: Jobs will be executed sequentially
Jun 19 01:01:01 weisshorn07 anacron[120134]: Normal exit (0 jobs run)
Jun 19 01:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1201 finished 0anacron
Jun 19 01:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1194 finished mcelog.cron
Jun 19 01:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1201 starting mcelog.cron
Jun 19 01:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1201 finished mcelog.cron
Jun 19 01:01:01 weisshorn08 CROND[118769]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 01:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1187 starting 0anacron
Jun 19 01:01:01 weisshorn10 CROND[120039]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 01:01:01 weisshorn08 anacron[118781]: Anacron started on 2012-06-19
Jun 19 01:01:01 weisshorn08 anacron[118781]: Jobs will be executed sequentially
Jun 19 01:01:01 weisshorn08 anacron[118781]: Normal exit (0 jobs run)
Jun 19 01:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1187 finished 0anacron
Jun 19 01:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1187 starting mcelog.cron
Jun 19 01:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1200 starting 0anacron
Jun 19 01:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1187 finished mcelog.cron
Jun 19 01:01:01 weisshorn10 anacron[120051]: Anacron started on 2012-06-19
Jun 19 01:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1200 finished 0anacron
Jun 19 01:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1200 starting mcelog.cron
Jun 19 01:01:01 weisshorn10 anacron[120051]: Jobs will be executed sequentially
Jun 19 01:01:01 weisshorn10 anacron[120051]: Normal exit (0 jobs run)
Jun 19 01:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1200 finished mcelog.cron
Jun 19 01:01:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:01:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:01:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:01:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:02:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:02:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:02:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:02:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:03:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:03:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:03:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:03:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:04:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:04:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:04:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:04:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:05:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:05:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:05:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:05:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:06:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:06:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:06:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:06:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:07:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:07:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:07:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:07:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:08:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:08:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:08:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:08:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:09:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:09:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:09:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:09:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:10:01 weisshorn12 CROND[120000]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:10:01 weisshorn09 CROND[119746]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:10:01 weisshorn03 CROND[120706]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:10:01 weisshorn14 CROND[48668]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:10:01 weisshorn11 CROND[120608]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:10:01 weisshorn01 CROND[46702]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:10:01 weisshorn01 CROND[46704]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 01:10:01 weisshorn05 CROND[120061]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:10:01 weisshorn06 CROND[120933]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:10:01 weisshorn02 CROND[19384]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:10:01 weisshorn04 CROND[119524]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:10:01 weisshorn13 CROND[123431]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:10:01 weisshorn07 CROND[120196]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:10:01 weisshorn14 sshd[48670]: Accepted publickey for root from 148.187.12.231 port 49550 ssh2
Jun 19 01:10:01 weisshorn14 sshd[48670]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 01:10:01 weisshorn09 sshd[119748]: Accepted publickey for root from 148.187.12.231 port 57876 ssh2
Jun 19 01:10:01 weisshorn09 sshd[119748]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 01:10:01 weisshorn10 sshd[120088]: Accepted publickey for root from 148.187.12.231 port 36910 ssh2
Jun 19 01:10:01 weisshorn10 sshd[120088]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 01:10:01 weisshorn06 sshd[120935]: Accepted publickey for root from 148.187.12.231 port 44702 ssh2
Jun 19 01:10:01 weisshorn03 sshd[120708]: Accepted publickey for root from 148.187.12.231 port 50435 ssh2
Jun 19 01:10:01 weisshorn06 sshd[120935]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 01:10:01 weisshorn03 sshd[120708]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 01:10:01 weisshorn04 sshd[119522]: Accepted publickey for root from 148.187.12.231 port 44193 ssh2
Jun 19 01:10:01 weisshorn04 sshd[119522]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 01:10:01 weisshorn13 sshd[123428]: Accepted publickey for root from 148.187.12.231 port 44602 ssh2
Jun 19 01:10:01 weisshorn11 sshd[120610]: Accepted publickey for root from 148.187.12.231 port 45124 ssh2
Jun 19 01:10:01 weisshorn12 sshd[120002]: Accepted publickey for root from 148.187.12.231 port 46344 ssh2
Jun 19 01:10:01 weisshorn08 sshd[118816]: Accepted publickey for root from 148.187.12.231 port 40237 ssh2
Jun 19 01:10:01 weisshorn13 sshd[123428]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 01:10:01 weisshorn12 sshd[120002]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 01:10:01 weisshorn08 sshd[118816]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 01:10:01 weisshorn11 sshd[120610]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 01:10:01 weisshorn14 sshd[48670]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 01:10:01 weisshorn14 sshd[48670]: pam_unix(sshd:session): session closed for user root
Jun 19 01:10:01 weisshorn07 sshd[120193]: Accepted publickey for root from 148.187.12.231 port 48252 ssh2
Jun 19 01:10:01 weisshorn05 sshd[120063]: Accepted publickey for root from 148.187.12.231 port 54647 ssh2
Jun 19 01:10:01 weisshorn07 sshd[120193]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 01:10:01 weisshorn08 CROND[118829]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:10:01 weisshorn05 sshd[120063]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 01:10:01 weisshorn09 sshd[119748]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 01:10:01 weisshorn09 sshd[119748]: pam_unix(sshd:session): session closed for user root
Jun 19 01:10:01 weisshorn04 sshd[119522]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 01:10:01 weisshorn04 sshd[119522]: pam_unix(sshd:session): session closed for user root
Jun 19 01:10:01 weisshorn06 sshd[120935]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 01:10:01 weisshorn06 sshd[120935]: pam_unix(sshd:session): session closed for user root
Jun 19 01:10:01 weisshorn03 sshd[120708]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 01:10:01 weisshorn03 sshd[120708]: pam_unix(sshd:session): session closed for user root
Jun 19 01:10:01 weisshorn13 sshd[123428]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 01:10:01 weisshorn13 sshd[123428]: pam_unix(sshd:session): session closed for user root
Jun 19 01:10:01 weisshorn10 CROND[120103]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:10:01 weisshorn12 sshd[120002]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 01:10:01 weisshorn12 sshd[120002]: pam_unix(sshd:session): session closed for user root
Jun 19 01:10:02 weisshorn11 sshd[120610]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 01:10:02 weisshorn11 sshd[120610]: pam_unix(sshd:session): session closed for user root
Jun 19 01:10:02 weisshorn07 sshd[120193]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 01:10:02 weisshorn07 sshd[120193]: pam_unix(sshd:session): session closed for user root
Jun 19 01:10:02 weisshorn10 sshd[120088]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 01:10:02 weisshorn10 sshd[120088]: pam_unix(sshd:session): session closed for user root
Jun 19 01:10:02 weisshorn08 sshd[118816]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 01:10:02 weisshorn08 sshd[118816]: pam_unix(sshd:session): session closed for user root
Jun 19 01:10:02 weisshorn05 sshd[120063]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 01:10:02 weisshorn05 sshd[120063]: pam_unix(sshd:session): session closed for user root
Jun 19 01:10:02 weisshorn01 postfix/pickup[46659]: 2A3CF92224F: uid=0 from=<root>
Jun 19 01:10:02 weisshorn01 postfix/cleanup[46734]: 2A3CF92224F: message-id=<20120618231002.2A3CF92224F@weisshorn01.admin.cscs.ch>
Jun 19 01:10:02 weisshorn01 postfix/qmgr[2710]: 2A3CF92224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 01:10:02 weisshorn01 postfix/local[46736]: 2A3CF92224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=0.39, delays=0.3/0.04/0/0.06, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 01:10:02 weisshorn01 postfix/qmgr[2710]: 2A3CF92224F: removed
Jun 19 01:10:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:10:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:10:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:10:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:11:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:11:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:11:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:11:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:12:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:12:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:12:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:12:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:13:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:13:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:13:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:13:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:14:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:14:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:14:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:14:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:15:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:15:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:15:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:15:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:16:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:16:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:16:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:16:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:17:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:17:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:17:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:17:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:18:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:18:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:18:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:18:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:19:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:19:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:19:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:19:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:20:01 weisshorn10 CROND[120129]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:20:01 weisshorn01 CROND[46794]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:20:01 weisshorn12 CROND[120040]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:20:01 weisshorn09 CROND[119791]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:20:01 weisshorn03 CROND[120756]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:20:01 weisshorn14 CROND[48714]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:20:01 weisshorn11 CROND[120653]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:20:01 weisshorn05 CROND[120111]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:20:01 weisshorn06 CROND[120973]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:20:01 weisshorn02 CROND[20302]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:20:01 weisshorn04 CROND[119573]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:20:01 weisshorn07 CROND[120239]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:20:01 weisshorn13 CROND[123469]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:20:02 weisshorn08 CROND[118862]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:20:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:20:16 weisshorn02 nslcd[2099]: [b5b1c6] ldap_result() failed: Can't contact LDAP server
Jun 19 01:20:16 weisshorn14 nslcd[3437]: [48f044] ldap_result() failed: Can't contact LDAP server
Jun 19 01:20:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:20:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:20:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:21:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:21:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:21:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:21:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:22:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:22:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:22:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:22:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:23:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:23:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:23:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:23:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:24:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:24:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:24:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:24:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:25:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:25:15 weisshorn09 nslcd[3954]: [1dd403] ldap_result() failed: Can't contact LDAP server
Jun 19 01:25:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:25:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:25:38 weisshorn08 nslcd[3946]: [36b29f] ldap_result() failed: Can't contact LDAP server
Jun 19 01:25:39 weisshorn06 nslcd[3953]: [c65e98] ldap_result() failed: Can't contact LDAP server
Jun 19 01:25:41 weisshorn04 nslcd[3885]: [1dd403] ldap_result() failed: Can't contact LDAP server
Jun 19 01:25:42 weisshorn07 nslcd[4297]: [36b29f] ldap_result() failed: Can't contact LDAP server
Jun 19 01:25:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:26:02 weisshorn12 nslcd[3954]: [36b29f] ldap_result() failed: Can't contact LDAP server
Jun 19 01:26:02 weisshorn11 nslcd[3954]: [66459b] ldap_result() failed: Can't contact LDAP server
Jun 19 01:26:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:26:04 weisshorn13 nslcd[4224]: [36b29f] ldap_result() failed: Can't contact LDAP server
Jun 19 01:26:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:26:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:26:41 weisshorn05 nslcd[4378]: [206b8e] ldap_result() failed: Can't contact LDAP server
Jun 19 01:26:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:27:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:27:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:27:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:27:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:28:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:28:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:28:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:28:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:29:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:29:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:29:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:29:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:30:01 weisshorn08 CROND[118930]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:30:01 weisshorn10 CROND[120199]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:30:01 weisshorn01 CROND[46889]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:30:01 weisshorn12 CROND[120070]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:30:01 weisshorn09 CROND[119865]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:30:01 weisshorn03 CROND[120829]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:30:01 weisshorn14 CROND[48738]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:30:01 weisshorn11 CROND[120691]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:30:01 weisshorn05 CROND[120180]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:30:01 weisshorn06 CROND[121017]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:30:01 weisshorn02 CROND[21266]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:30:01 weisshorn04 CROND[119647]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:30:01 weisshorn07 CROND[120283]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:30:01 weisshorn13 CROND[123499]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:30:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:30:17 weisshorn10 nslcd[4294]: [206b8e] ldap_result() failed: Can't contact LDAP server
Jun 19 01:30:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:30:31 weisshorn05 nslcd[4378]: [34d3d4] ldap_result() failed: Can't contact LDAP server
Jun 19 01:30:31 weisshorn12 nslcd[3954]: [206b8e] ldap_result() failed: Can't contact LDAP server
Jun 19 01:30:31 weisshorn04 nslcd[3885]: [0386d1] ldap_result() failed: Can't contact LDAP server
Jun 19 01:30:31 weisshorn09 nslcd[3954]: [0386d1] ldap_result() failed: Can't contact LDAP server
Jun 19 01:30:31 weisshorn03 nslcd[3944]: [0386d1] ldap_result() failed: Can't contact LDAP server
Jun 19 01:30:31 weisshorn13 nslcd[4224]: [206b8e] ldap_result() failed: Can't contact LDAP server
Jun 19 01:30:31 weisshorn08 nslcd[3946]: [206b8e] ldap_result() failed: Can't contact LDAP server
Jun 19 01:30:31 weisshorn07 nslcd[4297]: [206b8e] ldap_result() failed: Can't contact LDAP server
Jun 19 01:30:31 weisshorn06 nslcd[3953]: [95e50a] ldap_result() failed: Can't contact LDAP server
Jun 19 01:30:32 weisshorn11 nslcd[3954]: [8b0cbf] ldap_result() failed: Can't contact LDAP server
Jun 19 01:30:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:30:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:31:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:31:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:31:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:31:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:32:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:32:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:32:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:32:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:33:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:33:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:33:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:33:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:34:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:34:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:34:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:34:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:35:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:35:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:35:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:35:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:36:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:36:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:36:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:36:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:37:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:37:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:37:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:37:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:38:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:38:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:38:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:38:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:39:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:39:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:39:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:39:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:40:01 weisshorn02 CROND[22183]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:40:01 weisshorn07 CROND[120337]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:40:01 weisshorn13 CROND[123575]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:40:01 weisshorn08 CROND[118965]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:40:01 weisshorn10 CROND[120231]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:40:01 weisshorn01 CROND[46939]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:40:01 weisshorn12 CROND[120143]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:40:01 weisshorn03 CROND[120853]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:40:01 weisshorn09 CROND[119894]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:40:01 weisshorn14 CROND[48813]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:40:01 weisshorn11 CROND[120751]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:40:01 weisshorn05 CROND[120209]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:40:01 weisshorn06 CROND[121076]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:40:01 weisshorn04 CROND[119671]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:40:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:40:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:40:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:40:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:41:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:41:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:41:31 weisshorn14 nslcd[3437]: [d0feac] ldap_result() failed: Can't contact LDAP server
Jun 19 01:41:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:41:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:42:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:42:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:42:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:42:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:43:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:43:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:43:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:43:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:44:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:44:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:44:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:44:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:45:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:45:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:45:21 weisshorn03 nslcd[3944]: [25f2bc] ldap_result() failed: Can't contact LDAP server
Jun 19 01:45:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:45:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:46:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:46:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:46:31 weisshorn10 nslcd[4294]: [34d3d4] ldap_result() failed: Can't contact LDAP server
Jun 19 01:46:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:46:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:47:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:47:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:47:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:47:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:48:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:48:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:48:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:48:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:49:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:49:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:49:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:49:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:50:01 weisshorn04 CROND[119731]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:50:01 weisshorn07 CROND[120362]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:50:01 weisshorn02 CROND[23135]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:50:01 weisshorn13 CROND[123595]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:50:01 weisshorn08 CROND[119025]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:50:01 weisshorn10 CROND[120292]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:50:01 weisshorn01 CROND[46983]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:50:01 weisshorn12 CROND[120168]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:50:01 weisshorn09 CROND[119919]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:50:01 weisshorn14 CROND[48839]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:50:01 weisshorn03 CROND[120914]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:50:01 weisshorn11 CROND[120781]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:50:01 weisshorn05 CROND[120269]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:50:01 weisshorn06 CROND[121101]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 01:50:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:50:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:50:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:50:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:50:51 weisshorn02 nslcd[2099]: [6be6b9] ldap_result() failed: Can't contact LDAP server
Jun 19 01:51:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:51:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:51:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:51:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:52:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:52:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:52:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:52:39 weisshorn08 ntpd[4445]: synchronized to 148.187.18.120, stratum 2
Jun 19 01:52:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:53:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:53:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:53:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:53:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:54:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:54:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:54:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:54:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:55:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:55:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:55:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:55:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:56:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:56:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:56:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:56:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:57:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:57:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:57:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:57:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:58:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:58:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:58:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:58:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:59:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:59:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:59:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 01:59:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:00:01 weisshorn04 CROND[119769]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:00:01 weisshorn07 CROND[120440]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:00:01 weisshorn02 CROND[24064]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:00:01 weisshorn13 CROND[123635]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:00:01 weisshorn08 CROND[119064]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:00:01 weisshorn10 CROND[120330]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:00:01 weisshorn01 CROND[47084]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:00:01 weisshorn12 CROND[120200]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:00:01 weisshorn03 CROND[120952]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:00:01 weisshorn14 CROND[48868]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:00:01 weisshorn09 CROND[119987]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:00:01 weisshorn11 CROND[120854]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:00:01 weisshorn05 CROND[120302]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:00:01 weisshorn06 CROND[121174]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:00:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:00:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:00:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:00:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:01:01 weisshorn04 CROND[119774]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 02:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1197 starting 0anacron
Jun 19 02:01:01 weisshorn04 anacron[119786]: Anacron started on 2012-06-19
Jun 19 02:01:01 weisshorn04 anacron[119786]: Jobs will be executed sequentially
Jun 19 02:01:01 weisshorn04 anacron[119786]: Normal exit (0 jobs run)
Jun 19 02:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1197 finished 0anacron
Jun 19 02:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1197 starting mcelog.cron
Jun 19 02:01:01 weisshorn07 CROND[120445]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 02:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1204 starting 0anacron
Jun 19 02:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1197 finished mcelog.cron
Jun 19 02:01:01 weisshorn07 anacron[120457]: Anacron started on 2012-06-19
Jun 19 02:01:01 weisshorn07 anacron[120457]: Jobs will be executed sequentially
Jun 19 02:01:01 weisshorn07 anacron[120457]: Normal exit (0 jobs run)
Jun 19 02:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1204 finished 0anacron
Jun 19 02:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1204 starting mcelog.cron
Jun 19 02:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1204 finished mcelog.cron
Jun 19 02:01:01 weisshorn02 CROND[24158]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 02:01:01 weisshorn02 run-parts(/etc/cron.hourly)[2415 starting 0anacron
Jun 19 02:01:01 weisshorn02 anacron[24169]: Anacron started on 2012-06-19
Jun 19 02:01:01 weisshorn02 anacron[24169]: Jobs will be executed sequentially
Jun 19 02:01:01 weisshorn02 anacron[24169]: Normal exit (0 jobs run)
Jun 19 02:01:01 weisshorn02 run-parts(/etc/cron.hourly)[2417 finished 0anacron
Jun 19 02:01:01 weisshorn02 run-parts(/etc/cron.hourly)[2415 starting mcelog.cron
Jun 19 02:01:01 weisshorn13 CROND[123640]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 02:01:01 weisshorn02 run-parts(/etc/cron.hourly)[2417 finished mcelog.cron
Jun 19 02:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1236 starting 0anacron
Jun 19 02:01:01 weisshorn08 CROND[119069]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 02:01:01 weisshorn13 anacron[123652]: Anacron started on 2012-06-19
Jun 19 02:01:01 weisshorn13 anacron[123652]: Jobs will be executed sequentially
Jun 19 02:01:01 weisshorn13 anacron[123652]: Normal exit (0 jobs run)
Jun 19 02:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1236 finished 0anacron
Jun 19 02:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1236 starting mcelog.cron
Jun 19 02:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1236 finished mcelog.cron
Jun 19 02:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1190 starting 0anacron
Jun 19 02:01:01 weisshorn10 CROND[120335]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 02:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1203 starting 0anacron
Jun 19 02:01:01 weisshorn10 anacron[120347]: Anacron started on 2012-06-19
Jun 19 02:01:01 weisshorn10 anacron[120347]: Jobs will be executed sequentially
Jun 19 02:01:01 weisshorn10 anacron[120347]: Normal exit (0 jobs run)
Jun 19 02:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1203 finished 0anacron
Jun 19 02:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1203 starting mcelog.cron
Jun 19 02:01:01 weisshorn08 anacron[119081]: Anacron started on 2012-06-19
Jun 19 02:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1190 finished 0anacron
Jun 19 02:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1190 starting mcelog.cron
Jun 19 02:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1203 finished mcelog.cron
Jun 19 02:01:01 weisshorn08 anacron[119081]: Jobs will be executed sequentially
Jun 19 02:01:01 weisshorn08 anacron[119081]: Normal exit (0 jobs run)
Jun 19 02:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1190 finished mcelog.cron
Jun 19 02:01:01 weisshorn01 CROND[47087]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 02:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4708 starting 0anacron
Jun 19 02:01:01 weisshorn12 CROND[120205]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 02:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1202 starting 0anacron
Jun 19 02:01:01 weisshorn01 anacron[47099]: Anacron started on 2012-06-19
Jun 19 02:01:01 weisshorn01 anacron[47099]: Jobs will be executed sequentially
Jun 19 02:01:01 weisshorn01 anacron[47099]: Normal exit (0 jobs run)
Jun 19 02:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4710 finished 0anacron
Jun 19 02:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4708 starting mcelog.cron
Jun 19 02:01:01 weisshorn12 anacron[120217]: Anacron started on 2012-06-19
Jun 19 02:01:01 weisshorn12 anacron[120217]: Jobs will be executed sequentially
Jun 19 02:01:01 weisshorn12 anacron[120217]: Normal exit (0 jobs run)
Jun 19 02:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1202 finished 0anacron
Jun 19 02:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1202 starting mcelog.cron
Jun 19 02:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4710 finished mcelog.cron
Jun 19 02:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1202 finished mcelog.cron
Jun 19 02:01:01 weisshorn03 CROND[120957]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 02:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1209 starting 0anacron
Jun 19 02:01:01 weisshorn03 anacron[120969]: Anacron started on 2012-06-19
Jun 19 02:01:01 weisshorn03 anacron[120969]: Will run job `cron.weekly' in 59 min.
Jun 19 02:01:01 weisshorn03 anacron[120969]: Jobs will be executed sequentially
Jun 19 02:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1209 finished 0anacron
Jun 19 02:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1209 starting mcelog.cron
Jun 19 02:01:01 weisshorn14 CROND[48874]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 02:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4887 starting 0anacron
Jun 19 02:01:01 weisshorn14 anacron[48886]: Anacron started on 2012-06-19
Jun 19 02:01:01 weisshorn14 anacron[48886]: Jobs will be executed sequentially
Jun 19 02:01:01 weisshorn14 anacron[48886]: Normal exit (0 jobs run)
Jun 19 02:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4888 finished 0anacron
Jun 19 02:01:01 weisshorn09 CROND[119992]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 02:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4887 starting mcelog.cron
Jun 19 02:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1199 starting 0anacron
Jun 19 02:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1209 finished mcelog.cron
Jun 19 02:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4889 finished mcelog.cron
Jun 19 02:01:01 weisshorn09 anacron[120004]: Anacron started on 2012-06-19
Jun 19 02:01:01 weisshorn09 anacron[120004]: Jobs will be executed sequentially
Jun 19 02:01:01 weisshorn09 anacron[120004]: Normal exit (0 jobs run)
Jun 19 02:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1200 finished 0anacron
Jun 19 02:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1199 starting mcelog.cron
Jun 19 02:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1200 finished mcelog.cron
Jun 19 02:01:01 weisshorn11 CROND[120859]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 02:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1208 starting 0anacron
Jun 19 02:01:01 weisshorn05 CROND[120307]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 02:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1203 starting 0anacron
Jun 19 02:01:01 weisshorn11 anacron[120871]: Anacron started on 2012-06-19
Jun 19 02:01:01 weisshorn11 anacron[120871]: Jobs will be executed sequentially
Jun 19 02:01:01 weisshorn11 anacron[120871]: Normal exit (0 jobs run)
Jun 19 02:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1208 finished 0anacron
Jun 19 02:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1208 starting mcelog.cron
Jun 19 02:01:01 weisshorn06 CROND[121179]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 02:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1211 starting 0anacron
Jun 19 02:01:02 weisshorn05 anacron[120319]: Anacron started on 2012-06-19
Jun 19 02:01:02 weisshorn05 run-parts(/etc/cron.hourly)[1203 finished 0anacron
Jun 19 02:01:02 weisshorn11 run-parts(/etc/cron.hourly)[1208 finished mcelog.cron
Jun 19 02:01:02 weisshorn05 run-parts(/etc/cron.hourly)[1203 starting mcelog.cron
Jun 19 02:01:02 weisshorn05 anacron[120319]: Jobs will be executed sequentially
Jun 19 02:01:02 weisshorn05 anacron[120319]: Normal exit (0 jobs run)
Jun 19 02:01:02 weisshorn06 anacron[121191]: Anacron started on 2012-06-19
Jun 19 02:01:02 weisshorn06 anacron[121191]: Jobs will be executed sequentially
Jun 19 02:01:02 weisshorn06 anacron[121191]: Normal exit (0 jobs run)
Jun 19 02:01:02 weisshorn06 run-parts(/etc/cron.hourly)[1211 finished 0anacron
Jun 19 02:01:02 weisshorn06 run-parts(/etc/cron.hourly)[1211 starting mcelog.cron
Jun 19 02:01:02 weisshorn05 run-parts(/etc/cron.hourly)[1203 finished mcelog.cron
Jun 19 02:01:02 weisshorn06 run-parts(/etc/cron.hourly)[1212 finished mcelog.cron
Jun 19 02:01:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:01:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:01:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:01:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:02:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:02:10 weisshorn07 rhnsd[120469]: running program /usr/sbin/rhn_check
Jun 19 02:02:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:02:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:02:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:03:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:03:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:03:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:03:37 weisshorn06 ntpd[4454]: synchronized to 148.187.18.120, stratum 2
Jun 19 02:03:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:04:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:04:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:04:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:04:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:05:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:05:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:05:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:05:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:06:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:06:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:06:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:06:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:07:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:07:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:07:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:07:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:08:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:08:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:08:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:08:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:09:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:09:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:09:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:09:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:10:01 weisshorn11 CROND[120896]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:10:01 weisshorn05 CROND[120354]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:10:01 weisshorn06 CROND[121222]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:10:01 weisshorn04 CROND[119816]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:10:01 weisshorn07 CROND[120484]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:10:01 weisshorn02 CROND[25003]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:10:01 weisshorn13 CROND[123723]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:10:01 weisshorn10 CROND[120377]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:10:01 weisshorn08 CROND[119111]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:10:01 weisshorn01 CROND[47152]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:10:01 weisshorn01 CROND[47154]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 02:10:01 weisshorn12 CROND[120288]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:10:01 weisshorn03 CROND[121001]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:10:01 weisshorn14 CROND[48964]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:10:01 weisshorn04 sshd[119818]: Accepted publickey for root from 148.187.12.231 port 44198 ssh2
Jun 19 02:10:01 weisshorn03 sshd[120998]: Accepted publickey for root from 148.187.12.231 port 50447 ssh2
Jun 19 02:10:01 weisshorn03 sshd[120998]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 02:10:01 weisshorn04 sshd[119818]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 02:10:01 weisshorn06 sshd[121224]: Accepted publickey for root from 148.187.12.231 port 44707 ssh2
Jun 19 02:10:01 weisshorn06 sshd[121224]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 02:10:01 weisshorn08 sshd[119113]: Accepted publickey for root from 148.187.12.231 port 40254 ssh2
Jun 19 02:10:01 weisshorn14 sshd[48961]: Accepted publickey for root from 148.187.12.231 port 49569 ssh2
Jun 19 02:10:01 weisshorn08 sshd[119113]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 02:10:01 weisshorn14 sshd[48961]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 02:10:01 weisshorn13 sshd[123725]: Accepted publickey for root from 148.187.12.231 port 44615 ssh2
Jun 19 02:10:01 weisshorn13 sshd[123725]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 02:10:01 weisshorn09 CROND[120041]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:10:01 weisshorn09 sshd[120038]: Accepted publickey for root from 148.187.12.231 port 57890 ssh2
Jun 19 02:10:01 weisshorn09 sshd[120038]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 02:10:01 weisshorn07 sshd[120486]: Accepted publickey for root from 148.187.12.231 port 48257 ssh2
Jun 19 02:10:01 weisshorn07 sshd[120486]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 02:10:01 weisshorn10 sshd[120379]: Accepted publickey for root from 148.187.12.231 port 36924 ssh2
Jun 19 02:10:01 weisshorn10 sshd[120379]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 02:10:01 weisshorn12 sshd[120290]: Accepted publickey for root from 148.187.12.231 port 46357 ssh2
Jun 19 02:10:01 weisshorn12 sshd[120290]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 02:10:01 weisshorn11 sshd[120898]: Accepted publickey for root from 148.187.12.231 port 45139 ssh2
Jun 19 02:10:01 weisshorn14 sshd[48961]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 02:10:01 weisshorn14 sshd[48961]: pam_unix(sshd:session): session closed for user root
Jun 19 02:10:01 weisshorn11 sshd[120898]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 02:10:01 weisshorn04 sshd[119818]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 02:10:01 weisshorn04 sshd[119818]: pam_unix(sshd:session): session closed for user root
Jun 19 02:10:01 weisshorn03 sshd[120998]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 02:10:01 weisshorn03 sshd[120998]: pam_unix(sshd:session): session closed for user root
Jun 19 02:10:01 weisshorn06 sshd[121224]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 02:10:01 weisshorn06 sshd[121224]: pam_unix(sshd:session): session closed for user root
Jun 19 02:10:01 weisshorn09 sshd[120038]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 02:10:01 weisshorn09 sshd[120038]: pam_unix(sshd:session): session closed for user root
Jun 19 02:10:01 weisshorn08 sshd[119113]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 02:10:01 weisshorn08 sshd[119113]: pam_unix(sshd:session): session closed for user root
Jun 19 02:10:01 weisshorn07 sshd[120486]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 02:10:01 weisshorn07 sshd[120486]: pam_unix(sshd:session): session closed for user root
Jun 19 02:10:02 weisshorn12 sshd[120290]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 02:10:02 weisshorn12 sshd[120290]: pam_unix(sshd:session): session closed for user root
Jun 19 02:10:02 weisshorn10 sshd[120379]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 02:10:02 weisshorn10 sshd[120379]: pam_unix(sshd:session): session closed for user root
Jun 19 02:10:02 weisshorn13 sshd[123725]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 02:10:02 weisshorn13 sshd[123725]: pam_unix(sshd:session): session closed for user root
Jun 19 02:10:02 weisshorn11 sshd[120898]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 02:10:02 weisshorn11 sshd[120898]: pam_unix(sshd:session): session closed for user root
Jun 19 02:10:02 weisshorn05 sshd[120356]: Accepted publickey for root from 148.187.12.231 port 54659 ssh2
Jun 19 02:10:02 weisshorn05 sshd[120356]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 02:10:03 weisshorn05 sshd[120356]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 02:10:03 weisshorn05 sshd[120356]: pam_unix(sshd:session): session closed for user root
Jun 19 02:10:03 weisshorn01 postfix/pickup[46659]: 20CA692224F: uid=0 from=<root>
Jun 19 02:10:03 weisshorn01 postfix/cleanup[47184]: 20CA692224F: message-id=<20120619001003.20CA692224F@weisshorn01.admin.cscs.ch>
Jun 19 02:10:03 weisshorn01 postfix/qmgr[2710]: 20CA692224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 02:10:03 weisshorn01 postfix/local[47186]: 20CA692224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=1.4, delays=1.3/0.04/0/0.06, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 02:10:03 weisshorn01 postfix/qmgr[2710]: 20CA692224F: removed
Jun 19 02:10:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:10:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:10:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:10:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:11:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:11:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:11:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:11:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:12:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:12:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:12:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:12:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:13:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:13:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:13:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:13:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:14:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:14:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:14:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:14:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:15:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:15:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:15:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:15:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:16:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:16:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:16:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:16:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:17:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:17:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:17:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:17:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:18:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:18:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:18:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:18:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:19:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:19:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:19:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:19:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:20:01 weisshorn06 CROND[121273]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:20:01 weisshorn05 CROND[120434]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:20:01 weisshorn11 CROND[120946]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:20:01 weisshorn01 CROND[47274]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:20:01 weisshorn04 CROND[119896]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:20:01 weisshorn07 CROND[120535]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:20:01 weisshorn02 CROND[25960]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:20:01 weisshorn13 CROND[123768]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:20:01 weisshorn10 CROND[120457]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:20:01 weisshorn08 CROND[119199]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:20:01 weisshorn12 CROND[120333]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:20:01 weisshorn14 CROND[49013]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:20:01 weisshorn03 CROND[121079]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:20:01 weisshorn09 CROND[120119]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:20:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:20:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:20:31 weisshorn02 nslcd[2099]: [234363] ldap_result() failed: Can't contact LDAP server
Jun 19 02:20:31 weisshorn14 nslcd[3437]: [ee5a5b] ldap_result() failed: Can't contact LDAP server
Jun 19 02:20:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:20:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:21:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:21:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:21:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:21:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:22:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:22:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:22:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:22:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:23:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:23:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:23:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:23:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:24:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:24:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:24:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:24:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:25:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:25:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:25:30 weisshorn09 nslcd[3954]: [25f2bc] ldap_result() failed: Can't contact LDAP server
Jun 19 02:25:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:25:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:25:53 weisshorn08 nslcd[3946]: [34d3d4] ldap_result() failed: Can't contact LDAP server
Jun 19 02:25:54 weisshorn06 nslcd[3953]: [5bcf61] ldap_result() failed: Can't contact LDAP server
Jun 19 02:25:56 weisshorn04 nslcd[3885]: [25f2bc] ldap_result() failed: Can't contact LDAP server
Jun 19 02:25:57 weisshorn07 nslcd[4297]: [34d3d4] ldap_result() failed: Can't contact LDAP server
Jun 19 02:26:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:26:17 weisshorn12 nslcd[3954]: [34d3d4] ldap_result() failed: Can't contact LDAP server
Jun 19 02:26:17 weisshorn11 nslcd[3954]: [e203ab] ldap_result() failed: Can't contact LDAP server
Jun 19 02:26:19 weisshorn13 nslcd[4224]: [34d3d4] ldap_result() failed: Can't contact LDAP server
Jun 19 02:26:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:26:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:26:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:26:56 weisshorn05 nslcd[4378]: [c65e98] ldap_result() failed: Can't contact LDAP server
Jun 19 02:27:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:27:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:27:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:27:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:28:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:28:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:28:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:28:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:29:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:29:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:29:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:29:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:30:01 weisshorn09 CROND[120153]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:30:01 weisshorn06 CROND[121341]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:30:01 weisshorn05 CROND[120468]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:30:01 weisshorn01 CROND[47332]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:30:01 weisshorn11 CROND[121020]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:30:01 weisshorn04 CROND[119935]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:30:01 weisshorn07 CROND[120608]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:30:01 weisshorn02 CROND[26884]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:30:01 weisshorn13 CROND[123804]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:30:01 weisshorn10 CROND[120495]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:30:01 weisshorn08 CROND[119230]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:30:01 weisshorn12 CROND[120366]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:30:01 weisshorn14 CROND[49037]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:30:01 weisshorn03 CROND[121117]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:30:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:30:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:30:32 weisshorn10 nslcd[4294]: [c65e98] ldap_result() failed: Can't contact LDAP server
Jun 19 02:30:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:30:46 weisshorn05 nslcd[4378]: [95e50a] ldap_result() failed: Can't contact LDAP server
Jun 19 02:30:46 weisshorn12 nslcd[3954]: [c65e98] ldap_result() failed: Can't contact LDAP server
Jun 19 02:30:46 weisshorn04 nslcd[3885]: [e8e1a7] ldap_result() failed: Can't contact LDAP server
Jun 19 02:30:46 weisshorn09 nslcd[3954]: [e8e1a7] ldap_result() failed: Can't contact LDAP server
Jun 19 02:30:46 weisshorn03 nslcd[3944]: [e8e1a7] ldap_result() failed: Can't contact LDAP server
Jun 19 02:30:46 weisshorn13 nslcd[4224]: [c65e98] ldap_result() failed: Can't contact LDAP server
Jun 19 02:30:46 weisshorn08 nslcd[3946]: [c65e98] ldap_result() failed: Can't contact LDAP server
Jun 19 02:30:46 weisshorn07 nslcd[4297]: [c65e98] ldap_result() failed: Can't contact LDAP server
Jun 19 02:30:46 weisshorn06 nslcd[3953]: [1dd403] ldap_result() failed: Can't contact LDAP server
Jun 19 02:30:47 weisshorn11 nslcd[3954]: [86d60e] ldap_result() failed: Can't contact LDAP server
Jun 19 02:30:50 weisshorn14 rhnsd[49040]: running program /usr/sbin/rhn_check
Jun 19 02:30:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:31:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:31:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:31:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:31:51 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:32:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:32:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:32:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:32:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:33:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:33:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:33:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:33:51 weisshorn13 rhnsd[123858]: running program /usr/sbin/rhn_check
Jun 19 02:33:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:34:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:34:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:34:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:34:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:35:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:35:20 weisshorn09 ntpd[4454]: synchronized to 148.187.18.145, stratum 2
Jun 19 02:35:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:35:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:35:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:36:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:36:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:36:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:36:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:37:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:37:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:37:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:37:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:38:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:38:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:38:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:38:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:39:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:39:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:39:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:39:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:40:01 weisshorn14 CROND[49114]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:40:01 weisshorn03 CROND[121141]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:40:01 weisshorn09 CROND[120183]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:40:01 weisshorn05 CROND[120497]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:40:01 weisshorn06 CROND[121365]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:40:01 weisshorn11 CROND[121044]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:40:01 weisshorn04 CROND[119959]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:40:01 weisshorn07 CROND[120627]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:40:01 weisshorn02 CROND[27817]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:40:01 weisshorn01 CROND[47385]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:40:01 weisshorn10 CROND[120519]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:40:01 weisshorn08 CROND[119254]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:40:01 weisshorn13 CROND[123871]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:40:01 weisshorn12 CROND[120431]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:40:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:40:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:40:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:40:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:41:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:41:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:41:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:41:46 weisshorn14 nslcd[3437]: [51b9f3] ldap_result() failed: Can't contact LDAP server
Jun 19 02:41:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:42:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:42:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:42:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:42:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:43:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:43:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:43:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:43:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:44:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:44:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:44:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:44:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:45:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:45:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:45:36 weisshorn03 nslcd[3944]: [9efb0d] ldap_result() failed: Can't contact LDAP server
Jun 19 02:45:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:45:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:46:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:46:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:46:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:46:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:47:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:47:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:47:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:47:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:48:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:48:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:48:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:48:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:49:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:49:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:49:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:49:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:50:01 weisshorn12 CROND[120455]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:50:01 weisshorn03 CROND[121201]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:50:01 weisshorn14 CROND[49139]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:50:01 weisshorn09 CROND[120242]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:50:01 weisshorn06 CROND[121395]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:50:01 weisshorn05 CROND[120556]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:50:01 weisshorn11 CROND[121068]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:50:01 weisshorn04 CROND[120018]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:50:01 weisshorn07 CROND[120657]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:50:01 weisshorn02 CROND[28769]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:50:01 weisshorn01 CROND[47458]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:50:01 weisshorn10 CROND[120578]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:50:01 weisshorn08 CROND[119321]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:50:02 weisshorn13 CROND[123895]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 02:50:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:50:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:50:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:50:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:51:06 weisshorn02 nslcd[2099]: [d385c9] ldap_result() failed: Can't contact LDAP server
Jun 19 02:51:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:51:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:51:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:51:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:52:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:52:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:52:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:52:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:53:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:53:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:53:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:53:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:54:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:54:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:54:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:54:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:55:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:55:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:55:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:55:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:56:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:56:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:56:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:56:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:57:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:57:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:57:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:57:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:58:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:58:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:58:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:58:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:59:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:59:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:59:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 02:59:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:00:01 weisshorn12 CROND[120487]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:00:01 weisshorn14 CROND[49163]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:00:01 weisshorn03 CROND[121239]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:00:01 weisshorn13 CROND[123930]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:00:01 weisshorn09 CROND[120275]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:00:01 weisshorn06 CROND[121462]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:00:01 weisshorn05 CROND[120589]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:00:01 weisshorn11 CROND[121141]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:00:01 weisshorn07 CROND[120729]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:00:01 weisshorn04 CROND[120056]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:00:01 weisshorn03 anacron[120969]: Job `cron.weekly' started
Jun 19 03:00:01 weisshorn03 anacron[120969]: Job `cron.weekly' terminated
Jun 19 03:00:01 weisshorn03 anacron[120969]: Normal exit (1 job run)
Jun 19 03:00:01 weisshorn02 CROND[29692]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:00:01 weisshorn01 CROND[47524]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:00:01 weisshorn10 CROND[120616]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:00:01 weisshorn08 CROND[119351]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:00:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:00:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:00:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:00:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:00:58 weisshorn08 ntpd[4445]: synchronized to 148.187.18.145, stratum 2
Jun 19 03:01:01 weisshorn12 CROND[120538]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 03:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1205 starting 0anacron
Jun 19 03:01:01 weisshorn12 anacron[120550]: Anacron started on 2012-06-19
Jun 19 03:01:01 weisshorn12 anacron[120550]: Will run job `cron.daily' in 6 min.
Jun 19 03:01:01 weisshorn12 anacron[120550]: Jobs will be executed sequentially
Jun 19 03:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1205 finished 0anacron
Jun 19 03:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1205 starting mcelog.cron
Jun 19 03:01:01 weisshorn03 CROND[121246]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 03:01:01 weisshorn14 CROND[49219]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 03:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1205 finished mcelog.cron
Jun 19 03:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4921 starting 0anacron
Jun 19 03:01:01 weisshorn14 anacron[49231]: Anacron started on 2012-06-19
Jun 19 03:01:01 weisshorn14 anacron[49231]: Will run job `cron.daily' in 38 min.
Jun 19 03:01:01 weisshorn14 anacron[49231]: Jobs will be executed sequentially
Jun 19 03:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4923 finished 0anacron
Jun 19 03:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1212 starting 0anacron
Jun 19 03:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4921 starting mcelog.cron
Jun 19 03:01:01 weisshorn13 CROND[123981]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 03:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4924 finished mcelog.cron
Jun 19 03:01:01 weisshorn03 anacron[121257]: Anacron started on 2012-06-19
Jun 19 03:01:01 weisshorn03 anacron[121257]: Will run job `cron.daily' in 37 min.
Jun 19 03:01:01 weisshorn03 anacron[121257]: Jobs will be executed sequentially
Jun 19 03:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1212 finished 0anacron
Jun 19 03:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1212 starting mcelog.cron
Jun 19 03:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1239 starting 0anacron
Jun 19 03:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1212 finished mcelog.cron
Jun 19 03:01:01 weisshorn09 CROND[120285]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 03:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1202 starting 0anacron
Jun 19 03:01:01 weisshorn09 anacron[120297]: Anacron started on 2012-06-19
Jun 19 03:01:01 weisshorn09 anacron[120297]: Will run job `cron.daily' in 7 min.
Jun 19 03:01:01 weisshorn09 anacron[120297]: Jobs will be executed sequentially
Jun 19 03:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1202 finished 0anacron
Jun 19 03:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1202 starting mcelog.cron
Jun 19 03:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1203 finished mcelog.cron
Jun 19 03:01:01 weisshorn13 anacron[123993]: Anacron started on 2012-06-19
Jun 19 03:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1239 finished 0anacron
Jun 19 03:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1239 starting mcelog.cron
Jun 19 03:01:01 weisshorn13 anacron[123993]: Will run job `cron.daily' in 46 min.
Jun 19 03:01:01 weisshorn13 anacron[123993]: Jobs will be executed sequentially
Jun 19 03:01:01 weisshorn06 CROND[121472]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 03:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1214 starting 0anacron
Jun 19 03:01:01 weisshorn06 anacron[121484]: Anacron started on 2012-06-19
Jun 19 03:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1240 finished mcelog.cron
Jun 19 03:01:01 weisshorn06 anacron[121484]: Will run job `cron.daily' in 6 min.
Jun 19 03:01:01 weisshorn06 anacron[121484]: Jobs will be executed sequentially
Jun 19 03:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1214 finished 0anacron
Jun 19 03:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1214 starting mcelog.cron
Jun 19 03:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1214 finished mcelog.cron
Jun 19 03:01:01 weisshorn05 CROND[120599]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 03:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1205 starting 0anacron
Jun 19 03:01:01 weisshorn05 anacron[120611]: Anacron started on 2012-06-19
Jun 19 03:01:01 weisshorn05 anacron[120611]: Will run job `cron.daily' in 19 min.
Jun 19 03:01:01 weisshorn05 anacron[120611]: Jobs will be executed sequentially
Jun 19 03:01:01 weisshorn11 CROND[121146]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 03:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1206 finished 0anacron
Jun 19 03:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1205 starting mcelog.cron
Jun 19 03:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1211 starting 0anacron
Jun 19 03:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1206 finished mcelog.cron
Jun 19 03:01:01 weisshorn11 anacron[121158]: Anacron started on 2012-06-19
Jun 19 03:01:01 weisshorn11 anacron[121158]: Will run job `cron.daily' in 46 min.
Jun 19 03:01:01 weisshorn11 anacron[121158]: Jobs will be executed sequentially
Jun 19 03:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1211 finished 0anacron
Jun 19 03:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1211 starting mcelog.cron
Jun 19 03:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1211 finished mcelog.cron
Jun 19 03:01:01 weisshorn07 CROND[120734]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 03:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1207 starting 0anacron
Jun 19 03:01:01 weisshorn04 CROND[120061]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 03:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1200 starting 0anacron
Jun 19 03:01:01 weisshorn07 anacron[120746]: Anacron started on 2012-06-19
Jun 19 03:01:01 weisshorn07 anacron[120746]: Will run job `cron.daily' in 14 min.
Jun 19 03:01:01 weisshorn07 anacron[120746]: Jobs will be executed sequentially
Jun 19 03:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1207 finished 0anacron
Jun 19 03:01:01 weisshorn04 anacron[120073]: Anacron started on 2012-06-19
Jun 19 03:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1207 starting mcelog.cron
Jun 19 03:01:01 weisshorn04 anacron[120073]: Will run job `cron.daily' in 21 min.
Jun 19 03:01:01 weisshorn04 anacron[120073]: Jobs will be executed sequentially
Jun 19 03:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1200 finished 0anacron
Jun 19 03:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1200 starting mcelog.cron
Jun 19 03:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1207 finished mcelog.cron
Jun 19 03:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1200 finished mcelog.cron
Jun 19 03:01:01 weisshorn02 CROND[29791]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 03:01:01 weisshorn02 run-parts(/etc/cron.hourly)[2979 starting 0anacron
Jun 19 03:01:01 weisshorn02 anacron[29802]: Anacron started on 2012-06-19
Jun 19 03:01:01 weisshorn02 anacron[29802]: Will run job `cron.daily' in 29 min.
Jun 19 03:01:01 weisshorn02 anacron[29802]: Jobs will be executed sequentially
Jun 19 03:01:01 weisshorn02 run-parts(/etc/cron.hourly)[2980 finished 0anacron
Jun 19 03:01:01 weisshorn02 run-parts(/etc/cron.hourly)[2979 starting mcelog.cron
Jun 19 03:01:01 weisshorn01 CROND[47531]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 03:01:01 weisshorn02 run-parts(/etc/cron.hourly)[2981 finished mcelog.cron
Jun 19 03:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4753 starting 0anacron
Jun 19 03:01:01 weisshorn10 CROND[120626]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 03:01:01 weisshorn08 CROND[119356]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 03:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1193 starting 0anacron
Jun 19 03:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1206 starting 0anacron
Jun 19 03:01:02 weisshorn01 anacron[47543]: Anacron started on 2012-06-19
Jun 19 03:01:02 weisshorn10 anacron[120638]: Anacron started on 2012-06-19
Jun 19 03:01:02 weisshorn01 run-parts(/etc/cron.hourly)[4754 finished 0anacron
Jun 19 03:01:02 weisshorn10 anacron[120638]: Will run job `cron.daily' in 18 min.
Jun 19 03:01:02 weisshorn10 anacron[120638]: Will run job `cron.weekly' in 38 min.
Jun 19 03:01:02 weisshorn10 anacron[120638]: Jobs will be executed sequentially
Jun 19 03:01:02 weisshorn10 run-parts(/etc/cron.hourly)[1206 finished 0anacron
Jun 19 03:01:02 weisshorn10 run-parts(/etc/cron.hourly)[1206 starting mcelog.cron
Jun 19 03:01:02 weisshorn01 run-parts(/etc/cron.hourly)[4753 starting mcelog.cron
Jun 19 03:01:02 weisshorn08 anacron[119367]: Anacron started on 2012-06-19
Jun 19 03:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1193 finished 0anacron
Jun 19 03:01:02 weisshorn01 anacron[47543]: Will run job `cron.daily' in 29 min.
Jun 19 03:01:02 weisshorn01 anacron[47543]: Jobs will be executed sequentially
Jun 19 03:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1193 starting mcelog.cron
Jun 19 03:01:02 weisshorn10 run-parts(/etc/cron.hourly)[1206 finished mcelog.cron
Jun 19 03:01:02 weisshorn01 run-parts(/etc/cron.hourly)[4755 finished mcelog.cron
Jun 19 03:01:02 weisshorn08 anacron[119367]: Will run job `cron.daily' in 29 min.
Jun 19 03:01:02 weisshorn08 anacron[119367]: Jobs will be executed sequentially
Jun 19 03:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1193 finished mcelog.cron
Jun 19 03:01:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:01:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:01:25 weisshorn10 rhnsd[120649]: running program /usr/sbin/rhn_check
Jun 19 03:01:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:01:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:02:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:02:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:02:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:02:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:03:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:03:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:03:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:03:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:04:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:04:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:04:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:04:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:05:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:05:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:05:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:05:55 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:06:10 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:06:25 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:06:40 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:06:55 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:07:01 weisshorn12 anacron[120550]: Job `cron.daily' started
Jun 19 03:07:01 weisshorn12 run-parts(/etc/cron.daily)[12056 starting cups
Jun 19 03:07:01 weisshorn12 run-parts(/etc/cron.daily)[12057 finished cups
Jun 19 03:07:01 weisshorn12 run-parts(/etc/cron.daily)[12056 starting logrotate
Jun 19 03:07:01 weisshorn12 run-parts(/etc/cron.daily)[12058 finished logrotate
Jun 19 03:07:01 weisshorn12 run-parts(/etc/cron.daily)[12056 starting makewhatis.cron
Jun 19 03:07:01 weisshorn06 anacron[121484]: Job `cron.daily' started
Jun 19 03:07:01 weisshorn06 run-parts(/etc/cron.daily)[12150 starting cups
Jun 19 03:07:01 weisshorn06 run-parts(/etc/cron.daily)[12151 finished cups
Jun 19 03:07:01 weisshorn06 run-parts(/etc/cron.daily)[12150 starting logrotate
Jun 19 03:07:01 weisshorn06 run-parts(/etc/cron.daily)[12152 finished logrotate
Jun 19 03:07:01 weisshorn06 run-parts(/etc/cron.daily)[12150 starting makewhatis.cron
Jun 19 03:07:01 weisshorn12 run-parts(/etc/cron.daily)[12071 finished makewhatis.cron
Jun 19 03:07:01 weisshorn12 run-parts(/etc/cron.daily)[12056 starting mlocate.cron
Jun 19 03:07:02 weisshorn06 run-parts(/etc/cron.daily)[12165 finished makewhatis.cron
Jun 19 03:07:02 weisshorn06 run-parts(/etc/cron.daily)[12150 starting mlocate.cron
Jun 19 03:07:04 weisshorn06 run-parts(/etc/cron.daily)[12166 finished mlocate.cron
Jun 19 03:07:04 weisshorn06 run-parts(/etc/cron.daily)[12150 starting prelink
Jun 19 03:07:04 weisshorn06 run-parts(/etc/cron.daily)[12167 finished prelink
Jun 19 03:07:04 weisshorn06 run-parts(/etc/cron.daily)[12150 starting readahead.cron
Jun 19 03:07:04 weisshorn06 run-parts(/etc/cron.daily)[12168 finished readahead.cron
Jun 19 03:07:04 weisshorn06 run-parts(/etc/cron.daily)[12150 starting rhsmd
Jun 19 03:07:05 weisshorn06 rhsmd: This system is registered to RHN Classic
Jun 19 03:07:05 weisshorn06 run-parts(/etc/cron.daily)[12169 finished rhsmd
Jun 19 03:07:05 weisshorn06 run-parts(/etc/cron.daily)[12150 starting tmpwatch
Jun 19 03:07:05 weisshorn06 run-parts(/etc/cron.daily)[12172 finished tmpwatch
Jun 19 03:07:05 weisshorn06 anacron[121484]: Job `cron.daily' terminated
Jun 19 03:07:05 weisshorn06 anacron[121484]: Normal exit (1 job run)
Jun 19 03:07:06 weisshorn12 run-parts(/etc/cron.daily)[12072 finished mlocate.cron
Jun 19 03:07:06 weisshorn12 run-parts(/etc/cron.daily)[12056 starting prelink
Jun 19 03:07:06 weisshorn12 run-parts(/etc/cron.daily)[12073 finished prelink
Jun 19 03:07:06 weisshorn12 run-parts(/etc/cron.daily)[12056 starting readahead.cron
Jun 19 03:07:06 weisshorn12 run-parts(/etc/cron.daily)[12074 finished readahead.cron
Jun 19 03:07:06 weisshorn12 run-parts(/etc/cron.daily)[12056 starting rhsmd
Jun 19 03:07:07 weisshorn12 rhsmd: This system is registered to RHN Classic
Jun 19 03:07:07 weisshorn12 run-parts(/etc/cron.daily)[12075 finished rhsmd
Jun 19 03:07:07 weisshorn12 run-parts(/etc/cron.daily)[12056 starting tmpwatch
Jun 19 03:07:07 weisshorn12 run-parts(/etc/cron.daily)[12079 finished tmpwatch
Jun 19 03:07:07 weisshorn12 anacron[120550]: Job `cron.daily' terminated
Jun 19 03:07:07 weisshorn12 anacron[120550]: Normal exit (1 job run)
Jun 19 03:07:10 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:07:25 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:07:40 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:07:55 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:08:01 weisshorn09 anacron[120297]: Job `cron.daily' started
Jun 19 03:08:01 weisshorn09 run-parts(/etc/cron.daily)[12032 starting cups
Jun 19 03:08:01 weisshorn09 run-parts(/etc/cron.daily)[12033 finished cups
Jun 19 03:08:01 weisshorn09 run-parts(/etc/cron.daily)[12032 starting logrotate
Jun 19 03:08:01 weisshorn09 run-parts(/etc/cron.daily)[12034 finished logrotate
Jun 19 03:08:01 weisshorn09 run-parts(/etc/cron.daily)[12032 starting makewhatis.cron
Jun 19 03:08:02 weisshorn09 run-parts(/etc/cron.daily)[12047 finished makewhatis.cron
Jun 19 03:08:02 weisshorn09 run-parts(/etc/cron.daily)[12032 starting mlocate.cron
Jun 19 03:08:04 weisshorn09 run-parts(/etc/cron.daily)[12048 finished mlocate.cron
Jun 19 03:08:04 weisshorn09 run-parts(/etc/cron.daily)[12032 starting prelink
Jun 19 03:08:04 weisshorn09 run-parts(/etc/cron.daily)[12049 finished prelink
Jun 19 03:08:04 weisshorn09 run-parts(/etc/cron.daily)[12032 starting readahead.cron
Jun 19 03:08:04 weisshorn09 run-parts(/etc/cron.daily)[12050 finished readahead.cron
Jun 19 03:08:04 weisshorn09 run-parts(/etc/cron.daily)[12032 starting rhsmd
Jun 19 03:08:04 weisshorn09 rhsmd: This system is registered to RHN Classic
Jun 19 03:08:04 weisshorn09 run-parts(/etc/cron.daily)[12051 finished rhsmd
Jun 19 03:08:04 weisshorn09 run-parts(/etc/cron.daily)[12032 starting tmpwatch
Jun 19 03:08:05 weisshorn09 run-parts(/etc/cron.daily)[12054 finished tmpwatch
Jun 19 03:08:05 weisshorn09 anacron[120297]: Job `cron.daily' terminated
Jun 19 03:08:05 weisshorn09 anacron[120297]: Normal exit (1 job run)
Jun 19 03:08:10 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:08:25 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:08:40 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:08:55 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:09:10 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:09:25 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:09:40 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:09:55 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:10:01 weisshorn10 CROND[120665]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:10:01 weisshorn01 CROND[47595]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:10:01 weisshorn01 CROND[47597]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 03:10:01 weisshorn08 CROND[119398]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:10:01 weisshorn14 sshd[49261]: Accepted publickey for root from 148.187.12.231 port 49580 ssh2
Jun 19 03:10:01 weisshorn14 sshd[49261]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 03:10:01 weisshorn09 sshd[120552]: Accepted publickey for root from 148.187.12.231 port 57902 ssh2
Jun 19 03:10:01 weisshorn03 sshd[121292]: Accepted publickey for root from 148.187.12.231 port 50461 ssh2
Jun 19 03:10:01 weisshorn09 sshd[120552]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 03:10:01 weisshorn04 sshd[120103]: Accepted publickey for root from 148.187.12.231 port 44210 ssh2
Jun 19 03:10:01 weisshorn07 sshd[120771]: Accepted publickey for root from 148.187.12.231 port 48270 ssh2
Jun 19 03:10:01 weisshorn03 sshd[121292]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 03:10:01 weisshorn04 sshd[120103]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 03:10:01 weisshorn07 sshd[120771]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 03:10:01 weisshorn08 sshd[119400]: Accepted publickey for root from 148.187.12.231 port 40269 ssh2
Jun 19 03:10:01 weisshorn08 sshd[119400]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 03:10:01 weisshorn14 CROND[49279]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:10:01 weisshorn14 sshd[49261]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 03:10:01 weisshorn14 sshd[49261]: pam_unix(sshd:session): session closed for user root
Jun 19 03:10:01 weisshorn05 sshd[120641]: Accepted publickey for root from 148.187.12.231 port 54666 ssh2
Jun 19 03:10:01 weisshorn05 sshd[120641]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 03:10:01 weisshorn11 sshd[121188]: Accepted publickey for root from 148.187.12.231 port 45146 ssh2
Jun 19 03:10:01 weisshorn11 sshd[121188]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 03:10:01 weisshorn03 CROND[121306]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:10:01 weisshorn06 sshd[121734]: Accepted publickey for root from 148.187.12.231 port 44718 ssh2
Jun 19 03:10:01 weisshorn06 sshd[121734]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 03:10:01 weisshorn10 sshd[120667]: Accepted publickey for root from 148.187.12.231 port 36938 ssh2
Jun 19 03:10:01 weisshorn10 sshd[120667]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 03:10:01 weisshorn09 sshd[120552]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 03:10:01 weisshorn09 sshd[120552]: pam_unix(sshd:session): session closed for user root
Jun 19 03:10:01 weisshorn04 sshd[120103]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 03:10:01 weisshorn04 sshd[120103]: pam_unix(sshd:session): session closed for user root
Jun 19 03:10:01 weisshorn07 sshd[120771]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 03:10:01 weisshorn07 sshd[120771]: pam_unix(sshd:session): session closed for user root
Jun 19 03:10:01 weisshorn03 sshd[121292]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 03:10:01 weisshorn03 sshd[121292]: pam_unix(sshd:session): session closed for user root
Jun 19 03:10:01 weisshorn09 CROND[120575]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:10:01 weisshorn08 sshd[119400]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 03:10:01 weisshorn08 sshd[119400]: pam_unix(sshd:session): session closed for user root
Jun 19 03:10:01 weisshorn12 CROND[120803]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:10:01 weisshorn12 sshd[120800]: Accepted publickey for root from 148.187.12.231 port 46367 ssh2
Jun 19 03:10:01 weisshorn12 sshd[120800]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 03:10:01 weisshorn06 sshd[121734]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 03:10:01 weisshorn06 sshd[121734]: pam_unix(sshd:session): session closed for user root
Jun 19 03:10:01 weisshorn11 sshd[121188]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 03:10:01 weisshorn11 sshd[121188]: pam_unix(sshd:session): session closed for user root
Jun 19 03:10:01 weisshorn13 sshd[124018]: Accepted publickey for root from 148.187.12.231 port 44632 ssh2
Jun 19 03:10:01 weisshorn05 sshd[120641]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 03:10:01 weisshorn05 sshd[120641]: pam_unix(sshd:session): session closed for user root
Jun 19 03:10:01 weisshorn13 sshd[124018]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 03:10:01 weisshorn13 CROND[124022]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:10:01 weisshorn06 CROND[121757]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:10:01 weisshorn10 sshd[120667]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 03:10:01 weisshorn10 sshd[120667]: pam_unix(sshd:session): session closed for user root
Jun 19 03:10:01 weisshorn05 CROND[120664]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:10:01 weisshorn11 CROND[121211]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:10:01 weisshorn12 sshd[120800]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 03:10:01 weisshorn12 sshd[120800]: pam_unix(sshd:session): session closed for user root
Jun 19 03:10:01 weisshorn07 CROND[120794]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:10:01 weisshorn04 CROND[120126]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:10:01 weisshorn02 CROND[30636]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:10:02 weisshorn13 sshd[124018]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 03:10:02 weisshorn13 sshd[124018]: pam_unix(sshd:session): session closed for user root
Jun 19 03:10:02 weisshorn01 postfix/pickup[47377]: 0BED092224F: uid=0 from=<root>
Jun 19 03:10:02 weisshorn01 postfix/cleanup[47627]: 0BED092224F: message-id=<20120619011002.0BED092224F@weisshorn01.admin.cscs.ch>
Jun 19 03:10:02 weisshorn01 postfix/qmgr[2710]: 0BED092224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 03:10:02 weisshorn01 postfix/local[47629]: 0BED092224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=0.8, delays=0.74/0.01/0/0.05, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 03:10:02 weisshorn01 postfix/qmgr[2710]: 0BED092224F: removed
Jun 19 03:10:10 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:10:25 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:10:40 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:10:55 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:11:03 weisshorn06 rhnsd[121760]: running program /usr/sbin/rhn_check
Jun 19 03:11:10 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:11:25 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:11:40 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:11:55 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:12:10 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:12:26 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:12:41 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:12:56 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:13:11 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:13:26 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:13:41 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:13:56 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:14:11 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:14:26 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:14:41 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:14:56 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:15:01 weisshorn07 anacron[120746]: Job `cron.daily' started
Jun 19 03:15:01 weisshorn07 run-parts(/etc/cron.daily)[12080 starting cups
Jun 19 03:15:01 weisshorn07 run-parts(/etc/cron.daily)[12081 finished cups
Jun 19 03:15:01 weisshorn07 run-parts(/etc/cron.daily)[12080 starting logrotate
Jun 19 03:15:02 weisshorn07 run-parts(/etc/cron.daily)[12082 finished logrotate
Jun 19 03:15:02 weisshorn07 run-parts(/etc/cron.daily)[12080 starting makewhatis.cron
Jun 19 03:15:02 weisshorn07 run-parts(/etc/cron.daily)[12095 finished makewhatis.cron
Jun 19 03:15:02 weisshorn07 run-parts(/etc/cron.daily)[12080 starting mlocate.cron
Jun 19 03:15:05 weisshorn07 run-parts(/etc/cron.daily)[12096 finished mlocate.cron
Jun 19 03:15:05 weisshorn07 run-parts(/etc/cron.daily)[12080 starting prelink
Jun 19 03:15:05 weisshorn07 run-parts(/etc/cron.daily)[12097 finished prelink
Jun 19 03:15:05 weisshorn07 run-parts(/etc/cron.daily)[12080 starting readahead.cron
Jun 19 03:15:05 weisshorn07 run-parts(/etc/cron.daily)[12098 finished readahead.cron
Jun 19 03:15:05 weisshorn07 run-parts(/etc/cron.daily)[12080 starting rhsmd
Jun 19 03:15:05 weisshorn07 rhsmd: This system is registered to RHN Classic
Jun 19 03:15:05 weisshorn07 run-parts(/etc/cron.daily)[12099 finished rhsmd
Jun 19 03:15:05 weisshorn07 run-parts(/etc/cron.daily)[12080 starting tmpwatch
Jun 19 03:15:05 weisshorn07 run-parts(/etc/cron.daily)[12103 finished tmpwatch
Jun 19 03:15:05 weisshorn07 anacron[120746]: Job `cron.daily' terminated
Jun 19 03:15:05 weisshorn07 anacron[120746]: Normal exit (1 job run)
Jun 19 03:15:11 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:15:26 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:15:41 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:15:56 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:16:11 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:16:26 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:16:41 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:16:56 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:17:11 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:17:26 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:17:41 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:17:56 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:18:11 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:18:26 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:18:41 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:18:56 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:19:02 weisshorn10 anacron[120638]: Job `cron.daily' started
Jun 19 03:19:02 weisshorn10 run-parts(/etc/cron.daily)[12074 starting cups
Jun 19 03:19:02 weisshorn10 run-parts(/etc/cron.daily)[12075 finished cups
Jun 19 03:19:02 weisshorn10 run-parts(/etc/cron.daily)[12074 starting logrotate
Jun 19 03:19:02 weisshorn10 run-parts(/etc/cron.daily)[12075 finished logrotate
Jun 19 03:19:02 weisshorn10 run-parts(/etc/cron.daily)[12074 starting makewhatis.cron
Jun 19 03:19:02 weisshorn10 run-parts(/etc/cron.daily)[12088 finished makewhatis.cron
Jun 19 03:19:02 weisshorn10 run-parts(/etc/cron.daily)[12074 starting mlocate.cron
Jun 19 03:19:05 weisshorn10 run-parts(/etc/cron.daily)[12089 finished mlocate.cron
Jun 19 03:19:05 weisshorn10 run-parts(/etc/cron.daily)[12074 starting prelink
Jun 19 03:19:05 weisshorn10 run-parts(/etc/cron.daily)[12091 finished prelink
Jun 19 03:19:05 weisshorn10 run-parts(/etc/cron.daily)[12074 starting readahead.cron
Jun 19 03:19:05 weisshorn10 run-parts(/etc/cron.daily)[12092 finished readahead.cron
Jun 19 03:19:05 weisshorn10 run-parts(/etc/cron.daily)[12074 starting rhsmd
Jun 19 03:19:06 weisshorn10 rhsmd: This system is registered to RHN Classic
Jun 19 03:19:06 weisshorn10 run-parts(/etc/cron.daily)[12092 finished rhsmd
Jun 19 03:19:06 weisshorn10 run-parts(/etc/cron.daily)[12074 starting tmpwatch
Jun 19 03:19:06 weisshorn10 run-parts(/etc/cron.daily)[12096 finished tmpwatch
Jun 19 03:19:06 weisshorn10 anacron[120638]: Job `cron.daily' terminated
Jun 19 03:19:11 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:19:26 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:19:41 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:19:56 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:20:01 weisshorn01 CROND[47728]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:20:01 weisshorn02 CROND[31595]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:20:01 weisshorn10 CROND[120981]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:20:01 weisshorn08 CROND[119486]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:20:01 weisshorn14 CROND[49307]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:20:01 weisshorn03 CROND[121379]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:20:01 weisshorn09 CROND[120639]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:20:01 weisshorn05 anacron[120611]: Job `cron.daily' started
Jun 19 03:20:01 weisshorn05 run-parts(/etc/cron.daily)[12072 starting cups
Jun 19 03:20:01 weisshorn06 CROND[121801]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:20:01 weisshorn05 run-parts(/etc/cron.daily)[12073 finished cups
Jun 19 03:20:01 weisshorn05 run-parts(/etc/cron.daily)[12072 starting logrotate
Jun 19 03:20:01 weisshorn12 CROND[120851]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:20:01 weisshorn11 CROND[121247]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:20:01 weisshorn05 CROND[120743]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:20:01 weisshorn04 CROND[120190]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:20:01 weisshorn13 CROND[124069]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:20:02 weisshorn05 run-parts(/etc/cron.daily)[12074 finished logrotate
Jun 19 03:20:02 weisshorn05 run-parts(/etc/cron.daily)[12072 starting makewhatis.cron
Jun 19 03:20:02 weisshorn07 CROND[121066]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:20:02 weisshorn05 run-parts(/etc/cron.daily)[12087 finished makewhatis.cron
Jun 19 03:20:02 weisshorn05 run-parts(/etc/cron.daily)[12072 starting mlocate.cron
Jun 19 03:20:04 weisshorn05 run-parts(/etc/cron.daily)[12088 finished mlocate.cron
Jun 19 03:20:04 weisshorn05 run-parts(/etc/cron.daily)[12072 starting prelink
Jun 19 03:20:04 weisshorn05 run-parts(/etc/cron.daily)[12089 finished prelink
Jun 19 03:20:04 weisshorn05 run-parts(/etc/cron.daily)[12072 starting readahead.cron
Jun 19 03:20:05 weisshorn05 run-parts(/etc/cron.daily)[12090 finished readahead.cron
Jun 19 03:20:05 weisshorn05 run-parts(/etc/cron.daily)[12072 starting rhsmd
Jun 19 03:20:06 weisshorn05 rhsmd: This system is registered to RHN Classic
Jun 19 03:20:06 weisshorn05 run-parts(/etc/cron.daily)[12091 finished rhsmd
Jun 19 03:20:06 weisshorn05 run-parts(/etc/cron.daily)[12072 starting tmpwatch
Jun 19 03:20:06 weisshorn05 run-parts(/etc/cron.daily)[12095 finished tmpwatch
Jun 19 03:20:06 weisshorn05 anacron[120611]: Job `cron.daily' terminated
Jun 19 03:20:06 weisshorn05 anacron[120611]: Normal exit (1 job run)
Jun 19 03:20:11 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:20:26 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:20:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:20:46 weisshorn02 nslcd[2099]: [6f6d8a] ldap_result() failed: Can't contact LDAP server
Jun 19 03:20:46 weisshorn14 nslcd[3437]: [f6ab8e] ldap_result() failed: Can't contact LDAP server
Jun 19 03:20:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:21:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:21:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:21:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:21:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:22:01 weisshorn04 anacron[120073]: Job `cron.daily' started
Jun 19 03:22:01 weisshorn04 run-parts(/etc/cron.daily)[12019 starting cups
Jun 19 03:22:01 weisshorn04 run-parts(/etc/cron.daily)[12020 finished cups
Jun 19 03:22:01 weisshorn04 run-parts(/etc/cron.daily)[12019 starting logrotate
Jun 19 03:22:02 weisshorn04 run-parts(/etc/cron.daily)[12021 finished logrotate
Jun 19 03:22:02 weisshorn04 run-parts(/etc/cron.daily)[12019 starting makewhatis.cron
Jun 19 03:22:02 weisshorn04 run-parts(/etc/cron.daily)[12034 finished makewhatis.cron
Jun 19 03:22:02 weisshorn04 run-parts(/etc/cron.daily)[12019 starting mlocate.cron
Jun 19 03:22:04 weisshorn04 run-parts(/etc/cron.daily)[12035 finished mlocate.cron
Jun 19 03:22:04 weisshorn04 run-parts(/etc/cron.daily)[12019 starting prelink
Jun 19 03:22:04 weisshorn04 run-parts(/etc/cron.daily)[12036 finished prelink
Jun 19 03:22:04 weisshorn04 run-parts(/etc/cron.daily)[12019 starting readahead.cron
Jun 19 03:22:04 weisshorn04 run-parts(/etc/cron.daily)[12037 finished readahead.cron
Jun 19 03:22:04 weisshorn04 run-parts(/etc/cron.daily)[12019 starting rhsmd
Jun 19 03:22:04 weisshorn04 rhsmd: This system is registered to RHN Classic
Jun 19 03:22:04 weisshorn04 run-parts(/etc/cron.daily)[12038 finished rhsmd
Jun 19 03:22:04 weisshorn04 run-parts(/etc/cron.daily)[12019 starting tmpwatch
Jun 19 03:22:05 weisshorn04 run-parts(/etc/cron.daily)[12042 finished tmpwatch
Jun 19 03:22:05 weisshorn04 anacron[120073]: Job `cron.daily' terminated
Jun 19 03:22:05 weisshorn04 anacron[120073]: Normal exit (1 job run)
Jun 19 03:22:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:22:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:22:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:22:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:23:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:23:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:23:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:23:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:24:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:24:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:24:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:24:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:25:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:25:16 weisshorn10 nslcd[4294]: [95e50a] ldap_result() failed: Can't contact LDAP server
Jun 19 03:25:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:25:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:25:45 weisshorn09 nslcd[3954]: [9efb0d] ldap_result() failed: Can't contact LDAP server
Jun 19 03:25:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:26:08 weisshorn08 nslcd[3946]: [95e50a] ldap_result() failed: Can't contact LDAP server
Jun 19 03:26:09 weisshorn06 nslcd[3953]: [0386d1] ldap_result() failed: Can't contact LDAP server
Jun 19 03:26:11 weisshorn04 nslcd[3885]: [9efb0d] ldap_result() failed: Can't contact LDAP server
Jun 19 03:26:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:26:12 weisshorn07 nslcd[4297]: [95e50a] ldap_result() failed: Can't contact LDAP server
Jun 19 03:26:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:26:32 weisshorn12 nslcd[3954]: [95e50a] ldap_result() failed: Can't contact LDAP server
Jun 19 03:26:32 weisshorn11 nslcd[3954]: [3da4d5] ldap_result() failed: Can't contact LDAP server
Jun 19 03:26:32 weisshorn11 nslcd[3954]: [3da4d5] ldap_abandon() failed to abandon search: Other (e.g., implementation specific) error
Jun 19 03:26:34 weisshorn13 nslcd[4224]: [95e50a] ldap_result() failed: Can't contact LDAP server
Jun 19 03:26:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:26:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:27:11 weisshorn05 nslcd[4378]: [5bcf61] ldap_result() failed: Can't contact LDAP server
Jun 19 03:27:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:27:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:27:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:27:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:28:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:28:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:28:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:28:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:29:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:29:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:29:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:29:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:30:01 weisshorn05 CROND[120986]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:30:01 weisshorn07 CROND[121122]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:30:01 weisshorn13 CROND[124147]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:30:01 weisshorn04 CROND[120449]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:30:01 weisshorn01 CROND[47775]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:30:01 weisshorn02 CROND[32517]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:30:01 weisshorn10 CROND[121014]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:30:01 weisshorn08 CROND[119518]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:30:01 weisshorn14 CROND[49382]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:30:01 weisshorn03 CROND[121407]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:30:01 weisshorn09 CROND[120673]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:30:01 weisshorn06 CROND[121862]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:30:01 weisshorn12 CROND[120926]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:30:01 weisshorn11 CROND[121309]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:30:01 weisshorn02 anacron[29802]: Job `cron.daily' started
Jun 19 03:30:01 weisshorn02 run-parts(/etc/cron.daily)[32521 starting cups
Jun 19 03:30:01 weisshorn02 run-parts(/etc/cron.daily)[32529 finished cups
Jun 19 03:30:01 weisshorn02 run-parts(/etc/cron.daily)[32521 starting logrotate
Jun 19 03:30:01 weisshorn02 run-parts(/etc/cron.daily)[32536 finished logrotate
Jun 19 03:30:01 weisshorn02 run-parts(/etc/cron.daily)[32521 starting makewhatis.cron
Jun 19 03:30:02 weisshorn01 anacron[47543]: Job `cron.daily' started
Jun 19 03:30:02 weisshorn08 anacron[119367]: Job `cron.daily' started
Jun 19 03:30:02 weisshorn01 run-parts(/etc/cron.daily)[47777 starting cups
Jun 19 03:30:02 weisshorn01 run-parts(/etc/cron.daily)[47785 finished cups
Jun 19 03:30:02 weisshorn01 run-parts(/etc/cron.daily)[47777 starting logrotate
Jun 19 03:30:02 weisshorn08 run-parts(/etc/cron.daily)[11952 starting cups
Jun 19 03:30:02 weisshorn08 run-parts(/etc/cron.daily)[11952 finished cups
Jun 19 03:30:02 weisshorn08 run-parts(/etc/cron.daily)[11952 starting logrotate
Jun 19 03:30:02 weisshorn01 run-parts(/etc/cron.daily)[47792 finished logrotate
Jun 19 03:30:02 weisshorn01 run-parts(/etc/cron.daily)[47777 starting makewhatis.cron
Jun 19 03:30:02 weisshorn08 run-parts(/etc/cron.daily)[11953 finished logrotate
Jun 19 03:30:02 weisshorn08 run-parts(/etc/cron.daily)[11952 starting makewhatis.cron
Jun 19 03:30:02 weisshorn02 run-parts(/etc/cron.daily)[32665 finished makewhatis.cron
Jun 19 03:30:02 weisshorn02 run-parts(/etc/cron.daily)[32521 starting mlocate.cron
Jun 19 03:30:02 weisshorn02 run-parts(/etc/cron.daily)[32676 finished mlocate.cron
Jun 19 03:30:02 weisshorn02 run-parts(/etc/cron.daily)[32521 starting prelink
Jun 19 03:30:02 weisshorn02 run-parts(/etc/cron.daily)[32688 finished prelink
Jun 19 03:30:02 weisshorn02 run-parts(/etc/cron.daily)[32521 starting readahead.cron
Jun 19 03:30:02 weisshorn02 run-parts(/etc/cron.daily)[32699 finished readahead.cron
Jun 19 03:30:02 weisshorn02 run-parts(/etc/cron.daily)[32521 starting rhsmd
Jun 19 03:30:02 weisshorn08 run-parts(/etc/cron.daily)[11966 finished makewhatis.cron
Jun 19 03:30:02 weisshorn08 run-parts(/etc/cron.daily)[11952 starting mlocate.cron
Jun 19 03:30:02 weisshorn01 run-parts(/etc/cron.daily)[47923 finished makewhatis.cron
Jun 19 03:30:02 weisshorn01 run-parts(/etc/cron.daily)[47777 starting mlocate.cron
Jun 19 03:30:03 weisshorn02 rhsmd: This system is registered to RHN Classic
Jun 19 03:30:03 weisshorn02 run-parts(/etc/cron.daily)[32706 finished rhsmd
Jun 19 03:30:03 weisshorn02 run-parts(/etc/cron.daily)[32521 starting tmpwatch
Jun 19 03:30:03 weisshorn02 run-parts(/etc/cron.daily)[32744 finished tmpwatch
Jun 19 03:30:03 weisshorn02 anacron[29802]: Job `cron.daily' terminated
Jun 19 03:30:03 weisshorn02 anacron[29802]: Normal exit (1 job run)
Jun 19 03:30:03 weisshorn01 run-parts(/etc/cron.daily)[47934 finished mlocate.cron
Jun 19 03:30:03 weisshorn01 run-parts(/etc/cron.daily)[47777 starting prelink
Jun 19 03:30:03 weisshorn01 run-parts(/etc/cron.daily)[47946 finished prelink
Jun 19 03:30:03 weisshorn01 run-parts(/etc/cron.daily)[47777 starting readahead.cron
Jun 19 03:30:03 weisshorn01 run-parts(/etc/cron.daily)[47957 finished readahead.cron
Jun 19 03:30:03 weisshorn01 run-parts(/etc/cron.daily)[47777 starting rhsmd
Jun 19 03:30:04 weisshorn01 rhsmd: This system is registered to RHN Classic
Jun 19 03:30:04 weisshorn01 run-parts(/etc/cron.daily)[47964 finished rhsmd
Jun 19 03:30:04 weisshorn01 run-parts(/etc/cron.daily)[47777 starting tmpwatch
Jun 19 03:30:04 weisshorn01 run-parts(/etc/cron.daily)[48002 finished tmpwatch
Jun 19 03:30:04 weisshorn01 anacron[47543]: Job `cron.daily' terminated
Jun 19 03:30:04 weisshorn01 anacron[47543]: Normal exit (1 job run)
Jun 19 03:30:05 weisshorn08 run-parts(/etc/cron.daily)[11967 finished mlocate.cron
Jun 19 03:30:05 weisshorn08 run-parts(/etc/cron.daily)[11952 starting prelink
Jun 19 03:30:05 weisshorn08 run-parts(/etc/cron.daily)[11968 finished prelink
Jun 19 03:30:05 weisshorn08 run-parts(/etc/cron.daily)[11952 starting readahead.cron
Jun 19 03:30:05 weisshorn08 run-parts(/etc/cron.daily)[11970 finished readahead.cron
Jun 19 03:30:05 weisshorn08 run-parts(/etc/cron.daily)[11952 starting rhsmd
Jun 19 03:30:06 weisshorn08 rhsmd: This system is registered to RHN Classic
Jun 19 03:30:06 weisshorn08 run-parts(/etc/cron.daily)[11970 finished rhsmd
Jun 19 03:30:06 weisshorn08 run-parts(/etc/cron.daily)[11952 starting tmpwatch
Jun 19 03:30:06 weisshorn08 run-parts(/etc/cron.daily)[11974 finished tmpwatch
Jun 19 03:30:06 weisshorn08 anacron[119367]: Job `cron.daily' terminated
Jun 19 03:30:06 weisshorn08 anacron[119367]: Normal exit (1 job run)
Jun 19 03:30:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:30:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:30:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:30:47 weisshorn10 nslcd[4294]: [5bcf61] ldap_result() failed: Can't contact LDAP server
Jun 19 03:30:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:31:01 weisshorn05 nslcd[4378]: [1dd403] ldap_result() failed: Can't contact LDAP server
Jun 19 03:31:01 weisshorn12 nslcd[3954]: [5bcf61] ldap_result() failed: Can't contact LDAP server
Jun 19 03:31:01 weisshorn04 nslcd[3885]: [0802be] ldap_result() failed: Can't contact LDAP server
Jun 19 03:31:01 weisshorn09 nslcd[3954]: [0802be] ldap_result() failed: Can't contact LDAP server
Jun 19 03:31:01 weisshorn03 nslcd[3944]: [0802be] ldap_result() failed: Can't contact LDAP server
Jun 19 03:31:01 weisshorn13 nslcd[4224]: [5bcf61] ldap_result() failed: Can't contact LDAP server
Jun 19 03:31:01 weisshorn08 nslcd[3946]: [5bcf61] ldap_result() failed: Can't contact LDAP server
Jun 19 03:31:01 weisshorn07 nslcd[4297]: [5bcf61] ldap_result() failed: Can't contact LDAP server
Jun 19 03:31:01 weisshorn06 nslcd[3953]: [25f2bc] ldap_result() failed: Can't contact LDAP server
Jun 19 03:31:02 weisshorn11 nslcd[3954]: [c93698] ldap_result() failed: Can't contact LDAP server
Jun 19 03:31:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:31:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:31:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:31:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:32:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:32:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:32:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:32:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:33:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:33:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:33:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:33:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:34:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:34:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:34:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:34:50 weisshorn11 ntpd[4454]: synchronized to 148.187.18.120, stratum 2
Jun 19 03:34:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:35:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:35:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:35:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:35:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:36:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:36:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:36:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:36:45 weisshorn11 rhnsd[121328]: running program /usr/sbin/rhn_check
Jun 19 03:36:50 weisshorn03 rhnsd[121425]: running program /usr/sbin/rhn_check
Jun 19 03:36:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:37:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:37:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:37:41 weisshorn04 rhnsd[120470]: running program /usr/sbin/rhn_check
Jun 19 03:37:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:37:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:38:01 weisshorn03 anacron[121257]: Job `cron.daily' started
Jun 19 03:38:01 weisshorn03 run-parts(/etc/cron.daily)[12142 starting cups
Jun 19 03:38:01 weisshorn03 run-parts(/etc/cron.daily)[12143 finished cups
Jun 19 03:38:01 weisshorn03 run-parts(/etc/cron.daily)[12142 starting logrotate
Jun 19 03:38:01 weisshorn03 run-parts(/etc/cron.daily)[12144 finished logrotate
Jun 19 03:38:01 weisshorn03 run-parts(/etc/cron.daily)[12142 starting makewhatis.cron
Jun 19 03:38:02 weisshorn03 run-parts(/etc/cron.daily)[12157 finished makewhatis.cron
Jun 19 03:38:02 weisshorn03 run-parts(/etc/cron.daily)[12142 starting mlocate.cron
Jun 19 03:38:04 weisshorn03 run-parts(/etc/cron.daily)[12158 finished mlocate.cron
Jun 19 03:38:04 weisshorn03 run-parts(/etc/cron.daily)[12142 starting prelink
Jun 19 03:38:04 weisshorn03 run-parts(/etc/cron.daily)[12159 finished prelink
Jun 19 03:38:04 weisshorn03 run-parts(/etc/cron.daily)[12142 starting readahead.cron
Jun 19 03:38:04 weisshorn03 run-parts(/etc/cron.daily)[12160 finished readahead.cron
Jun 19 03:38:04 weisshorn03 run-parts(/etc/cron.daily)[12142 starting rhsmd
Jun 19 03:38:04 weisshorn03 rhsmd: This system is registered to RHN Classic
Jun 19 03:38:04 weisshorn03 run-parts(/etc/cron.daily)[12161 finished rhsmd
Jun 19 03:38:04 weisshorn03 run-parts(/etc/cron.daily)[12142 starting tmpwatch
Jun 19 03:38:05 weisshorn03 run-parts(/etc/cron.daily)[12165 finished tmpwatch
Jun 19 03:38:05 weisshorn03 anacron[121257]: Job `cron.daily' terminated
Jun 19 03:38:05 weisshorn03 anacron[121257]: Normal exit (1 job run)
Jun 19 03:38:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:38:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:38:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:38:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:39:01 weisshorn14 anacron[49231]: Job `cron.daily' started
Jun 19 03:39:01 weisshorn14 run-parts(/etc/cron.daily)[49404 starting cups
Jun 19 03:39:01 weisshorn14 run-parts(/etc/cron.daily)[49413 finished cups
Jun 19 03:39:01 weisshorn14 run-parts(/etc/cron.daily)[49404 starting logrotate
Jun 19 03:39:01 weisshorn14 run-parts(/etc/cron.daily)[49420 finished logrotate
Jun 19 03:39:01 weisshorn14 run-parts(/etc/cron.daily)[49404 starting makewhatis.cron
Jun 19 03:39:01 weisshorn14 run-parts(/etc/cron.daily)[49549 finished makewhatis.cron
Jun 19 03:39:01 weisshorn14 run-parts(/etc/cron.daily)[49404 starting mlocate.cron
Jun 19 03:39:01 weisshorn14 run-parts(/etc/cron.daily)[49560 finished mlocate.cron
Jun 19 03:39:01 weisshorn14 run-parts(/etc/cron.daily)[49404 starting prelink
Jun 19 03:39:01 weisshorn14 run-parts(/etc/cron.daily)[49572 finished prelink
Jun 19 03:39:01 weisshorn14 run-parts(/etc/cron.daily)[49404 starting readahead.cron
Jun 19 03:39:01 weisshorn14 run-parts(/etc/cron.daily)[49583 finished readahead.cron
Jun 19 03:39:01 weisshorn14 run-parts(/etc/cron.daily)[49404 starting rhsmd
Jun 19 03:39:01 weisshorn14 rhsmd: This system is registered to RHN Classic
Jun 19 03:39:01 weisshorn14 run-parts(/etc/cron.daily)[49590 finished rhsmd
Jun 19 03:39:01 weisshorn14 run-parts(/etc/cron.daily)[49404 starting tmpwatch
Jun 19 03:39:01 weisshorn14 run-parts(/etc/cron.daily)[49628 finished tmpwatch
Jun 19 03:39:01 weisshorn14 anacron[49231]: Job `cron.daily' terminated
Jun 19 03:39:01 weisshorn14 anacron[49231]: Normal exit (1 job run)
Jun 19 03:39:02 weisshorn10 anacron[120638]: Job `cron.weekly' started
Jun 19 03:39:02 weisshorn10 anacron[120638]: Job `cron.weekly' terminated
Jun 19 03:39:02 weisshorn10 anacron[120638]: Normal exit (2 jobs run)
Jun 19 03:39:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:39:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:39:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:39:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:40:01 weisshorn04 CROND[120481]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:40:01 weisshorn13 CROND[124167]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:40:01 weisshorn07 CROND[121147]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:40:01 weisshorn01 CROND[48058]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:40:01 weisshorn02 CROND[33694]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:40:01 weisshorn05 CROND[121016]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:40:01 weisshorn10 CROND[121041]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:40:01 weisshorn08 CROND[119771]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:40:01 weisshorn14 CROND[49632]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:40:01 weisshorn09 CROND[120702]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:40:01 weisshorn03 CROND[121663]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:40:01 weisshorn06 auditd[3928]: Audit daemon rotating log files
Jun 19 03:40:01 weisshorn06 CROND[121888]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:40:01 weisshorn12 CROND[120945]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:40:02 weisshorn11 CROND[121337]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:40:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:40:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:40:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:40:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:41:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:41:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:41:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:41:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:42:01 weisshorn14 nslcd[3437]: [4c574c] ldap_result() failed: Can't contact LDAP server
Jun 19 03:42:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:42:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:42:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:42:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:43:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:43:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:43:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:43:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:44:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:44:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:44:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:44:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:45:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:45:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:45:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:45:51 weisshorn03 nslcd[3944]: [66459b] ldap_result() failed: Can't contact LDAP server
Jun 19 03:45:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:46:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:46:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:46:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:46:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:47:01 weisshorn13 anacron[123993]: Job `cron.daily' started
Jun 19 03:47:01 weisshorn13 run-parts(/etc/cron.daily)[12418 starting cups
Jun 19 03:47:01 weisshorn13 run-parts(/etc/cron.daily)[12419 finished cups
Jun 19 03:47:01 weisshorn13 run-parts(/etc/cron.daily)[12418 starting logrotate
Jun 19 03:47:01 weisshorn11 anacron[121158]: Job `cron.daily' started
Jun 19 03:47:01 weisshorn11 run-parts(/etc/cron.daily)[12135 starting cups
Jun 19 03:47:01 weisshorn13 run-parts(/etc/cron.daily)[12420 finished logrotate
Jun 19 03:47:01 weisshorn13 run-parts(/etc/cron.daily)[12418 starting makewhatis.cron
Jun 19 03:47:01 weisshorn11 run-parts(/etc/cron.daily)[12136 finished cups
Jun 19 03:47:01 weisshorn11 run-parts(/etc/cron.daily)[12135 starting logrotate
Jun 19 03:47:01 weisshorn11 run-parts(/etc/cron.daily)[12137 finished logrotate
Jun 19 03:47:01 weisshorn11 run-parts(/etc/cron.daily)[12135 starting makewhatis.cron
Jun 19 03:47:02 weisshorn13 run-parts(/etc/cron.daily)[12433 finished makewhatis.cron
Jun 19 03:47:02 weisshorn13 run-parts(/etc/cron.daily)[12418 starting mlocate.cron
Jun 19 03:47:02 weisshorn11 run-parts(/etc/cron.daily)[12149 finished makewhatis.cron
Jun 19 03:47:02 weisshorn11 run-parts(/etc/cron.daily)[12135 starting mlocate.cron
Jun 19 03:47:04 weisshorn13 run-parts(/etc/cron.daily)[12434 finished mlocate.cron
Jun 19 03:47:04 weisshorn13 run-parts(/etc/cron.daily)[12418 starting prelink
Jun 19 03:47:04 weisshorn13 run-parts(/etc/cron.daily)[12435 finished prelink
Jun 19 03:47:04 weisshorn13 run-parts(/etc/cron.daily)[12418 starting readahead.cron
Jun 19 03:47:04 weisshorn13 run-parts(/etc/cron.daily)[12436 finished readahead.cron
Jun 19 03:47:04 weisshorn13 run-parts(/etc/cron.daily)[12418 starting rhsmd
Jun 19 03:47:04 weisshorn11 run-parts(/etc/cron.daily)[12151 finished mlocate.cron
Jun 19 03:47:04 weisshorn11 run-parts(/etc/cron.daily)[12135 starting prelink
Jun 19 03:47:05 weisshorn13 rhsmd: This system is registered to RHN Classic
Jun 19 03:47:05 weisshorn13 run-parts(/etc/cron.daily)[12437 finished rhsmd
Jun 19 03:47:05 weisshorn13 run-parts(/etc/cron.daily)[12418 starting tmpwatch
Jun 19 03:47:05 weisshorn13 run-parts(/etc/cron.daily)[12441 finished tmpwatch
Jun 19 03:47:05 weisshorn13 anacron[123993]: Job `cron.daily' terminated
Jun 19 03:47:05 weisshorn13 anacron[123993]: Normal exit (1 job run)
Jun 19 03:47:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:47:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:47:30 weisshorn11 run-parts(/etc/cron.daily)[12397 finished prelink
Jun 19 03:47:30 weisshorn11 run-parts(/etc/cron.daily)[12135 starting readahead.cron
Jun 19 03:47:30 weisshorn11 run-parts(/etc/cron.daily)[12398 finished readahead.cron
Jun 19 03:47:30 weisshorn11 run-parts(/etc/cron.daily)[12135 starting rhsmd
Jun 19 03:47:31 weisshorn11 rhsmd: This system is registered to RHN Classic
Jun 19 03:47:31 weisshorn11 run-parts(/etc/cron.daily)[12399 finished rhsmd
Jun 19 03:47:31 weisshorn11 run-parts(/etc/cron.daily)[12135 starting tmpwatch
Jun 19 03:47:32 weisshorn11 run-parts(/etc/cron.daily)[12402 finished tmpwatch
Jun 19 03:47:32 weisshorn11 anacron[121158]: Job `cron.daily' terminated
Jun 19 03:47:32 weisshorn11 anacron[121158]: Normal exit (1 job run)
Jun 19 03:47:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:47:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:48:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:48:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:48:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:48:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:49:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:49:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:49:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:49:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:50:01 weisshorn11 CROND[124048]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:50:01 weisshorn04 CROND[120541]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:50:01 weisshorn07 CROND[121190]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:50:01 weisshorn01 CROND[48141]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:50:01 weisshorn05 CROND[121076]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:50:01 weisshorn02 CROND[34617]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:50:01 weisshorn13 CROND[124421]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:50:01 weisshorn10 CROND[121105]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:50:01 weisshorn14 CROND[49658]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:50:01 weisshorn08 CROND[119833]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:50:01 weisshorn09 CROND[120762]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:50:01 weisshorn03 CROND[121729]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:50:01 weisshorn06 CROND[121925]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:50:01 weisshorn12 CROND[120974]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 03:50:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:50:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:50:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:50:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:51:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:51:21 weisshorn02 nslcd[2099]: [49361f] ldap_result() failed: Can't contact LDAP server
Jun 19 03:51:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:51:36 weisshorn12 rhnsd[120986]: running program /usr/sbin/rhn_check
Jun 19 03:51:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:51:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:52:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:52:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:52:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:52:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:53:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:53:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:53:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:53:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:54:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:54:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:54:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:54:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:55:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:55:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:55:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:55:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:56:14 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:56:29 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:56:44 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:56:59 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:57:14 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:57:29 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:57:44 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:57:59 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:58:14 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:58:29 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:58:44 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:58:59 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:59:14 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:59:29 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:59:44 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 03:59:59 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:00:01 weisshorn12 CROND[121049]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:00:01 weisshorn11 CROND[124108]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:00:01 weisshorn04 CROND[120578]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:00:01 weisshorn07 CROND[121244]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:00:01 weisshorn01 CROND[48196]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:00:01 weisshorn05 CROND[121108]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:00:01 weisshorn02 CROND[35539]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:00:01 weisshorn13 CROND[124497]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:00:01 weisshorn10 CROND[121137]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:00:01 weisshorn14 CROND[49733]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:00:01 weisshorn09 CROND[120794]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:00:01 weisshorn08 CROND[119863]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:00:01 weisshorn03 CROND[121761]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:00:02 weisshorn06 CROND[121984]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:00:14 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:00:29 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:00:44 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:00:59 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:01:01 weisshorn08 CROND[119868]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 04:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1198 starting 0anacron
Jun 19 04:01:01 weisshorn03 CROND[121766]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 04:01:01 weisshorn06 CROND[121989]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 04:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1219 starting 0anacron
Jun 19 04:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1217 starting 0anacron
Jun 19 04:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1198 finished 0anacron
Jun 19 04:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1198 starting mcelog.cron
Jun 19 04:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1219 finished 0anacron
Jun 19 04:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1219 starting mcelog.cron
Jun 19 04:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1217 finished 0anacron
Jun 19 04:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1217 starting mcelog.cron
Jun 19 04:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1220 finished mcelog.cron
Jun 19 04:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1217 finished mcelog.cron
Jun 19 04:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1198 finished mcelog.cron
Jun 19 04:01:01 weisshorn12 CROND[121054]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 04:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1210 starting 0anacron
Jun 19 04:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1210 finished 0anacron
Jun 19 04:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1210 starting mcelog.cron
Jun 19 04:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1210 finished mcelog.cron
Jun 19 04:01:01 weisshorn11 CROND[124113]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 04:01:01 weisshorn04 CROND[120583]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 04:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1205 starting 0anacron
Jun 19 04:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1241 starting 0anacron
Jun 19 04:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1205 finished 0anacron
Jun 19 04:01:01 weisshorn07 CROND[121249]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 04:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1205 starting mcelog.cron
Jun 19 04:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1241 finished 0anacron
Jun 19 04:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1241 starting mcelog.cron
Jun 19 04:01:01 weisshorn01 CROND[48201]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 04:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1206 finished mcelog.cron
Jun 19 04:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1212 starting 0anacron
Jun 19 04:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4820 starting 0anacron
Jun 19 04:01:01 weisshorn05 CROND[121113]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 04:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1241 finished mcelog.cron
Jun 19 04:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4821 finished 0anacron
Jun 19 04:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4820 starting mcelog.cron
Jun 19 04:01:01 weisshorn02 CROND[35633]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 04:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1212 finished 0anacron
Jun 19 04:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1212 starting mcelog.cron
Jun 19 04:01:01 weisshorn02 run-parts(/etc/cron.hourly)[3563 starting 0anacron
Jun 19 04:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4821 finished mcelog.cron
Jun 19 04:01:01 weisshorn02 run-parts(/etc/cron.hourly)[3564 finished 0anacron
Jun 19 04:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1211 starting 0anacron
Jun 19 04:01:01 weisshorn02 run-parts(/etc/cron.hourly)[3563 starting mcelog.cron
Jun 19 04:01:01 weisshorn02 run-parts(/etc/cron.hourly)[3564 finished mcelog.cron
Jun 19 04:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1211 finished 0anacron
Jun 19 04:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1212 finished mcelog.cron
Jun 19 04:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1211 starting mcelog.cron
Jun 19 04:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1211 finished mcelog.cron
Jun 19 04:01:01 weisshorn13 CROND[124502]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 04:01:01 weisshorn10 CROND[121142]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 04:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1245 starting 0anacron
Jun 19 04:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1245 finished 0anacron
Jun 19 04:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1245 starting mcelog.cron
Jun 19 04:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1211 starting 0anacron
Jun 19 04:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1211 finished 0anacron
Jun 19 04:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1211 starting mcelog.cron
Jun 19 04:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1245 finished mcelog.cron
Jun 19 04:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1211 finished mcelog.cron
Jun 19 04:01:01 weisshorn14 CROND[49738]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 04:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4973 starting 0anacron
Jun 19 04:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4974 finished 0anacron
Jun 19 04:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4973 starting mcelog.cron
Jun 19 04:01:01 weisshorn14 run-parts(/etc/cron.hourly)[4975 finished mcelog.cron
Jun 19 04:01:01 weisshorn09 CROND[120799]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 04:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1207 starting 0anacron
Jun 19 04:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1208 finished 0anacron
Jun 19 04:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1207 starting mcelog.cron
Jun 19 04:01:02 weisshorn09 run-parts(/etc/cron.hourly)[1208 finished mcelog.cron
Jun 19 04:01:14 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:01:29 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:01:44 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:01:59 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:02:14 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:02:29 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:02:44 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:02:59 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:03:14 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:03:29 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:03:44 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:03:59 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:04:14 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:04:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:04:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:05:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:05:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:05:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:05:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:06:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:06:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:06:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:06:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:07:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:07:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:07:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:07:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:08:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:08:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:08:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:08:30 weisshorn14 ntpd[3906]: synchronized to 148.187.18.120, stratum 2
Jun 19 04:08:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:09:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:09:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:09:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:09:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:10:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:10:01 weisshorn09 CROND[120843]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Jun 19 04:10:01 weisshorn09 CROND[120844]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:10:01 weisshorn08 CROND[119946]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:10:01 weisshorn08 CROND[119947]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Jun 19 04:10:01 weisshorn03 CROND[121839]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:10:01 weisshorn03 CROND[121840]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Jun 19 04:10:01 weisshorn06 CROND[122027]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Jun 19 04:10:01 weisshorn06 CROND[122029]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:10:01 weisshorn12 CROND[121093]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Jun 19 04:10:01 weisshorn12 CROND[121095]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:10:01 weisshorn04 CROND[120658]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:10:01 weisshorn04 CROND[120657]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Jun 19 04:10:01 weisshorn11 CROND[124157]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Jun 19 04:10:01 weisshorn11 CROND[124158]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:10:01 weisshorn01 CROND[48267]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 04:10:01 weisshorn01 CROND[48268]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Jun 19 04:10:01 weisshorn01 CROND[48273]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:10:01 weisshorn07 CROND[121289]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Jun 19 04:10:01 weisshorn02 CROND[36510]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:10:01 weisshorn07 CROND[121291]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:10:01 weisshorn02 CROND[36511]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Jun 19 04:10:01 weisshorn05 CROND[121193]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:10:01 weisshorn05 CROND[121195]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Jun 19 04:10:01 weisshorn04 sshd[120663]: Accepted publickey for root from 148.187.12.231 port 44223 ssh2
Jun 19 04:10:01 weisshorn04 sshd[120663]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 04:10:01 weisshorn14 sshd[49777]: Accepted publickey for root from 148.187.12.231 port 49593 ssh2
Jun 19 04:10:01 weisshorn14 sshd[49777]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 04:10:01 weisshorn09 sshd[120850]: Accepted publickey for root from 148.187.12.231 port 57913 ssh2
Jun 19 04:10:01 weisshorn03 sshd[121846]: Accepted publickey for root from 148.187.12.231 port 50472 ssh2
Jun 19 04:10:01 weisshorn09 sshd[120850]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 04:10:01 weisshorn03 sshd[121846]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 04:10:01 weisshorn06 sshd[122034]: Accepted publickey for root from 148.187.12.231 port 44729 ssh2
Jun 19 04:10:01 weisshorn08 sshd[119953]: Accepted publickey for root from 148.187.12.231 port 40280 ssh2
Jun 19 04:10:01 weisshorn06 sshd[122034]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 04:10:01 weisshorn10 sshd[121214]: Accepted publickey for root from 148.187.12.231 port 36946 ssh2
Jun 19 04:10:01 weisshorn14 sshd[49777]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 04:10:01 weisshorn14 sshd[49777]: pam_unix(sshd:session): session closed for user root
Jun 19 04:10:01 weisshorn08 sshd[119953]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 04:10:01 weisshorn10 sshd[121214]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 04:10:01 weisshorn12 sshd[121100]: Accepted publickey for root from 148.187.12.231 port 46383 ssh2
Jun 19 04:10:01 weisshorn13 CROND[124538]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Jun 19 04:10:01 weisshorn13 CROND[124539]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:10:01 weisshorn04 sshd[120663]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 04:10:01 weisshorn04 sshd[120663]: pam_unix(sshd:session): session closed for user root
Jun 19 04:10:01 weisshorn12 sshd[121100]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 04:10:01 weisshorn10 CROND[121224]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:10:01 weisshorn10 CROND[121227]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Jun 19 04:10:01 weisshorn13 sshd[124534]: Accepted publickey for root from 148.187.12.231 port 44641 ssh2
Jun 19 04:10:01 weisshorn13 sshd[124534]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 04:10:01 weisshorn11 sshd[124162]: Accepted publickey for root from 148.187.12.231 port 45165 ssh2
Jun 19 04:10:01 weisshorn09 sshd[120850]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 04:10:01 weisshorn09 sshd[120850]: pam_unix(sshd:session): session closed for user root
Jun 19 04:10:01 weisshorn14 CROND[49801]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log)
Jun 19 04:10:01 weisshorn14 CROND[49802]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:10:01 weisshorn11 sshd[124162]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 04:10:01 weisshorn06 sshd[122034]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 04:10:01 weisshorn06 sshd[122034]: pam_unix(sshd:session): session closed for user root
Jun 19 04:10:01 weisshorn03 sshd[121846]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 04:10:01 weisshorn03 sshd[121846]: pam_unix(sshd:session): session closed for user root
Jun 19 04:10:01 weisshorn08 sshd[119953]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 04:10:01 weisshorn08 sshd[119953]: pam_unix(sshd:session): session closed for user root
Jun 19 04:10:01 weisshorn05 sshd[121190]: Accepted publickey for root from 148.187.12.231 port 54676 ssh2
Jun 19 04:10:01 weisshorn07 sshd[121288]: Accepted publickey for root from 148.187.12.231 port 48283 ssh2
Jun 19 04:10:01 weisshorn07 sshd[121288]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 04:10:01 weisshorn05 sshd[121190]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 04:10:02 weisshorn12 sshd[121100]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 04:10:02 weisshorn12 sshd[121100]: pam_unix(sshd:session): session closed for user root
Jun 19 04:10:02 weisshorn11 sshd[124162]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 04:10:02 weisshorn11 sshd[124162]: pam_unix(sshd:session): session closed for user root
Jun 19 04:10:02 weisshorn10 sshd[121214]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 04:10:02 weisshorn10 sshd[121214]: pam_unix(sshd:session): session closed for user root
Jun 19 04:10:02 weisshorn07 sshd[121288]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 04:10:02 weisshorn07 sshd[121288]: pam_unix(sshd:session): session closed for user root
Jun 19 04:10:02 weisshorn13 sshd[124534]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 04:10:02 weisshorn13 sshd[124534]: pam_unix(sshd:session): session closed for user root
Jun 19 04:10:02 weisshorn05 sshd[121190]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 04:10:02 weisshorn05 sshd[121190]: pam_unix(sshd:session): session closed for user root
Jun 19 04:10:02 weisshorn01 postfix/pickup[47377]: 43FA192224F: uid=0 from=<root>
Jun 19 04:10:02 weisshorn01 postfix/cleanup[48304]: 43FA192224F: message-id=<20120619021002.43FA192224F@weisshorn01.admin.cscs.ch>
Jun 19 04:10:02 weisshorn01 postfix/qmgr[2710]: 43FA192224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 04:10:02 weisshorn01 postfix/local[48306]: 43FA192224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=0.6, delays=0.54/0.01/0/0.04, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 04:10:02 weisshorn01 postfix/qmgr[2710]: 43FA192224F: removed
Jun 19 04:10:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:10:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:10:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:11:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:11:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:11:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:11:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:12:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:12:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:12:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:12:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:13:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:13:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:13:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:13:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:14:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:14:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:14:30 weisshorn08 ntpd[4445]: synchronized to 148.187.18.120, stratum 2
Jun 19 04:14:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:14:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:15:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:15:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:15:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:15:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:16:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:16:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:16:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:16:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:17:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:17:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:17:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:17:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:18:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:18:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:18:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:18:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:19:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:19:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:19:22 weisshorn08 rhnsd[120010]: running program /usr/sbin/rhn_check
Jun 19 04:19:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:19:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:20:01 weisshorn13 CROND[124602]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:20:01 weisshorn10 CROND[121280]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:20:01 weisshorn01 CROND[48407]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:20:01 weisshorn09 CROND[120937]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:20:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:20:01 weisshorn03 CROND[121903]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:20:01 weisshorn08 CROND[120013]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:20:01 weisshorn06 CROND[122126]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:20:01 weisshorn12 CROND[121151]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:20:01 weisshorn02 CROND[37443]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:20:01 weisshorn04 CROND[120721]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:20:01 weisshorn11 CROND[124256]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:20:01 weisshorn14 CROND[49834]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:20:01 weisshorn07 CROND[121392]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:20:02 weisshorn05 CROND[121251]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:20:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:20:26 weisshorn01 rhnsd[48411]: running program /usr/sbin/rhn_check
Jun 19 04:20:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:20:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:21:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:21:01 weisshorn02 nslcd[2099]: [bc6770] ldap_result() failed: Can't contact LDAP server
Jun 19 04:21:01 weisshorn14 nslcd[3437]: [e99dd7] ldap_result() failed: Can't contact LDAP server
Jun 19 04:21:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:21:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:21:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:22:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:22:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:22:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:22:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:23:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:23:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:23:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:23:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:24:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:24:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:24:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:24:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:25:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:25:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:25:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:25:31 weisshorn10 nslcd[4294]: [1dd403] ldap_result() failed: Can't contact LDAP server
Jun 19 04:25:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:25:57 weisshorn01 nslcd[2168]: [9c0e94] ldap_result() failed: Can't contact LDAP server
Jun 19 04:26:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:26:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:26:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:26:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:27:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:27:05 weisshorn02 rhnsd[38094]: running program /usr/sbin/rhn_check
Jun 19 04:27:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:27:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:27:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:28:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:28:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:28:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:28:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:29:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:29:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:29:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:29:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:30:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:30:01 weisshorn14 CROND[49904]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:30:01 weisshorn07 CROND[121411]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:30:01 weisshorn05 CROND[121280]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:30:01 weisshorn13 CROND[124667]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:30:01 weisshorn10 CROND[121305]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:30:01 weisshorn01 CROND[48451]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:30:01 weisshorn09 CROND[120966]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:30:01 weisshorn03 CROND[121927]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:30:01 weisshorn08 CROND[120036]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:30:01 weisshorn06 CROND[122150]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:30:01 weisshorn12 CROND[121216]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:30:01 weisshorn02 CROND[38364]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:30:01 weisshorn11 CROND[124275]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:30:01 weisshorn04 CROND[120745]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:30:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:30:26 weisshorn05 rhnsd[121283]: running program /usr/sbin/rhn_check
Jun 19 04:30:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:30:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:31:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:31:02 weisshorn10 nslcd[4294]: [0386d1] ldap_result() failed: Can't contact LDAP server
Jun 19 04:31:16 weisshorn05 nslcd[4378]: [0386d1] ldap_result() failed: Can't contact LDAP server
Jun 19 04:31:16 weisshorn12 nslcd[3954]: [1dd403] ldap_result() failed: Can't contact LDAP server
Jun 19 04:31:16 weisshorn04 nslcd[3885]: [66459b] ldap_result() failed: Can't contact LDAP server
Jun 19 04:31:16 weisshorn09 nslcd[3954]: [66459b] ldap_result() failed: Can't contact LDAP server
Jun 19 04:31:16 weisshorn03 nslcd[3944]: [8b0cbf] ldap_result() failed: Can't contact LDAP server
Jun 19 04:31:16 weisshorn13 nslcd[4224]: [1dd403] ldap_result() failed: Can't contact LDAP server
Jun 19 04:31:16 weisshorn08 nslcd[3946]: [1dd403] ldap_result() failed: Can't contact LDAP server
Jun 19 04:31:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:31:16 weisshorn07 nslcd[4297]: [1dd403] ldap_result() failed: Can't contact LDAP server
Jun 19 04:31:16 weisshorn06 nslcd[3953]: [e8e1a7] ldap_result() failed: Can't contact LDAP server
Jun 19 04:31:17 weisshorn11 nslcd[3954]: [781401] ldap_result() failed: Can't contact LDAP server
Jun 19 04:31:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:31:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:32:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:32:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:32:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:32:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:33:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:33:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:33:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:33:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:34:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:34:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:34:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:34:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:35:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:35:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:35:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:35:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:36:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:36:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:36:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:36:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:37:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:37:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:37:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:37:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:38:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:38:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:38:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:38:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:39:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:39:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:39:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:39:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:40:01 weisshorn04 CROND[120805]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:40:01 weisshorn14 CROND[49929]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:40:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:40:01 weisshorn07 CROND[121436]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:40:01 weisshorn05 CROND[121340]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:40:01 weisshorn13 CROND[124687]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:40:01 weisshorn01 CROND[48536]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:40:01 weisshorn10 CROND[121365]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:40:01 weisshorn09 CROND[121026]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:40:01 weisshorn03 CROND[121987]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:40:01 weisshorn08 CROND[120096]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:40:01 weisshorn06 CROND[122175]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:40:01 weisshorn12 CROND[121241]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:40:01 weisshorn02 CROND[39316]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:40:01 weisshorn11 CROND[124305]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:40:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:40:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:40:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:41:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:41:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:41:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:41:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:42:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:42:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:42:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:42:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:43:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:43:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:43:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:43:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:44:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:44:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:44:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:44:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:45:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:45:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:45:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:45:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:45:58 weisshorn01 nslcd[2168]: [e611bf] ldap_result() failed: Can't contact LDAP server
Jun 19 04:45:58 weisshorn01 nslcd[2168]: [e611bf] ldap_abandon() failed to abandon search: Other (e.g., implementation specific) error
Jun 19 04:46:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:46:06 weisshorn03 nslcd[3944]: [e203ab] ldap_result() failed: Can't contact LDAP server
Jun 19 04:46:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:46:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:46:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:47:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:47:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:47:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:47:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:48:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:48:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:48:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:48:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:49:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:49:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:49:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:49:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:50:01 weisshorn11 CROND[124378]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:50:01 weisshorn04 CROND[120843]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:50:01 weisshorn14 CROND[49958]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:50:01 weisshorn07 CROND[121514]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:50:01 weisshorn05 CROND[121373]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:50:01 weisshorn13 CROND[124727]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:50:01 weisshorn10 CROND[121403]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:50:01 weisshorn01 CROND[48595]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:50:01 weisshorn09 CROND[121059]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:50:01 weisshorn03 CROND[122026]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:50:01 weisshorn08 CROND[120134]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:50:01 weisshorn06 CROND[122248]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:50:01 weisshorn12 CROND[121273]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:50:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:50:01 weisshorn02 CROND[40244]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 04:50:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:50:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:50:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:51:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:51:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:51:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:51:36 weisshorn02 nslcd[2099]: [0e6897] ldap_result() failed: Can't contact LDAP server
Jun 19 04:51:36 weisshorn02 nslcd[2099]: [0e6897] ldap_abandon() failed to abandon search: Other (e.g., implementation specific) error
Jun 19 04:51:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:52:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:52:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:52:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:52:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:53:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:53:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:53:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:53:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:54:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:54:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:54:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:54:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:55:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:55:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:55:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:55:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:56:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:56:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:56:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:56:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:56:59 weisshorn09 rhnsd[121078]: running program /usr/sbin/rhn_check
Jun 19 04:57:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:57:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:57:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:57:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:58:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:58:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:58:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:58:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:59:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:59:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:59:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 04:59:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:00:01 weisshorn02 CROND[41170]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:00:01 weisshorn04 CROND[120867]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:00:01 weisshorn11 CROND[124402]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:00:01 weisshorn14 CROND[50034]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:00:01 weisshorn07 CROND[121533]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:00:01 weisshorn05 CROND[121402]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:00:01 weisshorn13 CROND[124792]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:00:01 weisshorn01 CROND[48645]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:00:01 weisshorn10 CROND[121427]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:00:01 weisshorn09 CROND[121090]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:00:01 weisshorn03 CROND[122050]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:00:01 weisshorn08 CROND[120158]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:00:01 weisshorn06 CROND[122272]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:00:02 weisshorn12 CROND[121338]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:00:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:00:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:00:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:00:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:01:01 weisshorn06 CROND[122277]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 05:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1222 starting 0anacron
Jun 19 05:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1222 finished 0anacron
Jun 19 05:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1222 starting mcelog.cron
Jun 19 05:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1222 finished mcelog.cron
Jun 19 05:01:01 weisshorn02 CROND[41264]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 05:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4126 starting 0anacron
Jun 19 05:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4127 finished 0anacron
Jun 19 05:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4126 starting mcelog.cron
Jun 19 05:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4128 finished mcelog.cron
Jun 19 05:01:01 weisshorn12 CROND[121343]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 05:01:01 weisshorn04 CROND[120872]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 05:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1208 starting 0anacron
Jun 19 05:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1213 starting 0anacron
Jun 19 05:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1208 finished 0anacron
Jun 19 05:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1208 starting mcelog.cron
Jun 19 05:01:01 weisshorn11 CROND[124407]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 05:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1208 finished mcelog.cron
Jun 19 05:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1244 starting 0anacron
Jun 19 05:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1213 finished 0anacron
Jun 19 05:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1213 starting mcelog.cron
Jun 19 05:01:01 weisshorn14 CROND[50039]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 05:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5003 starting 0anacron
Jun 19 05:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1244 finished 0anacron
Jun 19 05:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5004 finished 0anacron
Jun 19 05:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1244 starting mcelog.cron
Jun 19 05:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5003 starting mcelog.cron
Jun 19 05:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1213 finished mcelog.cron
Jun 19 05:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5005 finished mcelog.cron
Jun 19 05:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1244 finished mcelog.cron
Jun 19 05:01:01 weisshorn07 CROND[121538]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 05:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1215 starting 0anacron
Jun 19 05:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1215 finished 0anacron
Jun 19 05:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1215 starting mcelog.cron
Jun 19 05:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1215 finished mcelog.cron
Jun 19 05:01:01 weisshorn05 CROND[121407]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 05:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1214 starting 0anacron
Jun 19 05:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1214 finished 0anacron
Jun 19 05:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1214 starting mcelog.cron
Jun 19 05:01:01 weisshorn10 CROND[121432]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 05:01:01 weisshorn01 CROND[48652]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 05:01:01 weisshorn13 CROND[124797]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 05:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1214 starting 0anacron
Jun 19 05:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4865 starting 0anacron
Jun 19 05:01:01 weisshorn09 CROND[121095]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 05:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1210 starting 0anacron
Jun 19 05:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1214 finished mcelog.cron
Jun 19 05:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4866 finished 0anacron
Jun 19 05:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4865 starting mcelog.cron
Jun 19 05:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1211 finished 0anacron
Jun 19 05:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1210 starting mcelog.cron
Jun 19 05:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1247 starting 0anacron
Jun 19 05:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1214 finished 0anacron
Jun 19 05:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4866 finished mcelog.cron
Jun 19 05:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1214 starting mcelog.cron
Jun 19 05:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1248 finished 0anacron
Jun 19 05:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1211 finished mcelog.cron
Jun 19 05:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1247 starting mcelog.cron
Jun 19 05:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1248 finished mcelog.cron
Jun 19 05:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1214 finished mcelog.cron
Jun 19 05:01:01 weisshorn03 CROND[122055]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 05:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1220 starting 0anacron
Jun 19 05:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1220 finished 0anacron
Jun 19 05:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1220 starting mcelog.cron
Jun 19 05:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1220 finished mcelog.cron
Jun 19 05:01:01 weisshorn08 CROND[120163]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 05:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1201 starting 0anacron
Jun 19 05:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1201 finished 0anacron
Jun 19 05:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1201 starting mcelog.cron
Jun 19 05:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1201 finished mcelog.cron
Jun 19 05:01:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:01:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:01:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:01:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:02:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:02:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:02:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:02:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:03:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:03:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:03:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:03:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:04:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:04:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:04:30 weisshorn09 nslcd[3954]: [8b0cbf] ldap_result() failed: Can't contact LDAP server
Jun 19 05:04:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:04:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:04:53 weisshorn08 nslcd[3946]: [0386d1] ldap_result() failed: Can't contact LDAP server
Jun 19 05:04:54 weisshorn06 nslcd[3953]: [9efb0d] ldap_result() failed: Can't contact LDAP server
Jun 19 05:04:56 weisshorn04 nslcd[3885]: [8b0cbf] ldap_result() failed: Can't contact LDAP server
Jun 19 05:04:57 weisshorn07 nslcd[4297]: [0386d1] ldap_result() failed: Can't contact LDAP server
Jun 19 05:05:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:05:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:05:17 weisshorn11 nslcd[3954]: [6928d6] ldap_result() failed: Can't contact LDAP server
Jun 19 05:05:18 weisshorn12 nslcd[3954]: [0386d1] ldap_result() failed: Can't contact LDAP server
Jun 19 05:05:19 weisshorn13 nslcd[4224]: [0386d1] ldap_result() failed: Can't contact LDAP server
Jun 19 05:05:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:05:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:05:52 weisshorn01 nslcd[2168]: [1030bc] ldap_result() failed: Can't contact LDAP server
Jun 19 05:05:56 weisshorn05 nslcd[4378]: [25f2bc] ldap_result() failed: Can't contact LDAP server
Jun 19 05:06:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:06:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:06:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:06:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:07:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:07:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:07:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:07:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:08:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:08:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:08:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:08:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:09:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:09:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:09:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:09:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:10:01 weisshorn06 CROND[122322]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:10:01 weisshorn08 CROND[120246]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:10:01 weisshorn02 CROND[42146]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:10:01 weisshorn04 CROND[120947]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:10:01 weisshorn12 CROND[121382]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:10:01 weisshorn14 CROND[50077]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:10:01 weisshorn11 CROND[124447]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:10:01 weisshorn07 CROND[121584]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:10:01 weisshorn09 CROND[121176]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:10:01 weisshorn01 CROND[48751]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:10:01 weisshorn01 CROND[48752]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 05:10:01 weisshorn05 CROND[121482]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:10:01 weisshorn13 CROND[124837]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:10:01 weisshorn10 CROND[121513]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:10:01 weisshorn03 sshd[122134]: Accepted publickey for root from 148.187.12.231 port 50493 ssh2
Jun 19 05:10:01 weisshorn06 sshd[122334]: Accepted publickey for root from 148.187.12.231 port 44757 ssh2
Jun 19 05:10:01 weisshorn03 sshd[122134]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 05:10:01 weisshorn09 sshd[121178]: Accepted publickey for root from 148.187.12.231 port 57936 ssh2
Jun 19 05:10:01 weisshorn09 sshd[121178]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 05:10:01 weisshorn14 sshd[50079]: Accepted publickey for root from 148.187.12.231 port 49617 ssh2
Jun 19 05:10:01 weisshorn14 sshd[50079]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 05:10:01 weisshorn03 CROND[122148]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:10:01 weisshorn04 sshd[120955]: Accepted publickey for root from 148.187.12.231 port 44246 ssh2
Jun 19 05:10:01 weisshorn06 sshd[122334]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 05:10:01 weisshorn04 sshd[120955]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 05:10:01 weisshorn12 sshd[121384]: Accepted publickey for root from 148.187.12.231 port 46405 ssh2
Jun 19 05:10:01 weisshorn14 sshd[50079]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 05:10:01 weisshorn14 sshd[50079]: pam_unix(sshd:session): session closed for user root
Jun 19 05:10:01 weisshorn08 sshd[120248]: Accepted publickey for root from 148.187.12.231 port 40303 ssh2
Jun 19 05:10:01 weisshorn07 sshd[121594]: Accepted publickey for root from 148.187.12.231 port 48305 ssh2
Jun 19 05:10:02 weisshorn11 sshd[124449]: Accepted publickey for root from 148.187.12.231 port 45185 ssh2
Jun 19 05:10:02 weisshorn13 sshd[124839]: Accepted publickey for root from 148.187.12.231 port 44664 ssh2
Jun 19 05:10:02 weisshorn12 sshd[121384]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 05:10:02 weisshorn13 sshd[124839]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 05:10:02 weisshorn07 sshd[121594]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 05:10:02 weisshorn08 sshd[120248]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 05:10:02 weisshorn06 sshd[122334]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 05:10:02 weisshorn06 sshd[122334]: pam_unix(sshd:session): session closed for user root
Jun 19 05:10:02 weisshorn03 sshd[122134]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 05:10:02 weisshorn03 sshd[122134]: pam_unix(sshd:session): session closed for user root
Jun 19 05:10:02 weisshorn09 sshd[121178]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 05:10:02 weisshorn09 sshd[121178]: pam_unix(sshd:session): session closed for user root
Jun 19 05:10:02 weisshorn05 sshd[121484]: Accepted publickey for root from 148.187.12.231 port 54697 ssh2
Jun 19 05:10:02 weisshorn04 sshd[120955]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 05:10:02 weisshorn04 sshd[120955]: pam_unix(sshd:session): session closed for user root
Jun 19 05:10:02 weisshorn11 sshd[124449]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 05:10:02 weisshorn10 sshd[121512]: Accepted publickey for root from 148.187.12.231 port 36972 ssh2
Jun 19 05:10:02 weisshorn05 sshd[121484]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 05:10:02 weisshorn10 sshd[121512]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 05:10:02 weisshorn13 sshd[124839]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 05:10:02 weisshorn13 sshd[124839]: pam_unix(sshd:session): session closed for user root
Jun 19 05:10:02 weisshorn07 sshd[121594]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 05:10:02 weisshorn07 sshd[121594]: pam_unix(sshd:session): session closed for user root
Jun 19 05:10:02 weisshorn12 sshd[121384]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 05:10:02 weisshorn12 sshd[121384]: pam_unix(sshd:session): session closed for user root
Jun 19 05:10:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:10:02 weisshorn08 sshd[120248]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 05:10:02 weisshorn08 sshd[120248]: pam_unix(sshd:session): session closed for user root
Jun 19 05:10:02 weisshorn11 sshd[124449]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 05:10:02 weisshorn11 sshd[124449]: pam_unix(sshd:session): session closed for user root
Jun 19 05:10:02 weisshorn10 sshd[121512]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 05:10:02 weisshorn10 sshd[121512]: pam_unix(sshd:session): session closed for user root
Jun 19 05:10:02 weisshorn05 sshd[121484]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 05:10:02 weisshorn05 sshd[121484]: pam_unix(sshd:session): session closed for user root
Jun 19 05:10:02 weisshorn01 postfix/pickup[48385]: 57D6892224F: uid=0 from=<root>
Jun 19 05:10:02 weisshorn01 postfix/cleanup[48783]: 57D6892224F: message-id=<20120619031002.57D6892224F@weisshorn01.admin.cscs.ch>
Jun 19 05:10:02 weisshorn01 postfix/qmgr[2710]: 57D6892224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 05:10:02 weisshorn01 postfix/local[48785]: 57D6892224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=0.46, delays=0.4/0.02/0/0.04, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 05:10:02 weisshorn01 postfix/qmgr[2710]: 57D6892224F: removed
Jun 19 05:10:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:10:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:10:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:11:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:11:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:11:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:11:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:12:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:12:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:12:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:12:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:13:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:13:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:13:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:13:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:14:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:14:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:14:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:14:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:15:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:15:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:15:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:15:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:16:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:16:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:16:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:16:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:17:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:17:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:17:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:17:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:18:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:18:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:18:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:18:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:19:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:19:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:19:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:19:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:20:01 weisshorn03 CROND[122188]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:20:01 weisshorn06 CROND[122410]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:20:01 weisshorn02 CROND[43064]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:20:01 weisshorn08 CROND[120297]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:20:01 weisshorn01 CROND[48842]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:20:01 weisshorn04 CROND[121006]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:20:01 weisshorn14 CROND[50122]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:20:01 weisshorn12 CROND[121435]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:20:01 weisshorn11 CROND[124541]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:20:01 weisshorn07 CROND[121677]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:20:01 weisshorn09 CROND[121224]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:20:01 weisshorn13 CROND[124893]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:20:02 weisshorn10 CROND[121565]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:20:02 weisshorn05 CROND[121536]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:20:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:20:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:20:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:20:46 weisshorn14 nslcd[3437]: [31b62d] ldap_result() failed: Can't contact LDAP server
Jun 19 05:20:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:21:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:21:16 weisshorn02 nslcd[2099]: [5138de] ldap_result() failed: Can't contact LDAP server
Jun 19 05:21:16 weisshorn14 nslcd[3437]: [49c29b] ldap_result() failed: Can't contact LDAP server
Jun 19 05:21:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:21:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:21:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:22:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:22:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:22:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:22:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:23:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:23:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:23:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:23:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:24:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:24:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:24:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:24:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:25:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:25:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:25:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:25:46 weisshorn10 nslcd[4294]: [25f2bc] ldap_result() failed: Can't contact LDAP server
Jun 19 05:25:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:26:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:26:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:26:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:26:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:27:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:27:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:27:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:27:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:28:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:28:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:28:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:28:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:29:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:29:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:29:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:29:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:30:01 weisshorn13 CROND[124958]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:30:01 weisshorn03 CROND[122212]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:30:01 weisshorn10 CROND[121590]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:30:01 weisshorn06 CROND[122434]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:30:01 weisshorn05 CROND[121565]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:30:01 weisshorn02 CROND[43989]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:30:01 weisshorn01 CROND[48886]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:30:01 weisshorn14 CROND[50199]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:30:01 weisshorn04 CROND[121030]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:30:01 weisshorn12 CROND[121500]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:30:01 weisshorn08 CROND[120321]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:30:01 weisshorn11 CROND[124565]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:30:01 weisshorn07 CROND[121696]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:30:01 weisshorn09 CROND[121253]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:30:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:30:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:30:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:30:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:31:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:31:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:31:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:31:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:32:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:32:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:32:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:32:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:33:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:33:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:33:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:33:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:34:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:34:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:34:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:34:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:35:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:35:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:35:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:35:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:36:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:36:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:36:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:36:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:37:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:37:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:37:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:37:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:38:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:38:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:38:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:38:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:39:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:39:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:39:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:39:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:40:01 weisshorn09 nslcd[3954]: [e203ab] ldap_result() failed: Can't contact LDAP server
Jun 19 05:40:01 weisshorn09 CROND[121319]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:40:01 weisshorn13 nslcd[4224]: [25f2bc] ldap_result() failed: Can't contact LDAP server
Jun 19 05:40:01 weisshorn13 CROND[124983]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:40:01 weisshorn03 nslcd[3944]: [86d60e] ldap_result() failed: Can't contact LDAP server
Jun 19 05:40:01 weisshorn03 CROND[122278]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:40:01 weisshorn10 nslcd[4294]: [e8e1a7] ldap_result() failed: Can't contact LDAP server
Jun 19 05:40:01 weisshorn10 CROND[121656]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:40:01 weisshorn06 nslcd[3953]: [0802be] ldap_result() failed: Can't contact LDAP server
Jun 19 05:40:01 weisshorn06 CROND[122478]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:40:01 weisshorn05 nslcd[4378]: [e8e1a7] ldap_result() failed: Can't contact LDAP server
Jun 19 05:40:01 weisshorn05 CROND[121631]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:40:01 weisshorn02 CROND[44947]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:40:01 weisshorn01 CROND[48977]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:40:01 weisshorn04 nslcd[3885]: [e203ab] ldap_result() failed: Can't contact LDAP server
Jun 19 05:40:01 weisshorn14 CROND[50223]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:40:01 weisshorn04 CROND[121096]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:40:01 weisshorn12 nslcd[3954]: [25f2bc] ldap_result() failed: Can't contact LDAP server
Jun 19 05:40:01 weisshorn12 CROND[121525]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:40:01 weisshorn08 nslcd[3946]: [25f2bc] ldap_result() failed: Can't contact LDAP server
Jun 19 05:40:01 weisshorn08 CROND[120389]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:40:01 weisshorn11 nslcd[3954]: [7195e5] ldap_result() failed: Can't contact LDAP server
Jun 19 05:40:01 weisshorn11 CROND[124598]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:40:01 weisshorn07 nslcd[4297]: [25f2bc] ldap_result() failed: Can't contact LDAP server
Jun 19 05:40:01 weisshorn07 CROND[121745]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:40:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:40:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:40:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:40:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:41:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:41:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:41:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:41:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:42:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:42:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:42:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:42:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:43:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:43:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:43:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:43:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:44:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:44:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:44:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:44:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:45:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:45:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:45:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:45:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:46:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:46:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:46:21 weisshorn03 nslcd[3944]: [3da4d5] ldap_result() failed: Can't contact LDAP server
Jun 19 05:46:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:46:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:47:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:47:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:47:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:47:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:48:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:48:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:48:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:48:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:49:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:49:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:49:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:49:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:50:01 weisshorn07 CROND[121798]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:50:01 weisshorn09 CROND[121350]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:50:01 weisshorn13 CROND[125058]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:50:01 weisshorn03 CROND[122310]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:50:01 weisshorn06 CROND[122536]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:50:01 weisshorn10 CROND[121687]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:50:01 weisshorn05 CROND[121662]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:50:01 weisshorn02 CROND[45869]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:50:01 weisshorn01 CROND[49029]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:50:01 weisshorn14 CROND[50298]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:50:01 weisshorn12 CROND[121602]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:50:01 weisshorn04 CROND[121127]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:50:01 weisshorn08 CROND[120418]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:50:01 weisshorn11 CROND[124662]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 05:50:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:50:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:50:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:50:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:51:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:51:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:51:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:51:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:51:51 weisshorn02 nslcd[2099]: [22ad0c] ldap_result() failed: Can't contact LDAP server
Jun 19 05:52:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:52:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:52:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:52:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:53:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:53:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:53:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:53:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:54:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:54:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:54:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:54:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:55:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:55:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:55:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:55:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:56:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:56:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:56:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:56:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:57:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:57:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:57:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:57:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:58:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:58:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:58:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:58:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:59:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:59:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:59:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 05:59:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:00:01 weisshorn07 CROND[121822]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:00:01 weisshorn09 CROND[121379]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:00:01 weisshorn03 CROND[122374]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:00:01 weisshorn13 CROND[125082]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:00:01 weisshorn06 CROND[122560]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:00:01 weisshorn10 CROND[121716]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:00:01 weisshorn05 CROND[121691]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:00:01 weisshorn02 CROND[46821]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:00:01 weisshorn01 CROND[49085]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:00:01 weisshorn14 CROND[50323]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:00:01 weisshorn12 CROND[121621]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:00:01 weisshorn08 CROND[120447]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:00:01 weisshorn04 CROND[121191]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:00:02 weisshorn11 CROND[124686]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:00:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:00:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:00:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:00:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:01:01 weisshorn11 CROND[124691]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 06:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1246 starting 0anacron
Jun 19 06:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1247 finished 0anacron
Jun 19 06:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1246 starting mcelog.cron
Jun 19 06:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1247 finished mcelog.cron
Jun 19 06:01:01 weisshorn07 CROND[121827]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 06:01:01 weisshorn09 CROND[121384]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 06:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1218 starting 0anacron
Jun 19 06:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1213 starting 0anacron
Jun 19 06:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1213 finished 0anacron
Jun 19 06:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1213 starting mcelog.cron
Jun 19 06:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1214 finished mcelog.cron
Jun 19 06:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1218 finished 0anacron
Jun 19 06:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1218 starting mcelog.cron
Jun 19 06:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1218 finished mcelog.cron
Jun 19 06:01:01 weisshorn03 CROND[122379]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 06:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1223 starting 0anacron
Jun 19 06:01:01 weisshorn13 CROND[125087]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 06:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1223 finished 0anacron
Jun 19 06:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1250 starting 0anacron
Jun 19 06:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1223 starting mcelog.cron
Jun 19 06:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1250 finished 0anacron
Jun 19 06:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1250 starting mcelog.cron
Jun 19 06:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1223 finished mcelog.cron
Jun 19 06:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1251 finished mcelog.cron
Jun 19 06:01:01 weisshorn06 CROND[122565]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 06:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1225 starting 0anacron
Jun 19 06:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1225 finished 0anacron
Jun 19 06:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1225 starting mcelog.cron
Jun 19 06:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1225 finished mcelog.cron
Jun 19 06:01:01 weisshorn10 CROND[121721]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 06:01:01 weisshorn05 CROND[121696]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 06:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1216 starting 0anacron
Jun 19 06:01:01 weisshorn02 CROND[46915]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 06:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1217 starting 0anacron
Jun 19 06:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4691 starting 0anacron
Jun 19 06:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4692 finished 0anacron
Jun 19 06:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4691 starting mcelog.cron
Jun 19 06:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1217 finished 0anacron
Jun 19 06:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1217 starting mcelog.cron
Jun 19 06:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4693 finished mcelog.cron
Jun 19 06:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1217 finished 0anacron
Jun 19 06:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1216 starting mcelog.cron
Jun 19 06:01:01 weisshorn01 CROND[49090]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 06:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1217 finished mcelog.cron
Jun 19 06:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4909 starting 0anacron
Jun 19 06:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1217 finished mcelog.cron
Jun 19 06:01:01 weisshorn14 CROND[50328]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 06:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5032 starting 0anacron
Jun 19 06:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4910 finished 0anacron
Jun 19 06:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5033 finished 0anacron
Jun 19 06:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4909 starting mcelog.cron
Jun 19 06:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5032 starting mcelog.cron
Jun 19 06:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5034 finished mcelog.cron
Jun 19 06:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4910 finished mcelog.cron
Jun 19 06:01:01 weisshorn12 CROND[121626]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 06:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1216 starting 0anacron
Jun 19 06:01:01 weisshorn04 CROND[121196]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 06:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1216 finished 0anacron
Jun 19 06:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1216 starting mcelog.cron
Jun 19 06:01:01 weisshorn08 CROND[120452]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 06:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1216 finished mcelog.cron
Jun 19 06:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1204 starting 0anacron
Jun 19 06:01:02 weisshorn04 run-parts(/etc/cron.hourly)[1211 starting 0anacron
Jun 19 06:01:02 weisshorn04 run-parts(/etc/cron.hourly)[1212 finished 0anacron
Jun 19 06:01:02 weisshorn04 run-parts(/etc/cron.hourly)[1211 starting mcelog.cron
Jun 19 06:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1204 finished 0anacron
Jun 19 06:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1204 starting mcelog.cron
Jun 19 06:01:02 weisshorn04 run-parts(/etc/cron.hourly)[1212 finished mcelog.cron
Jun 19 06:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1204 finished mcelog.cron
Jun 19 06:01:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:01:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:01:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:01:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:02:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:02:10 weisshorn07 rhnsd[121847]: running program /usr/sbin/rhn_check
Jun 19 06:02:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:02:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:02:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:03:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:03:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:03:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:03:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:04:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:04:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:04:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:04:45 weisshorn09 nslcd[3954]: [86d60e] ldap_result() failed: Can't contact LDAP server
Jun 19 06:04:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:05:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:05:08 weisshorn08 nslcd[3946]: [e8e1a7] ldap_result() failed: Can't contact LDAP server
Jun 19 06:05:09 weisshorn06 nslcd[3953]: [66459b] ldap_result() failed: Can't contact LDAP server
Jun 19 06:05:11 weisshorn04 nslcd[3885]: [86d60e] ldap_result() failed: Can't contact LDAP server
Jun 19 06:05:12 weisshorn07 nslcd[4297]: [e8e1a7] ldap_result() failed: Can't contact LDAP server
Jun 19 06:05:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:05:32 weisshorn11 nslcd[3954]: [46a5f2] ldap_result() failed: Can't contact LDAP server
Jun 19 06:05:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:05:33 weisshorn12 nslcd[3954]: [e8e1a7] ldap_result() failed: Can't contact LDAP server
Jun 19 06:05:34 weisshorn13 nslcd[4224]: [e8e1a7] ldap_result() failed: Can't contact LDAP server
Jun 19 06:05:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:06:00 weisshorn01 nslcd[2168]: [b8d7b1] ldap_result() failed: Can't contact LDAP server
Jun 19 06:06:00 weisshorn01 nslcd[2168]: [b8d7b1] ldap_abandon() failed to abandon search: Other (e.g., implementation specific) error
Jun 19 06:06:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:06:04 weisshorn01 nslcd[2168]: [f50d9f] ldap_result() failed: Can't contact LDAP server
Jun 19 06:06:11 weisshorn05 nslcd[4378]: [9efb0d] ldap_result() failed: Can't contact LDAP server
Jun 19 06:06:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:06:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:06:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:07:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:07:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:07:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:07:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:08:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:08:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:08:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:08:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:09:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:09:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:09:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:09:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:10:01 weisshorn12 CROND[121670]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:10:01 weisshorn04 CROND[121236]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:10:01 weisshorn08 CROND[120529]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:10:01 weisshorn11 CROND[124742]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:10:01 weisshorn09 CROND[121459]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:10:01 weisshorn07 CROND[121887]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:10:01 weisshorn03 CROND[122423]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:10:01 weisshorn13 CROND[125131]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:10:01 weisshorn06 CROND[122618]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:10:01 weisshorn02 CROND[47762]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:10:01 weisshorn10 CROND[121800]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:10:01 weisshorn05 CROND[121771]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:10:01 weisshorn14 CROND[50366]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:10:01 weisshorn01 CROND[337]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:10:01 weisshorn01 CROND[339]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 06:10:02 weisshorn13 sshd[125133]: Accepted publickey for root from 148.187.12.231 port 44674 ssh2
Jun 19 06:10:02 weisshorn12 sshd[121672]: Accepted publickey for root from 148.187.12.231 port 46417 ssh2
Jun 19 06:10:02 weisshorn12 sshd[121672]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 06:10:02 weisshorn14 sshd[50368]: Accepted publickey for root from 148.187.12.231 port 49631 ssh2
Jun 19 06:10:02 weisshorn14 sshd[50368]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 06:10:02 weisshorn03 sshd[122425]: Accepted publickey for root from 148.187.12.231 port 50507 ssh2
Jun 19 06:10:02 weisshorn13 sshd[125133]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 06:10:02 weisshorn03 sshd[122425]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 06:10:02 weisshorn06 sshd[122620]: Accepted publickey for root from 148.187.12.231 port 44769 ssh2
Jun 19 06:10:02 weisshorn04 sshd[121238]: Accepted publickey for root from 148.187.12.231 port 44260 ssh2
Jun 19 06:10:02 weisshorn04 sshd[121238]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 06:10:02 weisshorn06 sshd[122620]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 06:10:02 weisshorn09 sshd[121461]: Accepted publickey for root from 148.187.12.231 port 57954 ssh2
Jun 19 06:10:02 weisshorn09 sshd[121461]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 06:10:02 weisshorn11 sshd[124744]: Accepted publickey for root from 148.187.12.231 port 45203 ssh2
Jun 19 06:10:02 weisshorn11 sshd[124744]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 06:10:02 weisshorn10 sshd[121802]: Accepted publickey for root from 148.187.12.231 port 36986 ssh2
Jun 19 06:10:02 weisshorn14 sshd[50368]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 06:10:02 weisshorn14 sshd[50368]: pam_unix(sshd:session): session closed for user root
Jun 19 06:10:02 weisshorn08 sshd[120531]: Accepted publickey for root from 148.187.12.231 port 40315 ssh2
Jun 19 06:10:02 weisshorn05 sshd[121773]: Accepted publickey for root from 148.187.12.231 port 54711 ssh2
Jun 19 06:10:02 weisshorn10 sshd[121802]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 06:10:02 weisshorn05 sshd[121773]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 06:10:02 weisshorn07 sshd[121889]: Accepted publickey for root from 148.187.12.231 port 48321 ssh2
Jun 19 06:10:02 weisshorn08 sshd[120531]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 06:10:02 weisshorn07 sshd[121889]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 06:10:02 weisshorn04 sshd[121238]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 06:10:02 weisshorn04 sshd[121238]: pam_unix(sshd:session): session closed for user root
Jun 19 06:10:02 weisshorn06 sshd[122620]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 06:10:02 weisshorn06 sshd[122620]: pam_unix(sshd:session): session closed for user root
Jun 19 06:10:02 weisshorn12 sshd[121672]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 06:10:02 weisshorn12 sshd[121672]: pam_unix(sshd:session): session closed for user root
Jun 19 06:10:02 weisshorn09 sshd[121461]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 06:10:02 weisshorn09 sshd[121461]: pam_unix(sshd:session): session closed for user root
Jun 19 06:10:02 weisshorn13 sshd[125133]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 06:10:02 weisshorn13 sshd[125133]: pam_unix(sshd:session): session closed for user root
Jun 19 06:10:02 weisshorn11 sshd[124744]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 06:10:02 weisshorn11 sshd[124744]: pam_unix(sshd:session): session closed for user root
Jun 19 06:10:02 weisshorn03 sshd[122425]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 06:10:02 weisshorn03 sshd[122425]: pam_unix(sshd:session): session closed for user root
Jun 19 06:10:02 weisshorn10 sshd[121802]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 06:10:02 weisshorn10 sshd[121802]: pam_unix(sshd:session): session closed for user root
Jun 19 06:10:02 weisshorn08 sshd[120531]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 06:10:02 weisshorn08 sshd[120531]: pam_unix(sshd:session): session closed for user root
Jun 19 06:10:02 weisshorn05 sshd[121773]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 06:10:02 weisshorn05 sshd[121773]: pam_unix(sshd:session): session closed for user root
Jun 19 06:10:02 weisshorn07 sshd[121889]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 06:10:02 weisshorn07 sshd[121889]: pam_unix(sshd:session): session closed for user root
Jun 19 06:10:02 weisshorn01 postfix/pickup[49060]: 7197A92224F: uid=0 from=<root>
Jun 19 06:10:02 weisshorn01 postfix/cleanup[369]: 7197A92224F: message-id=<20120619041002.7197A92224F@weisshorn01.admin.cscs.ch>
Jun 19 06:10:02 weisshorn01 postfix/qmgr[2710]: 7197A92224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 06:10:02 weisshorn01 postfix/local[371]: 7197A92224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=0.45, delays=0.4/0.01/0/0.04, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 06:10:02 weisshorn01 postfix/qmgr[2710]: 7197A92224F: removed
Jun 19 06:10:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:10:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:10:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:10:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:11:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:11:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:11:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:11:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:12:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:12:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:12:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:12:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:13:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:13:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:13:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:13:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:14:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:14:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:14:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:14:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:15:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:15:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:15:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:15:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:16:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:16:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:16:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:16:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:17:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:17:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:17:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:17:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:18:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:18:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:18:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:18:45 weisshorn01 kernel: Lustre: 5268:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046e5:0xd1dc:0x0]] [valid 68719476736]
Jun 19 06:18:45 weisshorn01 kernel: Lustre: 5268:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 3 previous similar messages
Jun 19 06:18:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:19:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:19:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:19:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:19:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:20:01 weisshorn12 CROND[121764]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:20:01 weisshorn04 CROND[121289]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:20:01 weisshorn08 CROND[120580]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:20:01 weisshorn11 CROND[124823]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:20:01 weisshorn09 CROND[121512]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:20:01 weisshorn07 CROND[121962]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:20:01 weisshorn01 CROND[432]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:20:01 weisshorn03 CROND[122471]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:20:01 weisshorn13 CROND[125228]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:20:01 weisshorn06 CROND[122698]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:20:01 weisshorn02 CROND[48684]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:20:01 weisshorn10 CROND[121853]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:20:01 weisshorn14 CROND[50462]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:20:02 weisshorn05 CROND[121824]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:20:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:20:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:20:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:20:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:21:01 weisshorn14 nslcd[3437]: [ff9d09] ldap_result() failed: Can't contact LDAP server
Jun 19 06:21:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:21:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:21:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:21:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:22:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:22:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:22:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:22:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:23:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:23:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:23:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:23:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:24:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:24:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:24:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:24:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:25:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:25:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:25:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:25:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:26:01 weisshorn10 nslcd[4294]: [9efb0d] ldap_result() failed: Can't contact LDAP server
Jun 19 06:26:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:26:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:26:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:26:34 weisshorn01 nslcd[2168]: [88dedd] ldap_result() failed: Can't contact LDAP server
Jun 19 06:26:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:27:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:27:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:27:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:27:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:28:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:28:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:28:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:28:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:29:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:29:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:29:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:29:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:30:01 weisshorn10 CROND[121913]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:30:01 weisshorn14 nslcd[3437]: [754342] ldap_result() failed: Can't contact LDAP server
Jun 19 06:30:01 weisshorn14 CROND[50488]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:30:01 weisshorn12 CROND[121788]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:30:01 weisshorn05 CROND[121888]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:30:01 weisshorn04 CROND[121353]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:30:01 weisshorn08 CROND[120644]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:30:01 weisshorn11 CROND[124852]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:30:01 weisshorn09 CROND[121541]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:30:01 weisshorn01 CROND[517]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:30:01 weisshorn07 CROND[121986]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:30:01 weisshorn13 CROND[125247]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:30:01 weisshorn06 CROND[122722]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:30:01 weisshorn03 CROND[122535]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:30:01 weisshorn02 nslcd[2099]: [997556] ldap_result() failed: Can't contact LDAP server
Jun 19 06:30:01 weisshorn02 CROND[841]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:30:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:30:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:30:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:30:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:30:50 weisshorn14 rhnsd[50490]: running program /usr/sbin/rhn_check
Jun 19 06:31:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:31:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:31:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:31:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:32:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:32:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:32:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:32:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:33:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:33:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:33:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:33:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:33:51 weisshorn13 rhnsd[125260]: running program /usr/sbin/rhn_check
Jun 19 06:34:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:34:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:34:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:34:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:34:51 weisshorn01 kernel: Lustre: 5147:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046ee:0x1115:0x0]] [valid 68719476736]
Jun 19 06:35:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:35:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:35:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:35:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:36:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:36:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:36:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:36:39 weisshorn01 kernel: Lustre: 5278:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046ee:0x1125:0x0]] [valid 68719476736]
Jun 19 06:36:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:37:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:37:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:37:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:37:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:38:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:38:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:38:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:38:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:39:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:39:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:39:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:39:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:40:01 weisshorn02 CROND[1848]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:40:01 weisshorn10 CROND[121943]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:40:01 weisshorn14 CROND[50512]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:40:01 weisshorn12 CROND[121812]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:40:01 weisshorn04 CROND[121378]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:40:01 weisshorn05 CROND[121913]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:40:01 weisshorn08 CROND[120671]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:40:01 weisshorn11 CROND[124884]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:40:01 weisshorn09 CROND[121601]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:40:01 weisshorn01 CROND[603]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:40:01 weisshorn07 CROND[122029]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:40:01 weisshorn06 CROND[122760]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:40:01 weisshorn03 CROND[122565]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:40:01 weisshorn13 CROND[125278]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:40:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:40:16 weisshorn10 nslcd[4294]: [0802be] ldap_result() failed: Can't contact LDAP server
Jun 19 06:40:16 weisshorn05 nslcd[4378]: [0802be] ldap_result() failed: Can't contact LDAP server
Jun 19 06:40:16 weisshorn12 nslcd[3954]: [9efb0d] ldap_result() failed: Can't contact LDAP server
Jun 19 06:40:16 weisshorn04 nslcd[3885]: [3da4d5] ldap_result() failed: Can't contact LDAP server
Jun 19 06:40:16 weisshorn09 nslcd[3954]: [3da4d5] ldap_result() failed: Can't contact LDAP server
Jun 19 06:40:16 weisshorn11 nslcd[3954]: [2dfed6] ldap_result() failed: Can't contact LDAP server
Jun 19 06:40:16 weisshorn03 nslcd[3944]: [c93698] ldap_result() failed: Can't contact LDAP server
Jun 19 06:40:16 weisshorn13 nslcd[4224]: [9efb0d] ldap_result() failed: Can't contact LDAP server
Jun 19 06:40:16 weisshorn08 nslcd[3946]: [9efb0d] ldap_result() failed: Can't contact LDAP server
Jun 19 06:40:16 weisshorn07 nslcd[4297]: [9efb0d] ldap_result() failed: Can't contact LDAP server
Jun 19 06:40:16 weisshorn06 nslcd[3953]: [8b0cbf] ldap_result() failed: Can't contact LDAP server
Jun 19 06:40:16 weisshorn06 nslcd[3953]: [8b0cbf] ldap_abandon() failed to abandon search: Other (e.g., implementation specific) error
Jun 19 06:40:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:40:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:40:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:41:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:41:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:41:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:41:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:42:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:42:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:42:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:42:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:43:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:43:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:43:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:43:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:44:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:44:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:44:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:44:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:45:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:45:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:45:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:45:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:46:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:46:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:46:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:46:36 weisshorn03 nslcd[3944]: [781401] ldap_result() failed: Can't contact LDAP server
Jun 19 06:46:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:47:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:47:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:47:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:47:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:48:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:48:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:48:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:48:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:49:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:49:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:49:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:49:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:50:01 weisshorn02 CROND[2954]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:50:01 weisshorn13 CROND[125355]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:50:01 weisshorn14 CROND[50587]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:50:01 weisshorn10 CROND[121976]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:50:01 weisshorn12 CROND[121886]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:50:01 weisshorn04 CROND[121416]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:50:01 weisshorn05 CROND[121951]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:50:01 weisshorn11 CROND[124945]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:50:01 weisshorn08 CROND[120707]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:50:01 weisshorn09 CROND[121639]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:50:01 weisshorn01 CROND[680]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:50:01 weisshorn07 CROND[122084]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:50:01 weisshorn06 CROND[122820]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:50:02 weisshorn03 CROND[122599]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 06:50:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:50:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:50:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:50:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:51:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:51:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:51:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:51:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:52:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:52:06 weisshorn02 nslcd[2099]: [333c89] ldap_result() failed: Can't contact LDAP server
Jun 19 06:52:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:52:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:52:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:53:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:53:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:53:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:53:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:54:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:54:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:54:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:54:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:55:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:55:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:55:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:55:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:56:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:56:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:56:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:56:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:57:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:57:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:57:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:57:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:58:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:58:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:58:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:58:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:59:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:59:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:59:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 06:59:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:00:01 weisshorn06 CROND[122858]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:00:01 weisshorn03 CROND[122664]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:00:01 weisshorn02 CROND[3919]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:00:01 weisshorn14 CROND[50617]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:00:01 weisshorn10 CROND[122041]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:00:01 weisshorn13 CROND[125379]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:00:01 weisshorn12 CROND[121910]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:00:01 weisshorn04 CROND[121481]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:00:01 weisshorn05 CROND[122016]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:00:01 weisshorn11 CROND[124974]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:00:01 weisshorn09 CROND[121704]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:00:01 weisshorn01 CROND[774]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:00:01 weisshorn08 CROND[120774]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:00:01 weisshorn07 CROND[122122]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:00:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:00:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:00:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:00:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:01:01 weisshorn07 CROND[122127]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 07:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1221 starting 0anacron
Jun 19 07:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1221 finished 0anacron
Jun 19 07:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1221 starting mcelog.cron
Jun 19 07:01:01 weisshorn06 CROND[122863]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 07:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1228 starting 0anacron
Jun 19 07:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1228 finished 0anacron
Jun 19 07:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1228 starting mcelog.cron
Jun 19 07:01:01 weisshorn03 CROND[122669]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 07:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1228 finished mcelog.cron
Jun 19 07:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1221 finished mcelog.cron
Jun 19 07:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1226 starting 0anacron
Jun 19 07:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1226 finished 0anacron
Jun 19 07:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1226 starting mcelog.cron
Jun 19 07:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1226 finished mcelog.cron
Jun 19 07:01:01 weisshorn02 CROND[4016]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 07:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4016 starting 0anacron
Jun 19 07:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4025 finished 0anacron
Jun 19 07:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4016 starting mcelog.cron
Jun 19 07:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4032 finished mcelog.cron
Jun 19 07:01:01 weisshorn14 CROND[50623]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 07:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5062 starting 0anacron
Jun 19 07:01:01 weisshorn10 CROND[122046]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 07:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5063 finished 0anacron
Jun 19 07:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5062 starting mcelog.cron
Jun 19 07:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1220 starting 0anacron
Jun 19 07:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5064 finished mcelog.cron
Jun 19 07:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1220 finished 0anacron
Jun 19 07:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1220 starting mcelog.cron
Jun 19 07:01:01 weisshorn13 CROND[125384]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 07:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1220 finished mcelog.cron
Jun 19 07:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1253 starting 0anacron
Jun 19 07:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1253 finished 0anacron
Jun 19 07:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1253 starting mcelog.cron
Jun 19 07:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1254 finished mcelog.cron
Jun 19 07:01:01 weisshorn04 CROND[121486]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 07:01:01 weisshorn12 CROND[121915]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 07:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1214 starting 0anacron
Jun 19 07:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1219 starting 0anacron
Jun 19 07:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1214 finished 0anacron
Jun 19 07:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1214 starting mcelog.cron
Jun 19 07:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1219 finished 0anacron
Jun 19 07:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1219 starting mcelog.cron
Jun 19 07:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1215 finished mcelog.cron
Jun 19 07:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1219 finished mcelog.cron
Jun 19 07:01:01 weisshorn05 CROND[122021]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 07:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1220 starting 0anacron
Jun 19 07:01:01 weisshorn11 CROND[124979]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 07:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1249 starting 0anacron
Jun 19 07:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1249 finished 0anacron
Jun 19 07:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1249 starting mcelog.cron
Jun 19 07:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1220 finished 0anacron
Jun 19 07:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1220 starting mcelog.cron
Jun 19 07:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1249 finished mcelog.cron
Jun 19 07:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1220 finished mcelog.cron
Jun 19 07:01:01 weisshorn09 CROND[121709]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 07:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1217 starting 0anacron
Jun 19 07:01:01 weisshorn01 CROND[779]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 07:01:01 weisshorn01 run-parts(/etc/cron.hourly)[779] starting 0anacron
Jun 19 07:01:01 weisshorn01 run-parts(/etc/cron.hourly)[789] finished 0anacron
Jun 19 07:01:01 weisshorn01 run-parts(/etc/cron.hourly)[779] starting mcelog.cron
Jun 19 07:01:01 weisshorn08 CROND[120779]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 07:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1217 finished 0anacron
Jun 19 07:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1217 starting mcelog.cron
Jun 19 07:01:01 weisshorn01 run-parts(/etc/cron.hourly)[796] finished mcelog.cron
Jun 19 07:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1217 finished mcelog.cron
Jun 19 07:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1207 starting 0anacron
Jun 19 07:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1207 finished 0anacron
Jun 19 07:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1207 starting mcelog.cron
Jun 19 07:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1207 finished mcelog.cron
Jun 19 07:01:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:01:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:01:25 weisshorn10 rhnsd[122065]: running program /usr/sbin/rhn_check
Jun 19 07:01:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:01:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:02:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:02:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:02:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:02:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:03:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:03:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:03:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:03:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:04:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:04:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:04:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:04:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:05:00 weisshorn09 nslcd[3954]: [c93698] ldap_result() failed: Can't contact LDAP server
Jun 19 07:05:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:05:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:05:23 weisshorn08 nslcd[3946]: [0802be] ldap_result() failed: Can't contact LDAP server
Jun 19 07:05:24 weisshorn06 nslcd[3953]: [e203ab] ldap_result() failed: Can't contact LDAP server
Jun 19 07:05:26 weisshorn04 nslcd[3885]: [c93698] ldap_result() failed: Can't contact LDAP server
Jun 19 07:05:27 weisshorn07 nslcd[4297]: [0802be] ldap_result() failed: Can't contact LDAP server
Jun 19 07:05:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:05:47 weisshorn11 nslcd[3954]: [6ed7f6] ldap_result() failed: Can't contact LDAP server
Jun 19 07:05:48 weisshorn12 nslcd[3954]: [0802be] ldap_result() failed: Can't contact LDAP server
Jun 19 07:05:49 weisshorn13 nslcd[4224]: [0802be] ldap_result() failed: Can't contact LDAP server
Jun 19 07:05:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:06:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:06:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:06:26 weisshorn05 nslcd[4378]: [66459b] ldap_result() failed: Can't contact LDAP server
Jun 19 07:06:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:06:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:07:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:07:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:07:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:07:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:08:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:08:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:08:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:08:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:09:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:09:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:09:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:09:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:10:01 weisshorn08 CROND[120825]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:10:01 weisshorn06 CROND[122938]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:10:01 weisshorn07 CROND[122206]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:10:01 weisshorn03 CROND[122716]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:10:01 weisshorn02 CROND[4860]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:10:01 weisshorn14 CROND[50661]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:10:01 weisshorn10 CROND[122094]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:10:01 weisshorn04 CROND[121534]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:10:01 weisshorn13 CROND[125435]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:10:01 weisshorn12 CROND[121963]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:10:01 weisshorn11 CROND[125068]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:10:01 weisshorn01 CROND[843]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:10:01 weisshorn01 CROND[845]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 07:10:01 weisshorn09 CROND[121753]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:10:02 weisshorn05 CROND[122065]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:10:02 weisshorn06 sshd[122940]: Accepted publickey for root from 148.187.12.231 port 44780 ssh2
Jun 19 07:10:02 weisshorn14 sshd[50663]: Accepted publickey for root from 148.187.12.231 port 49638 ssh2
Jun 19 07:10:02 weisshorn06 sshd[122940]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 07:10:02 weisshorn14 sshd[50663]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 07:10:02 weisshorn04 sshd[121536]: Accepted publickey for root from 148.187.12.231 port 44274 ssh2
Jun 19 07:10:02 weisshorn04 sshd[121536]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 07:10:02 weisshorn09 sshd[121752]: Accepted publickey for root from 148.187.12.231 port 57967 ssh2
Jun 19 07:10:02 weisshorn09 sshd[121752]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 07:10:02 weisshorn14 sshd[50663]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 07:10:02 weisshorn14 sshd[50663]: pam_unix(sshd:session): session closed for user root
Jun 19 07:10:02 weisshorn10 sshd[122096]: Accepted publickey for root from 148.187.12.231 port 36998 ssh2
Jun 19 07:10:02 weisshorn11 sshd[125070]: Accepted publickey for root from 148.187.12.231 port 45215 ssh2
Jun 19 07:10:02 weisshorn08 sshd[120827]: Accepted publickey for root from 148.187.12.231 port 40327 ssh2
Jun 19 07:10:02 weisshorn08 sshd[120827]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 07:10:02 weisshorn12 sshd[121965]: Accepted publickey for root from 148.187.12.231 port 46433 ssh2
Jun 19 07:10:02 weisshorn11 sshd[125070]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 07:10:02 weisshorn10 sshd[122096]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 07:10:02 weisshorn03 sshd[122718]: Accepted publickey for root from 148.187.12.231 port 50520 ssh2
Jun 19 07:10:02 weisshorn03 sshd[122718]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 07:10:02 weisshorn07 sshd[122208]: Accepted publickey for root from 148.187.12.231 port 48334 ssh2
Jun 19 07:10:02 weisshorn13 sshd[125437]: Accepted publickey for root from 148.187.12.231 port 44693 ssh2
Jun 19 07:10:02 weisshorn07 sshd[122208]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 07:10:02 weisshorn12 sshd[121965]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 07:10:02 weisshorn13 sshd[125437]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 07:10:02 weisshorn04 sshd[121536]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 07:10:02 weisshorn04 sshd[121536]: pam_unix(sshd:session): session closed for user root
Jun 19 07:10:02 weisshorn09 sshd[121752]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 07:10:02 weisshorn09 sshd[121752]: pam_unix(sshd:session): session closed for user root
Jun 19 07:10:02 weisshorn06 sshd[122940]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 07:10:02 weisshorn06 sshd[122940]: pam_unix(sshd:session): session closed for user root
Jun 19 07:10:02 weisshorn05 sshd[122064]: Accepted publickey for root from 148.187.12.231 port 54725 ssh2
Jun 19 07:10:02 weisshorn05 sshd[122064]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 07:10:02 weisshorn07 sshd[122208]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 07:10:02 weisshorn07 sshd[122208]: pam_unix(sshd:session): session closed for user root
Jun 19 07:10:02 weisshorn11 sshd[125070]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 07:10:02 weisshorn11 sshd[125070]: pam_unix(sshd:session): session closed for user root
Jun 19 07:10:02 weisshorn08 sshd[120827]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 07:10:02 weisshorn08 sshd[120827]: pam_unix(sshd:session): session closed for user root
Jun 19 07:10:02 weisshorn03 sshd[122718]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 07:10:02 weisshorn03 sshd[122718]: pam_unix(sshd:session): session closed for user root
Jun 19 07:10:02 weisshorn10 sshd[122096]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 07:10:02 weisshorn10 sshd[122096]: pam_unix(sshd:session): session closed for user root
Jun 19 07:10:02 weisshorn12 sshd[121965]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 07:10:02 weisshorn12 sshd[121965]: pam_unix(sshd:session): session closed for user root
Jun 19 07:10:02 weisshorn13 sshd[125437]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 07:10:02 weisshorn13 sshd[125437]: pam_unix(sshd:session): session closed for user root
Jun 19 07:10:02 weisshorn05 sshd[122064]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 07:10:02 weisshorn05 sshd[122064]: pam_unix(sshd:session): session closed for user root
Jun 19 07:10:02 weisshorn01 postfix/pickup[49060]: A09BD92224F: uid=0 from=<root>
Jun 19 07:10:02 weisshorn01 postfix/cleanup[877]: A09BD92224F: message-id=<20120619051002.A09BD92224F@weisshorn01.admin.cscs.ch>
Jun 19 07:10:02 weisshorn01 postfix/qmgr[2710]: A09BD92224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 07:10:02 weisshorn01 postfix/local[879]: A09BD92224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=0.62, delays=0.56/0.01/0/0.04, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 07:10:02 weisshorn01 postfix/qmgr[2710]: A09BD92224F: removed
Jun 19 07:10:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:10:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:10:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:10:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:11:03 weisshorn06 rhnsd[122963]: running program /usr/sbin/rhn_check
Jun 19 07:11:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:11:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:11:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:11:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:12:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:12:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:12:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:12:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:13:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:13:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:13:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:13:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:14:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:14:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:14:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:14:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:15:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:15:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:15:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:15:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:16:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:16:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:16:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:16:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:17:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:17:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:17:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:17:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:18:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:18:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:18:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:18:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:19:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:19:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:19:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:19:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:20:01 weisshorn09 CROND[121802]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:20:01 weisshorn05 CROND[122114]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:20:01 weisshorn06 CROND[122985]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:20:01 weisshorn07 CROND[122246]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:20:01 weisshorn08 CROND[120870]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:20:01 weisshorn02 CROND[5783]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:20:01 weisshorn14 CROND[50758]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:20:01 weisshorn03 CROND[122761]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:20:01 weisshorn01 CROND[930]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:20:01 weisshorn10 CROND[122139]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:20:01 weisshorn04 CROND[121579]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:20:01 weisshorn12 CROND[122049]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:20:01 weisshorn13 CROND[125521]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:20:02 weisshorn11 CROND[125113]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:20:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:20:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:20:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:20:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:21:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:21:16 weisshorn14 nslcd[3437]: [e7f3e5] ldap_result() failed: Can't contact LDAP server
Jun 19 07:21:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:21:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:21:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:22:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:22:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:22:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:22:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:23:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:23:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:23:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:23:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:24:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:24:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:24:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:24:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:25:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:25:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:25:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:25:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:26:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:26:16 weisshorn10 nslcd[4294]: [66459b] ldap_result() failed: Can't contact LDAP server
Jun 19 07:26:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:26:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:26:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:27:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:27:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:27:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:27:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:28:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:28:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:28:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:28:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:29:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:29:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:29:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:29:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:30:01 weisshorn12 CROND[122073]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:30:01 weisshorn13 CROND[125545]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:30:01 weisshorn11 CROND[125145]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:30:01 weisshorn09 CROND[121867]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:30:01 weisshorn05 CROND[122179]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:30:01 weisshorn06 CROND[123028]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:30:01 weisshorn07 CROND[122294]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:30:01 weisshorn08 CROND[120937]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:30:01 weisshorn14 CROND[50783]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:30:01 weisshorn02 CROND[6741]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:30:01 weisshorn03 CROND[122826]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:30:01 weisshorn04 CROND[121644]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:30:01 weisshorn01 CROND[1011]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:30:01 weisshorn10 CROND[122205]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:30:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:30:16 weisshorn02 nslcd[2099]: [a9831a] ldap_result() failed: Can't contact LDAP server
Jun 19 07:30:16 weisshorn14 nslcd[3437]: [6de806] ldap_result() failed: Can't contact LDAP server
Jun 19 07:30:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:30:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:30:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:31:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:31:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:31:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:31:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:32:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:32:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:32:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:32:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:33:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:33:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:33:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:33:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:34:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:34:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:34:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:34:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:35:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:35:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:35:31 weisshorn01 nslcd[2168]: [399349] ldap_result() failed: Can't contact LDAP server
Jun 19 07:35:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:35:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:36:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:36:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:36:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:36:45 weisshorn11 rhnsd[125162]: running program /usr/sbin/rhn_check
Jun 19 07:36:50 weisshorn03 rhnsd[122852]: running program /usr/sbin/rhn_check
Jun 19 07:36:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:37:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:37:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:37:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:37:41 weisshorn04 rhnsd[121672]: running program /usr/sbin/rhn_check
Jun 19 07:37:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:38:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:38:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:38:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:38:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:39:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:39:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:39:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:39:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:40:01 weisshorn01 CROND[1072]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:40:01 weisshorn12 CROND[122105]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:40:01 weisshorn13 CROND[125580]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:40:01 weisshorn09 CROND[121894]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:40:01 weisshorn11 CROND[125212]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:40:01 weisshorn05 CROND[122206]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:40:01 weisshorn06 CROND[123082]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:40:01 weisshorn07 CROND[122348]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:40:01 weisshorn08 CROND[120967]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:40:01 weisshorn02 CROND[7658]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:40:01 weisshorn14 CROND[50807]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:40:01 weisshorn03 CROND[122860]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:40:01 weisshorn04 CROND[121678]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:40:01 weisshorn10 CROND[122237]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:40:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:40:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:40:31 weisshorn10 nslcd[4294]: [8b0cbf] ldap_result() failed: Can't contact LDAP server
Jun 19 07:40:31 weisshorn05 nslcd[4378]: [8b0cbf] ldap_result() failed: Can't contact LDAP server
Jun 19 07:40:31 weisshorn12 nslcd[3954]: [66459b] ldap_result() failed: Can't contact LDAP server
Jun 19 07:40:31 weisshorn04 nslcd[3885]: [781401] ldap_result() failed: Can't contact LDAP server
Jun 19 07:40:31 weisshorn09 nslcd[3954]: [781401] ldap_result() failed: Can't contact LDAP server
Jun 19 07:40:31 weisshorn11 nslcd[3954]: [9df57d] ldap_result() failed: Can't contact LDAP server
Jun 19 07:40:31 weisshorn03 nslcd[3944]: [6928d6] ldap_result() failed: Can't contact LDAP server
Jun 19 07:40:31 weisshorn13 nslcd[4224]: [66459b] ldap_result() failed: Can't contact LDAP server
Jun 19 07:40:31 weisshorn08 nslcd[3946]: [66459b] ldap_result() failed: Can't contact LDAP server
Jun 19 07:40:31 weisshorn07 nslcd[4297]: [66459b] ldap_result() failed: Can't contact LDAP server
Jun 19 07:40:31 weisshorn06 nslcd[3953]: [86d60e] ldap_result() failed: Can't contact LDAP server
Jun 19 07:40:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:40:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:41:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:41:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:41:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:41:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:42:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:42:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:42:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:42:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:43:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:43:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:43:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:43:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:44:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:44:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:44:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:44:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:45:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:45:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:45:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:45:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:46:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:46:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:46:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:46:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:47:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:47:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:47:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:47:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:48:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:48:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:48:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:48:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:49:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:49:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:49:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:49:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:50:01 weisshorn01 CROND[1116]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:50:01 weisshorn10 CROND[122261]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:50:01 weisshorn12 CROND[122170]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:50:01 weisshorn13 CROND[125645]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:50:01 weisshorn09 CROND[121923]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:50:01 weisshorn11 CROND[125236]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:50:01 weisshorn05 CROND[122235]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:50:01 weisshorn07 CROND[122367]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:50:01 weisshorn06 CROND[123106]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:50:01 weisshorn02 CROND[8580]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:50:01 weisshorn14 CROND[50885]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:50:01 weisshorn08 CROND[120991]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:50:01 weisshorn03 CROND[122884]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:50:02 weisshorn04 CROND[121702]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 07:50:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:50:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:50:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:50:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:51:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:51:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:51:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:51:36 weisshorn12 rhnsd[122174]: running program /usr/sbin/rhn_check
Jun 19 07:51:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:52:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:52:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:52:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:52:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:53:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:53:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:53:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:53:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:54:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:54:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:54:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:54:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:55:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:55:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:55:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:55:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:56:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:56:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:56:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:56:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:57:04 weisshorn01 nslcd[2168]: [593232] ldap_result() failed: Can't contact LDAP server
Jun 19 07:57:04 weisshorn01 nslcd[2168]: [9efef7] ldap_result() failed: Can't contact LDAP server
Jun 19 07:57:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:57:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:57:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:57:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:58:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:58:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:58:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:58:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:59:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:59:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:59:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 07:59:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:00:01 weisshorn04 CROND[121767]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:00:01 weisshorn01 CROND[1208]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:00:01 weisshorn10 CROND[122326]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:00:01 weisshorn12 CROND[122196]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:00:01 weisshorn13 CROND[125669]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:00:01 weisshorn09 CROND[121988]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:00:01 weisshorn11 CROND[125268]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:00:01 weisshorn05 CROND[122300]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:00:01 weisshorn07 CROND[122415]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:00:01 weisshorn06 CROND[123149]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:00:01 weisshorn14 CROND[50909]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:00:01 weisshorn02 CROND[9543]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:00:01 weisshorn08 CROND[121058]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:00:02 weisshorn03 CROND[122949]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:00:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:00:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:00:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:00:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:01:01 weisshorn03 CROND[122954]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 08:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1229 starting 0anacron
Jun 19 08:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1229 finished 0anacron
Jun 19 08:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1229 starting mcelog.cron
Jun 19 08:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1229 finished mcelog.cron
Jun 19 08:01:01 weisshorn04 CROND[121772]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 08:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1217 starting 0anacron
Jun 19 08:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1217 finished 0anacron
Jun 19 08:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1217 starting mcelog.cron
Jun 19 08:01:01 weisshorn01 CROND[1213]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 08:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1217 finished mcelog.cron
Jun 19 08:01:01 weisshorn01 run-parts(/etc/cron.hourly)[1213 starting 0anacron
Jun 19 08:01:01 weisshorn10 CROND[122331]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 08:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1223 starting 0anacron
Jun 19 08:01:01 weisshorn01 run-parts(/etc/cron.hourly)[1223 finished 0anacron
Jun 19 08:01:01 weisshorn01 run-parts(/etc/cron.hourly)[1213 starting mcelog.cron
Jun 19 08:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1223 finished 0anacron
Jun 19 08:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1223 starting mcelog.cron
Jun 19 08:01:01 weisshorn01 run-parts(/etc/cron.hourly)[1230 finished mcelog.cron
Jun 19 08:01:01 weisshorn13 CROND[125674]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 08:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1223 finished mcelog.cron
Jun 19 08:01:01 weisshorn12 CROND[122201]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 08:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1256 starting 0anacron
Jun 19 08:01:01 weisshorn09 CROND[121993]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 08:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1219 starting 0anacron
Jun 19 08:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1222 starting 0anacron
Jun 19 08:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1220 finished 0anacron
Jun 19 08:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1219 starting mcelog.cron
Jun 19 08:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1256 finished 0anacron
Jun 19 08:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1256 starting mcelog.cron
Jun 19 08:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1220 finished mcelog.cron
Jun 19 08:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1256 finished mcelog.cron
Jun 19 08:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1222 finished 0anacron
Jun 19 08:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1222 starting mcelog.cron
Jun 19 08:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1222 finished mcelog.cron
Jun 19 08:01:01 weisshorn11 CROND[125273]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 08:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1252 starting 0anacron
Jun 19 08:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1252 finished 0anacron
Jun 19 08:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1252 starting mcelog.cron
Jun 19 08:01:01 weisshorn05 CROND[122305]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 08:01:01 weisshorn07 CROND[122420]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 08:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1224 starting 0anacron
Jun 19 08:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1252 finished mcelog.cron
Jun 19 08:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1223 starting 0anacron
Jun 19 08:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1224 finished 0anacron
Jun 19 08:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1224 starting mcelog.cron
Jun 19 08:01:01 weisshorn06 CROND[123154]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 08:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1224 finished mcelog.cron
Jun 19 08:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1223 finished 0anacron
Jun 19 08:01:01 weisshorn14 CROND[50914]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 08:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1223 starting mcelog.cron
Jun 19 08:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5091 starting 0anacron
Jun 19 08:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5092 finished 0anacron
Jun 19 08:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1231 starting 0anacron
Jun 19 08:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5091 starting mcelog.cron
Jun 19 08:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1231 finished 0anacron
Jun 19 08:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1231 starting mcelog.cron
Jun 19 08:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5093 finished mcelog.cron
Jun 19 08:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1223 finished mcelog.cron
Jun 19 08:01:01 weisshorn02 CROND[9637]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 08:01:01 weisshorn08 CROND[121063]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 08:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1231 finished mcelog.cron
Jun 19 08:01:01 weisshorn02 run-parts(/etc/cron.hourly)[9637 starting 0anacron
Jun 19 08:01:01 weisshorn02 run-parts(/etc/cron.hourly)[9646 finished 0anacron
Jun 19 08:01:01 weisshorn02 run-parts(/etc/cron.hourly)[9637 starting mcelog.cron
Jun 19 08:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1210 starting 0anacron
Jun 19 08:01:01 weisshorn02 run-parts(/etc/cron.hourly)[9653 finished mcelog.cron
Jun 19 08:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1210 finished 0anacron
Jun 19 08:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1210 starting mcelog.cron
Jun 19 08:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1210 finished mcelog.cron
Jun 19 08:01:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:01:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:01:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:01:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:02:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:02:21 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:02:36 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:02:51 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:03:06 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:03:21 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:03:36 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:03:51 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:04:06 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:04:21 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:04:36 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:04:51 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:05:06 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:05:15 weisshorn09 nslcd[3954]: [6928d6] ldap_result() failed: Can't contact LDAP server
Jun 19 08:05:21 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:05:36 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:05:38 weisshorn08 nslcd[3946]: [8b0cbf] ldap_result() failed: Can't contact LDAP server
Jun 19 08:05:39 weisshorn06 nslcd[3953]: [3da4d5] ldap_result() failed: Can't contact LDAP server
Jun 19 08:05:41 weisshorn04 nslcd[3885]: [6928d6] ldap_result() failed: Can't contact LDAP server
Jun 19 08:05:42 weisshorn07 nslcd[4297]: [8b0cbf] ldap_result() failed: Can't contact LDAP server
Jun 19 08:05:51 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:06:02 weisshorn11 nslcd[3954]: [bd7a3e] ldap_result() failed: Can't contact LDAP server
Jun 19 08:06:03 weisshorn12 nslcd[3954]: [8b0cbf] ldap_result() failed: Can't contact LDAP server
Jun 19 08:06:04 weisshorn13 nslcd[4224]: [8b0cbf] ldap_result() failed: Can't contact LDAP server
Jun 19 08:06:06 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:06:21 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:06:36 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:06:41 weisshorn05 nslcd[4378]: [e203ab] ldap_result() failed: Can't contact LDAP server
Jun 19 08:06:51 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:07:06 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:07:21 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:07:36 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:07:51 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:08:06 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:08:21 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:08:36 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:08:51 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:09:06 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:09:21 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:09:36 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:09:51 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:10:01 weisshorn08 CROND[121151]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:10:01 weisshorn03 CROND[123034]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:10:01 weisshorn04 CROND[121827]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:10:01 weisshorn01 auditd[2143]: Audit daemon rotating log files
Jun 19 08:10:01 weisshorn01 CROND[1286]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:10:01 weisshorn01 CROND[1287]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 08:10:01 weisshorn10 CROND[122385]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:10:01 weisshorn09 CROND[122070]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:10:01 weisshorn12 CROND[122240]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:10:01 weisshorn04 sshd[121829]: Accepted publickey for root from 148.187.12.231 port 44287 ssh2
Jun 19 08:10:01 weisshorn04 sshd[121829]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 08:10:01 weisshorn14 sshd[50951]: Accepted publickey for root from 148.187.12.231 port 49655 ssh2
Jun 19 08:10:01 weisshorn14 sshd[50951]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 08:10:01 weisshorn09 sshd[122072]: Accepted publickey for root from 148.187.12.231 port 57985 ssh2
Jun 19 08:10:01 weisshorn09 sshd[122072]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 08:10:01 weisshorn07 sshd[122494]: Accepted publickey for root from 148.187.12.231 port 48346 ssh2
Jun 19 08:10:01 weisshorn07 sshd[122494]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 08:10:01 weisshorn05 sshd[122354]: Accepted publickey for root from 148.187.12.231 port 54739 ssh2
Jun 19 08:10:01 weisshorn11 CROND[125349]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:10:01 weisshorn05 sshd[122354]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 08:10:01 weisshorn14 sshd[50951]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 08:10:01 weisshorn14 sshd[50951]: pam_unix(sshd:session): session closed for user root
Jun 19 08:10:01 weisshorn08 sshd[121153]: Accepted publickey for root from 148.187.12.231 port 40347 ssh2
Jun 19 08:10:01 weisshorn03 sshd[123036]: Accepted publickey for root from 148.187.12.231 port 50543 ssh2
Jun 19 08:10:01 weisshorn03 sshd[123036]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 08:10:01 weisshorn11 sshd[125346]: Accepted publickey for root from 148.187.12.231 port 45222 ssh2
Jun 19 08:10:01 weisshorn13 CROND[125714]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:10:01 weisshorn08 sshd[121153]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 08:10:01 weisshorn10 sshd[122387]: Accepted publickey for root from 148.187.12.231 port 37010 ssh2
Jun 19 08:10:01 weisshorn10 sshd[122387]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 08:10:01 weisshorn11 sshd[125346]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 08:10:01 weisshorn07 CROND[122508]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:10:01 weisshorn04 sshd[121829]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 08:10:01 weisshorn04 sshd[121829]: pam_unix(sshd:session): session closed for user root
Jun 19 08:10:01 weisshorn06 sshd[123228]: Accepted publickey for root from 148.187.12.231 port 44800 ssh2
Jun 19 08:10:01 weisshorn09 sshd[122072]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 08:10:01 weisshorn09 sshd[122072]: pam_unix(sshd:session): session closed for user root
Jun 19 08:10:01 weisshorn06 sshd[123228]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 08:10:01 weisshorn12 sshd[122242]: Accepted publickey for root from 148.187.12.231 port 46440 ssh2
Jun 19 08:10:01 weisshorn14 CROND[50974]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:10:01 weisshorn05 CROND[122368]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:10:01 weisshorn12 sshd[122242]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 08:10:01 weisshorn06 CROND[123242]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:10:01 weisshorn07 sshd[122494]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 08:10:01 weisshorn07 sshd[122494]: pam_unix(sshd:session): session closed for user root
Jun 19 08:10:01 weisshorn02 CROND[10489]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:10:01 weisshorn05 sshd[122354]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 08:10:01 weisshorn05 sshd[122354]: pam_unix(sshd:session): session closed for user root
Jun 19 08:10:01 weisshorn06 sshd[123228]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 08:10:01 weisshorn06 sshd[123228]: pam_unix(sshd:session): session closed for user root
Jun 19 08:10:02 weisshorn03 sshd[123036]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 08:10:02 weisshorn03 sshd[123036]: pam_unix(sshd:session): session closed for user root
Jun 19 08:10:02 weisshorn13 sshd[125713]: Accepted publickey for root from 148.187.12.231 port 44709 ssh2
Jun 19 08:10:02 weisshorn10 sshd[122387]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 08:10:02 weisshorn10 sshd[122387]: pam_unix(sshd:session): session closed for user root
Jun 19 08:10:02 weisshorn13 sshd[125713]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 08:10:02 weisshorn08 sshd[121153]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 08:10:02 weisshorn08 sshd[121153]: pam_unix(sshd:session): session closed for user root
Jun 19 08:10:02 weisshorn11 sshd[125346]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 08:10:02 weisshorn11 sshd[125346]: pam_unix(sshd:session): session closed for user root
Jun 19 08:10:02 weisshorn12 sshd[122242]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 08:10:02 weisshorn12 sshd[122242]: pam_unix(sshd:session): session closed for user root
Jun 19 08:10:02 weisshorn13 sshd[125713]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 08:10:02 weisshorn13 sshd[125713]: pam_unix(sshd:session): session closed for user root
Jun 19 08:10:02 weisshorn01 postfix/pickup[1032]: 71E3292224F: uid=0 from=<root>
Jun 19 08:10:02 weisshorn01 postfix/cleanup[1317]: 71E3292224F: message-id=<20120619061002.71E3292224F@weisshorn01.admin.cscs.ch>
Jun 19 08:10:02 weisshorn01 postfix/qmgr[2710]: 71E3292224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 08:10:02 weisshorn01 postfix/local[1320]: 71E3292224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=0.76, delays=0.68/0.03/0/0.04, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 08:10:02 weisshorn01 postfix/qmgr[2710]: 71E3292224F: removed
Jun 19 08:10:06 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:10:21 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:10:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:10:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:11:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:11:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:11:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:11:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:12:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:12:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:12:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:12:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:13:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:13:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:13:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:13:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:14:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:14:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:14:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:14:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:15:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:15:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:15:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:15:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:16:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:16:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:16:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:16:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:17:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:17:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:17:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:17:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:18:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:18:22 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:18:37 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:18:52 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:19:07 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:19:22 weisshorn08 rhnsd[121201]: running program /usr/sbin/rhn_check
Jun 19 08:19:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:19:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:19:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:20:01 weisshorn02 CROND[11456]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:20:01 weisshorn13 CROND[125763]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:20:01 weisshorn08 CROND[121204]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:20:01 weisshorn04 CROND[121878]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:20:01 weisshorn03 CROND[123088]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:20:01 weisshorn01 CROND[1436]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:20:01 weisshorn10 CROND[122436]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:20:01 weisshorn09 CROND[122129]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:20:01 weisshorn12 CROND[122290]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:20:01 weisshorn11 CROND[125400]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:20:01 weisshorn07 CROND[122593]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:20:01 weisshorn14 CROND[51007]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:20:02 weisshorn05 CROND[122406]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:20:02 weisshorn06 CROND[123282]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:20:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:20:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:20:23 weisshorn09 ntpd[4454]: synchronized to 148.187.18.120, stratum 2
Jun 19 08:20:26 weisshorn01 rhnsd[1438]: running program /usr/sbin/rhn_check
Jun 19 08:20:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:20:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:21:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:21:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:21:31 weisshorn14 nslcd[3437]: [16f8c4] ldap_result() failed: Can't contact LDAP server
Jun 19 08:21:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:21:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:22:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:22:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:22:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:22:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:23:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:23:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:23:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:23:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:24:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:24:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:24:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:24:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:25:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:25:21 weisshorn03 nslcd[3944]: [7195e5] ldap_result() failed: Can't contact LDAP server
Jun 19 08:25:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:25:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:25:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:26:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:26:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:26:31 weisshorn10 nslcd[4294]: [e203ab] ldap_result() failed: Can't contact LDAP server
Jun 19 08:26:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:26:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:27:05 weisshorn02 rhnsd[12102]: running program /usr/sbin/rhn_check
Jun 19 08:27:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:27:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:27:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:27:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:28:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:28:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:28:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:28:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:29:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:29:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:29:38 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:29:53 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:30:01 weisshorn14 CROND[51073]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:30:01 weisshorn05 CROND[122436]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:30:01 weisshorn02 CROND[12369]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:30:01 weisshorn06 CROND[123342]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:30:01 weisshorn13 CROND[125834]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:30:01 weisshorn08 CROND[121228]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:30:01 weisshorn04 CROND[121908]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:30:01 weisshorn03 CROND[123119]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:30:01 weisshorn01 CROND[1509]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:30:01 weisshorn10 CROND[122467]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:30:01 weisshorn09 CROND[122154]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:30:01 weisshorn12 CROND[122358]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:30:01 weisshorn11 CROND[125465]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:30:02 weisshorn07 CROND[122613]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:30:08 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:30:23 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:30:26 weisshorn05 rhnsd[122439]: running program /usr/sbin/rhn_check
Jun 19 08:30:31 weisshorn02 nslcd[2099]: [d71a2b] ldap_result() failed: Can't contact LDAP server
Jun 19 08:30:31 weisshorn01 kernel: Lustre: 5242:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004695:0xb5e5:0x0]] [valid 68719476736]
Jun 19 08:30:31 weisshorn14 nslcd[3437]: [df2233] ldap_result() failed: Can't contact LDAP server
Jun 19 08:30:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:30:52 weisshorn02 nslcd[2099]: [fc7321] ldap_result() failed: Can't contact LDAP server
Jun 19 08:30:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:31:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:31:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:31:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:31:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:32:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:32:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:32:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:32:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:33:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:33:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:33:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:33:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:34:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:34:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:34:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:34:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:35:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:35:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:35:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:35:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:36:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:36:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:36:36 weisshorn01 kernel: Lustre: 5147:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046ee:0x1160:0x0]] [valid 68719476736]
Jun 19 08:36:36 weisshorn01 kernel: Lustre: 5147:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 3 previous similar messages
Jun 19 08:36:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:36:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:37:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:37:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:37:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:37:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:38:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:38:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:38:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:38:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:39:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:39:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:39:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:39:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:40:01 weisshorn11 CROND[125495]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:40:01 weisshorn14 CROND[51098]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:40:01 weisshorn07 CROND[122638]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:40:01 weisshorn02 CROND[13297]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:40:01 weisshorn06 CROND[123372]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:40:01 weisshorn13 CROND[125859]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:40:01 weisshorn05 CROND[122499]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:40:01 weisshorn08 CROND[121295]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:40:01 weisshorn04 CROND[121970]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:40:01 weisshorn01 CROND[1564]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:40:01 weisshorn03 CROND[123178]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:40:01 weisshorn10 CROND[122529]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:40:01 weisshorn09 CROND[122213]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:40:01 weisshorn12 CROND[122388]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:40:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:40:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:40:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:40:46 weisshorn10 nslcd[4294]: [86d60e] ldap_result() failed: Can't contact LDAP server
Jun 19 08:40:46 weisshorn05 nslcd[4378]: [86d60e] ldap_result() failed: Can't contact LDAP server
Jun 19 08:40:46 weisshorn12 nslcd[3954]: [e203ab] ldap_result() failed: Can't contact LDAP server
Jun 19 08:40:46 weisshorn04 nslcd[3885]: [7195e5] ldap_result() failed: Can't contact LDAP server
Jun 19 08:40:46 weisshorn09 nslcd[3954]: [7195e5] ldap_result() failed: Can't contact LDAP server
Jun 19 08:40:46 weisshorn11 nslcd[3954]: [90b7c5] ldap_result() failed: Can't contact LDAP server
Jun 19 08:40:46 weisshorn03 nslcd[3944]: [46a5f2] ldap_result() failed: Can't contact LDAP server
Jun 19 08:40:46 weisshorn13 nslcd[4224]: [e203ab] ldap_result() failed: Can't contact LDAP server
Jun 19 08:40:46 weisshorn08 nslcd[3946]: [e203ab] ldap_result() failed: Can't contact LDAP server
Jun 19 08:40:46 weisshorn07 nslcd[4297]: [e203ab] ldap_result() failed: Can't contact LDAP server
Jun 19 08:40:46 weisshorn06 nslcd[3953]: [c93698] ldap_result() failed: Can't contact LDAP server
Jun 19 08:40:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:41:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:41:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:41:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:41:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:42:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:42:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:42:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:42:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:43:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:43:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:43:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:43:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:44:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:44:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:44:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:44:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:45:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:45:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:45:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:45:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:46:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:46:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:46:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:46:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:47:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:47:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:47:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:47:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:48:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:48:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:48:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:48:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:49:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:49:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:49:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:49:44 weisshorn01 kernel: Lustre: 5239:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004631:0x150eb:0x0]] [valid 68719476736]
Jun 19 08:49:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:50:01 weisshorn12 CROND[122412]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:50:01 weisshorn01 kernel: Lustre: 5426:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x1614:0x0]] [valid 68719476736]
Jun 19 08:50:01 weisshorn14 CROND[51133]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:50:01 weisshorn11 CROND[125522]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:50:01 weisshorn02 CROND[14255]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:50:01 weisshorn06 CROND[123404]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:50:01 weisshorn07 CROND[122715]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:50:01 weisshorn13 CROND[125888]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:50:01 weisshorn05 CROND[122529]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:50:01 weisshorn08 CROND[121325]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:50:01 weisshorn01 CROND[1695]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:50:01 weisshorn04 CROND[122000]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:50:01 weisshorn03 CROND[123211]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:50:01 weisshorn09 CROND[122251]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:50:01 weisshorn10 CROND[122559]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 08:50:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:50:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:50:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:50:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:51:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:51:13 weisshorn01 kernel: Lustre: 4719:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004631:0x150f0:0x0]] [valid 68719476736]
Jun 19 08:51:16 weisshorn01 kernel: Lustre: 4695:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x1619:0x0]] [valid 68719476736]
Jun 19 08:51:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:51:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:51:44 weisshorn01 kernel: Lustre: 4698:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x161e:0x0]] [valid 68719476736]
Jun 19 08:51:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:52:01 weisshorn01 kernel: Lustre: 5320:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x1626:0x0]] [valid 68719476736]
Jun 19 08:52:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:52:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:52:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:52:44 weisshorn01 kernel: Lustre: 5461:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x1631:0x0]] [valid 68719476736]
Jun 19 08:52:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:53:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:53:16 weisshorn01 kernel: Lustre: 5291:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x163c:0x0]] [valid 68719476736]
Jun 19 08:53:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:53:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:53:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:54:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:54:13 weisshorn01 kernel: Lustre: 5314:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x1641:0x0]] [valid 68719476736]
Jun 19 08:54:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:54:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:54:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:55:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:55:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:55:32 weisshorn03 kernel: Lustre: scratch-OST0043: haven't heard from client e98629d7-d8d6-058f-2cb0-835e3d81c535 (at 1196@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881027f3d800, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn03 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 08:55:32 weisshorn13 kernel: Lustre: scratch-OST0005: haven't heard from client 8dc459e1-7a2a-f1d1-5f90-1a43bf55749c (at 1197@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a7354fc00, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn13 kernel: Lustre: Skipped 11 previous similar messages
Jun 19 08:55:32 weisshorn12 kernel: Lustre: scratch-OST001c: haven't heard from client 95c45cd3-2a10-e4d6-4e7a-919797ae466a (at 1235@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806fc9a5c00, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn12 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 08:55:32 weisshorn11 kernel: Lustre: scratch-OST0021: haven't heard from client 8dc459e1-7a2a-f1d1-5f90-1a43bf55749c (at 1197@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080374a400, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn07 kernel: Lustre: scratch-OST002b: haven't heard from client 8dc459e1-7a2a-f1d1-5f90-1a43bf55749c (at 1197@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810147f3400, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn11 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 08:55:32 weisshorn07 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 08:55:32 weisshorn08 kernel: Lustre: scratch-OST003b: haven't heard from client e98629d7-d8d6-058f-2cb0-835e3d81c535 (at 1196@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e16dd1c00, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn08 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 08:55:32 weisshorn04 kernel: Lustre: scratch-OST0045: haven't heard from client 95c45cd3-2a10-e4d6-4e7a-919797ae466a (at 1235@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880759e65c00, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn04 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 08:55:32 weisshorn05 kernel: Lustre: scratch-OST0046: haven't heard from client e98629d7-d8d6-058f-2cb0-835e3d81c535 (at 1196@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880147d22000, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn05 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 08:55:32 weisshorn10 kernel: Lustre: scratch-OST0001: haven't heard from client 8dc459e1-7a2a-f1d1-5f90-1a43bf55749c (at 1197@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881014cec400, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn10 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 08:55:32 weisshorn05 kernel: Lustre: scratch-OST003e: haven't heard from client e98629d7-d8d6-058f-2cb0-835e3d81c535 (at 1196@gni) in 226 seconds. I think it's dead, and I am evicting it. exp ffff8809734e3400, cur 1340088932 expire 1340088782 last 1340088706
Jun 19 08:55:32 weisshorn05 kernel: Lustre: Skipped 9 previous similar messages
Jun 19 08:55:32 weisshorn13 kernel: Lustre: scratch-OST0003: haven't heard from client 95c45cd3-2a10-e4d6-4e7a-919797ae466a (at 1235@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803ec68d800, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn13 kernel: Lustre: Skipped 10 previous similar messages
Jun 19 08:55:32 weisshorn08 kernel: Lustre: scratch-OST002f: haven't heard from client 61528081-1b92-1b81-5e28-281d2c476787 (at 1234@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88075fa9ec00, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn08 kernel: Lustre: Skipped 11 previous similar messages
Jun 19 08:55:32 weisshorn06 kernel: Lustre: scratch-OST0007: haven't heard from client 95c45cd3-2a10-e4d6-4e7a-919797ae466a (at 1235@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e4e61400, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn06 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 08:55:32 weisshorn09 kernel: Lustre: scratch-OST0000: haven't heard from client 95c45cd3-2a10-e4d6-4e7a-919797ae466a (at 1235@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881016116c00, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn09 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 08:55:32 weisshorn07 kernel: Lustre: scratch-OST0042: haven't heard from client 95c45cd3-2a10-e4d6-4e7a-919797ae466a (at 1235@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e0b071000, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn07 kernel: Lustre: Skipped 11 previous similar messages
Jun 19 08:55:32 weisshorn06 kernel: Lustre: scratch-OST001d: haven't heard from client 61528081-1b92-1b81-5e28-281d2c476787 (at 1234@gni) in 225 seconds. I think it's dead, and I am evicting it. exp ffff880805964800, cur 1340088932 expire 1340088782 last 1340088707
Jun 19 08:55:32 weisshorn06 kernel: Lustre: Skipped 10 previous similar messages
Jun 19 08:55:32 weisshorn12 kernel: Lustre: scratch-OST0028: haven't heard from client 8dc459e1-7a2a-f1d1-5f90-1a43bf55749c (at 1197@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807da531800, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn12 kernel: Lustre: Skipped 15 previous similar messages
Jun 19 08:55:32 weisshorn04 kernel: Lustre: scratch-OST0019: haven't heard from client 61528081-1b92-1b81-5e28-281d2c476787 (at 1234@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881019fa8400, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn04 kernel: Lustre: Skipped 7 previous similar messages
Jun 19 08:55:32 weisshorn03 kernel: Lustre: scratch-OST002c: haven't heard from client 95c45cd3-2a10-e4d6-4e7a-919797ae466a (at 1235@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881011768c00, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn03 kernel: Lustre: Skipped 11 previous similar messages
Jun 19 08:55:32 weisshorn01 kernel: Lustre: MGS: haven't heard from client efb94377-ed81-eefa-9983-ee98bd243910 (at 1235@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88030e912800, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn09 kernel: Lustre: scratch-OST0004: haven't heard from client 95c45cd3-2a10-e4d6-4e7a-919797ae466a (at 1235@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aa0a90000, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn09 kernel: Lustre: Skipped 15 previous similar messages
Jun 19 08:55:32 weisshorn06 kernel: Lustre: scratch-OST0012: haven't heard from client 61528081-1b92-1b81-5e28-281d2c476787 (at 1234@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080ac4b000, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn06 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 08:55:32 weisshorn07 kernel: Lustre: scratch-OST0013: haven't heard from client 95c45cd3-2a10-e4d6-4e7a-919797ae466a (at 1235@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88037ab3f800, cur 1340088932 expire 1340088782 last 1340088705
Jun 19 08:55:32 weisshorn07 kernel: Lustre: Skipped 7 previous similar messages
Jun 19 08:55:33 weisshorn01 kernel: Lustre: scratch-MDT0000: haven't heard from client e98629d7-d8d6-058f-2cb0-835e3d81c535 (at 1196@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b02d19400, cur 1340088933 expire 1340088783 last 1340088706
Jun 19 08:55:33 weisshorn01 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 08:55:33 weisshorn11 kernel: Lustre: scratch-OST0015: haven't heard from client 8dc459e1-7a2a-f1d1-5f90-1a43bf55749c (at 1197@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801fe0f4400, cur 1340088933 expire 1340088783 last 1340088706
Jun 19 08:55:33 weisshorn11 kernel: Lustre: Skipped 19 previous similar messages
Jun 19 08:55:33 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client e98629d7-d8d6-058f-2cb0-835e3d81c535 (at 1196@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e5799d400, cur 1340088933 expire 1340088783 last 1340088706
Jun 19 08:55:33 weisshorn08 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 08:55:33 weisshorn13 kernel: Lustre: scratch-OST0030: haven't heard from client e98629d7-d8d6-058f-2cb0-835e3d81c535 (at 1196@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880590da9000, cur 1340088933 expire 1340088783 last 1340088706
Jun 19 08:55:33 weisshorn13 kernel: Lustre: Skipped 16 previous similar messages
Jun 19 08:55:33 weisshorn05 kernel: Lustre: scratch-OST001b: haven't heard from client 8dc459e1-7a2a-f1d1-5f90-1a43bf55749c (at 1197@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802c2ea8c00, cur 1340088933 expire 1340088783 last 1340088706
Jun 19 08:55:33 weisshorn05 kernel: Lustre: Skipped 9 previous similar messages
Jun 19 08:55:33 weisshorn12 kernel: Lustre: scratch-OST000f: haven't heard from client 8dc459e1-7a2a-f1d1-5f90-1a43bf55749c (at 1197@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802cdd25800, cur 1340088933 expire 1340088783 last 1340088706
Jun 19 08:55:33 weisshorn12 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 08:55:33 weisshorn10 kernel: Lustre: scratch-OST000a: haven't heard from client 8dc459e1-7a2a-f1d1-5f90-1a43bf55749c (at 1197@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880807e00400, cur 1340088933 expire 1340088783 last 1340088706
Jun 19 08:55:33 weisshorn10 kernel: Lustre: Skipped 19 previous similar messages
Jun 19 08:55:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:55:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:56:04 weisshorn01 kernel: LustreError: 4544:0:(mdt_handler.c:2822:mdt_recovery()) operation 400 on unconnected MDS from 12345-1197@gni
Jun 19 08:56:04 weisshorn01 kernel: LustreError: 4544:0:(mdt_handler.c:2822:mdt_recovery()) operation 400 on unconnected MDS from 12345-1197@gni
Jun 19 08:56:04 weisshorn01 kernel: LustreError: 4544:0:(mdt_handler.c:2822:mdt_recovery()) Skipped 2 previous similar messages
Jun 19 08:56:08 weisshorn01 kernel: Lustre: 5256:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x1653:0x0]] [valid 68719476736]
Jun 19 08:56:08 weisshorn01 kernel: Lustre: 5256:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 2 previous similar messages
Jun 19 08:56:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:56:17 weisshorn01 kernel: LustreError: 4719:0:(mdt_handler.c:2822:mdt_recovery()) operation 400 on unconnected MDS from 12345-1234@gni
Jun 19 08:56:17 weisshorn01 kernel: LustreError: 4719:0:(mdt_handler.c:2822:mdt_recovery()) Skipped 8 previous similar messages
Jun 19 08:56:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:56:32 weisshorn01 kernel: LustreError: 5325:0:(mdt_handler.c:2822:mdt_recovery()) operation 400 on unconnected MDS from 12345-1235@gni
Jun 19 08:56:32 weisshorn01 kernel: LustreError: 5325:0:(mdt_handler.c:2822:mdt_recovery()) Skipped 9 previous similar messages
Jun 19 08:56:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:56:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:56:59 weisshorn09 rhnsd[122266]: running program /usr/sbin/rhn_check
Jun 19 08:57:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:57:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:57:29 weisshorn01 kernel: LustreError: 5314:0:(mdt_handler.c:2822:mdt_recovery()) operation 400 on unconnected MDS from 12345-1196@gni
Jun 19 08:57:29 weisshorn01 kernel: LustreError: 5314:0:(mdt_handler.c:2822:mdt_recovery()) Skipped 11 previous similar messages
Jun 19 08:57:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:57:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:58:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:58:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:58:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:58:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:59:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:59:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:59:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 08:59:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:00:01 weisshorn10 CROND[122584]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:00:01 weisshorn12 CROND[122472]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:00:01 weisshorn14 CROND[51187]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:00:01 weisshorn11 CROND[125552]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:00:01 weisshorn06 CROND[123435]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:00:01 weisshorn07 CROND[122741]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:00:01 weisshorn02 CROND[15172]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:00:01 weisshorn13 CROND[125948]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:00:01 weisshorn05 CROND[122554]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:00:01 weisshorn08 CROND[121353]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:00:01 weisshorn01 CROND[1752]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:00:01 weisshorn04 CROND[122025]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:00:01 weisshorn03 CROND[123236]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:00:01 weisshorn09 CROND[122273]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:00:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:00:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:00:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:00:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:01:01 weisshorn10 CROND[122589]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 09:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1225 starting 0anacron
Jun 19 09:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1225 finished 0anacron
Jun 19 09:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1225 starting mcelog.cron
Jun 19 09:01:01 weisshorn12 CROND[122477]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 09:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1224 starting 0anacron
Jun 19 09:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1224 finished 0anacron
Jun 19 09:01:01 weisshorn14 CROND[51192]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 09:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1224 starting mcelog.cron
Jun 19 09:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5119 starting 0anacron
Jun 19 09:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5120 finished 0anacron
Jun 19 09:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5119 starting mcelog.cron
Jun 19 09:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1226 finished mcelog.cron
Jun 19 09:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1224 finished mcelog.cron
Jun 19 09:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5120 finished mcelog.cron
Jun 19 09:01:01 weisshorn11 CROND[125557]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 09:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1255 starting 0anacron
Jun 19 09:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1255 finished 0anacron
Jun 19 09:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1255 starting mcelog.cron
Jun 19 09:01:01 weisshorn06 CROND[123440]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 09:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1234 starting 0anacron
Jun 19 09:01:01 weisshorn07 CROND[122746]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 09:01:01 weisshorn02 CROND[15266]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 09:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1234 finished 0anacron
Jun 19 09:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1234 starting mcelog.cron
Jun 19 09:01:01 weisshorn02 run-parts(/etc/cron.hourly)[1526 starting 0anacron
Jun 19 09:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1227 starting 0anacron
Jun 19 09:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1255 finished mcelog.cron
Jun 19 09:01:01 weisshorn02 run-parts(/etc/cron.hourly)[1527 finished 0anacron
Jun 19 09:01:01 weisshorn02 run-parts(/etc/cron.hourly)[1526 starting mcelog.cron
Jun 19 09:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1234 finished mcelog.cron
Jun 19 09:01:01 weisshorn02 run-parts(/etc/cron.hourly)[1528 finished mcelog.cron
Jun 19 09:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1227 finished 0anacron
Jun 19 09:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1227 starting mcelog.cron
Jun 19 09:01:01 weisshorn13 CROND[125953]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 09:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1227 finished mcelog.cron
Jun 19 09:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1259 starting 0anacron
Jun 19 09:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1259 finished 0anacron
Jun 19 09:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1259 starting mcelog.cron
Jun 19 09:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1259 finished mcelog.cron
Jun 19 09:01:01 weisshorn05 CROND[122559]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 09:01:01 weisshorn08 CROND[121358]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 09:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1225 starting 0anacron
Jun 19 09:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1213 starting 0anacron
Jun 19 09:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1213 finished 0anacron
Jun 19 09:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1213 starting mcelog.cron
Jun 19 09:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1225 finished 0anacron
Jun 19 09:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1225 starting mcelog.cron
Jun 19 09:01:01 weisshorn01 CROND[1763]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 09:01:01 weisshorn01 run-parts(/etc/cron.hourly)[1763 starting 0anacron
Jun 19 09:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1213 finished mcelog.cron
Jun 19 09:01:01 weisshorn01 run-parts(/etc/cron.hourly)[1773 finished 0anacron
Jun 19 09:01:01 weisshorn01 run-parts(/etc/cron.hourly)[1763 starting mcelog.cron
Jun 19 09:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1225 finished mcelog.cron
Jun 19 09:01:01 weisshorn03 CROND[123241]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 09:01:01 weisshorn04 CROND[122030]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 09:01:01 weisshorn01 run-parts(/etc/cron.hourly)[1780 finished mcelog.cron
Jun 19 09:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1232 starting 0anacron
Jun 19 09:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1220 starting 0anacron
Jun 19 09:01:01 weisshorn09 CROND[122278]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 09:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1222 starting 0anacron
Jun 19 09:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1232 finished 0anacron
Jun 19 09:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1232 starting mcelog.cron
Jun 19 09:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1222 finished 0anacron
Jun 19 09:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1222 starting mcelog.cron
Jun 19 09:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1220 finished 0anacron
Jun 19 09:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1220 starting mcelog.cron
Jun 19 09:01:02 weisshorn09 run-parts(/etc/cron.hourly)[1222 finished mcelog.cron
Jun 19 09:01:02 weisshorn04 run-parts(/etc/cron.hourly)[1220 finished mcelog.cron
Jun 19 09:01:02 weisshorn03 run-parts(/etc/cron.hourly)[1232 finished mcelog.cron
Jun 19 09:01:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:01:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:01:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:01:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:02:06 weisshorn01 kernel: Lustre: 5464:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x1662:0x0]] [valid 68719476736]
Jun 19 09:02:06 weisshorn01 kernel: Lustre: 5464:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 1 previous similar message
Jun 19 09:02:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:02:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:02:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:02:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:03:03 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 09:03:03 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 7 active RPCs
Jun 19 09:03:03 weisshorn11 kernel: Lustre: Skipped 1 previous similar message
Jun 19 09:03:03 weisshorn11 kernel: LustreError: 7658:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc -107  req@ffff8810190bdc00 x1403838203911626/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340089426 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 09:03:03 weisshorn11 kernel: Lustre: scratch-OST0008: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107
Jun 19 09:03:03 weisshorn11 kernel: LustreError: 7658:0:(ldlm_lib.c:2725:target_bulk_io()) Skipped 2 previous similar messages
Jun 19 09:03:03 weisshorn11 kernel: Lustre: scratch-OST0008: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107
Jun 19 09:03:03 weisshorn11 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 09:03:03 weisshorn11 kernel: LustreError: 6137:0:(ldlm_lib.c:2706:target_bulk_io()) @@@ Reconnect on bulk GET  req@ffff881018c7d000 x1403838203911480/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340089426 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 09:03:03 weisshorn11 kernel: LustreError: 6137:0:(ldlm_lib.c:2706:target_bulk_io()) Skipped 3 previous similar messages
Jun 19 09:03:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:03:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:03:27 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 09:03:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:03:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:04:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:04:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:04:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:04:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:04:56 weisshorn11 kernel: Lustre: scratch-OST002d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 09:04:56 weisshorn11 kernel: Lustre: scratch-OST002d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 8 active RPCs
Jun 19 09:04:56 weisshorn11 kernel: LustreError: 7658:0:(ldlm_lib.c:2706:target_bulk_io()) @@@ Reconnect on bulk GET  req@ffff88080b296850 x1403838203920820/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340089539 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 09:04:56 weisshorn11 kernel: Lustre: scratch-OST002d: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -110
Jun 19 09:04:56 weisshorn11 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 09:05:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:05:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:05:30 weisshorn09 nslcd[3954]: [46a5f2] ldap_result() failed: Can't contact LDAP server
Jun 19 09:05:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:05:39 weisshorn11 kernel: Lustre: scratch-OST0044: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 09:05:39 weisshorn11 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 09:05:39 weisshorn11 kernel: Lustre: scratch-OST0044: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 8 active RPCs
Jun 19 09:05:39 weisshorn11 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 09:05:39 weisshorn11 kernel: LustreError: 7916:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc -107  req@ffff88080afda850 x1403838203922150/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340089582 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 09:05:39 weisshorn11 kernel: Lustre: scratch-OST0044: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107
Jun 19 09:05:39 weisshorn11 kernel: Lustre: Skipped 17 previous similar messages
Jun 19 09:05:39 weisshorn11 kernel: LustreError: 7916:0:(ldlm_lib.c:2725:target_bulk_io()) Skipped 4 previous similar messages
Jun 19 09:05:53 weisshorn08 nslcd[3946]: [86d60e] ldap_result() failed: Can't contact LDAP server
Jun 19 09:05:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:05:54 weisshorn06 nslcd[3953]: [781401] ldap_result() failed: Can't contact LDAP server
Jun 19 09:05:56 weisshorn04 nslcd[3885]: [46a5f2] ldap_result() failed: Can't contact LDAP server
Jun 19 09:05:57 weisshorn07 nslcd[4297]: [86d60e] ldap_result() failed: Can't contact LDAP server
Jun 19 09:06:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:06:17 weisshorn11 nslcd[3954]: [03a216] ldap_result() failed: Can't contact LDAP server
Jun 19 09:06:18 weisshorn12 nslcd[3954]: [86d60e] ldap_result() failed: Can't contact LDAP server
Jun 19 09:06:19 weisshorn13 nslcd[4224]: [86d60e] ldap_result() failed: Can't contact LDAP server
Jun 19 09:06:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:06:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:06:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:06:56 weisshorn05 nslcd[4378]: [3da4d5] ldap_result() failed: Can't contact LDAP server
Jun 19 09:07:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:07:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:07:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:07:50 weisshorn01 kernel: Lustre: 5325:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x166c:0x0]] [valid 68719476736]
Jun 19 09:07:50 weisshorn01 kernel: Lustre: 5325:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 1 previous similar message
Jun 19 09:07:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:08:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:08:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:08:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:08:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:09:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:09:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:09:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:09:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:10:01 weisshorn09 CROND[122357]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:10:01 weisshorn04 CROND[122112]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:10:01 weisshorn03 CROND[123319]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:10:01 weisshorn10 CROND[122670]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:10:01 weisshorn14 CROND[51242]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:10:01 weisshorn12 CROND[122530]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:10:01 weisshorn11 CROND[125638]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:10:01 weisshorn06 CROND[123514]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:10:01 weisshorn02 CROND[16112]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:10:01 weisshorn07 CROND[122785]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:10:01 weisshorn13 CROND[126004]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:10:01 weisshorn08 CROND[121437]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:10:01 weisshorn05 CROND[122641]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:10:01 weisshorn01 CROND[1840]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:10:01 weisshorn01 CROND[1842]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 09:10:02 weisshorn09 sshd[122359]: Accepted publickey for root from 148.187.12.231 port 58000 ssh2
Jun 19 09:10:02 weisshorn14 sshd[51244]: Accepted publickey for root from 148.187.12.231 port 49678 ssh2
Jun 19 09:10:02 weisshorn12 sshd[122532]: Accepted publickey for root from 148.187.12.231 port 46465 ssh2
Jun 19 09:10:02 weisshorn09 sshd[122359]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 09:10:02 weisshorn14 sshd[51244]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 09:10:02 weisshorn12 sshd[122532]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 09:10:02 weisshorn13 sshd[126006]: Accepted publickey for root from 148.187.12.231 port 44727 ssh2
Jun 19 09:10:02 weisshorn13 sshd[126006]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 09:10:02 weisshorn04 sshd[122114]: Accepted publickey for root from 148.187.12.231 port 44309 ssh2
Jun 19 09:10:02 weisshorn04 sshd[122114]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 09:10:02 weisshorn14 sshd[51244]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 09:10:02 weisshorn14 sshd[51244]: pam_unix(sshd:session): session closed for user root
Jun 19 09:10:02 weisshorn10 sshd[122672]: Accepted publickey for root from 148.187.12.231 port 37031 ssh2
Jun 19 09:10:02 weisshorn03 sshd[123321]: Accepted publickey for root from 148.187.12.231 port 50556 ssh2
Jun 19 09:10:02 weisshorn08 sshd[121439]: Accepted publickey for root from 148.187.12.231 port 40361 ssh2
Jun 19 09:10:02 weisshorn10 sshd[122672]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 09:10:02 weisshorn11 sshd[125640]: Accepted publickey for root from 148.187.12.231 port 45248 ssh2
Jun 19 09:10:02 weisshorn11 sshd[125640]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 09:10:02 weisshorn09 sshd[122359]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 09:10:02 weisshorn09 sshd[122359]: pam_unix(sshd:session): session closed for user root
Jun 19 09:10:02 weisshorn07 sshd[122787]: Accepted publickey for root from 148.187.12.231 port 48367 ssh2
Jun 19 09:10:02 weisshorn03 sshd[123321]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 09:10:02 weisshorn08 sshd[121439]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 09:10:02 weisshorn05 sshd[122643]: Accepted publickey for root from 148.187.12.231 port 54757 ssh2
Jun 19 09:10:02 weisshorn07 sshd[122787]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 09:10:02 weisshorn05 sshd[122643]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 09:10:02 weisshorn12 sshd[122532]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 09:10:02 weisshorn12 sshd[122532]: pam_unix(sshd:session): session closed for user root
Jun 19 09:10:02 weisshorn04 sshd[122114]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 09:10:02 weisshorn04 sshd[122114]: pam_unix(sshd:session): session closed for user root
Jun 19 09:10:02 weisshorn13 sshd[126006]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 09:10:02 weisshorn13 sshd[126006]: pam_unix(sshd:session): session closed for user root
Jun 19 09:10:02 weisshorn11 sshd[125640]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 09:10:02 weisshorn11 sshd[125640]: pam_unix(sshd:session): session closed for user root
Jun 19 09:10:02 weisshorn03 sshd[123321]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 09:10:02 weisshorn03 sshd[123321]: pam_unix(sshd:session): session closed for user root
Jun 19 09:10:02 weisshorn10 sshd[122672]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 09:10:02 weisshorn10 sshd[122672]: pam_unix(sshd:session): session closed for user root
Jun 19 09:10:02 weisshorn08 sshd[121439]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 09:10:02 weisshorn08 sshd[121439]: pam_unix(sshd:session): session closed for user root
Jun 19 09:10:02 weisshorn07 sshd[122787]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 09:10:02 weisshorn07 sshd[122787]: pam_unix(sshd:session): session closed for user root
Jun 19 09:10:02 weisshorn05 sshd[122643]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 09:10:02 weisshorn05 sshd[122643]: pam_unix(sshd:session): session closed for user root
Jun 19 09:10:03 weisshorn06 sshd[123516]: Accepted publickey for root from 148.187.12.231 port 44825 ssh2
Jun 19 09:10:03 weisshorn06 sshd[123516]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 09:10:03 weisshorn06 sshd[123516]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 09:10:03 weisshorn06 sshd[123516]: pam_unix(sshd:session): session closed for user root
Jun 19 09:10:03 weisshorn01 postfix/pickup[1032]: 72DE492224F: uid=0 from=<root>
Jun 19 09:10:03 weisshorn01 postfix/cleanup[1872]: 72DE492224F: message-id=<20120619071003.72DE492224F@weisshorn01.admin.cscs.ch>
Jun 19 09:10:03 weisshorn01 postfix/qmgr[2710]: 72DE492224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 09:10:03 weisshorn01 postfix/local[1874]: 72DE492224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=1.5, delays=1.4/0.04/0/0.05, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 09:10:03 weisshorn01 postfix/qmgr[2710]: 72DE492224F: removed
Jun 19 09:10:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:10:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:10:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:10:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:11:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:11:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:11:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:11:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:12:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:12:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:12:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:12:53 weisshorn11 kernel: Lustre: 6010:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340089966/real 1340089966]  req@ffff880803848800 x1403849891133751/t0(0) o106->scratch-OST0015@148.187.6.8@o2ib2:15/16 lens 296/232 e 0 to 1 dl 1340089973 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jun 19 09:12:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:13:00 weisshorn11 kernel: Lustre: 6010:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340089973/real 1340089973]  req@ffff880803848800 x1403849891133751/t0(0) o106->scratch-OST0015@148.187.6.8@o2ib2:15/16 lens 296/232 e 0 to 1 dl 1340089980 ref 2 fl Rpc:X/2/ffffffff rc 0/-1
Jun 19 09:13:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:13:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:13:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:13:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:14:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:14:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:14:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:14:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:15:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:15:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:15:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:15:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:16:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:16:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:16:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:16:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:17:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:17:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:17:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:17:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:18:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:18:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:18:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:18:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:19:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:19:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:19:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:19:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:20:01 weisshorn05 CROND[122692]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:20:01 weisshorn09 CROND[122416]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:20:01 weisshorn04 CROND[122163]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:20:01 weisshorn03 CROND[123373]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:20:01 weisshorn14 CROND[51297]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:20:01 weisshorn10 CROND[122721]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:20:01 weisshorn12 CROND[122575]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:20:01 weisshorn01 CROND[1979]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:20:01 weisshorn06 CROND[123567]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:20:01 weisshorn02 CROND[17073]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:20:01 weisshorn07 CROND[122878]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:20:01 weisshorn11 CROND[125687]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:20:01 weisshorn13 CROND[126054]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:20:02 weisshorn08 CROND[121488]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:20:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:20:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:20:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:20:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:21:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:21:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:21:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:21:46 weisshorn14 nslcd[3437]: [b49daf] ldap_result() failed: Can't contact LDAP server
Jun 19 09:21:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:22:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:22:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:22:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:22:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:23:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:23:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:23:39 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:23:54 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:24:09 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:24:24 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:24:40 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:24:55 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:25:10 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:25:25 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:25:36 weisshorn03 nslcd[3944]: [2dfed6] ldap_result() failed: Can't contact LDAP server
Jun 19 09:25:40 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:25:55 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:26:10 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:26:25 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:26:40 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:26:52 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 09:26:52 weisshorn11 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 09:26:52 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 9 active RPCs
Jun 19 09:26:55 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:27:10 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:27:16 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 09:27:25 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:27:40 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:27:55 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:28:10 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:28:25 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:28:40 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:28:55 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:29:10 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:29:25 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:29:40 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:29:55 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:30:01 weisshorn08 CROND[121555]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:30:01 weisshorn09 CROND[122475]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:30:01 weisshorn04 CROND[122200]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:30:01 weisshorn03 CROND[123438]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:30:01 weisshorn05 CROND[122729]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:30:01 weisshorn14 CROND[51363]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:30:01 weisshorn12 CROND[122642]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:30:01 weisshorn10 CROND[122758]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:30:01 weisshorn01 CROND[2054]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:30:01 weisshorn06 CROND[123632]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:30:01 weisshorn02 CROND[17995]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:30:01 weisshorn07 CROND[122903]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:30:01 weisshorn11 CROND[125752]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:30:01 weisshorn13 CROND[126124]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:30:10 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:30:25 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:30:40 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:30:46 weisshorn02 nslcd[2099]: [21971b] ldap_result() failed: Can't contact LDAP server
Jun 19 09:30:46 weisshorn14 nslcd[3437]: [9f82cd] ldap_result() failed: Can't contact LDAP server
Jun 19 09:30:55 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:31:01 weisshorn02 nslcd[2099]: [404301] ldap_result() failed: Can't contact LDAP server
Jun 19 09:31:10 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:31:25 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:31:40 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:31:50 weisshorn06 kernel: Lustre: scratch-OST002a: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 09:31:50 weisshorn06 kernel: Lustre: scratch-OST001d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 7 active RPCs
Jun 19 09:31:50 weisshorn06 kernel: LustreError: 6363:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc -107  req@ffff8810110b8400 x1403838204198115/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340091120 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 09:31:50 weisshorn06 kernel: Lustre: scratch-OST001d: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107
Jun 19 09:31:50 weisshorn06 kernel: LustreError: 6363:0:(ldlm_lib.c:2725:target_bulk_io()) Skipped 5 previous similar messages
Jun 19 09:31:55 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:32:00 weisshorn05 kernel: LustreError: 137-5: scratch-OST0007: Not available for connect from 148.187.6.8@o2ib2 (no target)
Jun 19 09:32:00 weisshorn05 kernel: LustreError: Skipped 1 previous similar message
Jun 19 09:32:10 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:32:25 weisshorn06 kernel: Lustre: scratch-OST0007: Client 77ed2fd6-7ad2-3010-3442-924b80b07058 (at 148.187.6.8@o2ib2) reconnecting
Jun 19 09:32:25 weisshorn06 kernel: Lustre: Skipped 11 previous similar messages
Jun 19 09:32:26 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:32:41 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:32:56 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:33:11 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:33:26 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:33:41 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:33:56 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:34:11 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:34:26 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:34:41 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:34:56 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:35:11 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:35:26 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:35:41 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:35:56 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:35:59 weisshorn06 kernel: Lustre: scratch-OST0041: Client 77ed2fd6-7ad2-3010-3442-924b80b07058 (at 148.187.6.8@o2ib2) reconnecting
Jun 19 09:35:59 weisshorn06 kernel: Lustre: Skipped 11 previous similar messages
Jun 19 09:35:59 weisshorn06 kernel: Lustre: scratch-OST001d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 9 active RPCs
Jun 19 09:35:59 weisshorn06 kernel: LustreError: 6270:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc -107  req@ffff881011d12000 x1403838204214710/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340091407 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 09:35:59 weisshorn06 kernel: Lustre: scratch-OST001d: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107
Jun 19 09:35:59 weisshorn06 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 09:35:59 weisshorn06 kernel: LustreError: 6270:0:(ldlm_lib.c:2725:target_bulk_io()) Skipped 6 previous similar messages
Jun 19 09:36:11 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:36:24 weisshorn06 kernel: Lustre: scratch-OST001d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 09:36:24 weisshorn06 kernel: Lustre: Skipped 9 previous similar messages
Jun 19 09:36:26 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:36:41 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:36:56 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:37:11 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:37:26 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:37:41 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:37:56 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:38:11 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:38:26 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:38:41 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:38:56 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:39:11 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:39:20 weisshorn01 kernel: Lustre: 5296:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046ee:0x1240:0x0]] [valid 68719476736]
Jun 19 09:39:20 weisshorn01 kernel: Lustre: 5296:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 5 previous similar messages
Jun 19 09:39:26 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:39:41 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:39:56 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:40:01 weisshorn13 CROND[126154]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:40:01 weisshorn08 CROND[121585]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:40:01 weisshorn09 CROND[122507]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:40:01 weisshorn04 CROND[122259]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:40:01 weisshorn03 CROND[123470]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:40:01 weisshorn05 CROND[122789]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:40:01 weisshorn14 CROND[51398]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:40:01 weisshorn12 CROND[122672]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:40:01 weisshorn10 CROND[122817]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:40:01 weisshorn01 CROND[2123]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:40:01 weisshorn06 CROND[123658]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:40:01 weisshorn02 CROND[18947]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:40:01 weisshorn07 CROND[122932]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:40:01 weisshorn11 CROND[125782]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:40:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:40:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:40:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:40:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:41:01 weisshorn10 nslcd[4294]: [3da4d5] ldap_result() failed: Can't contact LDAP server
Jun 19 09:41:01 weisshorn05 nslcd[4378]: [c93698] ldap_result() failed: Can't contact LDAP server
Jun 19 09:41:01 weisshorn12 nslcd[3954]: [3da4d5] ldap_result() failed: Can't contact LDAP server
Jun 19 09:41:01 weisshorn04 nslcd[3885]: [2dfed6] ldap_result() failed: Can't contact LDAP server
Jun 19 09:41:01 weisshorn09 nslcd[3954]: [2dfed6] ldap_result() failed: Can't contact LDAP server
Jun 19 09:41:01 weisshorn11 nslcd[3954]: [b8a08e] ldap_result() failed: Can't contact LDAP server
Jun 19 09:41:01 weisshorn03 nslcd[3944]: [6ed7f6] ldap_result() failed: Can't contact LDAP server
Jun 19 09:41:01 weisshorn13 nslcd[4224]: [3da4d5] ldap_result() failed: Can't contact LDAP server
Jun 19 09:41:01 weisshorn08 nslcd[3946]: [3da4d5] ldap_result() failed: Can't contact LDAP server
Jun 19 09:41:01 weisshorn07 nslcd[4297]: [3da4d5] ldap_result() failed: Can't contact LDAP server
Jun 19 09:41:01 weisshorn06 nslcd[3953]: [6928d6] ldap_result() failed: Can't contact LDAP server
Jun 19 09:41:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:41:13 weisshorn06 kernel: Lustre: scratch-OST002a: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 09:41:13 weisshorn06 kernel: Lustre: scratch-OST001d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 8 active RPCs
Jun 19 09:41:13 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 09:41:13 weisshorn06 kernel: LustreError: 6372:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc -107  req@ffff881014097c00 x1403838204234820/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340091698 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 09:41:13 weisshorn06 kernel: Lustre: scratch-OST001d: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107
Jun 19 09:41:13 weisshorn06 kernel: Lustre: Skipped 6 previous similar messages
Jun 19 09:41:13 weisshorn06 kernel: LustreError: 6372:0:(ldlm_lib.c:2725:target_bulk_io()) Skipped 5 previous similar messages
Jun 19 09:41:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:41:36 weisshorn06 kernel: Lustre: scratch-OST001d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 09:41:36 weisshorn06 kernel: Lustre: Skipped 10 previous similar messages
Jun 19 09:41:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:41:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:42:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:42:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:42:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:42:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:43:06 weisshorn06 kernel: Lustre: scratch-OST0036: Client 77ed2fd6-7ad2-3010-3442-924b80b07058 (at 148.187.6.8@o2ib2) reconnecting
Jun 19 09:43:06 weisshorn06 kernel: Lustre: scratch-OST0036: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 1 active RPCs
Jun 19 09:43:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:43:25 weisshorn06 kernel: Lustre: scratch-OST0036: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 09:43:25 weisshorn06 kernel: Lustre: Skipped 10 previous similar messages
Jun 19 09:43:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:43:29 weisshorn01 kernel: Lustre: 5251:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x1841:0x0]] [valid 68719476736]
Jun 19 09:43:29 weisshorn01 kernel: Lustre: 5251:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 3 previous similar messages
Jun 19 09:43:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:43:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:44:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:44:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:44:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:44:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:45:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:45:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:45:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:45:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:46:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:46:17 weisshorn01 kernel: Lustre: 4682:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x1851:0x0]] [valid 68719476736]
Jun 19 09:46:17 weisshorn01 kernel: Lustre: 4682:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 1 previous similar message
Jun 19 09:46:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:46:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:46:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:47:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:47:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:47:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:47:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:48:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:48:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:48:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:48:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:49:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:49:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:49:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:49:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:50:01 weisshorn07 CROND[123005]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:50:01 weisshorn11 CROND[125814]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:50:01 weisshorn13 CROND[126184]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:50:01 weisshorn08 CROND[121618]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:50:01 weisshorn09 CROND[122538]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:50:01 weisshorn04 CROND[122290]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:50:01 weisshorn03 CROND[123496]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:50:01 weisshorn05 CROND[122819]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:50:01 weisshorn14 CROND[51422]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:50:01 weisshorn12 CROND[122696]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:50:01 weisshorn10 CROND[122847]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:50:01 weisshorn01 CROND[2260]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:50:01 weisshorn06 CROND[123703]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:50:01 weisshorn02 CROND[19876]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 09:50:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:50:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:50:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:50:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:51:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:51:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:51:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:51:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:52:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:52:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:52:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:52:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:53:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:53:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:53:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:53:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:54:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:54:20 weisshorn01 kernel: Lustre: 5236:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x1881:0x0]] [valid 68719476736]
Jun 19 09:54:20 weisshorn01 kernel: Lustre: 5236:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 3 previous similar messages
Jun 19 09:54:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:54:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:54:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:55:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:55:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:55:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:55:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:56:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:56:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:56:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:56:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:57:10 weisshorn06 kernel: Lustre: scratch-OST0036: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 09:57:10 weisshorn06 kernel: Lustre: scratch-OST002a: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 09:57:10 weisshorn06 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 09:57:10 weisshorn06 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 09:57:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:57:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:57:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:57:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:58:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:58:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:58:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:58:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:59:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:59:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:59:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 09:59:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:00:01 weisshorn07 CROND[123024]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:00:01 weisshorn11 CROND[125873]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:00:01 weisshorn13 CROND[126249]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:00:01 weisshorn08 CROND[121677]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:00:01 weisshorn09 CROND[122597]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:00:01 weisshorn04 CROND[122322]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:00:01 weisshorn03 CROND[123560]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:00:01 weisshorn14 CROND[51487]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:00:01 weisshorn05 CROND[122851]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:00:01 weisshorn12 CROND[122763]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:00:01 weisshorn10 CROND[122879]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:00:01 weisshorn01 CROND[2346]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:00:01 weisshorn06 CROND[123758]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:00:01 weisshorn02 CROND[20793]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:00:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:00:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:00:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:00:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:01:01 weisshorn02 CROND[20887]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 10:01:01 weisshorn02 run-parts(/etc/cron.hourly)[2088 starting 0anacron
Jun 19 10:01:01 weisshorn02 run-parts(/etc/cron.hourly)[2089 finished 0anacron
Jun 19 10:01:01 weisshorn02 run-parts(/etc/cron.hourly)[2088 starting mcelog.cron
Jun 19 10:01:01 weisshorn02 run-parts(/etc/cron.hourly)[2090 finished mcelog.cron
Jun 19 10:01:01 weisshorn07 CROND[123029]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 10:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1230 starting 0anacron
Jun 19 10:01:01 weisshorn11 CROND[125878]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 10:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1230 finished 0anacron
Jun 19 10:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1230 starting mcelog.cron
Jun 19 10:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1258 starting 0anacron
Jun 19 10:01:01 weisshorn13 CROND[126254]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 10:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1262 starting 0anacron
Jun 19 10:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1262 finished 0anacron
Jun 19 10:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1262 starting mcelog.cron
Jun 19 10:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1258 finished 0anacron
Jun 19 10:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1258 starting mcelog.cron
Jun 19 10:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1230 finished mcelog.cron
Jun 19 10:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1262 finished mcelog.cron
Jun 19 10:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1258 finished mcelog.cron
Jun 19 10:01:01 weisshorn08 CROND[121682]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 10:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1216 starting 0anacron
Jun 19 10:01:01 weisshorn09 CROND[122602]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 10:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1216 finished 0anacron
Jun 19 10:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1216 starting mcelog.cron
Jun 19 10:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1226 starting 0anacron
Jun 19 10:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1226 finished 0anacron
Jun 19 10:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1226 starting mcelog.cron
Jun 19 10:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1216 finished mcelog.cron
Jun 19 10:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1226 finished mcelog.cron
Jun 19 10:01:01 weisshorn04 CROND[122327]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 10:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1223 starting 0anacron
Jun 19 10:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1223 finished 0anacron
Jun 19 10:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1223 starting mcelog.cron
Jun 19 10:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1223 finished mcelog.cron
Jun 19 10:01:01 weisshorn03 CROND[123565]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 10:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1235 starting 0anacron
Jun 19 10:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1235 finished 0anacron
Jun 19 10:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1235 starting mcelog.cron
Jun 19 10:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1235 finished mcelog.cron
Jun 19 10:01:01 weisshorn14 CROND[51492]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 10:01:01 weisshorn05 CROND[122856]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 10:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5149 starting 0anacron
Jun 19 10:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5150 finished 0anacron
Jun 19 10:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5149 starting mcelog.cron
Jun 19 10:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5150 finished mcelog.cron
Jun 19 10:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1228 starting 0anacron
Jun 19 10:01:01 weisshorn10 CROND[122884]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 10:01:01 weisshorn12 CROND[122768]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 10:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1228 starting 0anacron
Jun 19 10:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1227 starting 0anacron
Jun 19 10:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1228 finished 0anacron
Jun 19 10:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1228 starting mcelog.cron
Jun 19 10:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1227 finished 0anacron
Jun 19 10:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1227 starting mcelog.cron
Jun 19 10:01:01 weisshorn01 CROND[2355]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 10:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1228 finished 0anacron
Jun 19 10:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1228 starting mcelog.cron
Jun 19 10:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1229 finished mcelog.cron
Jun 19 10:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1227 finished mcelog.cron
Jun 19 10:01:01 weisshorn06 CROND[123763]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 10:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1228 finished mcelog.cron
Jun 19 10:01:01 weisshorn01 run-parts(/etc/cron.hourly)[2355 starting 0anacron
Jun 19 10:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1237 starting 0anacron
Jun 19 10:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1237 finished 0anacron
Jun 19 10:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1237 starting mcelog.cron
Jun 19 10:01:02 weisshorn06 run-parts(/etc/cron.hourly)[1237 finished mcelog.cron
Jun 19 10:01:02 weisshorn01 run-parts(/etc/cron.hourly)[2365 finished 0anacron
Jun 19 10:01:02 weisshorn01 run-parts(/etc/cron.hourly)[2355 starting mcelog.cron
Jun 19 10:01:02 weisshorn01 run-parts(/etc/cron.hourly)[2372 finished mcelog.cron
Jun 19 10:01:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:01:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:01:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:01:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:02:10 weisshorn07 rhnsd[123049]: running program /usr/sbin/rhn_check
Jun 19 10:02:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:02:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:02:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:02:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:03:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:03:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:03:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:03:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:04:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:04:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:04:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:04:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:05:01 weisshorn01 kernel: Lustre: 4722:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004631:0x15163:0x0]] [valid 68719476736]
Jun 19 10:05:01 weisshorn01 kernel: Lustre: 4722:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 6 previous similar messages
Jun 19 10:05:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:05:16 weisshorn10 nslcd[4294]: [c93698] ldap_result() failed: Can't contact LDAP server
Jun 19 10:05:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:05:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:05:45 weisshorn09 nslcd[3954]: [6ed7f6] ldap_result() failed: Can't contact LDAP server
Jun 19 10:05:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:06:08 weisshorn08 nslcd[3946]: [c93698] ldap_result() failed: Can't contact LDAP server
Jun 19 10:06:09 weisshorn06 nslcd[3953]: [7195e5] ldap_result() failed: Can't contact LDAP server
Jun 19 10:06:11 weisshorn04 nslcd[3885]: [6ed7f6] ldap_result() failed: Can't contact LDAP server
Jun 19 10:06:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:06:12 weisshorn07 nslcd[4297]: [c93698] ldap_result() failed: Can't contact LDAP server
Jun 19 10:06:13 weisshorn11 kernel: Lustre: scratch-OST0044: Client 77ed2fd6-7ad2-3010-3442-924b80b07058 (at 148.187.6.8@o2ib2) reconnecting
Jun 19 10:06:13 weisshorn11 kernel: Lustre: scratch-OST0044: Client 77ed2fd6-7ad2-3010-3442-924b80b07058 (at 148.187.6.8@o2ib2) refused reconnection, still busy with 1 active RPCs
Jun 19 10:06:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:06:32 weisshorn11 nslcd[3954]: [b59621] ldap_result() failed: Can't contact LDAP server
Jun 19 10:06:33 weisshorn12 nslcd[3954]: [c93698] ldap_result() failed: Can't contact LDAP server
Jun 19 10:06:34 weisshorn13 nslcd[4224]: [c93698] ldap_result() failed: Can't contact LDAP server
Jun 19 10:06:37 weisshorn11 kernel: Lustre: scratch-OST0044: Client 77ed2fd6-7ad2-3010-3442-924b80b07058 (at 148.187.6.8@o2ib2) reconnecting
Jun 19 10:06:37 weisshorn11 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 10:06:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:06:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:07:11 weisshorn05 nslcd[4378]: [781401] ldap_result() failed: Can't contact LDAP server
Jun 19 10:07:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:07:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:07:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:07:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:08:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:08:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:08:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:08:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:09:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:09:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:09:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:09:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:10:01 weisshorn06 CROND[123803]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:10:01 weisshorn01 CROND[2448]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:10:01 weisshorn01 CROND[2449]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 10:10:01 weisshorn02 CROND[21770]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:10:01 weisshorn14 sshd[51540]: Accepted publickey for root from 148.187.12.231 port 49683 ssh2
Jun 19 10:10:01 weisshorn14 sshd[51540]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 10:10:01 weisshorn10 sshd[122958]: Accepted publickey for root from 148.187.12.231 port 37039 ssh2
Jun 19 10:10:01 weisshorn12 sshd[122813]: Accepted publickey for root from 148.187.12.231 port 46479 ssh2
Jun 19 10:10:01 weisshorn10 sshd[122958]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 10:10:01 weisshorn12 sshd[122813]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 10:10:01 weisshorn13 sshd[126299]: Accepted publickey for root from 148.187.12.231 port 44734 ssh2
Jun 19 10:10:01 weisshorn04 sshd[122407]: Accepted publickey for root from 148.187.12.231 port 44327 ssh2
Jun 19 10:10:01 weisshorn04 sshd[122407]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 10:10:01 weisshorn06 sshd[123805]: Accepted publickey for root from 148.187.12.231 port 44833 ssh2
Jun 19 10:10:01 weisshorn06 sshd[123805]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 10:10:01 weisshorn09 sshd[122649]: Accepted publickey for root from 148.187.12.231 port 58007 ssh2
Jun 19 10:10:01 weisshorn09 sshd[122649]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 10:10:01 weisshorn14 sshd[51540]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 10:10:01 weisshorn14 sshd[51540]: pam_unix(sshd:session): session closed for user root
Jun 19 10:10:01 weisshorn05 sshd[122936]: Accepted publickey for root from 148.187.12.231 port 54775 ssh2
Jun 19 10:10:01 weisshorn13 sshd[126299]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 10:10:01 weisshorn05 sshd[122936]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 10:10:01 weisshorn03 sshd[123617]: Accepted publickey for root from 148.187.12.231 port 50578 ssh2
Jun 19 10:10:01 weisshorn13 CROND[126313]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:10:01 weisshorn03 sshd[123617]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 10:10:01 weisshorn11 sshd[125923]: Accepted publickey for root from 148.187.12.231 port 45262 ssh2
Jun 19 10:10:01 weisshorn11 sshd[125923]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 10:10:01 weisshorn08 sshd[121727]: Accepted publickey for root from 148.187.12.231 port 40371 ssh2
Jun 19 10:10:01 weisshorn11 CROND[125928]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:10:01 weisshorn04 sshd[122407]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 10:10:01 weisshorn04 sshd[122407]: pam_unix(sshd:session): session closed for user root
Jun 19 10:10:01 weisshorn06 sshd[123805]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 10:10:01 weisshorn06 sshd[123805]: pam_unix(sshd:session): session closed for user root
Jun 19 10:10:01 weisshorn09 sshd[122649]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 10:10:01 weisshorn09 sshd[122649]: pam_unix(sshd:session): session closed for user root
Jun 19 10:10:01 weisshorn12 sshd[122813]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 10:10:01 weisshorn12 sshd[122813]: pam_unix(sshd:session): session closed for user root
Jun 19 10:10:01 weisshorn08 sshd[121727]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 10:10:01 weisshorn10 sshd[122958]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 10:10:01 weisshorn10 sshd[122958]: pam_unix(sshd:session): session closed for user root
Jun 19 10:10:01 weisshorn13 sshd[126299]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 10:10:01 weisshorn13 sshd[126299]: pam_unix(sshd:session): session closed for user root
Jun 19 10:10:01 weisshorn07 CROND[123086]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:10:01 weisshorn08 CROND[121738]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:10:01 weisshorn09 CROND[122678]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:10:01 weisshorn05 sshd[122936]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 10:10:01 weisshorn05 sshd[122936]: pam_unix(sshd:session): session closed for user root
Jun 19 10:10:01 weisshorn03 sshd[123617]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 10:10:01 weisshorn03 sshd[123617]: pam_unix(sshd:session): session closed for user root
Jun 19 10:10:01 weisshorn04 CROND[122430]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:10:01 weisshorn11 sshd[125923]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 10:10:01 weisshorn11 sshd[125923]: pam_unix(sshd:session): session closed for user root
Jun 19 10:10:01 weisshorn07 sshd[123075]: Accepted publickey for root from 148.187.12.231 port 48380 ssh2
Jun 19 10:10:01 weisshorn07 sshd[123075]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 10:10:01 weisshorn08 sshd[121727]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 10:10:01 weisshorn08 sshd[121727]: pam_unix(sshd:session): session closed for user root
Jun 19 10:10:01 weisshorn03 CROND[123640]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:10:01 weisshorn14 CROND[51563]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:10:01 weisshorn07 sshd[123075]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 10:10:01 weisshorn07 sshd[123075]: pam_unix(sshd:session): session closed for user root
Jun 19 10:10:01 weisshorn01 postfix/pickup[1900]: E95B092224F: uid=0 from=<root>
Jun 19 10:10:01 weisshorn10 CROND[122987]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:10:01 weisshorn12 CROND[122836]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:10:01 weisshorn01 postfix/cleanup[2483]: E95B092224F: message-id=<20120619081001.E95B092224F@weisshorn01.admin.cscs.ch>
Jun 19 10:10:01 weisshorn05 CROND[122959]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:10:02 weisshorn01 postfix/qmgr[2710]: E95B092224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 10:10:02 weisshorn01 postfix/local[2485]: E95B092224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=0.72, delays=0.67/0.02/0/0.03, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 10:10:02 weisshorn01 postfix/qmgr[2710]: E95B092224F: removed
Jun 19 10:10:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:10:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:10:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:10:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:11:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:11:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:11:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:11:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:12:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:12:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:12:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:12:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:13:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:13:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:13:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:13:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:14:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:14:23 weisshorn14 rhnsd[51576]: running program /usr/sbin/rhn_check
Jun 19 10:14:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:14:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:14:48 weisshorn06 kernel: Lustre: scratch-OST001d: Client 77ed2fd6-7ad2-3010-3442-924b80b07058 (at 148.187.6.8@o2ib2) reconnecting
Jun 19 10:14:48 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 10:14:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:15:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:15:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:15:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:15:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:16:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:16:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:16:32 weisshorn06 kernel: Lustre: scratch-OST001d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 10:16:32 weisshorn06 kernel: Lustre: Skipped 9 previous similar messages
Jun 19 10:16:32 weisshorn06 kernel: Lustre: scratch-OST001d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 5 active RPCs
Jun 19 10:16:32 weisshorn06 kernel: LustreError: 8287:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc -107  req@ffff8810118c8800 x1403838204442511/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340093847 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 10:16:32 weisshorn06 kernel: Lustre: scratch-OST001d: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107
Jun 19 10:16:32 weisshorn06 kernel: Lustre: Skipped 6 previous similar messages
Jun 19 10:16:32 weisshorn06 kernel: LustreError: 8287:0:(ldlm_lib.c:2725:target_bulk_io()) Skipped 3 previous similar messages
Jun 19 10:16:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:16:56 weisshorn06 kernel: Lustre: scratch-OST001d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 10:16:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:17:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:17:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:17:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:17:51 weisshorn01 kernel: Lustre: 5240:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046ee:0x12c6:0x0]] [valid 68719476736]
Jun 19 10:17:51 weisshorn01 kernel: Lustre: 5240:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 18 previous similar messages
Jun 19 10:17:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:18:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:18:17 weisshorn06 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 252s: evicting client at 148.187.6.8@o2ib2  ns: filter-scratch-OST001d_UUID lock: ffff8805019bb480/0x38a5357f482a3f66 lrc: 3/0,0 mode: PR/PR res: 8749182/0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0x2416669ca458dd57 expref: 1314 pid: 7622 timeout 5573505404
Jun 19 10:18:17 weisshorn06 kernel: LustreError: 8025:0:(ldlm_lib.c:2701:target_bulk_io()) @@@ Eviction on bulk GET  req@ffff88088bc49400 x1403838204447090/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340093929 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 10:18:24 weisshorn06 kernel: Lustre: scratch-OST001d: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107
Jun 19 10:18:24 weisshorn06 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 10:18:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:18:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:18:47 weisshorn11 ntpd[4454]: synchronized to 148.187.18.145, stratum 2
Jun 19 10:18:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:19:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:19:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:19:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:19:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:20:01 weisshorn10 CROND[123010]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:20:01 weisshorn12 CROND[122899]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:20:01 weisshorn05 CROND[122982]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:20:01 weisshorn06 CROND[123870]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:20:01 weisshorn02 CROND[22693]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:20:01 weisshorn13 CROND[126385]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:20:01 weisshorn11 CROND[125983]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:20:01 weisshorn07 CROND[123170]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:20:01 weisshorn09 CROND[122701]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:20:01 weisshorn04 CROND[122453]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:20:01 weisshorn08 CROND[121781]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:20:01 weisshorn03 CROND[123663]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:20:01 weisshorn14 CROND[51623]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:20:01 weisshorn01 CROND[2604]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:20:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:20:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:20:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:20:56 weisshorn06 kernel: Lustre: scratch-OST0036: Client 77ed2fd6-7ad2-3010-3442-924b80b07058 (at 148.187.6.8@o2ib2) reconnecting
Jun 19 10:20:56 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 10:20:56 weisshorn06 kernel: Lustre: scratch-OST0036: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 1 active RPCs
Jun 19 10:20:56 weisshorn06 kernel: LustreError: 8307:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk PUT failed: rc -107  req@ffff88058d2bb000 x1403838204532139/t0(0) o3->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/400 e 0 to 0 dl 1340094071 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 10:20:56 weisshorn06 kernel: Lustre: scratch-OST0036: Bulk IO read error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107
Jun 19 10:20:56 weisshorn06 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 10:20:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:21:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:21:15 weisshorn06 kernel: Lustre: scratch-OST0036: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 10:21:15 weisshorn06 kernel: Lustre: Skipped 9 previous similar messages
Jun 19 10:21:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:21:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:21:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:22:01 weisshorn14 nslcd[3437]: [e74ea3] ldap_result() failed: Can't contact LDAP server
Jun 19 10:22:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:22:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:22:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:22:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:23:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:23:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:23:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:23:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:24:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:24:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:24:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:24:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:25:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:25:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:25:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:25:51 weisshorn03 nslcd[3944]: [9df57d] ldap_result() failed: Can't contact LDAP server
Jun 19 10:25:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:26:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:26:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:26:32 weisshorn11 kernel: Lustre: 5780:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340094378/real 1340094378]  req@ffff88050664a400 x1403849892120495/t0(0) o104->scratch-OST0008@148.187.6.8@o2ib2:15/16 lens 296/192 e 0 to 1 dl 1340094392 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 10:26:32 weisshorn11 kernel: LustreError: 138-a: scratch-OST0008: A client on nid 148.187.6.8@o2ib2 was evicted due to a lock blocking callback time out: rc -107
Jun 19 10:26:32 weisshorn11 kernel: LustreError: Skipped 1 previous similar message
Jun 19 10:26:36 weisshorn11 kernel: Lustre: 5807:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340094380/real 1340094380]  req@ffff880e1abe9000 x1403849892120952/t0(0) o104->scratch-OST0021@148.187.6.8@o2ib2:15/16 lens 296/192 e 0 to 1 dl 1340094394 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 10:26:36 weisshorn11 kernel: LustreError: 138-a: scratch-OST0021: A client on nid 148.187.6.8@o2ib2 was evicted due to a lock blocking callback time out: rc -107
Jun 19 10:26:36 weisshorn11 kernel: LustreError: Skipped 7 previous similar messages
Jun 19 10:26:37 weisshorn11 kernel: Lustre: 5778:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340094380/real 0]  req@ffff881013625800 x1403849892120956/t0(0) o104->scratch-OST002d@148.187.6.8@o2ib2:15/16 lens 296/192 e 0 to 1 dl 1340094394 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 10:26:37 weisshorn11 kernel: LustreError: 138-a: scratch-OST002d: A client on nid 148.187.6.8@o2ib2 was evicted due to a lock blocking callback time out: rc -107
Jun 19 10:26:37 weisshorn11 kernel: LustreError: Skipped 3 previous similar messages
Jun 19 10:26:38 weisshorn11 kernel: LustreError: 8487:0:(ldlm_lockd.c:2077:ldlm_cancel_handler()) ldlm_cancel from 148.187.6.8@o2ib2 arrived at 1340094398 with bad export cookie 13982053158071586955
Jun 19 10:26:38 weisshorn11 kernel: LustreError: 8487:0:(ldlm_lockd.c:2077:ldlm_cancel_handler()) Skipped 1 previous similar message
Jun 19 10:26:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:26:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:27:10 weisshorn06 ntpd[4454]: synchronized to 148.187.18.145, stratum 2
Jun 19 10:27:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:27:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:27:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:27:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:28:09 weisshorn06 kernel: Lustre: scratch-OST002a: Client 77ed2fd6-7ad2-3010-3442-924b80b07058 (at 148.187.6.8@o2ib2) reconnecting
Jun 19 10:28:09 weisshorn06 kernel: Lustre: scratch-OST002a: Client 77ed2fd6-7ad2-3010-3442-924b80b07058 (at 148.187.6.8@o2ib2) refused reconnection, still busy with 1 active RPCs
Jun 19 10:28:09 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 10:28:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:28:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:28:33 weisshorn06 kernel: Lustre: scratch-OST002a: Client 77ed2fd6-7ad2-3010-3442-924b80b07058 (at 148.187.6.8@o2ib2) reconnecting
Jun 19 10:28:33 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 10:28:34 weisshorn01 kernel: Lustre: 4694:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046e5:0xd65e:0x0]] [valid 68719476736]
Jun 19 10:28:34 weisshorn01 kernel: Lustre: 4694:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 6 previous similar messages
Jun 19 10:28:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:28:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:29:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:29:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:29:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:29:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:30:01 weisshorn01 CROND[2679]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:30:01 weisshorn10 CROND[123042]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:30:01 weisshorn12 CROND[122931]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:30:01 weisshorn05 CROND[123014]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:30:01 weisshorn06 CROND[123925]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:30:01 weisshorn02 CROND[23610]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:30:01 weisshorn11 CROND[126043]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:30:01 weisshorn13 CROND[126415]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:30:01 weisshorn07 CROND[123189]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:30:01 weisshorn09 CROND[122760]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:30:01 weisshorn04 CROND[122485]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:30:01 weisshorn08 CROND[121840]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:30:01 weisshorn03 CROND[123723]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:30:01 weisshorn14 CROND[51655]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:30:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:30:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:30:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:30:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:31:01 weisshorn02 nslcd[2099]: [6e0907] ldap_result() failed: Can't contact LDAP server
Jun 19 10:31:01 weisshorn14 nslcd[3437]: [7b4d84] ldap_result() failed: Can't contact LDAP server
Jun 19 10:31:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:31:16 weisshorn02 nslcd[2099]: [683d0d] ldap_result() failed: Can't contact LDAP server
Jun 19 10:31:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:31:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:31:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:32:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:32:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:32:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:32:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:33:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:33:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:33:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:33:51 weisshorn13 rhnsd[126429]: running program /usr/sbin/rhn_check
Jun 19 10:33:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:34:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:34:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:34:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:34:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:35:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:35:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:35:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:35:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:36:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:36:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:36:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:36:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:37:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:37:14 weisshorn02 ntpd[2541]: synchronized to 148.187.18.145, stratum 2
Jun 19 10:37:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:37:32 weisshorn13 ntpd[4726]: synchronized to 148.187.18.145, stratum 2
Jun 19 10:37:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:37:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:38:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:38:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:38:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:38:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:39:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:39:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:39:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:39:43 weisshorn01 kernel: Lustre: 5256:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046e5:0xd6b5:0x0]] [valid 68719476736]
Jun 19 10:39:43 weisshorn01 kernel: Lustre: 5256:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 21 previous similar messages
Jun 19 10:39:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:40:01 weisshorn14 CROND[51695]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:40:01 weisshorn01 CROND[2797]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:40:01 weisshorn03 CROND[123761]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:40:01 weisshorn12 CROND[122961]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:40:01 weisshorn10 CROND[123107]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:40:01 weisshorn05 CROND[123079]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:40:01 weisshorn06 CROND[123962]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:40:01 weisshorn02 CROND[24573]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:40:01 weisshorn11 CROND[126076]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:40:01 weisshorn09 CROND[122803]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:40:01 weisshorn07 CROND[123266]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:40:01 weisshorn13 CROND[126453]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:40:01 weisshorn08 CROND[121875]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:40:01 weisshorn04 CROND[122550]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:40:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:40:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:40:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:40:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:41:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:41:16 weisshorn10 nslcd[4294]: [781401] ldap_result() failed: Can't contact LDAP server
Jun 19 10:41:16 weisshorn05 nslcd[4378]: [6928d6] ldap_result() failed: Can't contact LDAP server
Jun 19 10:41:16 weisshorn12 nslcd[3954]: [781401] ldap_result() failed: Can't contact LDAP server
Jun 19 10:41:16 weisshorn04 nslcd[3885]: [9df57d] ldap_result() failed: Can't contact LDAP server
Jun 19 10:41:16 weisshorn09 nslcd[3954]: [9df57d] ldap_result() failed: Can't contact LDAP server
Jun 19 10:41:16 weisshorn11 nslcd[3954]: [92a02e] ldap_result() failed: Can't contact LDAP server
Jun 19 10:41:16 weisshorn03 nslcd[3944]: [bd7a3e] ldap_result() failed: Can't contact LDAP server
Jun 19 10:41:16 weisshorn13 nslcd[4224]: [781401] ldap_result() failed: Can't contact LDAP server
Jun 19 10:41:16 weisshorn08 nslcd[3946]: [781401] ldap_result() failed: Can't contact LDAP server
Jun 19 10:41:16 weisshorn07 nslcd[4297]: [781401] ldap_result() failed: Can't contact LDAP server
Jun 19 10:41:16 weisshorn06 nslcd[3953]: [46a5f2] ldap_result() failed: Can't contact LDAP server
Jun 19 10:41:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:41:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:41:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:42:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:42:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:42:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:42:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:43:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:43:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:43:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:43:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:44:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:44:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:44:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:44:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:45:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:45:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:45:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:45:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:45:59 weisshorn05 ntpd[4877]: synchronized to 148.187.18.145, stratum 2
Jun 19 10:46:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:46:13 weisshorn06 kernel: Lustre: scratch-OST001d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 10:46:13 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 10:46:13 weisshorn06 kernel: Lustre: scratch-OST001d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 8 active RPCs
Jun 19 10:46:13 weisshorn06 kernel: LustreError: 8317:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc -107  req@ffff8810119b0800 x1403838204660506/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340095628 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 10:46:13 weisshorn06 kernel: Lustre: scratch-OST001d: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107
Jun 19 10:46:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:46:37 weisshorn06 kernel: Lustre: scratch-OST001d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 10:46:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:46:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:47:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:47:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:47:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:47:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:48:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:48:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:48:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:48:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:49:03 weisshorn07 ntpd[4804]: synchronized to 148.187.18.145, stratum 2
Jun 19 10:49:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:49:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:49:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:49:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:50:01 weisshorn14 CROND[51749]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:50:01 weisshorn01 CROND[2873]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:50:01 weisshorn03 CROND[123786]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:50:01 weisshorn12 CROND[123021]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:50:01 weisshorn10 CROND[123132]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:50:01 weisshorn05 CROND[123105]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:50:01 weisshorn06 CROND[123994]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:50:01 weisshorn02 CROND[25490]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:50:01 weisshorn11 CROND[126106]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:50:01 weisshorn09 CROND[122823]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:50:01 weisshorn07 CROND[123293]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:50:01 weisshorn08 CROND[121903]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:50:01 weisshorn04 CROND[122575]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:50:02 weisshorn13 CROND[126513]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 10:50:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:50:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:50:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:50:42 weisshorn01 kernel: Lustre: 5270:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046e5:0xd6d6:0x0]] [valid 68719476736]
Jun 19 10:50:42 weisshorn01 kernel: Lustre: 5270:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 4 previous similar messages
Jun 19 10:50:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:51:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:51:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:51:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:51:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:52:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:52:23 weisshorn06 kernel: Lustre: scratch-OST0036: Client 77ed2fd6-7ad2-3010-3442-924b80b07058 (at 148.187.6.8@o2ib2) reconnecting
Jun 19 10:52:23 weisshorn06 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 10:52:23 weisshorn06 kernel: Lustre: scratch-OST001d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 9 active RPCs
Jun 19 10:52:23 weisshorn06 kernel: LustreError: 8067:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc -107  req@ffff880ef6800000 x1403838204689464/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340095974 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 10:52:23 weisshorn06 kernel: Lustre: scratch-OST001d: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107
Jun 19 10:52:23 weisshorn06 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 10:52:23 weisshorn06 kernel: LustreError: 8067:0:(ldlm_lib.c:2725:target_bulk_io()) Skipped 11 previous similar messages
Jun 19 10:52:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:52:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:52:45 weisshorn06 kernel: Lustre: scratch-OST001d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 10:52:45 weisshorn06 kernel: Lustre: Skipped 8 previous similar messages
Jun 19 10:52:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:53:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:53:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:53:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:53:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:54:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:54:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:54:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:54:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:55:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:55:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:55:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:55:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:56:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:56:18 weisshorn06 kernel: Lustre: 7784:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340096167/real 1340096167]  req@ffff8810172ba400 x1403849888822103/t0(0) o106->scratch-OST001d@148.187.6.8@o2ib2:15/16 lens 296/232 e 0 to 1 dl 1340096178 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jun 19 10:56:21 weisshorn06 kernel: LustreError: 5795:0:(events.c:389:server_bulk_callback()) event type 4, status -103, desc ffff880c36aee000
Jun 19 10:56:21 weisshorn06 kernel: LustreError: 5795:0:(events.c:389:server_bulk_callback()) event type 2, status -103, desc ffff880c36aee000
Jun 19 10:56:21 weisshorn06 kernel: LustreError: 8221:0:(ldlm_lib.c:2717:target_bulk_io()) @@@ network error on bulk GET 0(1048576)  req@ffff881013dc7800 x1403838204710175/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340096192 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 10:56:21 weisshorn06 kernel: Lustre: scratch-OST001d: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -110
Jun 19 10:56:21 weisshorn06 kernel: Lustre: Skipped 6 previous similar messages
Jun 19 10:56:21 weisshorn06 kernel: Lustre: scratch-OST001d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 10:56:21 weisshorn06 kernel: Lustre: scratch-OST001d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 2 active RPCs
Jun 19 10:56:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:56:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:56:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:57:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:57:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:57:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:57:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:58:03 weisshorn06 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 1866s: evicting client at 148.187.6.8@o2ib2  ns: filter-scratch-OST001d_UUID lock: ffff8809bf130480/0x38a5357f4838bc6a lrc: 3/0,0 mode: PR/PR res: 8761635/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0x2416669ca45e17a7 expref: 419 pid: 7753 timeout 5575891308
Jun 19 10:58:03 weisshorn06 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) Skipped 2 previous similar messages
Jun 19 10:58:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:58:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:58:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:58:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:59:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:59:27 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:59:42 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 10:59:57 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:00:01 weisshorn14 CROND[51785]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:00:01 weisshorn13 CROND[126549]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:00:01 weisshorn01 CROND[3021]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:00:01 weisshorn12 CROND[123059]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:00:01 weisshorn03 CROND[123850]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:00:01 weisshorn10 CROND[123199]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:00:01 weisshorn05 CROND[123172]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:00:01 weisshorn06 CROND[124056]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:00:01 weisshorn02 CROND[26413]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:00:01 weisshorn11 CROND[126171]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:00:01 weisshorn09 CROND[122887]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:00:01 weisshorn07 CROND[123311]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:00:01 weisshorn08 CROND[121967]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:00:02 weisshorn04 CROND[122642]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:00:12 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:00:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:00:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:00:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:01:01 weisshorn08 CROND[121972]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 11:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1219 starting 0anacron
Jun 19 11:01:01 weisshorn04 CROND[122647]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 11:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1226 starting 0anacron
Jun 19 11:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1226 finished 0anacron
Jun 19 11:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1226 starting mcelog.cron
Jun 19 11:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1219 finished 0anacron
Jun 19 11:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1219 starting mcelog.cron
Jun 19 11:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1226 finished mcelog.cron
Jun 19 11:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1219 finished mcelog.cron
Jun 19 11:01:01 weisshorn14 CROND[51795]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 11:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5179 starting 0anacron
Jun 19 11:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5180 finished 0anacron
Jun 19 11:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5179 starting mcelog.cron
Jun 19 11:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5181 finished mcelog.cron
Jun 19 11:01:01 weisshorn13 CROND[126554]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 11:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1265 starting 0anacron
Jun 19 11:01:01 weisshorn01 CROND[3026]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 11:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3026 starting 0anacron
Jun 19 11:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1265 finished 0anacron
Jun 19 11:01:01 weisshorn03 CROND[123855]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 11:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1265 starting mcelog.cron
Jun 19 11:01:01 weisshorn12 CROND[123064]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 11:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3036 finished 0anacron
Jun 19 11:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3026 starting mcelog.cron
Jun 19 11:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1238 starting 0anacron
Jun 19 11:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1230 starting 0anacron
Jun 19 11:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1230 finished 0anacron
Jun 19 11:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1230 starting mcelog.cron
Jun 19 11:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3043 finished mcelog.cron
Jun 19 11:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1265 finished mcelog.cron
Jun 19 11:01:01 weisshorn10 CROND[123204]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 11:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1230 finished mcelog.cron
Jun 19 11:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1238 finished 0anacron
Jun 19 11:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1232 starting 0anacron
Jun 19 11:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1238 starting mcelog.cron
Jun 19 11:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1232 finished 0anacron
Jun 19 11:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1232 starting mcelog.cron
Jun 19 11:01:01 weisshorn05 CROND[123177]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 11:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1231 starting 0anacron
Jun 19 11:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1231 finished 0anacron
Jun 19 11:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1231 starting mcelog.cron
Jun 19 11:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1232 finished mcelog.cron
Jun 19 11:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1238 finished mcelog.cron
Jun 19 11:01:01 weisshorn06 CROND[124061]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 11:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1231 finished mcelog.cron
Jun 19 11:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1240 starting 0anacron
Jun 19 11:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1240 finished 0anacron
Jun 19 11:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1240 starting mcelog.cron
Jun 19 11:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1240 finished mcelog.cron
Jun 19 11:01:01 weisshorn02 CROND[26507]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 11:01:01 weisshorn02 run-parts(/etc/cron.hourly)[2650 starting 0anacron
Jun 19 11:01:01 weisshorn02 run-parts(/etc/cron.hourly)[2651 finished 0anacron
Jun 19 11:01:01 weisshorn02 run-parts(/etc/cron.hourly)[2650 starting mcelog.cron
Jun 19 11:01:01 weisshorn02 run-parts(/etc/cron.hourly)[2652 finished mcelog.cron
Jun 19 11:01:01 weisshorn11 CROND[126176]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 11:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1261 starting 0anacron
Jun 19 11:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1261 finished 0anacron
Jun 19 11:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1261 starting mcelog.cron
Jun 19 11:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1261 finished mcelog.cron
Jun 19 11:01:01 weisshorn09 CROND[122892]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 11:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1228 starting 0anacron
Jun 19 11:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1229 finished 0anacron
Jun 19 11:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1228 starting mcelog.cron
Jun 19 11:01:01 weisshorn07 CROND[123321]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 11:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1229 finished mcelog.cron
Jun 19 11:01:02 weisshorn07 run-parts(/etc/cron.hourly)[1233 starting 0anacron
Jun 19 11:01:02 weisshorn07 run-parts(/etc/cron.hourly)[1233 finished 0anacron
Jun 19 11:01:02 weisshorn07 run-parts(/etc/cron.hourly)[1233 starting mcelog.cron
Jun 19 11:01:02 weisshorn07 run-parts(/etc/cron.hourly)[1233 finished mcelog.cron
Jun 19 11:01:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:01:25 weisshorn10 rhnsd[123223]: running program /usr/sbin/rhn_check
Jun 19 11:01:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:01:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:01:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:02:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:02:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:02:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:02:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:03:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:03:26 weisshorn01 kernel: Lustre: 5245:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x1d14:0x0]] [valid 68719476736]
Jun 19 11:03:26 weisshorn01 kernel: Lustre: 5245:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 8 previous similar messages
Jun 19 11:03:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:03:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:03:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:04:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:04:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:04:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:04:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:05:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:05:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:05:31 weisshorn10 nslcd[4294]: [6928d6] ldap_result() failed: Can't contact LDAP server
Jun 19 11:05:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:05:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:06:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:06:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:06:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:06:58 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:07:13 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:07:28 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:07:43 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:07:59 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:08:14 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:08:29 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:08:44 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:08:59 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:09:14 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:09:29 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:09:44 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:09:59 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:10:01 weisshorn09 CROND[122945]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:10:01 weisshorn04 CROND[122691]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:10:01 weisshorn08 CROND[122016]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:10:01 weisshorn07 CROND[123408]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:10:01 weisshorn14 CROND[51838]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:10:01 weisshorn01 CROND[3154]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:10:01 weisshorn01 CROND[3155]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 11:10:01 weisshorn12 CROND[123103]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:10:01 weisshorn06 CROND[124110]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:10:01 weisshorn09 sshd[122947]: Accepted publickey for root from 148.187.12.231 port 58024 ssh2
Jun 19 11:10:01 weisshorn09 sshd[122947]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 11:10:01 weisshorn14 sshd[51840]: Accepted publickey for root from 148.187.12.231 port 49703 ssh2
Jun 19 11:10:01 weisshorn14 sshd[51840]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 11:10:01 weisshorn06 sshd[124107]: Accepted publickey for root from 148.187.12.231 port 44841 ssh2
Jun 19 11:10:01 weisshorn06 sshd[124107]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 11:10:01 weisshorn04 sshd[122693]: Accepted publickey for root from 148.187.12.231 port 44331 ssh2
Jun 19 11:10:01 weisshorn04 sshd[122693]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 11:10:01 weisshorn10 CROND[123252]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:10:01 weisshorn14 sshd[51840]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 11:10:01 weisshorn14 sshd[51840]: pam_unix(sshd:session): session closed for user root
Jun 19 11:10:01 weisshorn07 sshd[123410]: Accepted publickey for root from 148.187.12.231 port 48388 ssh2
Jun 19 11:10:01 weisshorn07 sshd[123410]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 11:10:01 weisshorn08 sshd[122018]: Accepted publickey for root from 148.187.12.231 port 40390 ssh2
Jun 19 11:10:01 weisshorn13 CROND[126600]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:10:01 weisshorn08 sshd[122018]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 11:10:01 weisshorn05 CROND[123223]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:10:01 weisshorn03 CROND[123905]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:10:01 weisshorn02 CROND[27389]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:10:01 weisshorn09 sshd[122947]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 11:10:01 weisshorn09 sshd[122947]: pam_unix(sshd:session): session closed for user root
Jun 19 11:10:01 weisshorn06 sshd[124107]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 11:10:01 weisshorn06 sshd[124107]: pam_unix(sshd:session): session closed for user root
Jun 19 11:10:01 weisshorn12 sshd[123105]: Accepted publickey for root from 148.187.12.231 port 46491 ssh2
Jun 19 11:10:01 weisshorn04 sshd[122693]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 11:10:01 weisshorn04 sshd[122693]: pam_unix(sshd:session): session closed for user root
Jun 19 11:10:01 weisshorn12 sshd[123105]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 11:10:01 weisshorn11 sshd[126217]: Accepted publickey for root from 148.187.12.231 port 45269 ssh2
Jun 19 11:10:01 weisshorn11 sshd[126217]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 11:10:01 weisshorn03 sshd[123902]: Accepted publickey for root from 148.187.12.231 port 50581 ssh2
Jun 19 11:10:01 weisshorn10 sshd[123250]: Accepted publickey for root from 148.187.12.231 port 37055 ssh2
Jun 19 11:10:01 weisshorn03 sshd[123902]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 11:10:01 weisshorn10 sshd[123250]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 11:10:01 weisshorn07 sshd[123410]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 11:10:01 weisshorn07 sshd[123410]: pam_unix(sshd:session): session closed for user root
Jun 19 11:10:01 weisshorn13 sshd[126598]: Accepted publickey for root from 148.187.12.231 port 44751 ssh2
Jun 19 11:10:01 weisshorn11 CROND[126231]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:10:01 weisshorn13 sshd[126598]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 11:10:01 weisshorn05 sshd[123221]: Accepted publickey for root from 148.187.12.231 port 54788 ssh2
Jun 19 11:10:01 weisshorn05 sshd[123221]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 11:10:01 weisshorn08 sshd[122018]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 11:10:01 weisshorn08 sshd[122018]: pam_unix(sshd:session): session closed for user root
Jun 19 11:10:02 weisshorn12 sshd[123105]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 11:10:02 weisshorn12 sshd[123105]: pam_unix(sshd:session): session closed for user root
Jun 19 11:10:02 weisshorn10 sshd[123250]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 11:10:02 weisshorn10 sshd[123250]: pam_unix(sshd:session): session closed for user root
Jun 19 11:10:02 weisshorn11 sshd[126217]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 11:10:02 weisshorn11 sshd[126217]: pam_unix(sshd:session): session closed for user root
Jun 19 11:10:02 weisshorn03 sshd[123902]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 11:10:02 weisshorn03 sshd[123902]: pam_unix(sshd:session): session closed for user root
Jun 19 11:10:02 weisshorn05 sshd[123221]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 11:10:02 weisshorn05 sshd[123221]: pam_unix(sshd:session): session closed for user root
Jun 19 11:10:02 weisshorn13 sshd[126598]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 11:10:02 weisshorn13 sshd[126598]: pam_unix(sshd:session): session closed for user root
Jun 19 11:10:02 weisshorn01 postfix/pickup[2885]: 45F1C92224F: uid=0 from=<root>
Jun 19 11:10:02 weisshorn01 postfix/cleanup[3192]: 45F1C92224F: message-id=<20120619091002.45F1C92224F@weisshorn01.admin.cscs.ch>
Jun 19 11:10:02 weisshorn01 postfix/qmgr[2710]: 45F1C92224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 11:10:02 weisshorn01 postfix/local[3194]: 45F1C92224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=0.69, delays=0.65/0.02/0/0.03, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 11:10:02 weisshorn01 postfix/qmgr[2710]: 45F1C92224F: removed
Jun 19 11:10:14 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:10:29 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:10:44 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:10:59 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:11:03 weisshorn06 rhnsd[124139]: running program /usr/sbin/rhn_check
Jun 19 11:11:14 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:11:29 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:11:44 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:11:59 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:12:14 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:12:29 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:12:44 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:12:59 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:13:14 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:13:29 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:13:44 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:13:59 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:14:14 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:14:29 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:14:44 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:14:59 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:15:14 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:15:29 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:15:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:16:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:16:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:16:26 weisshorn06 kernel: Lustre: scratch-OST001d: Client 77ed2fd6-7ad2-3010-3442-924b80b07058 (at 148.187.6.8@o2ib2) reconnecting
Jun 19 11:16:26 weisshorn06 kernel: Lustre: Skipped 12 previous similar messages
Jun 19 11:16:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:16:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:17:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:17:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:17:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:17:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:18:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:18:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:18:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:18:43 weisshorn06 kernel: Lustre: scratch-OST0041: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 11:18:43 weisshorn06 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 11:18:43 weisshorn06 kernel: Lustre: scratch-OST001d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 9 active RPCs
Jun 19 11:18:43 weisshorn06 kernel: LustreError: 8257:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc -107  req@ffff8810153a3000 x1403838204833880/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340097573 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 11:18:43 weisshorn06 kernel: Lustre: scratch-OST001d: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107
Jun 19 11:18:43 weisshorn06 kernel: LustreError: 8257:0:(ldlm_lib.c:2725:target_bulk_io()) Skipped 4 previous similar messages
Jun 19 11:18:43 weisshorn06 kernel: Lustre: scratch-OST002a: Bulk IO read error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107
Jun 19 11:18:43 weisshorn06 kernel: Lustre: scratch-OST001d: Bulk IO read error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107
Jun 19 11:18:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:19:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:19:06 weisshorn06 kernel: Lustre: scratch-OST001d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 11:19:06 weisshorn06 kernel: Lustre: Skipped 10 previous similar messages
Jun 19 11:19:06 weisshorn05 kernel: LustreError: 137-5: scratch-OST0041: Not available for connect from 148.187.6.8@o2ib2 (no target)
Jun 19 11:19:06 weisshorn05 kernel: LustreError: Skipped 10 previous similar messages
Jun 19 11:19:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:19:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:19:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:20:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:20:01 weisshorn11 CROND[126268]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:20:01 weisshorn09 CROND[122985]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:20:01 weisshorn04 CROND[122736]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:20:01 weisshorn01 CROND[3269]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:20:01 weisshorn07 CROND[123454]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:20:01 weisshorn14 CROND[51913]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:20:01 weisshorn08 CROND[122064]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:20:01 weisshorn12 CROND[123183]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:20:01 weisshorn06 CROND[124163]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:20:01 weisshorn02 CROND[28318]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:20:01 weisshorn03 CROND[123948]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:20:01 weisshorn10 CROND[123296]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:20:01 weisshorn05 CROND[123268]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:20:01 weisshorn13 CROND[126678]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:20:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:20:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:20:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:21:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:21:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:21:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:21:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:22:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:22:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:22:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:22:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:23:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:23:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:23:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:23:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:23:54 weisshorn06 kernel: Lustre: scratch-OST0036: Client 77ed2fd6-7ad2-3010-3442-924b80b07058 (at 148.187.6.8@o2ib2) reconnecting
Jun 19 11:23:54 weisshorn06 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 11:24:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:24:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:24:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:24:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:25:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:25:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:25:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:25:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:26:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:26:06 weisshorn03 nslcd[3944]: [90b7c5] ldap_result() failed: Can't contact LDAP server
Jun 19 11:26:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:26:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:26:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:27:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:27:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:27:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:27:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:27:59 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 11:27:59 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 8 active RPCs
Jun 19 11:27:59 weisshorn11 kernel: LustreError: 7941:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc -107  req@ffff881016502400 x1403838204880428/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340098120 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 11:27:59 weisshorn11 kernel: LustreError: 7941:0:(ldlm_lib.c:2725:target_bulk_io()) Skipped 9 previous similar messages
Jun 19 11:27:59 weisshorn11 kernel: Lustre: scratch-OST0008: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107
Jun 19 11:27:59 weisshorn11 kernel: Lustre: Skipped 13 previous similar messages
Jun 19 11:28:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:28:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:28:22 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 11:28:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:28:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:29:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:29:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:29:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:29:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:30:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:30:01 weisshorn11 CROND[126334]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:30:01 weisshorn09 CROND[123049]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:30:01 weisshorn04 CROND[122803]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:30:01 weisshorn01 CROND[3333]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:30:01 weisshorn14 CROND[51954]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:30:01 weisshorn08 CROND[122128]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:30:01 weisshorn07 CROND[123478]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:30:01 weisshorn12 CROND[123221]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:30:01 weisshorn06 CROND[124222]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:30:01 weisshorn10 CROND[123363]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:30:01 weisshorn02 CROND[29243]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:30:01 weisshorn03 CROND[124013]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:30:01 weisshorn05 CROND[123334]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:30:01 weisshorn13 CROND[126714]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:30:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:30:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:30:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:31:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:31:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:31:16 weisshorn02 nslcd[2099]: [6e41d8] ldap_result() failed: Can't contact LDAP server
Jun 19 11:31:16 weisshorn14 nslcd[3437]: [819e7f] ldap_result() failed: Can't contact LDAP server
Jun 19 11:31:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:31:31 weisshorn02 nslcd[2099]: [dce0fd] ldap_result() failed: Can't contact LDAP server
Jun 19 11:31:31 weisshorn02 nslcd[2099]: [dce0fd] ldap_abandon() failed to abandon search: Other (e.g., implementation specific) error
Jun 19 11:31:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:32:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:32:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:32:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:32:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:33:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:33:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:33:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:33:32 weisshorn01 kernel: Lustre: 5294:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004631:0x151e1:0x0]] [valid 68719476736]
Jun 19 11:33:32 weisshorn01 kernel: Lustre: 5294:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 4 previous similar messages
Jun 19 11:33:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:33:52 weisshorn06 kernel: Lustre: scratch-OST0041: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 11:33:52 weisshorn06 kernel: Lustre: Skipped 6 previous similar messages
Jun 19 11:33:52 weisshorn06 kernel: Lustre: scratch-OST0012: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 1 active RPCs
Jun 19 11:33:52 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 11:33:52 weisshorn06 kernel: LustreError: 8126:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk PUT failed: rc -107  req@ffff8810151a5400 x1403838204916742/t0(0) o3->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/400 e 0 to 0 dl 1340098450 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 11:33:52 weisshorn06 kernel: LustreError: 8126:0:(ldlm_lib.c:2725:target_bulk_io()) Skipped 3 previous similar messages
Jun 19 11:33:52 weisshorn06 kernel: Lustre: scratch-OST0012: Bulk IO read error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107
Jun 19 11:33:52 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 11:33:52 weisshorn14 ntpd[3906]: synchronized to 148.187.18.145, stratum 2
Jun 19 11:34:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:34:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:34:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:34:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:35:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:35:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:35:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:35:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:36:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:36:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:36:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:36:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:36:45 weisshorn11 rhnsd[126354]: running program /usr/sbin/rhn_check
Jun 19 11:36:50 weisshorn03 rhnsd[124040]: running program /usr/sbin/rhn_check
Jun 19 11:37:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:37:06 weisshorn01 kernel: Lustre: 5425:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004631:0x1576c:0x0]] [valid 68719476736]
Jun 19 11:37:06 weisshorn01 kernel: Lustre: 5425:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 1 previous similar message
Jun 19 11:37:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:37:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:37:41 weisshorn04 rhnsd[122829]: running program /usr/sbin/rhn_check
Jun 19 11:37:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:38:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:38:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:38:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:38:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:39:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:39:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:39:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:39:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:39:47 weisshorn11 kernel: Lustre: scratch-OST0015: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 11:39:47 weisshorn11 kernel: Lustre: scratch-OST0044: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 1 active RPCs
Jun 19 11:39:47 weisshorn11 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 11:40:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:40:01 weisshorn13 CROND[126743]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:40:01 weisshorn11 CROND[126364]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:40:01 weisshorn09 CROND[123087]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:40:01 weisshorn04 CROND[122835]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:40:01 weisshorn01 CROND[3438]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:40:01 weisshorn14 CROND[51985]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:40:01 weisshorn08 CROND[122158]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:40:01 weisshorn07 CROND[123550]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:40:01 weisshorn12 CROND[123245]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:40:01 weisshorn06 CROND[124255]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:40:01 weisshorn10 CROND[123393]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:40:01 weisshorn05 CROND[123364]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:40:02 weisshorn02 CROND[30201]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:40:02 weisshorn03 CROND[124048]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:40:11 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting
Jun 19 11:40:11 weisshorn11 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 11:40:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:40:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:40:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:41:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:41:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:41:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:41:44 weisshorn09 ntpd[4454]: synchronized to 148.187.18.145, stratum 2
Jun 19 11:41:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:42:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:42:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:42:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:42:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:43:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:43:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:43:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:43:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:44:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:44:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:44:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:44:30 weisshorn09 nslcd[3954]: [bd7a3e] ldap_result() failed: Can't contact LDAP server
Jun 19 11:44:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:44:53 weisshorn08 nslcd[3946]: [6928d6] ldap_result() failed: Can't contact LDAP server
Jun 19 11:44:54 weisshorn06 nslcd[3953]: [2dfed6] ldap_result() failed: Can't contact LDAP server
Jun 19 11:44:56 weisshorn04 nslcd[3885]: [bd7a3e] ldap_result() failed: Can't contact LDAP server
Jun 19 11:44:57 weisshorn07 nslcd[4297]: [6928d6] ldap_result() failed: Can't contact LDAP server
Jun 19 11:45:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:45:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:45:17 weisshorn11 nslcd[3954]: [ef532d] ldap_result() failed: Can't contact LDAP server
Jun 19 11:45:17 weisshorn12 nslcd[3954]: [6928d6] ldap_result() failed: Can't contact LDAP server
Jun 19 11:45:19 weisshorn13 nslcd[4224]: [6928d6] ldap_result() failed: Can't contact LDAP server
Jun 19 11:45:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:45:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:45:56 weisshorn05 nslcd[4378]: [7195e5] ldap_result() failed: Can't contact LDAP server
Jun 19 11:46:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:46:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:46:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:46:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:47:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:47:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:47:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:47:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:48:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:48:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:48:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:48:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:49:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:49:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:49:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:49:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:50:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:50:01 weisshorn05 nslcd[4378]: [46a5f2] ldap_result() failed: Can't contact LDAP server
Jun 19 11:50:01 weisshorn05 CROND[123404]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:50:01 weisshorn10 nslcd[4294]: [7195e5] ldap_result() failed: Can't contact LDAP server
Jun 19 11:50:01 weisshorn10 CROND[123432]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:50:01 weisshorn02 CROND[31120]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:50:01 weisshorn13 nslcd[4224]: [7195e5] ldap_result() failed: Can't contact LDAP server
Jun 19 11:50:01 weisshorn13 CROND[126816]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:50:01 weisshorn03 nslcd[3944]: [03a216] ldap_result() failed: Can't contact LDAP server
Jun 19 11:50:01 weisshorn03 CROND[124114]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:50:01 weisshorn09 nslcd[3954]: [90b7c5] ldap_result() failed: Can't contact LDAP server
Jun 19 11:50:01 weisshorn09 CROND[123150]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:50:01 weisshorn04 nslcd[3885]: [90b7c5] ldap_result() failed: Can't contact LDAP server
Jun 19 11:50:01 weisshorn04 CROND[122875]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:50:01 weisshorn11 nslcd[3954]: [d601af] ldap_result() failed: Can't contact LDAP server
Jun 19 11:50:01 weisshorn11 CROND[126430]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:50:01 weisshorn01 CROND[3523]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:50:01 weisshorn08 nslcd[3946]: [7195e5] ldap_result() failed: Can't contact LDAP server
Jun 19 11:50:01 weisshorn08 CROND[122228]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:50:01 weisshorn14 CROND[52051]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:50:01 weisshorn07 nslcd[4297]: [7195e5] ldap_result() failed: Can't contact LDAP server
Jun 19 11:50:01 weisshorn07 CROND[123578]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:50:01 weisshorn12 nslcd[3954]: [7195e5] ldap_result() failed: Can't contact LDAP server
Jun 19 11:50:01 weisshorn12 CROND[123315]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:50:02 weisshorn06 nslcd[3953]: [6ed7f6] ldap_result() failed: Can't contact LDAP server
Jun 19 11:50:02 weisshorn06 CROND[124323]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 11:50:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:50:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:50:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:51:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:51:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:51:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:51:36 weisshorn12 rhnsd[123318]: running program /usr/sbin/rhn_check
Jun 19 11:51:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:52:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:52:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:52:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:52:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:53:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:53:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:53:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:53:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:54:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:54:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:54:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:54:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:54:53 weisshorn01 kernel: Lustre: 4569:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004631:0x16057:0x0]] [valid 68719476736]
Jun 19 11:54:53 weisshorn01 kernel: Lustre: 4569:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 3 previous similar messages
Jun 19 11:55:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:55:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:55:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:55:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:56:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:56:05 weisshorn01 kernel: Lustre: 4784:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x2914:0x0]] [valid 68719476736]
Jun 19 11:56:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:56:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:56:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:57:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:57:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:57:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:57:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:57:52 weisshorn01 kernel: Lustre: 4565:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x292a:0x0]] [valid 68719476736]
Jun 19 11:57:52 weisshorn01 kernel: Lustre: 4565:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 1 previous similar message
Jun 19 11:58:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:58:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:58:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:58:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:59:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:59:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:59:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:59:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 11:59:50 weisshorn01 kernel: Lustre: 5473:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046ee:0x12ed:0x0]] [valid 68719476736]
Jun 19 11:59:50 weisshorn01 kernel: Lustre: 5473:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 3 previous similar messages
Jun 19 12:00:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:00:01 weisshorn06 CROND[124346]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:00:01 weisshorn05 CROND[123462]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:00:01 weisshorn10 CROND[123490]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:00:01 weisshorn02 CROND[32073]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:00:01 weisshorn13 CROND[126845]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:00:01 weisshorn03 CROND[124145]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:00:01 weisshorn09 CROND[123181]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:00:01 weisshorn11 CROND[126459]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:00:01 weisshorn04 CROND[122933]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:00:01 weisshorn01 CROND[3595]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:00:01 weisshorn14 CROND[52086]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:00:01 weisshorn07 CROND[123606]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:00:01 weisshorn08 CROND[122257]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:00:02 weisshorn12 CROND[123346]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:00:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:00:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:00:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:00:46 weisshorn14 nslcd[3437]: [c7d42d] ldap_result() failed: Can't contact LDAP server
Jun 19 12:01:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:01:01 weisshorn12 CROND[123351]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 12:01:01 weisshorn06 CROND[124364]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 12:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1233 starting 0anacron
Jun 19 12:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1243 starting 0anacron
Jun 19 12:01:01 weisshorn05 CROND[123473]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 12:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1243 finished 0anacron
Jun 19 12:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1243 starting mcelog.cron
Jun 19 12:01:01 weisshorn10 CROND[123501]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 12:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1233 finished 0anacron
Jun 19 12:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1233 starting mcelog.cron
Jun 19 12:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1234 starting 0anacron
Jun 19 12:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1243 finished mcelog.cron
Jun 19 12:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1235 starting 0anacron
Jun 19 12:01:01 weisshorn02 CROND[32173]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 12:01:01 weisshorn02 run-parts(/etc/cron.hourly)[3217 starting 0anacron
Jun 19 12:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1235 finished 0anacron
Jun 19 12:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1235 starting mcelog.cron
Jun 19 12:01:01 weisshorn02 run-parts(/etc/cron.hourly)[3218 finished 0anacron
Jun 19 12:01:01 weisshorn02 run-parts(/etc/cron.hourly)[3217 starting mcelog.cron
Jun 19 12:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1233 finished mcelog.cron
Jun 19 12:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1234 finished 0anacron
Jun 19 12:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1234 starting mcelog.cron
Jun 19 12:01:01 weisshorn02 run-parts(/etc/cron.hourly)[3218 finished mcelog.cron
Jun 19 12:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1235 finished mcelog.cron
Jun 19 12:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1234 finished mcelog.cron
Jun 19 12:01:01 weisshorn13 CROND[126850]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 12:01:01 weisshorn03 CROND[124156]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 12:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1241 starting 0anacron
Jun 19 12:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1268 starting 0anacron
Jun 19 12:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1268 finished 0anacron
Jun 19 12:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1241 finished 0anacron
Jun 19 12:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1268 starting mcelog.cron
Jun 19 12:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1241 starting mcelog.cron
Jun 19 12:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1268 finished mcelog.cron
Jun 19 12:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1241 finished mcelog.cron
Jun 19 12:01:01 weisshorn09 CROND[123192]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 12:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1231 starting 0anacron
Jun 19 12:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1232 finished 0anacron
Jun 19 12:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1231 starting mcelog.cron
Jun 19 12:01:01 weisshorn11 CROND[126472]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 12:01:01 weisshorn04 CROND[122944]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 12:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1232 finished mcelog.cron
Jun 19 12:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1229 starting 0anacron
Jun 19 12:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1264 starting 0anacron
Jun 19 12:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1229 finished 0anacron
Jun 19 12:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1229 starting mcelog.cron
Jun 19 12:01:01 weisshorn01 CROND[3603]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 12:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1229 finished mcelog.cron
Jun 19 12:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1264 finished 0anacron
Jun 19 12:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1264 starting mcelog.cron
Jun 19 12:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3603 starting 0anacron
Jun 19 12:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3613 finished 0anacron
Jun 19 12:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3603 starting mcelog.cron
Jun 19 12:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1264 finished mcelog.cron
Jun 19 12:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3620 finished mcelog.cron
Jun 19 12:01:01 weisshorn14 CROND[52092]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 12:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5209 starting 0anacron
Jun 19 12:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5210 finished 0anacron
Jun 19 12:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5209 starting mcelog.cron
Jun 19 12:01:01 weisshorn07 CROND[123624]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 12:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5210 finished mcelog.cron
Jun 19 12:01:01 weisshorn08 CROND[122262]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 12:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1236 starting 0anacron
Jun 19 12:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1222 starting 0anacron
Jun 19 12:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1236 finished 0anacron
Jun 19 12:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1236 starting mcelog.cron
Jun 19 12:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1222 finished 0anacron
Jun 19 12:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1222 starting mcelog.cron
Jun 19 12:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1236 finished mcelog.cron
Jun 19 12:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1222 finished mcelog.cron
Jun 19 12:01:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:01:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:01:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:02:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:02:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:02:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:02:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:03:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:03:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:03:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:03:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:04:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:04:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:04:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:04:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:05:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:05:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:05:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:05:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:05:46 weisshorn10 nslcd[4294]: [46a5f2] ldap_result() failed: Can't contact LDAP server
Jun 19 12:06:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:06:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:06:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:06:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:07:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:07:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:07:19 weisshorn08 ntpd[4445]: synchronized to 148.187.18.145, stratum 2
Jun 19 12:07:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:07:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:08:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:08:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:08:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:08:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:09:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:09:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:09:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:09:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:10:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:10:01 weisshorn08 CROND[122309]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:10:01 weisshorn06 CROND[124408]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:10:01 weisshorn10 CROND[123540]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:10:01 weisshorn02 CROND[33019]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:10:01 weisshorn12 CROND[123428]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:10:01 weisshorn05 CROND[123511]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:10:01 weisshorn03 CROND[124194]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:10:01 weisshorn13 CROND[126928]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:10:01 weisshorn09 CROND[123230]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:10:01 weisshorn04 CROND[122982]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:10:01 weisshorn11 CROND[126515]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:10:01 weisshorn01 CROND[3725]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 12:10:01 weisshorn01 CROND[3728]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:10:01 weisshorn14 CROND[52166]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:10:01 weisshorn05 sshd[123513]: Accepted publickey for root from 148.187.12.231 port 54795 ssh2
Jun 19 12:10:01 weisshorn10 sshd[123542]: Accepted publickey for root from 148.187.12.231 port 37070 ssh2
Jun 19 12:10:01 weisshorn10 sshd[123542]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 12:10:01 weisshorn14 sshd[52163]: Accepted publickey for root from 148.187.12.231 port 49715 ssh2
Jun 19 12:10:01 weisshorn14 sshd[52163]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 12:10:01 weisshorn04 sshd[122984]: Accepted publickey for root from 148.187.12.231 port 44343 ssh2
Jun 19 12:10:01 weisshorn09 sshd[123232]: Accepted publickey for root from 148.187.12.231 port 58039 ssh2
Jun 19 12:10:01 weisshorn05 sshd[123513]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 12:10:01 weisshorn09 sshd[123232]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 12:10:01 weisshorn04 sshd[122984]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 12:10:01 weisshorn13 sshd[126930]: Accepted publickey for root from 148.187.12.231 port 44764 ssh2
Jun 19 12:10:01 weisshorn13 sshd[126930]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 12:10:01 weisshorn06 sshd[124410]: Accepted publickey for root from 148.187.12.231 port 44853 ssh2
Jun 19 12:10:01 weisshorn06 sshd[124410]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 12:10:01 weisshorn03 sshd[124196]: Accepted publickey for root from 148.187.12.231 port 50592 ssh2
Jun 19 12:10:01 weisshorn03 sshd[124196]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 12:10:01 weisshorn14 sshd[52163]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 12:10:01 weisshorn14 sshd[52163]: pam_unix(sshd:session): session closed for user root
Jun 19 12:10:01 weisshorn07 CROND[123700]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:10:01 weisshorn12 sshd[123430]: Accepted publickey for root from 148.187.12.231 port 46498 ssh2
Jun 19 12:10:01 weisshorn08 sshd[122311]: Accepted publickey for root from 148.187.12.231 port 40400 ssh2
Jun 19 12:10:02 weisshorn12 sshd[123430]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 12:10:02 weisshorn08 sshd[122311]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 12:10:02 weisshorn07 sshd[123697]: Accepted publickey for root from 148.187.12.231 port 48403 ssh2
Jun 19 12:10:02 weisshorn10 sshd[123542]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 12:10:02 weisshorn10 sshd[123542]: pam_unix(sshd:session): session closed for user root
Jun 19 12:10:02 weisshorn04 sshd[122984]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 12:10:02 weisshorn04 sshd[122984]: pam_unix(sshd:session): session closed for user root
Jun 19 12:10:02 weisshorn11 sshd[126517]: Accepted publickey for root from 148.187.12.231 port 45284 ssh2
Jun 19 12:10:02 weisshorn07 sshd[123697]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 12:10:02 weisshorn05 sshd[123513]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 12:10:02 weisshorn05 sshd[123513]: pam_unix(sshd:session): session closed for user root
Jun 19 12:10:02 weisshorn09 sshd[123232]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 12:10:02 weisshorn09 sshd[123232]: pam_unix(sshd:session): session closed for user root
Jun 19 12:10:02 weisshorn11 sshd[126517]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 12:10:02 weisshorn06 sshd[124410]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 12:10:02 weisshorn06 sshd[124410]: pam_unix(sshd:session): session closed for user root
Jun 19 12:10:02 weisshorn13 sshd[126930]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 12:10:02 weisshorn13 sshd[126930]: pam_unix(sshd:session): session closed for user root
Jun 19 12:10:02 weisshorn03 sshd[124196]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 12:10:02 weisshorn03 sshd[124196]: pam_unix(sshd:session): session closed for user root
Jun 19 12:10:02 weisshorn12 sshd[123430]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 12:10:02 weisshorn12 sshd[123430]: pam_unix(sshd:session): session closed for user root
Jun 19 12:10:02 weisshorn11 sshd[126517]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 12:10:02 weisshorn11 sshd[126517]: pam_unix(sshd:session): session closed for user root
Jun 19 12:10:02 weisshorn07 sshd[123697]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 12:10:02 weisshorn07 sshd[123697]: pam_unix(sshd:session): session closed for user root
Jun 19 12:10:02 weisshorn08 sshd[122311]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 12:10:02 weisshorn08 sshd[122311]: pam_unix(sshd:session): session closed for user root
Jun 19 12:10:02 weisshorn01 postfix/pickup[2885]: 5483592224F: uid=0 from=<root>
Jun 19 12:10:02 weisshorn01 postfix/cleanup[3758]: 5483592224F: message-id=<20120619101002.5483592224F@weisshorn01.admin.cscs.ch>
Jun 19 12:10:02 weisshorn01 postfix/qmgr[2710]: 5483592224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 12:10:02 weisshorn01 postfix/local[3760]: 5483592224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=0.44, delays=0.39/0.02/0/0.04, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 12:10:02 weisshorn01 postfix/qmgr[2710]: 5483592224F: removed
Jun 19 12:10:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:10:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:10:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:11:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:11:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:11:17 weisshorn03 ntpd[4444]: synchronized to 148.187.18.145, stratum 2
Jun 19 12:11:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:11:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:12:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:12:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:12:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:12:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:13:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:13:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:13:28 weisshorn01 kernel: Lustre: 4684:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046e5:0xdbc9:0x0]] [valid 68719476736]
Jun 19 12:13:28 weisshorn01 kernel: Lustre: 4684:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 2 previous similar messages
Jun 19 12:13:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:13:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:14:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:14:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:14:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:14:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:15:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:15:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:15:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:15:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:16:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:16:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:16:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:16:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:17:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:17:15 weisshorn10 ntpd[4792]: synchronized to 148.187.18.145, stratum 2
Jun 19 12:17:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:17:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:17:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:18:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:18:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:18:15 weisshorn01 kernel: Lustre: 5301:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004631:0x16761:0x0]] [valid 68719476736]
Jun 19 12:18:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:18:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:19:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:19:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:19:22 weisshorn08 rhnsd[122388]: running program /usr/sbin/rhn_check
Jun 19 12:19:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:19:30 weisshorn01 kernel: Lustre: 5145:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004631:0x16767:0x0]] [valid 68719476736]
Jun 19 12:19:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:20:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:20:01 weisshorn07 CROND[123738]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:20:01 weisshorn08 CROND[122391]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:20:01 weisshorn06 CROND[124483]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:20:01 weisshorn01 CROND[3840]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:20:01 weisshorn02 CROND[33936]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:20:01 weisshorn12 CROND[123476]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:20:01 weisshorn05 CROND[123564]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:20:01 weisshorn10 CROND[123594]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:20:01 weisshorn03 CROND[124275]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:20:01 weisshorn09 CROND[123310]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:20:01 weisshorn13 CROND[126979]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:20:01 weisshorn04 CROND[123035]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:20:01 weisshorn11 CROND[126590]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:20:01 weisshorn14 CROND[52215]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:20:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:20:26 weisshorn01 rhnsd[3841]: running program /usr/sbin/rhn_check
Jun 19 12:20:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:20:35 weisshorn06 kernel: Lustre: 7764:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340101142/real 1340101142]  req@ffff8807e4e60800 x1403849889591921/t0(0) o106->scratch-OST001d@6@gni:15/16 lens 296/232 e 0 to 1 dl 1340101235 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jun 19 12:20:35 weisshorn10 kernel: Lustre: 6224:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340101142/real 1340101142]  req@ffff880996e42800 x1403849933940394/t0(0) o106->scratch-OST0022@6@gni:15/16 lens 296/232 e 0 to 1 dl 1340101235 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jun 19 12:20:35 weisshorn08 kernel: Lustre: 8208:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340101142/real 1340101142]  req@ffff880357d62800 x1403849816093256/t0(0) o106->scratch-OST0017@6@gni:15/16 lens 296/232 e 0 to 1 dl 1340101235 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jun 19 12:20:35 weisshorn10 kernel: Lustre: 6224:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 43 previous similar messages
Jun 19 12:20:35 weisshorn04 kernel: Lustre: 8262:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340101142/real 1340101142]  req@ffff880b068fa000 x1403849855890316/t0(0) o106->scratch-OST0031@6@gni:15/16 lens 296/232 e 0 to 1 dl 1340101235 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jun 19 12:20:35 weisshorn04 kernel: Lustre: 8262:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 37 previous similar messages
Jun 19 12:20:35 weisshorn13 kernel: Lustre: 8592:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340101142/real 1340101142]  req@ffff8801bf03a800 x1403849965509747/t0(0) o106->scratch-OST001e@6@gni:15/16 lens 296/232 e 0 to 1 dl 1340101235 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jun 19 12:20:35 weisshorn09 kernel: Lustre: 8320:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340101142/real 1340101142]  req@ffff880d794ccc00 x1403849914423017/t0(0) o106->scratch-OST0010@6@gni:15/16 lens 296/232 e 0 to 1 dl 1340101235 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jun 19 12:20:35 weisshorn13 kernel: Lustre: 8592:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 183 previous similar messages
Jun 19 12:20:35 weisshorn09 kernel: Lustre: 8320:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 36 previous similar messages
Jun 19 12:20:35 weisshorn07 kernel: Lustre: 6301:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340101142/real 1340101142]  req@ffff8805a0d73400 x1403849864753396/t0(0) o106->scratch-OST001f@6@gni:15/16 lens 296/232 e 0 to 1 dl 1340101235 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jun 19 12:20:35 weisshorn07 kernel: Lustre: 6301:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 72 previous similar messages
Jun 19 12:20:35 weisshorn12 kernel: Lustre: 8184:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340101142/real 1340101142]  req@ffff8801d25dc400 x1403849942697836/t0(0) o106->scratch-OST0032@6@gni:15/16 lens 296/232 e 0 to 1 dl 1340101235 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jun 19 12:20:35 weisshorn12 kernel: Lustre: 8184:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 37 previous similar messages
Jun 19 12:20:35 weisshorn03 kernel: Lustre: 7746:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340101142/real 1340101142]  req@ffff880b6616b000 x1403849863799571/t0(0) o106->scratch-OST002c@6@gni:15/16 lens 296/232 e 0 to 1 dl 1340101235 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jun 19 12:20:35 weisshorn05 kernel: Lustre: 7980:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340101142/real 1340101142]  req@ffff8803e2302800 x1403849626237056/t0(0) o106->scratch-OST0027@6@gni:15/16 lens 296/232 e 0 to 1 dl 1340101235 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jun 19 12:20:35 weisshorn03 kernel: Lustre: 7746:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 36 previous similar messages
Jun 19 12:20:35 weisshorn05 kernel: Lustre: 7980:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 41 previous similar messages
Jun 19 12:20:35 weisshorn11 kernel: Lustre: 6030:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340101142/real 1340101142]  req@ffff880803e5a400 x1403849893611355/t0(0) o106->scratch-OST0021@6@gni:15/16 lens 296/232 e 0 to 1 dl 1340101235 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jun 19 12:20:35 weisshorn11 kernel: Lustre: 6030:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Jun 19 12:20:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:21:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:21:13 weisshorn01 kernel: Lustre: 4718:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046ee:0x12fe:0x0]] [valid 68719476736]
Jun 19 12:21:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:21:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:21:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:22:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:22:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:22:22 weisshorn13 kernel: LNet: Service thread pid 8132 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Jun 19 12:22:22 weisshorn13 kernel: LNet: Skipped 1 previous similar message
Jun 19 12:22:22 weisshorn13 kernel: Pid: 8132, comm: ll_ost_io_332
Jun 19 12:22:22 weisshorn13 kernel:
Jun 19 12:22:22 weisshorn13 kernel: Call Trace:
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffff8107bf8c>] ? lock_timer_base+0x3c/0x70
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffff814edc52>] schedule_timeout+0x192/0x2e0
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffff8107c0a0>] ? process_timeout+0x0/0x10
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffffa03a55c1>] cfs_waitq_timedwait+0x11/0x20 [libcfs]
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffffa08780ad>] target_bulk_io+0x38d/0x8b0 [ptlrpc]
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffff8105e7f0>] ? default_wake_function+0x0/0x20
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffffa0d15792>] ost_brw_write+0x1172/0x1380 [ost]
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffff81263133>] ? cfq_free_io_context+0x13/0x40
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffffa086f4a0>] ? target_bulk_timeout+0x0/0x80 [ptlrpc]
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffffa0d197c4>] ost_handle+0x2764/0x39e0 [ost]
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffffa08a5af4>] ? lustre_msg_get_transno+0x54/0x90 [ptlrpc]
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffffa08abb15>] ? ptlrpc_update_export_timer+0x55/0x360 [ptlrpc]
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffffa08b13c1>] ptlrpc_server_handle_request+0x3c1/0xcb0 [ptlrpc]
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffffa03a54ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffffa03afef9>] ? lc_watchdog_touch+0x79/0x110 [libcfs]
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffffa08ab462>] ? ptlrpc_wait_event+0xb2/0x2c0 [ptlrpc]
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffff810519c3>] ? __wake_up+0x53/0x70
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffffa08b23cf>] ptlrpc_main+0x71f/0x1210 [ptlrpc]
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffffa08b1cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffffa08b1cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffffa08b1cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 12:22:22 weisshorn13 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Jun 19 12:22:22 weisshorn13 kernel:
Jun 19 12:22:22 weisshorn13 kernel: LustreError: dumping log to /tmp/lustre-log.1340101342.8132
Jun 19 12:22:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:22:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:23:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:23:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:23:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:23:45 weisshorn01 kernel: Lustre: 4724:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046ee:0x1307:0x0]] [valid 68719476736]
Jun 19 12:23:45 weisshorn01 kernel: Lustre: 4724:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 2 previous similar messages
Jun 19 12:23:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:24:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:24:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:24:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:24:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:25:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:25:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:25:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:25:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:26:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:26:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:26:21 weisshorn03 nslcd[3944]: [b8a08e] ldap_result() failed: Can't contact LDAP server
Jun 19 12:26:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:26:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:27:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:27:05 weisshorn02 rhnsd[34622]: running program /usr/sbin/rhn_check
Jun 19 12:27:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:27:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:27:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:28:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:28:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:28:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:28:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:29:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:29:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:29:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:29:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:30:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:30:01 weisshorn08 CROND[122420]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:30:01 weisshorn07 CROND[123780]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:30:01 weisshorn01 CROND[3921]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:30:01 weisshorn06 CROND[124520]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:30:01 weisshorn02 CROND[34895]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:30:01 weisshorn12 CROND[123506]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:30:01 weisshorn10 CROND[123659]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:30:01 weisshorn05 CROND[123629]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:30:01 weisshorn03 CROND[124314]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:30:01 weisshorn09 CROND[123348]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:30:01 weisshorn04 CROND[123100]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:30:01 weisshorn13 CROND[127012]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:30:01 weisshorn11 CROND[126628]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:30:01 weisshorn14 CROND[52250]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:30:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:30:26 weisshorn05 rhnsd[123632]: running program /usr/sbin/rhn_check
Jun 19 12:30:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:30:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:31:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:31:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:31:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:31:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:31:46 weisshorn02 nslcd[2099]: [06328b] ldap_result() failed: Can't contact LDAP server
Jun 19 12:31:53 weisshorn01 kernel: Lustre: 5482:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046ee:0x1312:0x0]] [valid 68719476736]
Jun 19 12:31:53 weisshorn01 kernel: Lustre: 5482:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 1 previous similar message
Jun 19 12:32:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:32:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:32:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:32:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:33:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:33:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:33:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:33:33 weisshorn13 kernel: Lustre: 8149:0:(service.c:1034:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-271), not sending early reply
Jun 19 12:33:33 weisshorn13 kernel:  req@ffff880c72523c00 x1403846443260526/t0(0) o4->b1f230ec-3313-9236-1eb0-f2dfcd7a1f42@6@gni:0/0 lens 448/416 e 3 to 0 dl 1340102018 ref 2 fl Interpret:/0/0 rc 0/0
Jun 19 12:33:38 weisshorn13 kernel: LustreError: 8132:0:(ldlm_lib.c:2697:target_bulk_io()) @@@ timeout on bulk GET after 876+0s  req@ffff880c72523c00 x1403846443260526/t0(0) o4->b1f230ec-3313-9236-1eb0-f2dfcd7a1f42@6@gni:0/0 lens 448/416 e 3 to 0 dl 1340102018 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 12:33:38 weisshorn13 kernel: LustreError: 8132:0:(ldlm_lib.c:2697:target_bulk_io()) Skipped 1 previous similar message
Jun 19 12:33:38 weisshorn13 kernel: Lustre: scratch-OST0029: Bulk IO write error with b1f230ec-3313-9236-1eb0-f2dfcd7a1f42 (at 6@gni), client will retry: rc -110
Jun 19 12:33:38 weisshorn13 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 12:33:38 weisshorn13 kernel: LNet: Service thread pid 8132 completed after 876.01s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Jun 19 12:33:38 weisshorn13 kernel: LNet: Skipped 1 previous similar message
Jun 19 12:33:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:34:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:34:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:34:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:34:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:35:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:35:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:35:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:35:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:36:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:36:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:36:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:36:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:36:46 weisshorn13 kernel: Lustre: scratch-OST0029: Client b1f230ec-3313-9236-1eb0-f2dfcd7a1f42 (at 6@gni) reconnecting
Jun 19 12:36:46 weisshorn13 kernel: Lustre: Skipped 1 previous similar message
Jun 19 12:37:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:37:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:37:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:37:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:38:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:38:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:38:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:38:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:39:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:39:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:39:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:39:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:40:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:40:01 weisshorn13 CROND[127078]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:40:01 weisshorn11 CROND[126657]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:40:01 weisshorn14 nslcd[3437]: [2167ad] ldap_result() failed: Can't contact LDAP server
Jun 19 12:40:01 weisshorn14 CROND[52310]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:40:01 weisshorn07 CROND[123840]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:40:01 weisshorn08 CROND[122452]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:40:01 weisshorn01 CROND[4044]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:40:01 weisshorn06 CROND[124550]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:40:01 weisshorn02 nslcd[2099]: [2bbc16] ldap_result() failed: Can't contact LDAP server
Jun 19 12:40:01 weisshorn10 CROND[123683]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:40:01 weisshorn02 CROND[35820]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:40:01 weisshorn12 CROND[123570]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:40:01 weisshorn05 CROND[123654]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:40:01 weisshorn03 CROND[124338]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:40:01 weisshorn09 CROND[123372]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:40:01 weisshorn04 CROND[123124]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:40:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:40:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:40:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:41:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:41:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:41:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:41:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:42:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:42:09 weisshorn01 kernel: Lustre: 4734:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046ee:0x135b:0x0]] [valid 68719476736]
Jun 19 12:42:09 weisshorn01 kernel: Lustre: 4734:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 41 previous similar messages
Jun 19 12:42:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:42:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:42:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:43:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:43:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:43:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:43:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:44:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:44:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:44:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:44:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:44:45 weisshorn09 nslcd[3954]: [03a216] ldap_result() failed: Can't contact LDAP server
Jun 19 12:45:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:45:08 weisshorn08 nslcd[3946]: [46a5f2] ldap_result() failed: Can't contact LDAP server
Jun 19 12:45:09 weisshorn06 nslcd[3953]: [9df57d] ldap_result() failed: Can't contact LDAP server
Jun 19 12:45:11 weisshorn04 nslcd[3885]: [03a216] ldap_result() failed: Can't contact LDAP server
Jun 19 12:45:12 weisshorn07 nslcd[4297]: [46a5f2] ldap_result() failed: Can't contact LDAP server
Jun 19 12:45:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:45:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:45:32 weisshorn11 nslcd[3954]: [c77402] ldap_result() failed: Can't contact LDAP server
Jun 19 12:45:32 weisshorn12 nslcd[3954]: [46a5f2] ldap_result() failed: Can't contact LDAP server
Jun 19 12:45:34 weisshorn13 nslcd[4224]: [46a5f2] ldap_result() failed: Can't contact LDAP server
Jun 19 12:45:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:46:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:46:11 weisshorn05 nslcd[4378]: [2dfed6] ldap_result() failed: Can't contact LDAP server
Jun 19 12:46:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:46:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:46:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:47:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:47:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:47:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:47:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:48:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:48:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:48:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:48:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:49:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:49:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:49:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:49:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:50:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:50:01 weisshorn04 CROND[123192]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:50:01 weisshorn11 CROND[126717]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:50:01 weisshorn13 CROND[127109]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:50:01 weisshorn14 CROND[52339]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:50:01 weisshorn07 CROND[123860]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:50:01 weisshorn08 CROND[122512]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:50:01 weisshorn01 CROND[4106]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:50:01 weisshorn06 CROND[124605]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:50:01 weisshorn02 CROND[36742]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:50:01 weisshorn10 CROND[123715]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:50:01 weisshorn12 CROND[123603]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:50:01 weisshorn03 CROND[124402]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:50:01 weisshorn05 CROND[123722]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:50:02 weisshorn13 nslcd[4224]: [2dfed6] ldap_result() failed: Can't contact LDAP server
Jun 19 12:50:02 weisshorn09 CROND[123432]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 12:50:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:50:16 weisshorn10 nslcd[4294]: [2dfed6] ldap_result() failed: Can't contact LDAP server
Jun 19 12:50:16 weisshorn11 nslcd[3954]: [b5b1c6] ldap_result() failed: Can't contact LDAP server
Jun 19 12:50:16 weisshorn12 nslcd[3954]: [2dfed6] ldap_result() failed: Can't contact LDAP server
Jun 19 12:50:16 weisshorn03 nslcd[3944]: [b59621] ldap_result() failed: Can't contact LDAP server
Jun 19 12:50:16 weisshorn05 nslcd[4378]: [6ed7f6] ldap_result() failed: Can't contact LDAP server
Jun 19 12:50:16 weisshorn04 nslcd[3885]: [b8a08e] ldap_result() failed: Can't contact LDAP server
Jun 19 12:50:16 weisshorn09 nslcd[3954]: [b8a08e] ldap_result() failed: Can't contact LDAP server
Jun 19 12:50:16 weisshorn07 nslcd[4297]: [2dfed6] ldap_result() failed: Can't contact LDAP server
Jun 19 12:50:16 weisshorn08 nslcd[3946]: [2dfed6] ldap_result() failed: Can't contact LDAP server
Jun 19 12:50:17 weisshorn06 nslcd[3953]: [bd7a3e] ldap_result() failed: Can't contact LDAP server
Jun 19 12:50:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:50:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:51:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:51:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:51:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:51:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:52:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:52:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:52:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:52:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:53:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:53:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:53:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:53:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:54:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:54:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:54:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:54:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:55:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:55:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:55:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:55:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:56:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:56:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:56:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:56:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:56:59 weisshorn09 rhnsd[123465]: running program /usr/sbin/rhn_check
Jun 19 12:57:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:57:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:57:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:57:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:58:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:58:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:58:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:58:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:59:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:59:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:59:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 12:59:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:00:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:00:01 weisshorn05 CROND[123752]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:00:01 weisshorn09 CROND[123477]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:00:01 weisshorn04 CROND[123222]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:00:01 weisshorn11 CROND[126750]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:00:01 weisshorn13 CROND[127144]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:00:01 weisshorn14 CROND[52379]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:00:01 weisshorn07 CROND[123937]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:00:01 weisshorn08 CROND[122547]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:00:01 weisshorn01 CROND[4205]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:00:01 weisshorn06 CROND[124642]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:00:01 weisshorn02 CROND[37700]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:00:01 weisshorn10 CROND[123780]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:00:01 weisshorn12 CROND[123633]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:00:01 weisshorn03 CROND[124435]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:00:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:00:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:00:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:01:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:01:01 weisshorn05 CROND[123757]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 13:01:01 weisshorn09 CROND[123482]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 13:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1234 starting 0anacron
Jun 19 13:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1234 finished 0anacron
Jun 19 13:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1234 starting mcelog.cron
Jun 19 13:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1237 starting 0anacron
Jun 19 13:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1234 finished mcelog.cron
Jun 19 13:01:01 weisshorn04 CROND[123227]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 13:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1237 finished 0anacron
Jun 19 13:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1237 starting mcelog.cron
Jun 19 13:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1232 starting 0anacron
Jun 19 13:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1232 finished 0anacron
Jun 19 13:01:01 weisshorn11 CROND[126755]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 13:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1232 starting mcelog.cron
Jun 19 13:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1267 starting 0anacron
Jun 19 13:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1267 finished 0anacron
Jun 19 13:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1267 starting mcelog.cron
Jun 19 13:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1232 finished mcelog.cron
Jun 19 13:01:01 weisshorn14 CROND[52384]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 13:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1237 finished mcelog.cron
Jun 19 13:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5238 starting 0anacron
Jun 19 13:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5239 finished 0anacron
Jun 19 13:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5238 starting mcelog.cron
Jun 19 13:01:01 weisshorn13 CROND[127149]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 13:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1267 finished mcelog.cron
Jun 19 13:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5240 finished mcelog.cron
Jun 19 13:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1271 starting 0anacron
Jun 19 13:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1271 finished 0anacron
Jun 19 13:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1271 starting mcelog.cron
Jun 19 13:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1271 finished mcelog.cron
Jun 19 13:01:01 weisshorn07 CROND[123942]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 13:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1239 starting 0anacron
Jun 19 13:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1239 finished 0anacron
Jun 19 13:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1239 starting mcelog.cron
Jun 19 13:01:01 weisshorn08 CROND[122552]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 13:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1239 finished mcelog.cron
Jun 19 13:01:01 weisshorn01 CROND[4212]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 13:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4212 starting 0anacron
Jun 19 13:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1225 starting 0anacron
Jun 19 13:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4222 finished 0anacron
Jun 19 13:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4212 starting mcelog.cron
Jun 19 13:01:01 weisshorn06 CROND[124647]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 13:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1246 starting 0anacron
Jun 19 13:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4229 finished mcelog.cron
Jun 19 13:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1225 finished 0anacron
Jun 19 13:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1246 finished 0anacron
Jun 19 13:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1225 starting mcelog.cron
Jun 19 13:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1246 starting mcelog.cron
Jun 19 13:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1246 finished mcelog.cron
Jun 19 13:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1225 finished mcelog.cron
Jun 19 13:01:01 weisshorn10 CROND[123785]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 13:01:01 weisshorn14 nslcd[3437]: [1b64d4] ldap_result() failed: Can't contact LDAP server
Jun 19 13:01:01 weisshorn02 CROND[37794]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 13:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1237 starting 0anacron
Jun 19 13:01:01 weisshorn02 run-parts(/etc/cron.hourly)[3779 starting 0anacron
Jun 19 13:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1237 finished 0anacron
Jun 19 13:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1237 starting mcelog.cron
Jun 19 13:01:01 weisshorn12 CROND[123638]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 13:01:01 weisshorn02 run-parts(/etc/cron.hourly)[3780 finished 0anacron
Jun 19 13:01:01 weisshorn02 run-parts(/etc/cron.hourly)[3779 starting mcelog.cron
Jun 19 13:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1238 finished mcelog.cron
Jun 19 13:01:01 weisshorn02 run-parts(/etc/cron.hourly)[3781 finished mcelog.cron
Jun 19 13:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1236 starting 0anacron
Jun 19 13:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1236 finished 0anacron
Jun 19 13:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1236 starting mcelog.cron
Jun 19 13:01:01 weisshorn03 CROND[124440]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 13:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1236 finished mcelog.cron
Jun 19 13:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1244 starting 0anacron
Jun 19 13:01:02 weisshorn03 run-parts(/etc/cron.hourly)[1244 finished 0anacron
Jun 19 13:01:02 weisshorn03 run-parts(/etc/cron.hourly)[1244 starting mcelog.cron
Jun 19 13:01:02 weisshorn03 run-parts(/etc/cron.hourly)[1244 finished mcelog.cron
Jun 19 13:01:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:01:26 weisshorn01 kernel: Lustre: 4774:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004631:0x16780:0x0]] [valid 68719476736]
Jun 19 13:01:26 weisshorn01 kernel: Lustre: 4774:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 18 previous similar messages
Jun 19 13:01:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:01:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:02:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:02:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:02:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:02:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:03:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:03:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:03:30 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:03:45 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:04:00 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:04:15 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:04:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:04:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:05:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:05:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:05:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:05:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:06:01 weisshorn10 nslcd[4294]: [6ed7f6] ldap_result() failed: Can't contact LDAP server
Jun 19 13:06:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:06:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:06:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:06:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:07:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:07:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:07:26 weisshorn11 ntpd[4454]: synchronized to 148.187.18.120, stratum 2
Jun 19 13:07:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:07:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:08:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:08:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:08:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:08:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:09:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:09:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:09:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:09:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:10:01 weisshorn12 CROND[123711]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:10:01 weisshorn03 CROND[124478]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:10:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:10:01 weisshorn09 CROND[123515]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:10:01 weisshorn04 CROND[123265]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:10:01 weisshorn05 CROND[123795]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:10:01 weisshorn11 CROND[126799]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:10:01 weisshorn14 CROND[52452]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:10:01 weisshorn13 CROND[127222]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:10:01 weisshorn07 CROND[123981]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:10:01 weisshorn06 CROND[124691]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:10:01 weisshorn01 CROND[4290]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:10:01 weisshorn01 CROND[4291]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 13:10:01 weisshorn08 CROND[122593]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:10:01 weisshorn10 CROND[123826]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:10:01 weisshorn02 CROND[38635]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:10:02 weisshorn14 sshd[52454]: Accepted publickey for root from 148.187.12.231 port 49728 ssh2
Jun 19 13:10:02 weisshorn14 sshd[52454]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 13:10:02 weisshorn12 sshd[123713]: Accepted publickey for root from 148.187.12.231 port 46518 ssh2
Jun 19 13:10:02 weisshorn04 sshd[123267]: Accepted publickey for root from 148.187.12.231 port 44371 ssh2
Jun 19 13:10:02 weisshorn12 sshd[123713]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 13:10:02 weisshorn04 sshd[123267]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 13:10:02 weisshorn11 sshd[126801]: Accepted publickey for root from 148.187.12.231 port 45298 ssh2
Jun 19 13:10:02 weisshorn11 sshd[126801]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 13:10:02 weisshorn03 sshd[124480]: Accepted publickey for root from 148.187.12.231 port 50620 ssh2
Jun 19 13:10:02 weisshorn06 sshd[124693]: Accepted publickey for root from 148.187.12.231 port 44876 ssh2
Jun 19 13:10:02 weisshorn06 sshd[124693]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 13:10:02 weisshorn03 sshd[124480]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 13:10:02 weisshorn10 sshd[123823]: Accepted publickey for root from 148.187.12.231 port 37088 ssh2
Jun 19 13:10:02 weisshorn10 sshd[123823]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 13:10:02 weisshorn09 sshd[123517]: Accepted publickey for root from 148.187.12.231 port 58061 ssh2
Jun 19 13:10:02 weisshorn09 sshd[123517]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 13:10:02 weisshorn14 sshd[52454]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 13:10:02 weisshorn14 sshd[52454]: pam_unix(sshd:session): session closed for user root
Jun 19 13:10:02 weisshorn05 sshd[123797]: Accepted publickey for root from 148.187.12.231 port 54812 ssh2
Jun 19 13:10:02 weisshorn05 sshd[123797]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 13:10:02 weisshorn04 sshd[123267]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 13:10:02 weisshorn04 sshd[123267]: pam_unix(sshd:session): session closed for user root
Jun 19 13:10:02 weisshorn11 sshd[126801]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 13:10:02 weisshorn11 sshd[126801]: pam_unix(sshd:session): session closed for user root
Jun 19 13:10:02 weisshorn06 sshd[124693]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 13:10:02 weisshorn06 sshd[124693]: pam_unix(sshd:session): session closed for user root
Jun 19 13:10:02 weisshorn07 sshd[123983]: Accepted publickey for root from 148.187.12.231 port 48419 ssh2
Jun 19 13:10:02 weisshorn08 sshd[122595]: Accepted publickey for root from 148.187.12.231 port 40416 ssh2
Jun 19 13:10:02 weisshorn12 sshd[123713]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 13:10:02 weisshorn12 sshd[123713]: pam_unix(sshd:session): session closed for user root
Jun 19 13:10:02 weisshorn08 sshd[122595]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 13:10:02 weisshorn07 sshd[123983]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 13:10:02 weisshorn13 sshd[127224]: Accepted publickey for root from 148.187.12.231 port 44777 ssh2
Jun 19 13:10:02 weisshorn03 sshd[124480]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 13:10:02 weisshorn03 sshd[124480]: pam_unix(sshd:session): session closed for user root
Jun 19 13:10:02 weisshorn13 sshd[127224]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 13:10:02 weisshorn09 sshd[123517]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 13:10:02 weisshorn09 sshd[123517]: pam_unix(sshd:session): session closed for user root
Jun 19 13:10:02 weisshorn10 sshd[123823]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 13:10:02 weisshorn10 sshd[123823]: pam_unix(sshd:session): session closed for user root
Jun 19 13:10:02 weisshorn07 sshd[123983]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 13:10:02 weisshorn07 sshd[123983]: pam_unix(sshd:session): session closed for user root
Jun 19 13:10:02 weisshorn05 sshd[123797]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 13:10:02 weisshorn05 sshd[123797]: pam_unix(sshd:session): session closed for user root
Jun 19 13:10:02 weisshorn08 sshd[122595]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 13:10:02 weisshorn08 sshd[122595]: pam_unix(sshd:session): session closed for user root
Jun 19 13:10:02 weisshorn13 sshd[127224]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 13:10:02 weisshorn13 sshd[127224]: pam_unix(sshd:session): session closed for user root
Jun 19 13:10:02 weisshorn01 postfix/pickup[3907]: 7288792224F: uid=0 from=<root>
Jun 19 13:10:02 weisshorn01 postfix/cleanup[4323]: 7288792224F: message-id=<20120619111002.7288792224F@weisshorn01.admin.cscs.ch>
Jun 19 13:10:02 weisshorn01 postfix/qmgr[2710]: 7288792224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 13:10:02 weisshorn01 postfix/local[4325]: 7288792224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=0.49, delays=0.44/0.02/0/0.03, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 13:10:02 weisshorn01 postfix/qmgr[2710]: 7288792224F: removed
Jun 19 13:10:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:10:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:10:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:11:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:11:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:11:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:11:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:12:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:12:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:12:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:12:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:13:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:13:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:13:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:13:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:14:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:14:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:14:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:14:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:15:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:15:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:15:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:15:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:16:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:16:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:16:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:16:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:17:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:17:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:17:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:17:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:18:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:18:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:18:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:18:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:19:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:19:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:19:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:19:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:20:01 weisshorn10 CROND[123912]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:20:01 weisshorn02 CROND[39554]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:20:01 weisshorn12 CROND[123770]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:20:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:20:01 weisshorn03 CROND[124563]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:20:01 weisshorn04 CROND[123353]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:20:01 weisshorn09 CROND[123600]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:20:01 weisshorn14 CROND[52509]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:20:01 weisshorn05 CROND[123883]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:20:01 weisshorn11 CROND[126885]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:20:01 weisshorn01 CROND[4398]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:20:01 weisshorn13 CROND[127279]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:20:01 weisshorn07 CROND[124021]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:20:01 weisshorn06 CROND[124771]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:20:02 weisshorn08 CROND[122678]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:20:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:20:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:20:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:21:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:21:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:21:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:21:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:22:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:22:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:22:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:22:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:22:48 weisshorn07 ntpd[4804]: synchronized to 148.187.18.120, stratum 2
Jun 19 13:22:50 weisshorn01 kernel: Lustre: 4699:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x294f:0x0]] [valid 68719476736]
Jun 19 13:23:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:23:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:23:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:23:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:24:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:24:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:24:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:24:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:25:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:25:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:25:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:25:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:26:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:26:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:26:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:26:31 weisshorn06 ntpd[4454]: synchronized to 148.187.18.120, stratum 2
Jun 19 13:26:36 weisshorn03 nslcd[3944]: [92a02e] ldap_result() failed: Can't contact LDAP server
Jun 19 13:26:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:27:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:27:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:27:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:27:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:28:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:28:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:28:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:28:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:29:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:29:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:29:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:29:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:30:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:30:01 weisshorn10 CROND[123943]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:30:01 weisshorn02 CROND[40512]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:30:01 weisshorn08 CROND[122709]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:30:01 weisshorn03 CROND[124598]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:30:01 weisshorn12 CROND[123795]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:30:01 weisshorn04 CROND[123384]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:30:01 weisshorn09 CROND[123639]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:30:01 weisshorn14 CROND[52543]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:30:01 weisshorn05 CROND[123914]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:30:01 weisshorn11 CROND[126913]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:30:01 weisshorn01 CROND[4505]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:30:01 weisshorn13 CROND[127309]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:30:01 weisshorn07 CROND[124100]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:30:02 weisshorn06 CROND[124805]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:30:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:30:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:30:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:31:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:31:05 weisshorn01 kernel: Lustre: 5316:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046e5:0xede1:0x0]] [valid 68719476736]
Jun 19 13:31:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:31:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:31:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:32:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:32:01 weisshorn02 nslcd[2099]: [6d98c2] ldap_result() failed: Can't contact LDAP server
Jun 19 13:32:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:32:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:32:35 weisshorn08 ntpd[4445]: synchronized to 148.187.18.120, stratum 2
Jun 19 13:32:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:33:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:33:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:33:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:33:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:34:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:34:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:34:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:34:38 weisshorn01 kernel: LustreError: 42589:0:(mdt_recovery.c:813:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 52534411515, new: 52534411514 replay: 0. see LU-617.
Jun 19 13:34:38 weisshorn01 kernel: LustreError: 42589:0:(mdt_recovery.c:813:mdt_last_rcvd_update()) Skipped 1 previous similar message
Jun 19 13:34:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:35:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:35:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:35:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:35:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:36:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:36:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:36:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:36:44 weisshorn05 ntpd[4877]: synchronized to 148.187.18.120, stratum 2
Jun 19 13:36:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:36:55 weisshorn13 ntpd[4726]: synchronized to 148.187.18.120, stratum 2
Jun 19 13:37:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:37:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:37:30 weisshorn02 ntpd[2541]: synchronized to 148.187.18.120, stratum 2
Jun 19 13:37:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:37:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:38:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:38:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:38:23 weisshorn01 kernel: Lustre: 4699:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046e5:0xee21:0x0]] [valid 68719476736]
Jun 19 13:38:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:38:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:39:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:39:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:39:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:39:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:40:01 weisshorn06 CROND[124835]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:40:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:40:01 weisshorn02 CROND[41445]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:40:01 weisshorn10 CROND[123975]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:40:01 weisshorn08 CROND[122772]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:40:01 weisshorn03 CROND[124657]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:40:01 weisshorn12 CROND[123854]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:40:01 weisshorn04 CROND[123416]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:40:01 weisshorn09 CROND[123693]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:40:01 weisshorn14 CROND[52597]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:40:01 weisshorn05 CROND[123947]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:40:01 weisshorn11 CROND[126942]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:40:01 weisshorn01 CROND[4614]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:40:01 weisshorn13 CROND[127369]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:40:02 weisshorn07 CROND[124125]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:40:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:40:16 weisshorn02 nslcd[2099]: [09d4a1] ldap_result() failed: Can't contact LDAP server
Jun 19 13:40:16 weisshorn14 nslcd[3437]: [b5e776] ldap_result() failed: Can't contact LDAP server
Jun 19 13:40:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:40:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:41:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:41:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:41:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:41:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:42:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:42:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:42:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:42:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:43:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:43:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:43:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:43:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:44:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:44:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:44:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:44:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:45:00 weisshorn09 nslcd[3954]: [b59621] ldap_result() failed: Can't contact LDAP server
Jun 19 13:45:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:45:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:45:23 weisshorn08 nslcd[3946]: [6ed7f6] ldap_result() failed: Can't contact LDAP server
Jun 19 13:45:24 weisshorn06 nslcd[3953]: [90b7c5] ldap_result() failed: Can't contact LDAP server
Jun 19 13:45:26 weisshorn04 nslcd[3885]: [b59621] ldap_result() failed: Can't contact LDAP server
Jun 19 13:45:27 weisshorn07 nslcd[4297]: [6ed7f6] ldap_result() failed: Can't contact LDAP server
Jun 19 13:45:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:45:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:45:47 weisshorn11 nslcd[3954]: [6be6b9] ldap_result() failed: Can't contact LDAP server
Jun 19 13:45:47 weisshorn12 nslcd[3954]: [6ed7f6] ldap_result() failed: Can't contact LDAP server
Jun 19 13:45:49 weisshorn13 nslcd[4224]: [6ed7f6] ldap_result() failed: Can't contact LDAP server
Jun 19 13:46:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:46:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:46:26 weisshorn05 nslcd[4378]: [9df57d] ldap_result() failed: Can't contact LDAP server
Jun 19 13:46:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:46:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:47:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:47:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:47:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:47:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:48:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:48:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:48:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:48:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:49:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:49:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:49:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:49:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:50:01 weisshorn13 CROND[127411]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:50:01 weisshorn07 CROND[124155]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:50:01 weisshorn06 CROND[124895]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:50:01 weisshorn02 CROND[42397]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:50:01 weisshorn10 CROND[124039]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:50:01 weisshorn08 CROND[122808]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:50:01 weisshorn03 CROND[124694]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:50:01 weisshorn12 CROND[123893]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:50:01 weisshorn04 CROND[123481]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:50:01 weisshorn09 CROND[123731]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:50:01 weisshorn14 CROND[52644]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:50:01 weisshorn05 CROND[124012]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:50:01 weisshorn11 CROND[127008]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:50:01 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:50:01 weisshorn01 CROND[4745]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 13:50:10 weisshorn01 kernel: Lustre: 5144:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046ee:0x1450:0x0]] [valid 68719476736]
Jun 19 13:50:10 weisshorn01 kernel: Lustre: 5144:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 1 previous similar message
Jun 19 13:50:16 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:50:17 weisshorn13 nslcd[4224]: [9df57d] ldap_result() failed: Can't contact LDAP server
Jun 19 13:50:31 weisshorn10 nslcd[4294]: [9df57d] ldap_result() failed: Can't contact LDAP server
Jun 19 13:50:31 weisshorn11 nslcd[3954]: [234363] ldap_result() failed: Can't contact LDAP server
Jun 19 13:50:31 weisshorn12 nslcd[3954]: [9df57d] ldap_result() failed: Can't contact LDAP server
Jun 19 13:50:31 weisshorn03 nslcd[3944]: [ef532d] ldap_result() failed: Can't contact LDAP server
Jun 19 13:50:31 weisshorn05 nslcd[4378]: [bd7a3e] ldap_result() failed: Can't contact LDAP server
Jun 19 13:50:31 weisshorn04 nslcd[3885]: [92a02e] ldap_result() failed: Can't contact LDAP server
Jun 19 13:50:31 weisshorn09 nslcd[3954]: [92a02e] ldap_result() failed: Can't contact LDAP server
Jun 19 13:50:31 weisshorn07 nslcd[4297]: [9df57d] ldap_result() failed: Can't contact LDAP server
Jun 19 13:50:31 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:50:31 weisshorn08 nslcd[3946]: [9df57d] ldap_result() failed: Can't contact LDAP server
Jun 19 13:50:32 weisshorn06 nslcd[3953]: [03a216] ldap_result() failed: Can't contact LDAP server
Jun 19 13:50:46 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:51:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:51:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:51:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:51:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:52:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:52:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:52:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:52:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:53:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:53:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:53:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:53:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:54:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:54:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:54:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:54:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:55:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:55:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:55:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:55:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:56:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:56:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:56:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:56:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:57:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:57:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:57:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:57:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:58:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:58:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:58:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:58:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:59:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:59:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:59:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 13:59:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:00:01 weisshorn07 CROND[124228]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:00:01 weisshorn13 CROND[127435]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:00:01 weisshorn06 CROND[124933]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:00:01 weisshorn02 CROND[43325]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:00:01 weisshorn10 CROND[124069]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:00:01 weisshorn08 CROND[122840]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:00:01 weisshorn03 CROND[124719]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:00:01 weisshorn12 CROND[123917]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:00:01 weisshorn04 CROND[123511]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:00:01 weisshorn09 CROND[123761]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:00:01 weisshorn14 CROND[52668]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:00:01 weisshorn05 CROND[124037]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:00:01 weisshorn11 CROND[127035]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:00:02 weisshorn01 CROND[4899]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:00:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:00:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:00:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:00:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:01:01 weisshorn01 CROND[4906]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 14:01:01 weisshorn07 CROND[124233]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 14:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1242 starting 0anacron
Jun 19 14:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1242 finished 0anacron
Jun 19 14:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1242 starting mcelog.cron
Jun 19 14:01:01 weisshorn13 CROND[127440]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 14:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4906 starting 0anacron
Jun 19 14:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1242 finished mcelog.cron
Jun 19 14:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1274 starting 0anacron
Jun 19 14:01:01 weisshorn06 CROND[124938]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 14:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1249 starting 0anacron
Jun 19 14:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1249 finished 0anacron
Jun 19 14:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4915 finished 0anacron
Jun 19 14:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1249 starting mcelog.cron
Jun 19 14:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4906 starting mcelog.cron
Jun 19 14:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1274 finished 0anacron
Jun 19 14:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1274 starting mcelog.cron
Jun 19 14:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1249 finished mcelog.cron
Jun 19 14:01:01 weisshorn01 run-parts(/etc/cron.hourly)[4922 finished mcelog.cron
Jun 19 14:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1274 finished mcelog.cron
Jun 19 14:01:01 weisshorn10 CROND[124074]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 14:01:01 weisshorn02 CROND[43418]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 14:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1240 starting 0anacron
Jun 19 14:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4341 starting 0anacron
Jun 19 14:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1240 finished 0anacron
Jun 19 14:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1240 starting mcelog.cron
Jun 19 14:01:01 weisshorn08 CROND[122845]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 14:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4342 finished 0anacron
Jun 19 14:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4341 starting mcelog.cron
Jun 19 14:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1240 finished mcelog.cron
Jun 19 14:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4343 finished mcelog.cron
Jun 19 14:01:01 weisshorn03 CROND[124724]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 14:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1228 starting 0anacron
Jun 19 14:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1247 starting 0anacron
Jun 19 14:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1247 finished 0anacron
Jun 19 14:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1247 starting mcelog.cron
Jun 19 14:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1228 finished 0anacron
Jun 19 14:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1228 starting mcelog.cron
Jun 19 14:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1247 finished mcelog.cron
Jun 19 14:01:01 weisshorn12 CROND[123922]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 14:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1228 finished mcelog.cron
Jun 19 14:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1239 starting 0anacron
Jun 19 14:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1239 finished 0anacron
Jun 19 14:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1239 starting mcelog.cron
Jun 19 14:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1239 finished mcelog.cron
Jun 19 14:01:01 weisshorn04 CROND[123516]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 14:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1235 starting 0anacron
Jun 19 14:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1235 finished 0anacron
Jun 19 14:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1235 starting mcelog.cron
Jun 19 14:01:01 weisshorn09 CROND[123766]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 14:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1235 finished mcelog.cron
Jun 19 14:01:01 weisshorn14 CROND[52673]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 14:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5267 starting 0anacron
Jun 19 14:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1237 starting 0anacron
Jun 19 14:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5268 finished 0anacron
Jun 19 14:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1237 finished 0anacron
Jun 19 14:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1237 starting mcelog.cron
Jun 19 14:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5267 starting mcelog.cron
Jun 19 14:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5269 finished mcelog.cron
Jun 19 14:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1237 finished mcelog.cron
Jun 19 14:01:01 weisshorn11 CROND[127040]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 14:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1270 starting 0anacron
Jun 19 14:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1270 finished 0anacron
Jun 19 14:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1270 starting mcelog.cron
Jun 19 14:01:01 weisshorn05 CROND[124042]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 14:01:02 weisshorn11 run-parts(/etc/cron.hourly)[1270 finished mcelog.cron
Jun 19 14:01:02 weisshorn05 run-parts(/etc/cron.hourly)[1240 starting 0anacron
Jun 19 14:01:02 weisshorn05 run-parts(/etc/cron.hourly)[1240 finished 0anacron
Jun 19 14:01:02 weisshorn05 run-parts(/etc/cron.hourly)[1240 starting mcelog.cron
Jun 19 14:01:02 weisshorn05 run-parts(/etc/cron.hourly)[1240 finished mcelog.cron
Jun 19 14:01:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:01:16 weisshorn14 nslcd[3437]: [486e47] ldap_result() failed: Can't contact LDAP server
Jun 19 14:01:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:01:25 weisshorn01 kernel: Lustre: 5426:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046ee:0x148b:0x0]] [valid 68719476736]
Jun 19 14:01:25 weisshorn01 kernel: Lustre: 5426:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 1 previous similar message
Jun 19 14:01:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:01:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:02:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:02:10 weisshorn07 rhnsd[124253]: running program /usr/sbin/rhn_check
Jun 19 14:02:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:02:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:02:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:03:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:03:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:03:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:03:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:04:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:04:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:04:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:04:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:05:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:05:17 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:05:32 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:05:47 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:06:02 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:06:16 weisshorn10 nslcd[4294]: [bd7a3e] ldap_result() failed: Can't contact LDAP server
Jun 19 14:06:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:06:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:06:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:07:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:07:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:07:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:07:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:08:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:08:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:08:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:08:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:09:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:09:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:09:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:09:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:10:01 weisshorn11 CROND[127118]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:10:01 weisshorn05 CROND[124093]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:10:01 weisshorn07 CROND[124268]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:10:01 weisshorn06 CROND[125011]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:10:01 weisshorn01 CROND[4980]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:10:01 weisshorn01 CROND[4982]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 14:10:01 weisshorn13 CROND[127524]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:10:01 weisshorn10 CROND[124116]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:10:01 weisshorn02 CROND[44260]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:10:01 weisshorn03 CROND[124802]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:10:01 weisshorn08 CROND[122919]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:10:01 weisshorn12 CROND[124005]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:10:01 weisshorn04 sshd[123561]: Accepted publickey for root from 148.187.12.231 port 44381 ssh2
Jun 19 14:10:01 weisshorn04 sshd[123561]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 14:10:01 weisshorn09 sshd[123838]: Accepted publickey for root from 148.187.12.231 port 58063 ssh2
Jun 19 14:10:01 weisshorn10 sshd[124118]: Accepted publickey for root from 148.187.12.231 port 37097 ssh2
Jun 19 14:10:01 weisshorn11 sshd[127120]: Accepted publickey for root from 148.187.12.231 port 45310 ssh2
Jun 19 14:10:01 weisshorn14 sshd[52751]: Accepted publickey for root from 148.187.12.231 port 49739 ssh2
Jun 19 14:10:01 weisshorn09 sshd[123838]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 14:10:01 weisshorn14 sshd[52751]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 14:10:01 weisshorn11 sshd[127120]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 14:10:01 weisshorn05 sshd[124095]: Accepted publickey for root from 148.187.12.231 port 54833 ssh2
Jun 19 14:10:01 weisshorn06 sshd[125013]: Accepted publickey for root from 148.187.12.231 port 44891 ssh2
Jun 19 14:10:01 weisshorn06 sshd[125013]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 14:10:01 weisshorn10 sshd[124118]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 14:10:01 weisshorn05 sshd[124095]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 14:10:01 weisshorn04 CROND[123575]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:10:01 weisshorn14 sshd[52751]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 14:10:01 weisshorn14 sshd[52751]: pam_unix(sshd:session): session closed for user root
Jun 19 14:10:01 weisshorn12 sshd[124002]: Accepted publickey for root from 148.187.12.231 port 46535 ssh2
Jun 19 14:10:01 weisshorn03 sshd[124804]: Accepted publickey for root from 148.187.12.231 port 50622 ssh2
Jun 19 14:10:01 weisshorn13 sshd[127526]: Accepted publickey for root from 148.187.12.231 port 44788 ssh2
Jun 19 14:10:01 weisshorn03 sshd[124804]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 14:10:01 weisshorn12 sshd[124002]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 14:10:01 weisshorn14 CROND[52774]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:10:01 weisshorn13 sshd[127526]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 14:10:01 weisshorn09 CROND[123859]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:10:01 weisshorn07 sshd[124270]: Accepted publickey for root from 148.187.12.231 port 48438 ssh2
Jun 19 14:10:01 weisshorn09 sshd[123838]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 14:10:01 weisshorn09 sshd[123838]: pam_unix(sshd:session): session closed for user root
Jun 19 14:10:01 weisshorn04 sshd[123561]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 14:10:01 weisshorn04 sshd[123561]: pam_unix(sshd:session): session closed for user root
Jun 19 14:10:01 weisshorn07 sshd[124270]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 14:10:01 weisshorn06 sshd[125013]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 14:10:01 weisshorn06 sshd[125013]: pam_unix(sshd:session): session closed for user root
Jun 19 14:10:01 weisshorn10 sshd[124118]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 14:10:01 weisshorn10 sshd[124118]: pam_unix(sshd:session): session closed for user root
Jun 19 14:10:01 weisshorn08 sshd[122917]: Accepted publickey for root from 148.187.12.231 port 40430 ssh2
Jun 19 14:10:01 weisshorn08 sshd[122917]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 14:10:01 weisshorn11 sshd[127120]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 14:10:01 weisshorn11 sshd[127120]: pam_unix(sshd:session): session closed for user root
Jun 19 14:10:01 weisshorn05 sshd[124095]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 14:10:01 weisshorn05 sshd[124095]: pam_unix(sshd:session): session closed for user root
Jun 19 14:10:01 weisshorn03 sshd[124804]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 14:10:01 weisshorn03 sshd[124804]: pam_unix(sshd:session): session closed for user root
Jun 19 14:10:01 weisshorn12 sshd[124002]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 14:10:01 weisshorn12 sshd[124002]: pam_unix(sshd:session): session closed for user root
Jun 19 14:10:02 weisshorn13 sshd[127526]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 14:10:02 weisshorn13 sshd[127526]: pam_unix(sshd:session): session closed for user root
Jun 19 14:10:02 weisshorn07 sshd[124270]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 14:10:02 weisshorn07 sshd[124270]: pam_unix(sshd:session): session closed for user root
Jun 19 14:10:02 weisshorn08 sshd[122917]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 14:10:02 weisshorn08 sshd[122917]: pam_unix(sshd:session): session closed for user root
Jun 19 14:10:02 weisshorn01 postfix/pickup[4971]: 2C7F592224F: uid=0 from=<root>
Jun 19 14:10:02 weisshorn01 postfix/cleanup[5103]: 2C7F592224F: message-id=<20120619121002.2C7F592224F@weisshorn01.admin.cscs.ch>
Jun 19 14:10:02 weisshorn01 postfix/qmgr[2710]: 2C7F592224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 14:10:02 weisshorn01 postfix/local[5136]: 2C7F592224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=0.48, delays=0.4/0.04/0/0.05, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 14:10:02 weisshorn01 postfix/qmgr[2710]: 2C7F592224F: removed
Jun 19 14:10:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:10:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:10:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:10:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:11:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:11:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:11:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:11:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:12:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:12:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:12:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:12:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:13:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:13:18 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:13:33 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:13:48 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:14:03 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:14:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:14:23 weisshorn14 rhnsd[52788]: running program /usr/sbin/rhn_check
Jun 19 14:14:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:14:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:15:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:15:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:15:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:15:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:16:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:16:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:16:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:16:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:17:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:17:04 weisshorn01 kernel: Lustre: 4722:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004631:0x1678f:0x0]] [valid 68719476736]
Jun 19 14:17:04 weisshorn01 kernel: Lustre: 4722:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 2 previous similar messages
Jun 19 14:17:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:17:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:17:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:18:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:18:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:18:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:18:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:19:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:19:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:19:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:19:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:20:01 weisshorn11 CROND[127169]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:20:01 weisshorn01 CROND[5526]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:20:01 weisshorn05 CROND[124173]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:20:01 weisshorn06 CROND[125056]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:20:01 weisshorn07 CROND[124318]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:20:01 weisshorn02 CROND[45212]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:20:01 weisshorn03 CROND[124855]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:20:01 weisshorn13 CROND[127575]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:20:01 weisshorn10 CROND[124201]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:20:01 weisshorn12 CROND[124054]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:20:01 weisshorn04 CROND[123642]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:20:01 weisshorn08 CROND[122969]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:20:01 weisshorn14 CROND[52810]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:20:01 weisshorn09 CROND[123892]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:20:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:20:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:20:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:20:49 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:21:04 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:21:19 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:21:34 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:21:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:22:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:22:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:22:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:22:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:23:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:23:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:23:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:23:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:24:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:24:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:24:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:24:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:25:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:25:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:25:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:25:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:26:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:26:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:26:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:26:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:27:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:27:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:27:29 weisshorn01 kernel: Lustre: 4716:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004631:0x167ab:0x0]] [valid 68719476736]
Jun 19 14:27:29 weisshorn01 kernel: Lustre: 4716:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 4 previous similar messages
Jun 19 14:27:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:27:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:28:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:28:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:28:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:28:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:29:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:29:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:29:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:29:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:30:01 weisshorn09 CROND[123922]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:30:01 weisshorn11 CROND[127206]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:30:01 weisshorn05 CROND[124203]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:30:01 weisshorn01 CROND[5638]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:30:01 weisshorn07 CROND[124391]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:30:01 weisshorn06 CROND[125099]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:30:01 weisshorn02 CROND[45397]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:30:01 weisshorn03 CROND[124885]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:30:01 weisshorn13 CROND[127639]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:30:01 weisshorn12 CROND[124118]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:30:01 weisshorn14 CROND[52869]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:30:02 weisshorn10 CROND[124231]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:30:02 weisshorn08 CROND[123001]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:30:02 weisshorn04 CROND[123672]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:30:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:30:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:30:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:30:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:31:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:31:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:31:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:31:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:32:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:32:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:32:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:32:50 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:33:05 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:33:20 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:33:35 weisshorn02 /usr/sbin/cerebrod[40936]: lmt_mysql: failed to connect to database
Jun 19 14:33:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:33:51 weisshorn13 rhnsd[127647]: running program /usr/sbin/rhn_check
Jun 19 14:34:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:34:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:34:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:34:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:35:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:35:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:35:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:35:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:36:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:36:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:36:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:36:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:37:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:37:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:37:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:37:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:38:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:38:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:38:30 weisshorn01 kernel: Lustre: 5431:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046e5:0xf10c:0x0]] [valid 68719476736]
Jun 19 14:38:30 weisshorn01 kernel: Lustre: 5431:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 4 previous similar messages
Jun 19 14:38:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:38:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:39:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:39:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:39:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:39:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:40:01 weisshorn14 CROND[52899]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:40:01 weisshorn09 CROND[123981]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:40:01 weisshorn10 CROND[124258]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:40:01 weisshorn12 CROND[124145]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:40:01 weisshorn08 CROND[123060]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:40:01 weisshorn04 CROND[123704]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:40:01 weisshorn11 CROND[127260]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:40:01 weisshorn05 CROND[124235]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:40:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:40:01 weisshorn01 CROND[5711]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:40:01 weisshorn07 CROND[124410]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:40:01 weisshorn06 CROND[125153]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:40:01 weisshorn02 CROND[45454]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:40:01 weisshorn03 CROND[124944]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:40:02 weisshorn13 CROND[127671]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:40:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:40:31 weisshorn02 nslcd[2099]: [e45ca4] ldap_result() failed: Can't contact LDAP server
Jun 19 14:40:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:40:31 weisshorn14 nslcd[3437]: [534cde] ldap_result() failed: Can't contact LDAP server
Jun 19 14:40:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:41:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:41:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:41:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:41:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:42:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:42:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:42:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:42:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:43:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:43:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:43:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:43:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:44:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:44:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:44:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:44:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:45:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:45:15 weisshorn09 nslcd[3954]: [ef532d] ldap_result() failed: Can't contact LDAP server
Jun 19 14:45:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:45:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:45:38 weisshorn08 nslcd[3946]: [bd7a3e] ldap_result() failed: Can't contact LDAP server
Jun 19 14:45:39 weisshorn06 nslcd[3953]: [b8a08e] ldap_result() failed: Can't contact LDAP server
Jun 19 14:45:41 weisshorn04 nslcd[3885]: [ef532d] ldap_result() failed: Can't contact LDAP server
Jun 19 14:45:42 weisshorn07 nslcd[4297]: [bd7a3e] ldap_result() failed: Can't contact LDAP server
Jun 19 14:45:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:46:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:46:02 weisshorn11 nslcd[3954]: [d385c9] ldap_result() failed: Can't contact LDAP server
Jun 19 14:46:02 weisshorn12 nslcd[3954]: [bd7a3e] ldap_result() failed: Can't contact LDAP server
Jun 19 14:46:04 weisshorn13 nslcd[4224]: [bd7a3e] ldap_result() failed: Can't contact LDAP server
Jun 19 14:46:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:46:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:46:41 weisshorn05 nslcd[4378]: [90b7c5] ldap_result() failed: Can't contact LDAP server
Jun 19 14:46:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:47:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:47:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:47:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:47:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:48:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:48:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:48:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:48:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:49:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:49:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:49:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:49:32 weisshorn09 ntpd[4454]: synchronized to 148.187.18.120, stratum 2
Jun 19 14:49:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:50:01 weisshorn14 CROND[52939]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:50:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:50:01 weisshorn03 CROND[124982]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:50:01 weisshorn09 CROND[124026]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:50:01 weisshorn10 CROND[124328]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:50:01 weisshorn13 CROND[127707]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:50:01 weisshorn08 CROND[123096]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:50:01 weisshorn12 CROND[124181]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:50:01 weisshorn04 CROND[123770]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:50:01 weisshorn11 CROND[127299]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:50:01 weisshorn05 CROND[124301]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:50:01 weisshorn07 CROND[124488]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:50:01 weisshorn01 CROND[5816]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:50:01 weisshorn02 CROND[45551]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:50:01 weisshorn06 CROND[125191]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 14:50:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:50:23 weisshorn01 kernel: Lustre: 4735:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004631:0x167c4:0x0]] [valid 68719476736]
Jun 19 14:50:23 weisshorn01 kernel: Lustre: 4735:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 22 previous similar messages
Jun 19 14:50:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:50:32 weisshorn13 nslcd[4224]: [90b7c5] ldap_result() failed: Can't contact LDAP server
Jun 19 14:50:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:50:46 weisshorn10 nslcd[4294]: [90b7c5] ldap_result() failed: Can't contact LDAP server
Jun 19 14:50:46 weisshorn11 nslcd[3954]: [6f6d8a] ldap_result() failed: Can't contact LDAP server
Jun 19 14:50:46 weisshorn12 nslcd[3954]: [90b7c5] ldap_result() failed: Can't contact LDAP server
Jun 19 14:50:46 weisshorn03 nslcd[3944]: [d601af] ldap_result() failed: Can't contact LDAP server
Jun 19 14:50:46 weisshorn05 nslcd[4378]: [03a216] ldap_result() failed: Can't contact LDAP server
Jun 19 14:50:46 weisshorn04 nslcd[3885]: [d601af] ldap_result() failed: Can't contact LDAP server
Jun 19 14:50:46 weisshorn09 nslcd[3954]: [d601af] ldap_result() failed: Can't contact LDAP server
Jun 19 14:50:46 weisshorn07 nslcd[4297]: [90b7c5] ldap_result() failed: Can't contact LDAP server
Jun 19 14:50:46 weisshorn08 nslcd[3946]: [90b7c5] ldap_result() failed: Can't contact LDAP server
Jun 19 14:50:47 weisshorn06 nslcd[3953]: [b59621] ldap_result() failed: Can't contact LDAP server
Jun 19 14:51:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:51:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:51:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:51:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:52:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:52:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:52:31 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:52:46 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:53:01 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:53:16 weisshorn02 /usr/sbin/cerebrod[45433]: lmt_mysql: failed to connect to database
Jun 19 14:53:30 weisshorn02 /usr/sbin/cerebrod[45613]: lmt_mysql: failed to connect to database
Jun 19 14:53:45 weisshorn02 /usr/sbin/cerebrod[45613]: lmt_mysql: failed to connect to database
Jun 19 14:54:00 weisshorn02 /usr/sbin/cerebrod[45613]: lmt_mysql: failed to connect to database
Jun 19 14:54:15 weisshorn02 /usr/sbin/cerebrod[45613]: lmt_mysql: failed to connect to database
Jun 19 14:54:30 weisshorn02 /usr/sbin/cerebrod[45613]: lmt_mysql: failed to connect to database
Jun 19 14:54:45 weisshorn02 /usr/sbin/cerebrod[45613]: lmt_mysql: failed to connect to database
Jun 19 14:55:00 weisshorn02 /usr/sbin/cerebrod[45613]: lmt_mysql: failed to connect to database
Jun 19 14:55:15 weisshorn02 /usr/sbin/cerebrod[45613]: lmt_mysql: failed to connect to database
Jun 19 14:55:30 weisshorn02 /usr/sbin/cerebrod[45613]: lmt_mysql: failed to connect to database
Jun 19 14:55:45 weisshorn02 /usr/sbin/cerebrod[45613]: lmt_mysql: failed to connect to database
Jun 19 14:56:00 weisshorn02 /usr/sbin/cerebrod[45613]: lmt_mysql: failed to connect to database
Jun 19 14:56:15 weisshorn02 /usr/sbin/cerebrod[45613]: lmt_mysql: failed to connect to database
Jun 19 14:56:30 weisshorn02 /usr/sbin/cerebrod[45613]: lmt_mysql: failed to connect to database
Jun 19 14:56:45 weisshorn02 /usr/sbin/cerebrod[45613]: lmt_mysql: failed to connect to database
Jun 19 14:57:00 weisshorn02 /usr/sbin/cerebrod[45613]: lmt_mysql: failed to connect to database
Jun 19 14:57:15 weisshorn02 /usr/sbin/cerebrod[45613]: lmt_mysql: failed to connect to database
Jun 19 14:57:30 weisshorn02 /usr/sbin/cerebrod[45613]: lmt_mysql: failed to connect to database
Jun 19 14:57:45 weisshorn02 /usr/sbin/cerebrod[45613]: lmt_mysql: failed to connect to database
Jun 19 14:58:30 weisshorn02 cerebrod[45692]: lmt_mysql: failed to connect to database
Jun 19 14:58:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 14:59:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 14:59:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 14:59:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 14:59:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:00:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:00:01 weisshorn02 CROND[45723]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:00:01 weisshorn06 CROND[125221]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:00:01 weisshorn01 CROND[5887]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:00:01 weisshorn14 CROND[52993]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:00:01 weisshorn09 CROND[124044]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:00:01 weisshorn03 CROND[125006]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:00:01 weisshorn10 CROND[124352]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:00:01 weisshorn13 CROND[127766]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:00:01 weisshorn12 CROND[124240]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:00:01 weisshorn08 CROND[123123]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:00:01 weisshorn04 CROND[123794]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:00:01 weisshorn11 CROND[127328]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:00:01 weisshorn07 CROND[124513]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:00:02 weisshorn05 CROND[124325]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:00:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:00:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:00:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:01:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:01:01 weisshorn07 CROND[124518]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 15:01:01 weisshorn05 CROND[124330]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 15:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1245 starting 0anacron
Jun 19 15:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1243 starting 0anacron
Jun 19 15:01:01 weisshorn02 CROND[45727]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 15:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4572 starting 0anacron
Jun 19 15:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1245 finished 0anacron
Jun 19 15:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1245 starting mcelog.cron
Jun 19 15:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4573 finished 0anacron
Jun 19 15:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4572 starting mcelog.cron
Jun 19 15:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1243 finished 0anacron
Jun 19 15:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4574 finished mcelog.cron
Jun 19 15:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1243 starting mcelog.cron
Jun 19 15:01:01 weisshorn06 CROND[125226]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 15:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1245 finished mcelog.cron
Jun 19 15:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1252 starting 0anacron
Jun 19 15:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1243 finished mcelog.cron
Jun 19 15:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1252 finished 0anacron
Jun 19 15:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1252 starting mcelog.cron
Jun 19 15:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1252 finished mcelog.cron
Jun 19 15:01:01 weisshorn14 CROND[52998]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 15:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5299 starting 0anacron
Jun 19 15:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5300 finished 0anacron
Jun 19 15:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5299 starting mcelog.cron
Jun 19 15:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5301 finished mcelog.cron
Jun 19 15:01:01 weisshorn01 CROND[5892]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 15:01:01 weisshorn01 run-parts(/etc/cron.hourly)[5892 starting 0anacron
Jun 19 15:01:01 weisshorn09 CROND[124049]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 15:01:01 weisshorn01 run-parts(/etc/cron.hourly)[5902 finished 0anacron
Jun 19 15:01:01 weisshorn03 CROND[125011]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 15:01:01 weisshorn01 run-parts(/etc/cron.hourly)[5892 starting mcelog.cron
Jun 19 15:01:01 weisshorn10 CROND[124357]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 15:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1240 starting 0anacron
Jun 19 15:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1243 starting 0anacron
Jun 19 15:01:01 weisshorn13 CROND[127771]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 15:01:01 weisshorn01 run-parts(/etc/cron.hourly)[5910 finished mcelog.cron
Jun 19 15:01:01 weisshorn12 CROND[124245]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 15:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1243 finished 0anacron
Jun 19 15:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1250 starting 0anacron
Jun 19 15:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1243 starting mcelog.cron
Jun 19 15:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1240 finished 0anacron
Jun 19 15:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1240 starting mcelog.cron
Jun 19 15:01:01 weisshorn08 CROND[123128]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 15:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1277 starting 0anacron
Jun 19 15:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1242 starting 0anacron
Jun 19 15:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1240 finished mcelog.cron
Jun 19 15:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1243 finished mcelog.cron
Jun 19 15:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1250 finished 0anacron
Jun 19 15:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1250 starting mcelog.cron
Jun 19 15:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1277 finished 0anacron
Jun 19 15:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1277 starting mcelog.cron
Jun 19 15:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1231 starting 0anacron
Jun 19 15:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1242 finished 0anacron
Jun 19 15:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1242 starting mcelog.cron
Jun 19 15:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1250 finished mcelog.cron
Jun 19 15:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1231 finished 0anacron
Jun 19 15:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1231 starting mcelog.cron
Jun 19 15:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1277 finished mcelog.cron
Jun 19 15:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1242 finished mcelog.cron
Jun 19 15:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1231 finished mcelog.cron
Jun 19 15:01:01 weisshorn04 CROND[123799]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 15:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1237 starting 0anacron
Jun 19 15:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1238 finished 0anacron
Jun 19 15:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1237 starting mcelog.cron
Jun 19 15:01:01 weisshorn11 CROND[127333]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 15:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1238 finished mcelog.cron
Jun 19 15:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1273 starting 0anacron
Jun 19 15:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1273 finished 0anacron
Jun 19 15:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1273 starting mcelog.cron
Jun 19 15:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1273 finished mcelog.cron
Jun 19 15:01:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:01:25 weisshorn10 rhnsd[124376]: running program /usr/sbin/rhn_check
Jun 19 15:01:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:01:31 weisshorn14 nslcd[3437]: [0dde32] ldap_result() failed: Can't contact LDAP server
Jun 19 15:01:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:02:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:02:10 weisshorn01 kernel: Lustre: 5431:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046e5:0xf189:0x0]] [valid 68719476736]
Jun 19 15:02:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:02:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:02:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:03:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:03:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:03:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:03:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:04:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:04:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:04:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:04:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:05:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:05:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:05:21 weisshorn03 nslcd[3944]: [c77402] ldap_result() failed: Can't contact LDAP server
Jun 19 15:05:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:05:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:06:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:06:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:06:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:06:31 weisshorn10 nslcd[4294]: [03a216] ldap_result() failed: Can't contact LDAP server
Jun 19 15:06:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:07:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:07:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:07:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:07:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:08:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:08:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:08:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:08:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:09:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:09:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:09:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:09:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:10:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:10:01 weisshorn02 CROND[45769]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:10:01 weisshorn06 CROND[125302]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:10:01 weisshorn07 CROND[124553]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:10:01 weisshorn14 CROND[53048]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:10:01 weisshorn05 CROND[124413]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:10:01 weisshorn01 CROND[5975]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 15:10:01 weisshorn01 CROND[5977]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:10:01 weisshorn09 CROND[124130]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:10:01 weisshorn10 CROND[124443]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:10:01 weisshorn03 CROND[125095]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:10:01 weisshorn12 CROND[124302]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:10:01 weisshorn14 sshd[53050]: Accepted publickey for root from 148.187.12.231 port 49753 ssh2
Jun 19 15:10:01 weisshorn14 sshd[53050]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 15:10:01 weisshorn14 sshd[53050]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 15:10:01 weisshorn14 sshd[53050]: pam_unix(sshd:session): session closed for user root
Jun 19 15:10:01 weisshorn06 sshd[125304]: Accepted publickey for root from 148.187.12.231 port 44902 ssh2
Jun 19 15:10:01 weisshorn05 sshd[124415]: Accepted publickey for root from 148.187.12.231 port 54847 ssh2
Jun 19 15:10:01 weisshorn07 sshd[124555]: Accepted publickey for root from 148.187.12.231 port 48444 ssh2
Jun 19 15:10:01 weisshorn05 sshd[124415]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 15:10:01 weisshorn07 sshd[124555]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 15:10:01 weisshorn13 CROND[127825]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:10:01 weisshorn06 sshd[125304]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 15:10:01 weisshorn09 sshd[124132]: Accepted publickey for root from 148.187.12.231 port 58075 ssh2
Jun 19 15:10:01 weisshorn09 sshd[124132]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 15:10:02 weisshorn12 sshd[124300]: Accepted publickey for root from 148.187.12.231 port 46541 ssh2
Jun 19 15:10:02 weisshorn04 CROND[123885]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:10:02 weisshorn11 CROND[127417]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:10:02 weisshorn12 sshd[124300]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 15:10:02 weisshorn03 sshd[125093]: Accepted publickey for root from 148.187.12.231 port 50641 ssh2
Jun 19 15:10:02 weisshorn10 sshd[124445]: Accepted publickey for root from 148.187.12.231 port 37112 ssh2
Jun 19 15:10:02 weisshorn03 sshd[125093]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 15:10:02 weisshorn08 CROND[123211]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:10:02 weisshorn10 sshd[124445]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 15:10:02 weisshorn11 sshd[127414]: Accepted publickey for root from 148.187.12.231 port 45323 ssh2
Jun 19 15:10:02 weisshorn04 sshd[123882]: Accepted publickey for root from 148.187.12.231 port 44386 ssh2
Jun 19 15:10:02 weisshorn11 sshd[127414]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 15:10:02 weisshorn04 sshd[123882]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 15:10:02 weisshorn13 sshd[127824]: Accepted publickey for root from 148.187.12.231 port 44801 ssh2
Jun 19 15:10:02 weisshorn13 sshd[127824]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 15:10:02 weisshorn05 sshd[124415]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 15:10:02 weisshorn05 sshd[124415]: pam_unix(sshd:session): session closed for user root
Jun 19 15:10:02 weisshorn06 sshd[125304]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 15:10:02 weisshorn06 sshd[125304]: pam_unix(sshd:session): session closed for user root
Jun 19 15:10:02 weisshorn08 sshd[123209]: Accepted publickey for root from 148.187.12.231 port 40447 ssh2
Jun 19 15:10:02 weisshorn08 sshd[123209]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 15:10:02 weisshorn12 sshd[124300]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 15:10:02 weisshorn12 sshd[124300]: pam_unix(sshd:session): session closed for user root
Jun 19 15:10:02 weisshorn07 sshd[124555]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 15:10:02 weisshorn07 sshd[124555]: pam_unix(sshd:session): session closed for user root
Jun 19 15:10:02 weisshorn09 sshd[124132]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 15:10:02 weisshorn09 sshd[124132]: pam_unix(sshd:session): session closed for user root
Jun 19 15:10:02 weisshorn10 sshd[124445]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 15:10:02 weisshorn10 sshd[124445]: pam_unix(sshd:session): session closed for user root
Jun 19 15:10:02 weisshorn03 sshd[125093]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 15:10:02 weisshorn03 sshd[125093]: pam_unix(sshd:session): session closed for user root
Jun 19 15:10:02 weisshorn04 sshd[123882]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 15:10:02 weisshorn04 sshd[123882]: pam_unix(sshd:session): session closed for user root
Jun 19 15:10:02 weisshorn11 sshd[127414]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 15:10:02 weisshorn11 sshd[127414]: pam_unix(sshd:session): session closed for user root
Jun 19 15:10:02 weisshorn13 sshd[127824]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 15:10:02 weisshorn13 sshd[127824]: pam_unix(sshd:session): session closed for user root
Jun 19 15:10:02 weisshorn08 sshd[123209]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 15:10:02 weisshorn08 sshd[123209]: pam_unix(sshd:session): session closed for user root
Jun 19 15:10:02 weisshorn01 postfix/pickup[4971]: 90D6892224F: uid=0 from=<root>
Jun 19 15:10:02 weisshorn01 postfix/cleanup[6007]: 90D6892224F: message-id=<20120619131002.90D6892224F@weisshorn01.admin.cscs.ch>
Jun 19 15:10:02 weisshorn01 postfix/qmgr[2710]: 90D6892224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 15:10:02 weisshorn01 postfix/local[6009]: 90D6892224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=0.79, delays=0.7/0.04/0/0.05, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 15:10:02 weisshorn01 postfix/qmgr[2710]: 90D6892224F: removed
Jun 19 15:10:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:10:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:10:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:10:51 weisshorn02 nslcd[2099]: [232ee3] ldap_result() failed: Can't contact LDAP server
Jun 19 15:11:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:11:03 weisshorn06 rhnsd[125327]: running program /usr/sbin/rhn_check
Jun 19 15:11:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:11:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:11:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:12:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:12:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:12:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:12:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:12:50 weisshorn01 kernel: Lustre: 4784:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046e5:0xf216:0x0]] [valid 68719476736]
Jun 19 15:12:50 weisshorn01 kernel: Lustre: 4784:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 9 previous similar messages
Jun 19 15:13:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:13:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:13:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:13:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:14:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:14:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:14:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:14:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:15:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:15:16 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:15:31 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:15:46 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:16:01 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:16:16 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:16:31 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:16:46 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:17:01 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:17:16 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:17:31 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:17:46 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:18:01 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:18:16 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:18:31 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:18:46 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:19:01 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:19:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:19:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:19:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:20:01 weisshorn13 CROND[127869]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:20:01 weisshorn04 CROND[123934]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:20:01 weisshorn11 CROND[127463]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:20:01 weisshorn02 CROND[45836]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:20:01 weisshorn08 CROND[123255]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:20:01 weisshorn14 CROND[53103]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:20:01 weisshorn06 CROND[125357]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:20:01 weisshorn07 CROND[124651]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:20:01 weisshorn01 CROND[6126]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:20:01 weisshorn05 CROND[124464]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:20:01 weisshorn09 CROND[124184]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:20:01 weisshorn10 CROND[124494]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:20:01 weisshorn03 CROND[125147]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:20:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:20:02 weisshorn12 CROND[124340]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:20:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:20:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:20:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:21:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:21:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:21:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:21:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:22:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:22:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:22:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:22:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:23:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:23:08 weisshorn01 kernel: Lustre: 4716:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046e5:0xf481:0x0]] [valid 68719476736]
Jun 19 15:23:08 weisshorn01 kernel: Lustre: 4716:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 28 previous similar messages
Jun 19 15:23:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:23:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:23:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:24:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:24:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:24:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:24:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:25:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:25:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:25:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:25:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:26:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:26:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:26:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:26:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:27:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:27:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:27:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:27:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:28:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:28:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:28:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:28:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:29:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:29:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:29:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:29:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:30:01 weisshorn10 CROND[124518]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:30:01 weisshorn13 CROND[127933]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:30:01 weisshorn03 CROND[125171]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:30:01 weisshorn12 CROND[124404]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:30:01 weisshorn04 CROND[123958]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:30:01 weisshorn11 CROND[127492]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:30:01 weisshorn02 CROND[45859]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:30:01 weisshorn08 CROND[123287]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:30:01 weisshorn14 CROND[53158]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:30:01 weisshorn07 CROND[124676]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:30:01 weisshorn06 CROND[125387]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:30:01 weisshorn05 CROND[124488]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:30:01 weisshorn09 CROND[124208]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:30:01 weisshorn01 CROND[6202]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:30:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:30:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:30:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:30:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:31:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:31:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:31:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:31:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:32:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:32:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:32:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:32:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:33:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:33:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:33:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:33:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:34:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:34:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:34:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:34:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:35:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:35:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:35:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:35:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:36:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:36:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:36:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:36:45 weisshorn11 rhnsd[127550]: running program /usr/sbin/rhn_check
Jun 19 15:36:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:36:50 weisshorn03 rhnsd[125229]: running program /usr/sbin/rhn_check
Jun 19 15:37:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:37:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:37:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:37:41 weisshorn04 rhnsd[124021]: running program /usr/sbin/rhn_check
Jun 19 15:37:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:38:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:38:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:38:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:38:48 weisshorn01 kernel: Lustre: 4717:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046ee:0x1564:0x0]] [valid 68719476736]
Jun 19 15:38:48 weisshorn01 kernel: Lustre: 4717:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 9 previous similar messages
Jun 19 15:38:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:39:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:39:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:39:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:39:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:40:01 weisshorn09 CROND[124272]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:40:01 weisshorn10 CROND[124585]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:40:01 weisshorn13 CROND[127969]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:40:01 weisshorn01 CROND[6277]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:40:01 weisshorn03 CROND[125237]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:40:01 weisshorn12 CROND[124442]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:40:01 weisshorn04 CROND[124027]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:40:01 weisshorn02 CROND[45887]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:40:01 weisshorn08 CROND[123351]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:40:01 weisshorn14 CROND[53194]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:40:01 weisshorn11 CROND[127559]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:40:01 weisshorn06 CROND[125446]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:40:01 weisshorn07 CROND[124695]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:40:01 weisshorn05 CROND[124555]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:40:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:40:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:40:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:40:46 weisshorn02 nslcd[2099]: [9c74cf] ldap_result() failed: Can't contact LDAP server
Jun 19 15:40:46 weisshorn14 nslcd[3437]: [968c1c] ldap_result() failed: Can't contact LDAP server
Jun 19 15:40:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:41:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:41:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:41:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:41:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:42:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:42:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:42:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:42:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:43:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:43:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:43:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:43:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:44:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:44:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:44:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:44:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:45:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:45:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:45:30 weisshorn09 nslcd[3954]: [c77402] ldap_result() failed: Can't contact LDAP server
Jun 19 15:45:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:45:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:45:53 weisshorn08 nslcd[3946]: [03a216] ldap_result() failed: Can't contact LDAP server
Jun 19 15:45:54 weisshorn06 nslcd[3953]: [92a02e] ldap_result() failed: Can't contact LDAP server
Jun 19 15:45:56 weisshorn04 nslcd[3885]: [c77402] ldap_result() failed: Can't contact LDAP server
Jun 19 15:45:57 weisshorn07 nslcd[4297]: [03a216] ldap_result() failed: Can't contact LDAP server
Jun 19 15:46:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:46:17 weisshorn11 nslcd[3954]: [49361f] ldap_result() failed: Can't contact LDAP server
Jun 19 15:46:17 weisshorn12 nslcd[3954]: [03a216] ldap_result() failed: Can't contact LDAP server
Jun 19 15:46:19 weisshorn13 nslcd[4224]: [03a216] ldap_result() failed: Can't contact LDAP server
Jun 19 15:46:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:46:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:46:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:46:56 weisshorn05 nslcd[4378]: [b8a08e] ldap_result() failed: Can't contact LDAP server
Jun 19 15:47:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:47:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:47:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:47:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:48:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:48:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:48:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:48:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:49:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:49:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:49:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:49:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:50:01 weisshorn05 CROND[124586]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:50:01 weisshorn09 CROND[124311]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:50:01 weisshorn10 CROND[124615]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:50:01 weisshorn13 CROND[127999]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:50:01 weisshorn12 CROND[124467]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:50:01 weisshorn02 CROND[45951]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:50:01 weisshorn01 CROND[6387]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:50:01 weisshorn03 CROND[125270]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:50:01 weisshorn04 CROND[124058]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:50:01 weisshorn14 CROND[53229]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:50:01 weisshorn08 CROND[123382]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:50:01 weisshorn11 CROND[127587]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:50:01 weisshorn06 CROND[125479]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:50:01 weisshorn07 CROND[124773]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 15:50:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:50:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:50:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:50:47 weisshorn13 nslcd[4224]: [b8a08e] ldap_result() failed: Can't contact LDAP server
Jun 19 15:50:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:51:01 weisshorn10 nslcd[4294]: [b8a08e] ldap_result() failed: Can't contact LDAP server
Jun 19 15:51:01 weisshorn11 nslcd[3954]: [bc6770] ldap_result() failed: Can't contact LDAP server
Jun 19 15:51:01 weisshorn12 nslcd[3954]: [b8a08e] ldap_result() failed: Can't contact LDAP server
Jun 19 15:51:01 weisshorn03 nslcd[3944]: [b5b1c6] ldap_result() failed: Can't contact LDAP server
Jun 19 15:51:01 weisshorn05 nslcd[4378]: [b59621] ldap_result() failed: Can't contact LDAP server
Jun 19 15:51:01 weisshorn04 nslcd[3885]: [b5b1c6] ldap_result() failed: Can't contact LDAP server
Jun 19 15:51:01 weisshorn09 nslcd[3954]: [b5b1c6] ldap_result() failed: Can't contact LDAP server
Jun 19 15:51:01 weisshorn07 nslcd[4297]: [b8a08e] ldap_result() failed: Can't contact LDAP server
Jun 19 15:51:01 weisshorn08 nslcd[3946]: [b8a08e] ldap_result() failed: Can't contact LDAP server
Jun 19 15:51:02 weisshorn06 nslcd[3953]: [ef532d] ldap_result() failed: Can't contact LDAP server
Jun 19 15:51:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:51:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:51:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:51:36 weisshorn12 rhnsd[124471]: running program /usr/sbin/rhn_check
Jun 19 15:51:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:52:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:52:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:52:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:52:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:53:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:53:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:53:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:53:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:54:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:54:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:54:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:54:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:55:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:55:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:55:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:55:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:56:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:56:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:56:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:56:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:57:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:57:16 weisshorn01 kernel: Lustre: 5456:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046ee:0x15d0:0x0]] [valid 68719476736]
Jun 19 15:57:16 weisshorn01 kernel: Lustre: 5456:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 17 previous similar messages
Jun 19 15:57:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:57:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:57:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:58:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:58:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:58:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:58:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:59:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:59:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:59:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 15:59:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:00:01 weisshorn09 CROND[124371]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:00:01 weisshorn05 CROND[124624]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:00:01 weisshorn10 CROND[124647]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:00:01 weisshorn13 CROND[128069]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:00:01 weisshorn02 CROND[45981]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:00:01 weisshorn12 CROND[124535]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:00:01 weisshorn14 CROND[53294]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:00:01 weisshorn03 CROND[125335]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:00:01 weisshorn08 CROND[123450]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:00:01 weisshorn04 CROND[124096]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:00:01 weisshorn01 CROND[6469]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:00:01 weisshorn11 CROND[127651]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:00:01 weisshorn06 CROND[125545]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:00:01 weisshorn07 CROND[124799]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:00:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:00:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:00:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:00:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:01:01 weisshorn07 CROND[124804]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 16:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1248 starting 0anacron
Jun 19 16:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1248 finished 0anacron
Jun 19 16:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1248 starting mcelog.cron
Jun 19 16:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1248 finished mcelog.cron
Jun 19 16:01:01 weisshorn09 CROND[124376]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 16:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1243 starting 0anacron
Jun 19 16:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1243 finished 0anacron
Jun 19 16:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1243 starting mcelog.cron
Jun 19 16:01:01 weisshorn05 CROND[124629]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 16:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1243 finished mcelog.cron
Jun 19 16:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1246 starting 0anacron
Jun 19 16:01:01 weisshorn10 CROND[124652]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 16:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1246 finished 0anacron
Jun 19 16:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1246 starting mcelog.cron
Jun 19 16:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1246 starting 0anacron
Jun 19 16:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1246 finished 0anacron
Jun 19 16:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1246 starting mcelog.cron
Jun 19 16:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1246 finished mcelog.cron
Jun 19 16:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1246 finished mcelog.cron
Jun 19 16:01:01 weisshorn13 CROND[128074]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 16:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1280 starting 0anacron
Jun 19 16:01:01 weisshorn02 CROND[45985]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 16:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4598 starting 0anacron
Jun 19 16:01:01 weisshorn14 CROND[53299]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 16:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4599 finished 0anacron
Jun 19 16:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4598 starting mcelog.cron
Jun 19 16:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5329 starting 0anacron
Jun 19 16:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1280 finished 0anacron
Jun 19 16:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1280 starting mcelog.cron
Jun 19 16:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5330 finished 0anacron
Jun 19 16:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5329 starting mcelog.cron
Jun 19 16:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4600 finished mcelog.cron
Jun 19 16:01:01 weisshorn03 CROND[125340]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 16:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5331 finished mcelog.cron
Jun 19 16:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1253 starting 0anacron
Jun 19 16:01:01 weisshorn04 CROND[124101]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 16:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1280 finished mcelog.cron
Jun 19 16:01:01 weisshorn12 CROND[124540]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 16:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1253 finished 0anacron
Jun 19 16:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1241 starting 0anacron
Jun 19 16:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1253 starting mcelog.cron
Jun 19 16:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1245 starting 0anacron
Jun 19 16:01:01 weisshorn08 CROND[123455]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 16:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1234 starting 0anacron
Jun 19 16:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1253 finished mcelog.cron
Jun 19 16:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1245 finished 0anacron
Jun 19 16:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1241 finished 0anacron
Jun 19 16:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1245 starting mcelog.cron
Jun 19 16:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1241 starting mcelog.cron
Jun 19 16:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1234 finished 0anacron
Jun 19 16:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1234 starting mcelog.cron
Jun 19 16:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1245 finished mcelog.cron
Jun 19 16:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1241 finished mcelog.cron
Jun 19 16:01:01 weisshorn11 CROND[127656]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 16:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1234 finished mcelog.cron
Jun 19 16:01:01 weisshorn06 CROND[125550]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 16:01:02 weisshorn11 run-parts(/etc/cron.hourly)[1276 starting 0anacron
Jun 19 16:01:02 weisshorn06 run-parts(/etc/cron.hourly)[1255 starting 0anacron
Jun 19 16:01:02 weisshorn01 CROND[6478]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 16:01:02 weisshorn06 run-parts(/etc/cron.hourly)[1255 finished 0anacron
Jun 19 16:01:02 weisshorn06 run-parts(/etc/cron.hourly)[1255 starting mcelog.cron
Jun 19 16:01:02 weisshorn11 run-parts(/etc/cron.hourly)[1276 finished 0anacron
Jun 19 16:01:02 weisshorn11 run-parts(/etc/cron.hourly)[1276 starting mcelog.cron
Jun 19 16:01:02 weisshorn01 run-parts(/etc/cron.hourly)[6478 starting 0anacron
Jun 19 16:01:02 weisshorn06 run-parts(/etc/cron.hourly)[1255 finished mcelog.cron
Jun 19 16:01:02 weisshorn11 run-parts(/etc/cron.hourly)[1276 finished mcelog.cron
Jun 19 16:01:02 weisshorn01 run-parts(/etc/cron.hourly)[6488 finished 0anacron
Jun 19 16:01:02 weisshorn01 run-parts(/etc/cron.hourly)[6478 starting mcelog.cron
Jun 19 16:01:02 weisshorn01 run-parts(/etc/cron.hourly)[6495 finished mcelog.cron
Jun 19 16:01:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:01:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:01:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:01:46 weisshorn14 nslcd[3437]: [263dec] ldap_result() failed: Can't contact LDAP server
Jun 19 16:01:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:02:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:02:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:02:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:02:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:03:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:03:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:03:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:03:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:04:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:04:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:04:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:04:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:05:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:05:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:05:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:05:36 weisshorn03 nslcd[3944]: [6be6b9] ldap_result() failed: Can't contact LDAP server
Jun 19 16:05:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:06:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:06:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:06:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:06:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:07:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:07:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:07:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:07:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:08:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:08:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:08:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:08:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:09:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:09:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:09:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:09:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:10:01 weisshorn06 CROND[125588]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:10:01 weisshorn11 CROND[127700]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:10:01 weisshorn07 CROND[124841]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:10:01 weisshorn01 CROND[6536]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 16:10:01 weisshorn01 CROND[6537]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:10:01 weisshorn09 CROND[124414]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:10:01 weisshorn05 CROND[124697]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:10:01 weisshorn10 CROND[124725]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:10:01 weisshorn02 CROND[46023]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:10:01 weisshorn14 CROND[53345]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:10:01 weisshorn13 CROND[128115]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:10:01 weisshorn14 sshd[53342]: Accepted publickey for root from 148.187.12.231 port 49760 ssh2
Jun 19 16:10:01 weisshorn10 sshd[124727]: Accepted publickey for root from 148.187.12.231 port 37125 ssh2
Jun 19 16:10:01 weisshorn09 sshd[124416]: Accepted publickey for root from 148.187.12.231 port 58091 ssh2
Jun 19 16:10:01 weisshorn14 sshd[53342]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 16:10:01 weisshorn10 sshd[124727]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 16:10:01 weisshorn09 sshd[124416]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 16:10:01 weisshorn13 sshd[128112]: Accepted publickey for root from 148.187.12.231 port 44815 ssh2
Jun 19 16:10:01 weisshorn03 CROND[125381]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:10:01 weisshorn08 sshd[123492]: Accepted publickey for root from 148.187.12.231 port 40452 ssh2
Jun 19 16:10:01 weisshorn08 sshd[123492]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 16:10:01 weisshorn03 sshd[125378]: Accepted publickey for root from 148.187.12.231 port 50656 ssh2
Jun 19 16:10:01 weisshorn05 sshd[124699]: Accepted publickey for root from 148.187.12.231 port 54859 ssh2
Jun 19 16:10:01 weisshorn05 sshd[124699]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 16:10:01 weisshorn06 sshd[125590]: Accepted publickey for root from 148.187.12.231 port 44905 ssh2
Jun 19 16:10:01 weisshorn13 sshd[128112]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 16:10:01 weisshorn12 sshd[124583]: Accepted publickey for root from 148.187.12.231 port 46556 ssh2
Jun 19 16:10:01 weisshorn06 sshd[125590]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 16:10:01 weisshorn14 sshd[53342]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 16:10:01 weisshorn14 sshd[53342]: pam_unix(sshd:session): session closed for user root
Jun 19 16:10:01 weisshorn04 sshd[124168]: Accepted publickey for root from 148.187.12.231 port 44402 ssh2
Jun 19 16:10:01 weisshorn04 CROND[124172]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:10:01 weisshorn12 sshd[124583]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 16:10:01 weisshorn12 CROND[124587]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:10:01 weisshorn07 sshd[124843]: Accepted publickey for root from 148.187.12.231 port 48461 ssh2
Jun 19 16:10:01 weisshorn08 CROND[123503]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:10:02 weisshorn04 sshd[124168]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 16:10:02 weisshorn03 sshd[125378]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 16:10:02 weisshorn07 sshd[124843]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 16:10:02 weisshorn11 sshd[127702]: Accepted publickey for root from 148.187.12.231 port 45334 ssh2
Jun 19 16:10:02 weisshorn11 sshd[127702]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 16:10:02 weisshorn09 sshd[124416]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 16:10:02 weisshorn09 sshd[124416]: pam_unix(sshd:session): session closed for user root
Jun 19 16:10:02 weisshorn10 sshd[124727]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 16:10:02 weisshorn10 sshd[124727]: pam_unix(sshd:session): session closed for user root
Jun 19 16:10:02 weisshorn08 sshd[123492]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 16:10:02 weisshorn08 sshd[123492]: pam_unix(sshd:session): session closed for user root
Jun 19 16:10:02 weisshorn06 sshd[125590]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 16:10:02 weisshorn06 sshd[125590]: pam_unix(sshd:session): session closed for user root
Jun 19 16:10:02 weisshorn04 sshd[124168]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 16:10:02 weisshorn04 sshd[124168]: pam_unix(sshd:session): session closed for user root
Jun 19 16:10:02 weisshorn11 sshd[127702]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 16:10:02 weisshorn11 sshd[127702]: pam_unix(sshd:session): session closed for user root
Jun 19 16:10:02 weisshorn03 sshd[125378]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 16:10:02 weisshorn03 sshd[125378]: pam_unix(sshd:session): session closed for user root
Jun 19 16:10:02 weisshorn07 sshd[124843]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 16:10:02 weisshorn07 sshd[124843]: pam_unix(sshd:session): session closed for user root
Jun 19 16:10:02 weisshorn13 sshd[128112]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 16:10:02 weisshorn13 sshd[128112]: pam_unix(sshd:session): session closed for user root
Jun 19 16:10:02 weisshorn12 sshd[124583]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 16:10:02 weisshorn12 sshd[124583]: pam_unix(sshd:session): session closed for user root
Jun 19 16:10:02 weisshorn05 sshd[124699]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 16:10:02 weisshorn05 sshd[124699]: pam_unix(sshd:session): session closed for user root
Jun 19 16:10:02 weisshorn01 postfix/pickup[6318]: 6EBD892224F: uid=0 from=<root>
Jun 19 16:10:02 weisshorn01 postfix/cleanup[6568]: 6EBD892224F: message-id=<20120619141002.6EBD892224F@weisshorn01.admin.cscs.ch>
Jun 19 16:10:02 weisshorn01 postfix/qmgr[2710]: 6EBD892224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 16:10:02 weisshorn01 postfix/local[6570]: 6EBD892224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=0.61, delays=0.49/0.06/0/0.06, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 16:10:02 weisshorn01 postfix/qmgr[2710]: 6EBD892224F: removed
Jun 19 16:10:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:10:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:10:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:10:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:11:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:11:06 weisshorn02 nslcd[2099]: [d3afd2] ldap_result() failed: Can't contact LDAP server
Jun 19 16:11:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:11:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:11:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:12:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:12:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:12:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:12:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:13:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:13:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:13:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:13:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:14:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:14:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:14:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:14:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:15:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:15:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:15:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:15:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:16:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:16:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:16:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:16:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:17:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:17:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:17:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:17:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:18:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:18:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:18:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:18:44 weisshorn02 ntpd[2541]: synchronized to 148.187.18.145, stratum 2
Jun 19 16:18:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:19:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:19:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:19:22 weisshorn08 rhnsd[123543]: running program /usr/sbin/rhn_check
Jun 19 16:19:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:19:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:20:01 weisshorn03 CROND[125433]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:20:01 weisshorn04 CROND[124225]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:20:01 weisshorn12 CROND[124629]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:20:01 weisshorn06 CROND[125647]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:20:01 weisshorn08 CROND[123554]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:20:01 weisshorn11 CROND[127748]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:20:01 weisshorn07 CROND[124940]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:20:01 weisshorn09 CROND[124473]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:20:01 weisshorn01 CROND[6696]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:20:01 weisshorn05 CROND[124748]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:20:01 weisshorn10 CROND[124781]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:20:01 weisshorn14 CROND[53393]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:20:01 weisshorn02 CROND[46092]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:20:01 weisshorn13 CROND[128163]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:20:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:20:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:20:26 weisshorn01 rhnsd[6700]: running program /usr/sbin/rhn_check
Jun 19 16:20:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:20:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:21:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:21:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:21:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:21:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:22:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:22:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:22:34 weisshorn01 kernel: Lustre: 5248:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046e5:0x107ac:0x0]] [valid 68719476736]
Jun 19 16:22:34 weisshorn01 kernel: Lustre: 5248:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 3 previous similar messages
Jun 19 16:22:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:22:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:23:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:23:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:23:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:23:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:24:02 weisshorn01 kernel: Lustre: 5465:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046ee:0x1673:0x0]] [valid 68719476736]
Jun 19 16:24:02 weisshorn01 kernel: Lustre: 5465:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 2 previous similar messages
Jun 19 16:24:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:24:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:24:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:24:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:25:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:25:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:25:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:25:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:26:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:26:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:26:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:26:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:26:55 weisshorn01 kernel: Lustre: 5456:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046e5:0x10917:0x0]] [valid 68719476736]
Jun 19 16:26:55 weisshorn01 kernel: Lustre: 5456:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 5 previous similar messages
Jun 19 16:27:05 weisshorn02 rhnsd[46107]: running program /usr/sbin/rhn_check
Jun 19 16:27:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:27:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:27:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:27:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:28:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:28:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:28:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:28:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:29:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:29:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:29:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:29:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:30:01 weisshorn13 CROND[128233]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:30:01 weisshorn03 CROND[125497]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:30:01 weisshorn04 CROND[124257]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:30:01 weisshorn12 CROND[124696]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:30:01 weisshorn06 CROND[125706]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:30:01 weisshorn11 CROND[127812]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:30:01 weisshorn08 CROND[123612]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:30:01 weisshorn07 CROND[124959]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:30:01 weisshorn09 CROND[124532]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:30:01 weisshorn10 CROND[124813]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:30:01 weisshorn05 CROND[124785]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:30:01 weisshorn01 CROND[6764]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:30:01 weisshorn14 CROND[53458]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:30:01 weisshorn02 CROND[46118]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:30:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:30:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:30:26 weisshorn05 rhnsd[124788]: running program /usr/sbin/rhn_check
Jun 19 16:30:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:30:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:31:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:31:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:31:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:31:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:32:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:32:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:32:31 weisshorn01 kernel: Lustre: 5444:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046e5:0x10920:0x0]] [valid 68719476736]
Jun 19 16:32:31 weisshorn01 kernel: Lustre: 5444:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 1 previous similar message
Jun 19 16:32:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:32:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:33:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:33:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:33:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:33:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:34:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:34:19 weisshorn11 ntpd[4454]: synchronized to 148.187.18.145, stratum 2
Jun 19 16:34:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:34:24 weisshorn06 ntpd[4454]: synchronized to 148.187.18.145, stratum 2
Jun 19 16:34:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:34:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:35:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:35:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:35:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:35:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:35:57 weisshorn12 ntpd[4454]: synchronized to 148.187.18.145, stratum 2
Jun 19 16:36:05 weisshorn13 ntpd[4726]: synchronized to 148.187.18.145, stratum 2
Jun 19 16:36:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:36:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:36:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:36:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:37:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:37:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:37:29 weisshorn04 ntpd[4387]: synchronized to 148.187.18.145, stratum 2
Jun 19 16:37:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:37:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:38:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:38:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:38:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:38:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:39:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:39:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:39:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:39:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:40:01 weisshorn14 CROND[53488]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:40:01 weisshorn02 CROND[46141]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:40:01 weisshorn13 CROND[128259]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:40:01 weisshorn03 CROND[125521]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:40:01 weisshorn04 CROND[124312]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:40:01 weisshorn12 CROND[124727]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:40:01 weisshorn06 CROND[125731]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:40:01 weisshorn11 CROND[127843]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:40:01 weisshorn08 CROND[123636]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:40:01 weisshorn07 CROND[124983]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:40:01 weisshorn09 CROND[124556]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:40:01 weisshorn10 CROND[124867]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:40:02 weisshorn05 CROND[124840]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:40:02 weisshorn01 CROND[6833]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:40:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:40:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:40:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:40:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:41:01 weisshorn02 nslcd[2099]: [f93092] ldap_result() failed: Can't contact LDAP server
Jun 19 16:41:01 weisshorn14 nslcd[3437]: [0d8c4a] ldap_result() failed: Can't contact LDAP server
Jun 19 16:41:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:41:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:41:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:41:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:42:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:42:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:42:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:42:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:43:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:43:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:43:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:43:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:44:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:44:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:44:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:44:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:45:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:45:17 weisshorn10 nslcd[4294]: [b59621] ldap_result() failed: Can't contact LDAP server
Jun 19 16:45:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:45:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:45:45 weisshorn09 nslcd[3954]: [6be6b9] ldap_result() failed: Can't contact LDAP server
Jun 19 16:45:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:46:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:46:08 weisshorn08 nslcd[3946]: [b59621] ldap_result() failed: Can't contact LDAP server
Jun 19 16:46:09 weisshorn06 nslcd[3953]: [d601af] ldap_result() failed: Can't contact LDAP server
Jun 19 16:46:11 weisshorn04 nslcd[3885]: [6be6b9] ldap_result() failed: Can't contact LDAP server
Jun 19 16:46:12 weisshorn07 nslcd[4297]: [b59621] ldap_result() failed: Can't contact LDAP server
Jun 19 16:46:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:46:32 weisshorn11 nslcd[3954]: [0e6897] ldap_result() failed: Can't contact LDAP server
Jun 19 16:46:32 weisshorn12 nslcd[3954]: [b59621] ldap_result() failed: Can't contact LDAP server
Jun 19 16:46:34 weisshorn13 nslcd[4224]: [b59621] ldap_result() failed: Can't contact LDAP server
Jun 19 16:46:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:46:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:47:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:47:11 weisshorn05 nslcd[4378]: [92a02e] ldap_result() failed: Can't contact LDAP server
Jun 19 16:47:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:47:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:47:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:48:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:48:20 weisshorn01 ntpd[2609]: synchronized to 148.187.18.145, stratum 2
Jun 19 16:48:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:48:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:48:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:48:59 weisshorn09 ntpd[4454]: synchronized to 148.187.18.145, stratum 2
Jun 19 16:49:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:49:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:49:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:49:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:50:01 weisshorn14 CROND[53553]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:50:01 weisshorn02 CROND[46210]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:50:01 weisshorn05 CROND[124877]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:50:01 weisshorn04 CROND[124349]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:50:01 weisshorn06 CROND[125776]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:50:01 weisshorn03 CROND[125560]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:50:01 weisshorn01 CROND[6938]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:50:01 weisshorn11 CROND[127882]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:50:01 weisshorn12 CROND[124793]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:50:01 weisshorn13 CROND[128330]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:50:01 weisshorn08 CROND[123676]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:50:01 weisshorn07 CROND[125063]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:50:01 weisshorn09 CROND[124597]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:50:01 weisshorn10 CROND[124904]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 16:50:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:50:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:50:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:50:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:51:02 weisshorn13 nslcd[4224]: [92a02e] ldap_result() failed: Can't contact LDAP server
Jun 19 16:51:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:51:16 weisshorn10 nslcd[4294]: [92a02e] ldap_result() failed: Can't contact LDAP server
Jun 19 16:51:16 weisshorn11 nslcd[3954]: [5138de] ldap_result() failed: Can't contact LDAP server
Jun 19 16:51:16 weisshorn12 nslcd[3954]: [92a02e] ldap_result() failed: Can't contact LDAP server
Jun 19 16:51:16 weisshorn03 nslcd[3944]: [234363] ldap_result() failed: Can't contact LDAP server
Jun 19 16:51:16 weisshorn05 nslcd[4378]: [ef532d] ldap_result() failed: Can't contact LDAP server
Jun 19 16:51:16 weisshorn04 nslcd[3885]: [234363] ldap_result() failed: Can't contact LDAP server
Jun 19 16:51:16 weisshorn09 nslcd[3954]: [234363] ldap_result() failed: Can't contact LDAP server
Jun 19 16:51:16 weisshorn07 nslcd[4297]: [92a02e] ldap_result() failed: Can't contact LDAP server
Jun 19 16:51:16 weisshorn08 nslcd[3946]: [92a02e] ldap_result() failed: Can't contact LDAP server
Jun 19 16:51:17 weisshorn06 nslcd[3953]: [c77402] ldap_result() failed: Can't contact LDAP server
Jun 19 16:51:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:51:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:51:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:52:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:52:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:52:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:52:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:53:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:53:10 weisshorn05 ntpd[4877]: synchronized to 148.187.18.145, stratum 2
Jun 19 16:53:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:53:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:53:42 weisshorn07 ntpd[4804]: synchronized to 148.187.18.145, stratum 2
Jun 19 16:53:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:54:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:54:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:54:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:54:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:55:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:55:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:55:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:55:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:56:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:56:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:56:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:56:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:56:59 weisshorn09 rhnsd[124652]: running program /usr/sbin/rhn_check
Jun 19 16:57:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:57:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:57:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:57:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:58:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:58:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:58:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:58:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:59:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:59:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:59:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 16:59:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:00:01 weisshorn10 CROND[124937]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:00:01 weisshorn14 CROND[53583]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:00:01 weisshorn02 CROND[46234]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:00:01 weisshorn05 CROND[124911]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:00:01 weisshorn04 CROND[124382]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:00:01 weisshorn06 CROND[125831]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:00:01 weisshorn11 CROND[127937]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:00:01 weisshorn03 CROND[125620]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:00:01 weisshorn01 CROND[7018]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:00:01 weisshorn13 CROND[128361]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:00:01 weisshorn12 CROND[124821]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:00:01 weisshorn08 CROND[123736]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:00:01 weisshorn07 CROND[125084]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:00:01 weisshorn09 CROND[124659]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:00:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:00:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:00:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:00:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:01:01 weisshorn10 CROND[124942]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 17:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1249 starting 0anacron
Jun 19 17:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1249 finished 0anacron
Jun 19 17:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1249 starting mcelog.cron
Jun 19 17:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1249 finished mcelog.cron
Jun 19 17:01:01 weisshorn14 CROND[53589]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 17:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5358 starting 0anacron
Jun 19 17:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5359 finished 0anacron
Jun 19 17:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5358 starting mcelog.cron
Jun 19 17:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5360 finished mcelog.cron
Jun 19 17:01:01 weisshorn02 CROND[46238]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 17:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4623 starting 0anacron
Jun 19 17:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4624 finished 0anacron
Jun 19 17:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4623 starting mcelog.cron
Jun 19 17:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4625 finished mcelog.cron
Jun 19 17:01:01 weisshorn05 CROND[124916]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 17:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1249 starting 0anacron
Jun 19 17:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1249 finished 0anacron
Jun 19 17:01:01 weisshorn04 CROND[124387]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 17:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1249 starting mcelog.cron
Jun 19 17:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1243 starting 0anacron
Jun 19 17:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1243 finished 0anacron
Jun 19 17:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1243 starting mcelog.cron
Jun 19 17:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1249 finished mcelog.cron
Jun 19 17:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1244 finished mcelog.cron
Jun 19 17:01:01 weisshorn06 CROND[125836]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 17:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1258 starting 0anacron
Jun 19 17:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1258 finished 0anacron
Jun 19 17:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1258 starting mcelog.cron
Jun 19 17:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1258 finished mcelog.cron
Jun 19 17:01:01 weisshorn11 CROND[127947]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 17:01:01 weisshorn03 CROND[125625]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 17:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1279 starting 0anacron
Jun 19 17:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1279 finished 0anacron
Jun 19 17:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1279 starting mcelog.cron
Jun 19 17:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1256 starting 0anacron
Jun 19 17:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1279 finished mcelog.cron
Jun 19 17:01:01 weisshorn13 CROND[128366]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 17:01:01 weisshorn12 CROND[124831]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 17:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1283 starting 0anacron
Jun 19 17:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1256 finished 0anacron
Jun 19 17:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1256 starting mcelog.cron
Jun 19 17:01:01 weisshorn01 CROND[7027]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 17:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1283 finished 0anacron
Jun 19 17:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1283 starting mcelog.cron
Jun 19 17:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1283 finished mcelog.cron
Jun 19 17:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1248 starting 0anacron
Jun 19 17:01:01 weisshorn01 run-parts(/etc/cron.hourly)[7027 starting 0anacron
Jun 19 17:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1256 finished mcelog.cron
Jun 19 17:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1248 finished 0anacron
Jun 19 17:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1248 starting mcelog.cron
Jun 19 17:01:01 weisshorn07 CROND[125089]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 17:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1250 starting 0anacron
Jun 19 17:01:01 weisshorn01 run-parts(/etc/cron.hourly)[7037 finished 0anacron
Jun 19 17:01:01 weisshorn01 run-parts(/etc/cron.hourly)[7027 starting mcelog.cron
Jun 19 17:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1248 finished mcelog.cron
Jun 19 17:01:01 weisshorn09 CROND[124664]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 17:01:01 weisshorn08 CROND[123741]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 17:01:01 weisshorn01 run-parts(/etc/cron.hourly)[7044 finished mcelog.cron
Jun 19 17:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1250 finished 0anacron
Jun 19 17:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1250 starting mcelog.cron
Jun 19 17:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1237 starting 0anacron
Jun 19 17:01:02 weisshorn09 run-parts(/etc/cron.hourly)[1246 starting 0anacron
Jun 19 17:01:02 weisshorn07 run-parts(/etc/cron.hourly)[1251 finished mcelog.cron
Jun 19 17:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1237 finished 0anacron
Jun 19 17:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1237 starting mcelog.cron
Jun 19 17:01:02 weisshorn09 run-parts(/etc/cron.hourly)[1246 finished 0anacron
Jun 19 17:01:02 weisshorn09 run-parts(/etc/cron.hourly)[1246 starting mcelog.cron
Jun 19 17:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1237 finished mcelog.cron
Jun 19 17:01:02 weisshorn09 run-parts(/etc/cron.hourly)[1246 finished mcelog.cron
Jun 19 17:01:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:01:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:01:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:01:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:02:01 weisshorn14 nslcd[3437]: [d4d3c4] ldap_result() failed: Can't contact LDAP server
Jun 19 17:02:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:02:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:02:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:02:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:02:55 weisshorn08 ntpd[4445]: synchronized to 148.187.18.145, stratum 2
Jun 19 17:03:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:03:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:03:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:03:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:04:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:04:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:04:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:04:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:05:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:05:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:05:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:05:51 weisshorn03 nslcd[3944]: [d385c9] ldap_result() failed: Can't contact LDAP server
Jun 19 17:05:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:06:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:06:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:06:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:06:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:07:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:07:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:07:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:07:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:08:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:08:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:08:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:08:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:09:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:09:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:09:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:09:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:10:01 weisshorn08 CROND[123786]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:10:01 weisshorn09 CROND[124716]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:10:01 weisshorn07 CROND[125145]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:10:01 weisshorn10 CROND[125021]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:10:01 weisshorn14 CROND[53639]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:10:01 weisshorn02 CROND[46316]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:10:01 weisshorn05 CROND[124995]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:10:01 weisshorn04 CROND[124466]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:10:01 weisshorn06 CROND[125887]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:10:01 weisshorn11 CROND[127994]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:10:01 weisshorn13 CROND[128410]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:10:01 weisshorn03 CROND[125678]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:10:01 weisshorn12 CROND[124870]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:10:01 weisshorn01 CROND[7100]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:10:01 weisshorn01 CROND[7102]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 17:10:02 weisshorn11 sshd[127996]: Accepted publickey for root from 148.187.12.231 port 45352 ssh2
Jun 19 17:10:02 weisshorn11 sshd[127996]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 17:10:02 weisshorn13 sshd[128412]: Accepted publickey for root from 148.187.12.231 port 44839 ssh2
Jun 19 17:10:02 weisshorn14 sshd[53641]: Accepted publickey for root from 148.187.12.231 port 49780 ssh2
Jun 19 17:10:02 weisshorn09 sshd[124718]: Accepted publickey for root from 148.187.12.231 port 58115 ssh2
Jun 19 17:10:02 weisshorn04 sshd[124468]: Accepted publickey for root from 148.187.12.231 port 44425 ssh2
Jun 19 17:10:02 weisshorn13 sshd[128412]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 17:10:02 weisshorn14 sshd[53641]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 17:10:02 weisshorn09 sshd[124718]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 17:10:02 weisshorn06 sshd[125889]: Accepted publickey for root from 148.187.12.231 port 44936 ssh2
Jun 19 17:10:02 weisshorn04 sshd[124468]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 17:10:02 weisshorn10 sshd[125023]: Accepted publickey for root from 148.187.12.231 port 37139 ssh2
Jun 19 17:10:02 weisshorn06 sshd[125889]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 17:10:02 weisshorn08 sshd[123788]: Accepted publickey for root from 148.187.12.231 port 40474 ssh2
Jun 19 17:10:02 weisshorn10 sshd[125023]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 17:10:02 weisshorn05 sshd[124997]: Accepted publickey for root from 148.187.12.231 port 54870 ssh2
Jun 19 17:10:02 weisshorn08 sshd[123788]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 17:10:02 weisshorn12 sshd[124872]: Accepted publickey for root from 148.187.12.231 port 46574 ssh2
Jun 19 17:10:02 weisshorn14 sshd[53641]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 17:10:02 weisshorn14 sshd[53641]: pam_unix(sshd:session): session closed for user root
Jun 19 17:10:02 weisshorn03 sshd[125680]: Accepted publickey for root from 148.187.12.231 port 50670 ssh2
Jun 19 17:10:02 weisshorn07 sshd[125147]: Accepted publickey for root from 148.187.12.231 port 48482 ssh2
Jun 19 17:10:02 weisshorn03 sshd[125680]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 17:10:02 weisshorn07 sshd[125147]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 17:10:02 weisshorn09 sshd[124718]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 17:10:02 weisshorn09 sshd[124718]: pam_unix(sshd:session): session closed for user root
Jun 19 17:10:02 weisshorn05 sshd[124997]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 17:10:02 weisshorn12 sshd[124872]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 17:10:02 weisshorn04 sshd[124468]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 17:10:02 weisshorn04 sshd[124468]: pam_unix(sshd:session): session closed for user root
Jun 19 17:10:02 weisshorn13 sshd[128412]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 17:10:02 weisshorn13 sshd[128412]: pam_unix(sshd:session): session closed for user root
Jun 19 17:10:02 weisshorn11 sshd[127996]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 17:10:02 weisshorn11 sshd[127996]: pam_unix(sshd:session): session closed for user root
Jun 19 17:10:02 weisshorn10 sshd[125023]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 17:10:02 weisshorn10 sshd[125023]: pam_unix(sshd:session): session closed for user root
Jun 19 17:10:02 weisshorn06 sshd[125889]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 17:10:02 weisshorn06 sshd[125889]: pam_unix(sshd:session): session closed for user root
Jun 19 17:10:02 weisshorn08 sshd[123788]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 17:10:02 weisshorn08 sshd[123788]: pam_unix(sshd:session): session closed for user root
Jun 19 17:10:02 weisshorn03 sshd[125680]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 17:10:02 weisshorn03 sshd[125680]: pam_unix(sshd:session): session closed for user root
Jun 19 17:10:02 weisshorn07 sshd[125147]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 17:10:02 weisshorn07 sshd[125147]: pam_unix(sshd:session): session closed for user root
Jun 19 17:10:02 weisshorn05 sshd[124997]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 17:10:02 weisshorn05 sshd[124997]: pam_unix(sshd:session): session closed for user root
Jun 19 17:10:02 weisshorn12 sshd[124872]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 17:10:02 weisshorn12 sshd[124872]: pam_unix(sshd:session): session closed for user root
Jun 19 17:10:02 weisshorn01 postfix/pickup[6318]: BE44992224F: uid=0 from=<root>
Jun 19 17:10:02 weisshorn01 postfix/cleanup[7132]: BE44992224F: message-id=<20120619151002.BE44992224F@weisshorn01.admin.cscs.ch>
Jun 19 17:10:02 weisshorn01 postfix/qmgr[2710]: BE44992224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 17:10:02 weisshorn01 postfix/local[7134]: BE44992224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=0.41, delays=0.32/0.04/0/0.05, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 17:10:02 weisshorn01 postfix/qmgr[2710]: BE44992224F: removed
Jun 19 17:10:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:10:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:10:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:10:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:11:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:11:21 weisshorn02 nslcd[2099]: [63e8d2] ldap_result() failed: Can't contact LDAP server
Jun 19 17:11:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:11:34 weisshorn01 kernel: Lustre: 5304:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046e5:0x10942:0x0]] [valid 68719476736]
Jun 19 17:11:34 weisshorn01 kernel: Lustre: 5304:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 1 previous similar message
Jun 19 17:11:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:11:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:12:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:12:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:12:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:12:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:13:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:13:13 weisshorn01 kernel: Lustre: 5431:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046ee:0x16a5:0x0]] [valid 68719476736]
Jun 19 17:13:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:13:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:13:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:14:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:14:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:14:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:14:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:15:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:15:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:15:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:15:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:15:57 weisshorn01 kernel: Lustre: 5436:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046e5:0x10952:0x0]] [valid 68719476736]
Jun 19 17:15:57 weisshorn01 kernel: Lustre: 5436:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 1 previous similar message
Jun 19 17:16:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:16:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:16:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:16:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:17:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:17:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:17:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:17:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:18:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:18:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:18:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:18:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:19:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:19:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:19:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:19:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:20:01 weisshorn12 CROND[124955]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:20:01 weisshorn09 CROND[124761]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:20:01 weisshorn08 CROND[123839]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:20:01 weisshorn10 CROND[125066]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:20:01 weisshorn07 CROND[125226]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:20:01 weisshorn14 CROND[53719]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:20:01 weisshorn02 CROND[46344]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:20:01 weisshorn04 CROND[124511]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:20:01 weisshorn01 CROND[7245]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:20:01 weisshorn06 CROND[125938]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:20:01 weisshorn05 CROND[125040]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:20:01 weisshorn11 CROND[128044]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:20:01 weisshorn03 CROND[125723]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:20:01 weisshorn13 CROND[128495]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:20:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:20:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:20:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:20:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:21:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:21:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:21:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:21:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:22:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:22:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:22:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:22:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:23:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:23:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:23:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:23:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:24:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:24:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:24:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:24:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:24:58 weisshorn01 kernel: Lustre: 4735:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004631:0x168dc:0x0]] [valid 68719476736]
Jun 19 17:24:58 weisshorn01 kernel: Lustre: 4735:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 5 previous similar messages
Jun 19 17:25:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:25:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:25:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:25:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:26:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:26:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:26:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:26:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:27:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:27:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:27:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:27:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:28:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:28:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:28:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:28:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:29:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:29:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:29:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:29:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:30:01 weisshorn03 CROND[125782]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:30:01 weisshorn11 CROND[128103]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:30:01 weisshorn13 CROND[128525]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:30:01 weisshorn12 CROND[124987]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:30:01 weisshorn09 CROND[124820]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:30:01 weisshorn07 CROND[125245]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:30:01 weisshorn14 CROND[53749]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:30:01 weisshorn10 CROND[125098]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:30:01 weisshorn08 CROND[123898]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:30:01 weisshorn02 CROND[46367]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:30:01 weisshorn04 CROND[124543]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:30:01 weisshorn01 CROND[7299]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:30:01 weisshorn06 CROND[125992]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:30:02 weisshorn05 CROND[125072]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:30:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:30:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:30:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:30:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:31:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:31:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:31:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:31:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:32:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:32:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:32:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:32:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:33:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:33:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:33:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:33:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:34:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:34:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:34:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:34:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:35:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:35:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:35:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:35:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:36:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:36:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:36:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:36:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:37:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:37:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:37:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:37:47 weisshorn01 kernel: Lustre: 4733:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x29a8:0x0]] [valid 68719476736]
Jun 19 17:37:47 weisshorn01 kernel: Lustre: 4733:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 5 previous similar messages
Jun 19 17:37:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:38:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:38:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:38:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:38:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:39:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:39:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:39:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:39:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:40:01 weisshorn03 CROND[125820]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:40:01 weisshorn11 CROND[128136]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:40:01 weisshorn13 CROND[128555]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:40:01 weisshorn05 CROND[125137]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:40:01 weisshorn12 CROND[125017]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:40:01 weisshorn09 CROND[124863]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:40:01 weisshorn07 CROND[125322]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:40:01 weisshorn14 CROND[53789]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:40:01 weisshorn10 CROND[125163]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:40:01 weisshorn08 CROND[123933]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:40:01 weisshorn02 CROND[46436]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:40:01 weisshorn04 CROND[124608]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:40:01 weisshorn01 CROND[7411]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:40:01 weisshorn06 CROND[126029]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:40:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:40:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:40:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:40:55 weisshorn13 kernel: Lustre: scratch-OST0035: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880804559400, cur 1340120455 expire 1340120305 last 1340120228
Jun 19 17:40:55 weisshorn13 kernel: Lustre: Skipped 19 previous similar messages
Jun 19 17:40:55 weisshorn11 kernel: Lustre: scratch-OST0044: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880800ab0800, cur 1340120455 expire 1340120305 last 1340120228
Jun 19 17:40:55 weisshorn11 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 17:40:55 weisshorn12 kernel: Lustre: scratch-OST0032: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eb8029400, cur 1340120455 expire 1340120305 last 1340120228
Jun 19 17:40:55 weisshorn12 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 17:40:55 weisshorn07 kernel: Lustre: scratch-OST0037: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808bce07c00, cur 1340120455 expire 1340120305 last 1340120228
Jun 19 17:40:55 weisshorn07 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 17:40:55 weisshorn05 kernel: Lustre: scratch-OST0027: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100ffa8c00, cur 1340120455 expire 1340120305 last 1340120228
Jun 19 17:40:55 weisshorn05 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 17:40:55 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810293c4c00, cur 1340120455 expire 1340120305 last 1340120228
Jun 19 17:40:55 weisshorn08 kernel: Lustre: Skipped 7 previous similar messages
Jun 19 17:40:55 weisshorn10 kernel: Lustre: scratch-OST0016: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c44391000, cur 1340120455 expire 1340120305 last 1340120228
Jun 19 17:40:55 weisshorn10 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 17:40:55 weisshorn03 kernel: Lustre: scratch-OST0009: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804dcbf3c00, cur 1340120455 expire 1340120305 last 1340120228
Jun 19 17:40:55 weisshorn03 kernel: Lustre: Skipped 11 previous similar messages
Jun 19 17:40:55 weisshorn04 kernel: Lustre: scratch-OST0019: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880876eef800, cur 1340120455 expire 1340120305 last 1340120228
Jun 19 17:40:55 weisshorn04 kernel: Lustre: Skipped 15 previous similar messages
Jun 19 17:40:55 weisshorn06 kernel: Lustre: scratch-OST0036: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804fd331000, cur 1340120455 expire 1340120305 last 1340120228
Jun 19 17:40:55 weisshorn06 kernel: Lustre: Skipped 7 previous similar messages
Jun 19 17:40:55 weisshorn09 kernel: Lustre: scratch-OST0026: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080996bc00, cur 1340120455 expire 1340120305 last 1340120228
Jun 19 17:40:55 weisshorn09 kernel: Lustre: Skipped 7 previous similar messages
Jun 19 17:40:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:41:04 weisshorn08 kernel: Lustre: scratch-OST0017: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d30b5d000, cur 1340120464 expire 1340120314 last 1340120237
Jun 19 17:41:04 weisshorn11 kernel: Lustre: scratch-OST0015: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880696b33400, cur 1340120464 expire 1340120314 last 1340120237
Jun 19 17:41:04 weisshorn11 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 17:41:04 weisshorn12 kernel: Lustre: scratch-OST000f: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a7bb95000, cur 1340120464 expire 1340120314 last 1340120237
Jun 19 17:41:04 weisshorn12 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:41:04 weisshorn10 kernel: Lustre: scratch-OST000a: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e9c1f5c00, cur 1340120464 expire 1340120314 last 1340120237
Jun 19 17:41:04 weisshorn09 kernel: Lustre: scratch-OST0000: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880819a85c00, cur 1340120464 expire 1340120314 last 1340120237
Jun 19 17:41:04 weisshorn09 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:41:04 weisshorn03 kernel: Lustre: scratch-OST0014: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88081da65c00, cur 1340120464 expire 1340120314 last 1340120237
Jun 19 17:41:04 weisshorn03 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:41:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:41:16 weisshorn02 nslcd[2099]: [896198] ldap_result() failed: Can't contact LDAP server
Jun 19 17:41:16 weisshorn14 nslcd[3437]: [6f2e30] ldap_result() failed: Can't contact LDAP server
Jun 19 17:41:20 weisshorn04 kernel: Lustre: scratch-OST0045: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fdce99000, cur 1340120480 expire 1340120330 last 1340120253
Jun 19 17:41:20 weisshorn04 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:41:20 weisshorn06 kernel: Lustre: scratch-OST0007: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cc6c29000, cur 1340120480 expire 1340120330 last 1340120253
Jun 19 17:41:20 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:41:20 weisshorn05 kernel: Lustre: scratch-OST0046: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080e02dc00, cur 1340120480 expire 1340120330 last 1340120253
Jun 19 17:41:20 weisshorn05 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:41:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:41:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:41:45 weisshorn12 kernel: Lustre: scratch-OST0047: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e95035400, cur 1340120505 expire 1340120355 last 1340120278
Jun 19 17:41:45 weisshorn13 kernel: Lustre: scratch-OST000b: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880514084c00, cur 1340120505 expire 1340120355 last 1340120278
Jun 19 17:41:45 weisshorn13 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 17:41:45 weisshorn03 kernel: Lustre: scratch-OST0038: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d3702e400, cur 1340120505 expire 1340120355 last 1340120278
Jun 19 17:41:45 weisshorn09 kernel: Lustre: scratch-OST001a: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080378bc00, cur 1340120505 expire 1340120355 last 1340120278
Jun 19 17:41:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:42:10 weisshorn11 kernel: Lustre: scratch-OST0039: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88064f251000, cur 1340120530 expire 1340120380 last 1340120303
Jun 19 17:42:10 weisshorn08 kernel: Lustre: scratch-OST000c: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080a7f3800, cur 1340120530 expire 1340120380 last 1340120303
Jun 19 17:42:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:42:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:42:35 weisshorn12 kernel: Lustre: scratch-OST001c: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b1751d000, cur 1340120555 expire 1340120405 last 1340120328
Jun 19 17:42:35 weisshorn10 kernel: Lustre: scratch-OST003a: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880135482000, cur 1340120555 expire 1340120405 last 1340120328
Jun 19 17:42:35 weisshorn04 kernel: Lustre: scratch-OST000d: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f441aec00, cur 1340120555 expire 1340120405 last 1340120328
Jun 19 17:42:35 weisshorn07 kernel: Lustre: scratch-OST002b: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d6ad53000, cur 1340120555 expire 1340120405 last 1340120328
Jun 19 17:42:35 weisshorn07 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:42:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:42:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:43:00 weisshorn06 kernel: Lustre: scratch-OST001d: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c1db9f800, cur 1340120580 expire 1340120430 last 1340120353
Jun 19 17:43:00 weisshorn01 kernel: Lustre: MGS: haven't heard from client aca8aee1-5c22-ca9e-453b-0ad0e5295f34 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802f32a5800, cur 1340120580 expire 1340120430 last 1340120353
Jun 19 17:43:00 weisshorn01 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 17:43:00 weisshorn05 kernel: Lustre: scratch-OST000e: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102a43dc00, cur 1340120580 expire 1340120430 last 1340120353
Jun 19 17:43:00 weisshorn03 kernel: Lustre: scratch-OST002c: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88081d0c6c00, cur 1340120580 expire 1340120430 last 1340120353
Jun 19 17:43:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:43:25 weisshorn10 kernel: Lustre: scratch-OST002e: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c768d4800, cur 1340120605 expire 1340120455 last 1340120378
Jun 19 17:43:25 weisshorn07 kernel: Lustre: scratch-OST001f: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080a2cd400, cur 1340120605 expire 1340120455 last 1340120378
Jun 19 17:43:25 weisshorn09 kernel: Lustre: scratch-OST0010: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f310c9c00, cur 1340120605 expire 1340120455 last 1340120378
Jun 19 17:43:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:43:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:43:50 weisshorn08 kernel: Lustre: scratch-OST002f: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880dfbad3000, cur 1340120630 expire 1340120480 last 1340120403
Jun 19 17:43:50 weisshorn03 kernel: Lustre: scratch-OST0020: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880458e53800, cur 1340120630 expire 1340120480 last 1340120403
Jun 19 17:43:50 weisshorn05 kernel: Lustre: scratch-OST003e: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880807242400, cur 1340120630 expire 1340120480 last 1340120403
Jun 19 17:43:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:44:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:44:15 weisshorn13 kernel: Lustre: scratch-OST0003: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cfb6c7400, cur 1340120655 expire 1340120505 last 1340120428
Jun 19 17:44:15 weisshorn13 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:44:15 weisshorn06 kernel: Lustre: scratch-OST0012: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880823b16400, cur 1340120655 expire 1340120505 last 1340120428
Jun 19 17:44:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:44:40 weisshorn10 kernel: Lustre: scratch-OST0022: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f2bd59800, cur 1340120680 expire 1340120530 last 1340120453
Jun 19 17:44:40 weisshorn09 kernel: Lustre: scratch-OST0004: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880be4649000, cur 1340120680 expire 1340120530 last 1340120453
Jun 19 17:44:40 weisshorn07 kernel: Lustre: scratch-OST0013: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 214 seconds. I think it's dead, and I am evicting it. exp ffff88081be0cc00, cur 1340120680 expire 1340120530 last 1340120466
Jun 19 17:44:40 weisshorn04 kernel: Lustre: scratch-OST0031: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b6dbb6c00, cur 1340120680 expire 1340120530 last 1340120453
Jun 19 17:44:40 weisshorn12 kernel: Lustre: scratch-OST0040: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a93cb2400, cur 1340120680 expire 1340120530 last 1340120453
Jun 19 17:44:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:44:53 weisshorn06 kernel: Lustre: scratch-OST0041: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cdb94d400, cur 1340120693 expire 1340120543 last 1340120466
Jun 19 17:44:53 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:44:53 weisshorn03 kernel: Lustre: scratch-OST0014: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880818c5dc00, cur 1340120693 expire 1340120543 last 1340120466
Jun 19 17:44:53 weisshorn03 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:44:53 weisshorn09 kernel: Lustre: scratch-OST0034: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881027379400, cur 1340120693 expire 1340120543 last 1340120466
Jun 19 17:44:53 weisshorn09 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:44:53 weisshorn07 kernel: Lustre: scratch-OST0042: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881023c69400, cur 1340120693 expire 1340120543 last 1340120466
Jun 19 17:44:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:45:05 weisshorn11 kernel: Lustre: scratch-OST0015: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881025cff800, cur 1340120705 expire 1340120555 last 1340120478
Jun 19 17:45:05 weisshorn05 kernel: Lustre: scratch-OST0033: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808087ea400, cur 1340120705 expire 1340120555 last 1340120478
Jun 19 17:45:09 weisshorn07 kernel: Lustre: scratch-OST0006: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881028986c00, cur 1340120709 expire 1340120559 last 1340120482
Jun 19 17:45:09 weisshorn07 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:45:09 weisshorn04 kernel: Lustre: scratch-OST0045: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88082aff3400, cur 1340120709 expire 1340120559 last 1340120482
Jun 19 17:45:09 weisshorn04 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:45:09 weisshorn10 kernel: Lustre: scratch-OST000a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881023792c00, cur 1340120709 expire 1340120559 last 1340120482
Jun 19 17:45:09 weisshorn10 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:45:09 weisshorn12 kernel: Lustre: scratch-OST0032: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881012994400, cur 1340120709 expire 1340120559 last 1340120482
Jun 19 17:45:09 weisshorn12 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:45:09 weisshorn11 kernel: Lustre: scratch-OST0021: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880623787000, cur 1340120709 expire 1340120559 last 1340120482
Jun 19 17:45:09 weisshorn11 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:45:09 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880815025000, cur 1340120709 expire 1340120559 last 1340120482
Jun 19 17:45:09 weisshorn13 kernel: Lustre: scratch-OST0035: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88028ef3e800, cur 1340120709 expire 1340120559 last 1340120482
Jun 19 17:45:09 weisshorn13 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 17:45:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:45:18 weisshorn06 kernel: Lustre: scratch-OST0007: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ca7068800, cur 1340120718 expire 1340120568 last 1340120491
Jun 19 17:45:18 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:45:18 weisshorn10 kernel: Lustre: scratch-OST0016: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080b55f000, cur 1340120718 expire 1340120568 last 1340120491
Jun 19 17:45:18 weisshorn03 kernel: Lustre: scratch-OST0043: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080cac3000, cur 1340120718 expire 1340120568 last 1340120491
Jun 19 17:45:18 weisshorn11 kernel: Lustre: scratch-OST0044: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880950f64800, cur 1340120718 expire 1340120568 last 1340120491
Jun 19 17:45:18 weisshorn11 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:45:18 weisshorn08 kernel: Lustre: scratch-OST0017: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880823866000, cur 1340120718 expire 1340120568 last 1340120491
Jun 19 17:45:18 weisshorn04 kernel: Lustre: scratch-OST0025: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806d2be0000, cur 1340120718 expire 1340120568 last 1340120491
Jun 19 17:45:18 weisshorn09 kernel: Lustre: scratch-OST0026: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88081554f000, cur 1340120718 expire 1340120568 last 1340120491
Jun 19 17:45:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:45:30 weisshorn05 kernel: Lustre: scratch-OST0027: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080603a800, cur 1340120730 expire 1340120580 last 1340120503
Jun 19 17:45:30 weisshorn05 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:45:30 weisshorn11 kernel: Lustre: scratch-OST0008: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880818106800, cur 1340120730 expire 1340120580 last 1340120503
Jun 19 17:45:30 weisshorn07 kernel: Lustre: scratch-OST0037: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880800594000, cur 1340120730 expire 1340120580 last 1340120503
Jun 19 17:45:32 weisshorn10 nslcd[4294]: [ef532d] ldap_result() failed: Can't contact LDAP server
Jun 19 17:45:39 weisshorn13 kernel: Lustre: scratch-OST0018: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 236 seconds. I think it's dead, and I am evicting it. exp ffff880295e1a800, cur 1340120739 expire 1340120589 last 1340120503
Jun 19 17:45:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:45:43 weisshorn03 kernel: Lustre: scratch-OST0009: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881027780c00, cur 1340120743 expire 1340120593 last 1340120516
Jun 19 17:45:43 weisshorn04 kernel: Lustre: scratch-OST0019: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810259f6c00, cur 1340120743 expire 1340120593 last 1340120516
Jun 19 17:45:43 weisshorn05 kernel: Lustre: scratch-OST0046: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880819cb2000, cur 1340120743 expire 1340120593 last 1340120516
Jun 19 17:45:43 weisshorn05 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:45:43 weisshorn12 kernel: Lustre: scratch-OST0028: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881022b3e800, cur 1340120743 expire 1340120593 last 1340120516
Jun 19 17:45:47 weisshorn13 kernel: Lustre: scratch-OST0011: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 194 seconds. I think it's dead, and I am evicting it. exp ffff88016976cc00, cur 1340120747 expire 1340120597 last 1340120553
Jun 19 17:45:47 weisshorn13 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 17:45:55 weisshorn06 kernel: Lustre: scratch-OST002a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 214 seconds. I think it's dead, and I am evicting it. exp ffff880dab0ca400, cur 1340120755 expire 1340120605 last 1340120541
Jun 19 17:45:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:46:08 weisshorn08 kernel: Lustre: scratch-OST000c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810293c4800, cur 1340120768 expire 1340120618 last 1340120541
Jun 19 17:46:08 weisshorn13 kernel: Lustre: scratch-OST0029: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880818e51800, cur 1340120768 expire 1340120618 last 1340120541
Jun 19 17:46:08 weisshorn13 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 17:46:08 weisshorn11 kernel: Lustre: scratch-OST0039: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880804b77c00, cur 1340120768 expire 1340120618 last 1340120541
Jun 19 17:46:08 weisshorn11 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:46:08 weisshorn09 kernel: Lustre: scratch-OST001a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881028f68000, cur 1340120768 expire 1340120618 last 1340120541
Jun 19 17:46:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:46:20 weisshorn06 kernel: Lustre: scratch-OST0041: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881025162800, cur 1340120780 expire 1340120630 last 1340120553
Jun 19 17:46:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:46:26 weisshorn13 kernel: Lustre: scratch-OST0024: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 158 seconds. I think it's dead, and I am evicting it. exp ffff8804a29eec00, cur 1340120786 expire 1340120636 last 1340120628
Jun 19 17:46:33 weisshorn12 kernel: Lustre: scratch-OST001c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881022b8ec00, cur 1340120793 expire 1340120643 last 1340120566
Jun 19 17:46:33 weisshorn10 kernel: Lustre: scratch-OST003a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880809153c00, cur 1340120793 expire 1340120643 last 1340120566
Jun 19 17:46:33 weisshorn04 kernel: Lustre: scratch-OST000d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102226f000, cur 1340120793 expire 1340120643 last 1340120566
Jun 19 17:46:33 weisshorn07 kernel: Lustre: scratch-OST002b: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080a006400, cur 1340120793 expire 1340120643 last 1340120566
Jun 19 17:46:33 weisshorn07 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:46:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:46:45 weisshorn08 kernel: Lustre: scratch-OST0017: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809edaf5000, cur 1340120805 expire 1340120655 last 1340120578
Jun 19 17:46:45 weisshorn08 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:46:45 weisshorn09 kernel: Lustre: scratch-OST0026: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88011beae000, cur 1340120805 expire 1340120655 last 1340120578
Jun 19 17:46:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:47:10 weisshorn06 kernel: Lustre: scratch-OST001d: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ab0205800, cur 1340120830 expire 1340120680 last 1340120603
Jun 19 17:47:10 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:47:10 weisshorn03 kernel: Lustre: scratch-OST002c: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880075acd400, cur 1340120830 expire 1340120680 last 1340120603
Jun 19 17:47:10 weisshorn04 kernel: Lustre: scratch-OST003d: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808fb1fd800, cur 1340120830 expire 1340120680 last 1340120603
Jun 19 17:47:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:47:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:47:35 weisshorn07 kernel: Lustre: scratch-OST0013: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880805b82400, cur 1340120855 expire 1340120705 last 1340120628
Jun 19 17:47:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:47:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:48:00 weisshorn13 kernel: Lustre: scratch-OST003c: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88013a09ac00, cur 1340120880 expire 1340120730 last 1340120653
Jun 19 17:48:00 weisshorn12 kernel: Lustre: scratch-OST001c: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807c7c3c400, cur 1340120880 expire 1340120730 last 1340120653
Jun 19 17:48:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:48:13 weisshorn08 kernel: Lustre: scratch-OST003b: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808091f6c00, cur 1340120893 expire 1340120743 last 1340120666
Jun 19 17:48:13 weisshorn11 kernel: Lustre: scratch-OST002d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d3b057c00, cur 1340120893 expire 1340120743 last 1340120666
Jun 19 17:48:25 weisshorn06 kernel: Lustre: scratch-OST0012: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808062e7000, cur 1340120905 expire 1340120755 last 1340120678
Jun 19 17:48:25 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:48:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:48:34 weisshorn01 kernel: Lustre: 4733:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004631:0x16baa:0x0]] [valid 68719476736]
Jun 19 17:48:34 weisshorn01 kernel: Lustre: 4733:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 17 previous similar messages
Jun 19 17:48:38 weisshorn10 kernel: Lustre: scratch-OST0001: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881029ac8400, cur 1340120918 expire 1340120768 last 1340120691
Jun 19 17:48:38 weisshorn04 kernel: Lustre: scratch-OST003d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808073f1000, cur 1340120918 expire 1340120768 last 1340120691
Jun 19 17:48:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:48:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:49:03 weisshorn05 kernel: Lustre: scratch-OST003e: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88081a0afc00, cur 1340120943 expire 1340120793 last 1340120716
Jun 19 17:49:03 weisshorn05 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 17:49:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:49:15 weisshorn09 kernel: Lustre: scratch-OST0004: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881016acfc00, cur 1340120955 expire 1340120805 last 1340120728
Jun 19 17:49:23 weisshorn01 kernel: Lustre: scratch-MDT0000: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ac422cc00, cur 1340120963 expire 1340120813 last 1340120736
Jun 19 17:49:23 weisshorn07 kernel: Lustre: scratch-OST0006: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe2f7a800, cur 1340120963 expire 1340120813 last 1340120736
Jun 19 17:49:23 weisshorn07 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:49:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:49:28 weisshorn03 kernel: Lustre: scratch-OST0014: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880afee55800, cur 1340120968 expire 1340120818 last 1340120741
Jun 19 17:49:28 weisshorn03 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:49:28 weisshorn13 kernel: Lustre: scratch-OST0005: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808047e3800, cur 1340120968 expire 1340120818 last 1340120741
Jun 19 17:49:28 weisshorn13 kernel: Lustre: Skipped 6 previous similar messages
Jun 19 17:49:28 weisshorn12 kernel: Lustre: scratch-OST0032: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88095e3a9800, cur 1340120968 expire 1340120818 last 1340120741
Jun 19 17:49:38 weisshorn01 kernel: LustreError: 5145:0:(mdt_handler.c:2822:mdt_recovery()) operation 400 on unconnected MDS from 12345-148.187.6.6@o2ib2
Jun 19 17:49:38 weisshorn01 kernel: LustreError: 5145:0:(mdt_handler.c:2822:mdt_recovery()) Skipped 9 previous similar messages
Jun 19 17:49:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:49:53 weisshorn11 kernel: Lustre: scratch-OST0008: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880802b5dc00, cur 1340120993 expire 1340120843 last 1340120766
Jun 19 17:49:53 weisshorn08 kernel: Lustre: scratch-OST0017: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880066342800, cur 1340120993 expire 1340120843 last 1340120766
Jun 19 17:49:53 weisshorn08 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 17:49:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:50:01 weisshorn06 CROND[126062]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:50:01 weisshorn11 CROND[128169]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:50:01 weisshorn03 CROND[125848]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:50:01 weisshorn05 CROND[125164]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:50:01 weisshorn13 CROND[128622]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:50:01 weisshorn09 CROND[124885]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:50:01 weisshorn12 CROND[125079]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:50:01 weisshorn14 CROND[53844]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:50:01 weisshorn07 CROND[125352]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:50:01 weisshorn10 CROND[125191]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:50:01 weisshorn08 CROND[123965]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:50:01 weisshorn02 CROND[46459]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:50:01 weisshorn04 CROND[124635]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:50:02 weisshorn01 CROND[7479]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 17:50:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:50:18 weisshorn10 kernel: Lustre: scratch-OST002e: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080b7f6800, cur 1340121018 expire 1340120868 last 1340120791
Jun 19 17:50:18 weisshorn10 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 17:50:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:50:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:50:43 weisshorn06 kernel: Lustre: scratch-OST0007: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881012449000, cur 1340121043 expire 1340120893 last 1340120816
Jun 19 17:50:43 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:50:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:51:08 weisshorn05 kernel: Lustre: scratch-OST0046: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880808251400, cur 1340121068 expire 1340120918 last 1340120841
Jun 19 17:51:08 weisshorn05 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 17:51:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:51:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:51:33 weisshorn04 kernel: Lustre: scratch-OST0045: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803f3653800, cur 1340121093 expire 1340120943 last 1340120866
Jun 19 17:51:33 weisshorn04 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 17:51:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:51:45 weisshorn03 kernel: Lustre: scratch-OST0020: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 189 seconds. I think it's dead, and I am evicting it. exp ffff88101272ec00, cur 1340121105 expire 1340120955 last 1340120916
Jun 19 17:51:45 weisshorn03 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 17:51:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:51:58 weisshorn07 kernel: Lustre: scratch-OST001f: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080b793400, cur 1340121118 expire 1340120968 last 1340120891
Jun 19 17:51:58 weisshorn07 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 17:52:10 weisshorn11 kernel: Lustre: scratch-OST0021: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101622c800, cur 1340121130 expire 1340120980 last 1340120903
Jun 19 17:52:10 weisshorn11 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 17:52:10 weisshorn09 kernel: Lustre: scratch-OST0034: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 164 seconds. I think it's dead, and I am evicting it. exp ffff880801c6e400, cur 1340121130 expire 1340120980 last 1340120966
Jun 19 17:52:10 weisshorn09 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:52:10 weisshorn12 kernel: Lustre: scratch-OST000f: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea28e5400, cur 1340121130 expire 1340120980 last 1340120903
Jun 19 17:52:10 weisshorn12 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 17:52:10 weisshorn13 kernel: Lustre: scratch-OST0035: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 164 seconds. I think it's dead, and I am evicting it. exp ffff8806a7b7d400, cur 1340121130 expire 1340120980 last 1340120966
Jun 19 17:52:10 weisshorn13 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:52:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:52:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:52:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:52:48 weisshorn01 kernel: Lustre: MGS: haven't heard from client 02e8b9ff-fc43-5841-d3bb-815052085829 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b1d8e3800, cur 1340121168 expire 1340121018 last 1340120941
Jun 19 17:52:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:53:00 weisshorn05 kernel: Lustre: scratch-OST0027: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88081a743c00, cur 1340121180 expire 1340121030 last 1340120953
Jun 19 17:53:00 weisshorn05 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 17:53:00 weisshorn10 kernel: Lustre: scratch-OST003a: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807cf8d4c00, cur 1340121180 expire 1340121030 last 1340120953
Jun 19 17:53:00 weisshorn10 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 17:53:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:53:13 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 165 seconds. I think it's dead, and I am evicting it. exp ffff880fd628a800, cur 1340121193 expire 1340121043 last 1340121028
Jun 19 17:53:13 weisshorn08 kernel: Lustre: Skipped 1 previous similar message
Jun 19 17:53:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:53:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:53:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:54:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:54:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:54:40 weisshorn01 kernel: Lustre: scratch-MDT0000: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ef1379c00, cur 1340121280 expire 1340121130 last 1340121053
Jun 19 17:54:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:54:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:55:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:55:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:55:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:55:43 weisshorn06 kernel: Lustre: scratch-OST0012: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a6f36bc00, cur 1340121343 expire 1340121193 last 1340121116
Jun 19 17:55:43 weisshorn06 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 17:55:43 weisshorn05 kernel: Lustre: scratch-OST0027: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802dc113c00, cur 1340121343 expire 1340121193 last 1340121116
Jun 19 17:55:43 weisshorn05 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 17:55:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:56:08 weisshorn03 kernel: Lustre: scratch-OST0038: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880810b3b000, cur 1340121368 expire 1340121218 last 1340121141
Jun 19 17:56:08 weisshorn03 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 17:56:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:56:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:56:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:56:43 weisshorn01 kernel: LustreError: 5304:0:(mdt_handler.c:2822:mdt_recovery()) operation 400 on unconnected MDS from 12345-148.187.6.6@o2ib2
Jun 19 17:56:45 weisshorn04 kernel: Lustre: scratch-OST0031: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808014e4000, cur 1340121405 expire 1340121255 last 1340121178
Jun 19 17:56:45 weisshorn04 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 17:56:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:56:58 weisshorn07 kernel: Lustre: scratch-OST002b: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880495d25c00, cur 1340121418 expire 1340121268 last 1340121191
Jun 19 17:56:58 weisshorn07 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 17:56:58 weisshorn13 kernel: Lustre: scratch-OST0005: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e96059000, cur 1340121418 expire 1340121268 last 1340121191
Jun 19 17:56:58 weisshorn13 kernel: Lustre: Skipped 13 previous similar messages
Jun 19 17:57:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:57:23 weisshorn09 kernel: Lustre: scratch-OST0004: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880818ec2000, cur 1340121443 expire 1340121293 last 1340121216
Jun 19 17:57:23 weisshorn09 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 17:57:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:57:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:57:44 weisshorn01 kernel: Lustre: scratch-MDT0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b03043000, cur 1340121464 expire 1340121314 last 1340121237
Jun 19 17:57:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:58:00 weisshorn10 kernel: Lustre: scratch-OST0016: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803386ad400, cur 1340121480 expire 1340121330 last 1340121253
Jun 19 17:58:00 weisshorn10 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 17:58:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:58:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:58:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:58:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:59:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:59:15 weisshorn11 kernel: Lustre: scratch-OST0021: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881016182800, cur 1340121555 expire 1340121405 last 1340121328
Jun 19 17:59:15 weisshorn11 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 17:59:15 weisshorn12 kernel: Lustre: scratch-OST0047: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b342d7c00, cur 1340121555 expire 1340121405 last 1340121328
Jun 19 17:59:15 weisshorn12 kernel: Lustre: Skipped 6 previous similar messages
Jun 19 17:59:27 weisshorn01 kernel: Lustre: 5434:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x2e43:0x0]] [valid 68719476736]
Jun 19 17:59:27 weisshorn01 kernel: Lustre: 5434:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 14 previous similar messages
Jun 19 17:59:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:59:28 weisshorn08 kernel: Lustre: scratch-OST002f: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e0d269c00, cur 1340121568 expire 1340121418 last 1340121341
Jun 19 17:59:28 weisshorn08 kernel: Lustre: Skipped 6 previous similar messages
Jun 19 17:59:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 17:59:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:00:01 weisshorn01 CROND[7541]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:00:01 weisshorn06 nslcd[3953]: [b5b1c6] ldap_result() failed: Can't contact LDAP server
Jun 19 18:00:01 weisshorn06 CROND[126123]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:00:01 weisshorn03 nslcd[3944]: [6f6d8a] ldap_result() failed: Can't contact LDAP server
Jun 19 18:00:01 weisshorn11 nslcd[3954]: [22ad0c] ldap_result() failed: Can't contact LDAP server
Jun 19 18:00:01 weisshorn03 CROND[125914]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:00:01 weisshorn11 CROND[128235]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:00:01 weisshorn05 nslcd[4378]: [d601af] ldap_result() failed: Can't contact LDAP server
Jun 19 18:00:01 weisshorn13 nslcd[4224]: [ef532d] ldap_result() failed: Can't contact LDAP server
Jun 19 18:00:01 weisshorn05 CROND[125234]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:00:01 weisshorn13 CROND[128660]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:00:01 weisshorn09 nslcd[3954]: [d385c9] ldap_result() failed: Can't contact LDAP server
Jun 19 18:00:01 weisshorn09 CROND[124951]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:00:01 weisshorn14 CROND[53880]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:00:01 weisshorn12 nslcd[3954]: [ef532d] ldap_result() failed: Can't contact LDAP server
Jun 19 18:00:01 weisshorn12 CROND[125118]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:00:01 weisshorn10 nslcd[4294]: [d601af] ldap_result() failed: Can't contact LDAP server
Jun 19 18:00:01 weisshorn10 CROND[125261]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:00:01 weisshorn07 nslcd[4297]: [ef532d] ldap_result() failed: Can't contact LDAP server
Jun 19 18:00:01 weisshorn07 CROND[125373]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:00:01 weisshorn08 nslcd[3946]: [ef532d] ldap_result() failed: Can't contact LDAP server
Jun 19 18:00:01 weisshorn08 CROND[124030]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:00:01 weisshorn02 CROND[46487]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:00:01 weisshorn04 nslcd[3885]: [d385c9] ldap_result() failed: Can't contact LDAP server
Jun 19 18:00:01 weisshorn04 CROND[124705]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:00:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:00:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:00:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:00:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:01:01 weisshorn04 CROND[124709]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 18:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1247 starting 0anacron
Jun 19 18:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1247 finished 0anacron
Jun 19 18:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1247 starting mcelog.cron
Jun 19 18:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1247 finished mcelog.cron
Jun 19 18:01:01 weisshorn01 CROND[7547]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 18:01:01 weisshorn01 run-parts(/etc/cron.hourly)[7547 starting 0anacron
Jun 19 18:01:01 weisshorn01 run-parts(/etc/cron.hourly)[7557 finished 0anacron
Jun 19 18:01:01 weisshorn01 run-parts(/etc/cron.hourly)[7547 starting mcelog.cron
Jun 19 18:01:01 weisshorn01 run-parts(/etc/cron.hourly)[7564 finished mcelog.cron
Jun 19 18:01:01 weisshorn06 CROND[126127]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 18:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1261 starting 0anacron
Jun 19 18:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1261 finished 0anacron
Jun 19 18:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1261 starting mcelog.cron
Jun 19 18:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1261 finished mcelog.cron
Jun 19 18:01:01 weisshorn03 CROND[125918]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 18:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1259 starting 0anacron
Jun 19 18:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1259 finished 0anacron
Jun 19 18:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1259 starting mcelog.cron
Jun 19 18:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1259 finished mcelog.cron
Jun 19 18:01:01 weisshorn13 CROND[128664]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 18:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1286 starting 0anacron
Jun 19 18:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1286 finished 0anacron
Jun 19 18:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1286 starting mcelog.cron
Jun 19 18:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1286 finished mcelog.cron
Jun 19 18:01:01 weisshorn05 CROND[125238]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 18:01:01 weisshorn11 CROND[128239]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 18:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1282 starting 0anacron
Jun 19 18:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1252 starting 0anacron
Jun 19 18:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1282 finished 0anacron
Jun 19 18:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1282 starting mcelog.cron
Jun 19 18:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1282 finished mcelog.cron
Jun 19 18:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1252 finished 0anacron
Jun 19 18:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1252 starting mcelog.cron
Jun 19 18:01:01 weisshorn09 CROND[124955]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 18:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1249 starting 0anacron
Jun 19 18:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1252 finished mcelog.cron
Jun 19 18:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1249 finished 0anacron
Jun 19 18:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1249 starting mcelog.cron
Jun 19 18:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1249 finished mcelog.cron
Jun 19 18:01:01 weisshorn14 CROND[53885]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 18:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5388 starting 0anacron
Jun 19 18:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5389 finished 0anacron
Jun 19 18:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5388 starting mcelog.cron
Jun 19 18:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5390 finished mcelog.cron
Jun 19 18:01:01 weisshorn12 CROND[125122]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 18:01:01 weisshorn07 CROND[125377]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 18:01:01 weisshorn10 CROND[125265]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 18:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1251 starting 0anacron
Jun 19 18:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1251 finished 0anacron
Jun 19 18:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1252 starting 0anacron
Jun 19 18:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1251 starting mcelog.cron
Jun 19 18:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1252 finished 0anacron
Jun 19 18:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1253 starting 0anacron
Jun 19 18:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1252 starting mcelog.cron
Jun 19 18:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1251 finished mcelog.cron
Jun 19 18:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1252 finished mcelog.cron
Jun 19 18:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1253 finished 0anacron
Jun 19 18:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1253 starting mcelog.cron
Jun 19 18:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1253 finished mcelog.cron
Jun 19 18:01:01 weisshorn08 CROND[124034]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 18:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1240 starting 0anacron
Jun 19 18:01:01 weisshorn02 CROND[46491]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 18:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4649 starting 0anacron
Jun 19 18:01:02 weisshorn02 run-parts(/etc/cron.hourly)[4650 finished 0anacron
Jun 19 18:01:02 weisshorn02 run-parts(/etc/cron.hourly)[4649 starting mcelog.cron
Jun 19 18:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1240 finished 0anacron
Jun 19 18:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1240 starting mcelog.cron
Jun 19 18:01:02 weisshorn02 run-parts(/etc/cron.hourly)[4650 finished mcelog.cron
Jun 19 18:01:02 weisshorn08 run-parts(/etc/cron.hourly)[1240 finished mcelog.cron
Jun 19 18:01:08 weisshorn05 kernel: Lustre: scratch-OST003e: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100f28c800, cur 1340121668 expire 1340121518 last 1340121441
Jun 19 18:01:08 weisshorn05 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 18:01:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:01:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:01:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:01:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:02:10 weisshorn07 rhnsd[125402]: running program /usr/sbin/rhn_check
Jun 19 18:02:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:02:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:02:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:02:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:03:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:03:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:03:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:03:47 weisshorn01 kernel: Lustre: MGS: haven't heard from client 02e8b9ff-fc43-5841-d3bb-815052085829 (at 148.187.6.7@o2ib2) in 229 seconds. I think it's dead, and I am evicting it. exp ffff880313867000, cur 1340121827 expire 1340121677 last 1340121598
Jun 19 18:03:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:04:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:04:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:04:28 weisshorn06 kernel: Lustre: scratch-OST0041: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808059c8400, cur 1340121868 expire 1340121718 last 1340121641
Jun 19 18:04:28 weisshorn06 kernel: Lustre: Skipped 10 previous similar messages
Jun 19 18:04:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:04:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:05:02 weisshorn01 kernel: LustreError: 5247:0:(mdt_handler.c:2822:mdt_recovery()) operation 41 on unconnected MDS from 12345-148.187.6.7@o2ib2
Jun 19 18:05:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:05:18 weisshorn03 kernel: Lustre: scratch-OST0014: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88061fc50c00, cur 1340121918 expire 1340121768 last 1340121691
Jun 19 18:05:18 weisshorn03 kernel: Lustre: Skipped 6 previous similar messages
Jun 19 18:05:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:05:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:05:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:06:06 weisshorn03 nslcd[3944]: [49361f] ldap_result() failed: Can't contact LDAP server
Jun 19 18:06:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:06:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:06:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:06:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:07:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:07:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:07:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:07:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:08:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:08:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:08:38 weisshorn09 kernel: Lustre: scratch-OST001a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080d951400, cur 1340122118 expire 1340121968 last 1340121891
Jun 19 18:08:38 weisshorn09 kernel: Lustre: Skipped 11 previous similar messages
Jun 19 18:08:38 weisshorn04 kernel: Lustre: scratch-OST0031: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88064b0a6800, cur 1340122118 expire 1340121968 last 1340121891
Jun 19 18:08:38 weisshorn04 kernel: Lustre: Skipped 7 previous similar messages
Jun 19 18:08:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:08:49 weisshorn01 kernel: Lustre: scratch-MDT0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ac7523c00, cur 1340122129 expire 1340121979 last 1340121902
Jun 19 18:08:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:09:01 weisshorn01 kernel: LustreError: 5305:0:(mdt_handler.c:2822:mdt_recovery()) operation 400 on unconnected MDS from 12345-148.187.6.7@o2ib2
Jun 19 18:09:03 weisshorn11 kernel: Lustre: scratch-OST0021: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b4946dc00, cur 1340122143 expire 1340121993 last 1340121916
Jun 19 18:09:03 weisshorn11 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 18:09:03 weisshorn07 kernel: Lustre: scratch-OST0042: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881015aa2400, cur 1340122143 expire 1340121993 last 1340121916
Jun 19 18:09:03 weisshorn07 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 18:09:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:09:28 weisshorn13 kernel: Lustre: scratch-OST0011: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881013e65800, cur 1340122168 expire 1340122018 last 1340121941
Jun 19 18:09:28 weisshorn13 kernel: Lustre: Skipped 10 previous similar messages
Jun 19 18:09:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:09:36 weisshorn01 kernel: Lustre: 4569:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d7:0x3092:0x0]] [valid 68719476736]
Jun 19 18:09:36 weisshorn01 kernel: Lustre: 4569:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 9 previous similar messages
Jun 19 18:09:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:09:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:10:01 weisshorn02 CROND[46574]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:10:01 weisshorn08 CROND[124086]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:10:01 weisshorn04 CROND[124753]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:10:01 weisshorn01 CROND[7658]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:10:01 weisshorn01 CROND[7659]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 18:10:01 weisshorn06 CROND[126182]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:10:01 weisshorn03 sshd[125965]: Accepted publickey for root from 148.187.12.231 port 50678 ssh2
Jun 19 18:10:01 weisshorn04 sshd[124755]: Accepted publickey for root from 148.187.12.231 port 44431 ssh2
Jun 19 18:10:01 weisshorn03 sshd[125965]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 18:10:01 weisshorn04 sshd[124755]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 18:10:01 weisshorn14 sshd[53932]: Accepted publickey for root from 148.187.12.231 port 49801 ssh2
Jun 19 18:10:01 weisshorn14 sshd[53932]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 18:10:01 weisshorn06 sshd[126179]: Accepted publickey for root from 148.187.12.231 port 44940 ssh2
Jun 19 18:10:01 weisshorn09 sshd[125007]: Accepted publickey for root from 148.187.12.231 port 58123 ssh2
Jun 19 18:10:01 weisshorn06 sshd[126179]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 18:10:01 weisshorn09 sshd[125007]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 18:10:01 weisshorn12 sshd[125159]: Accepted publickey for root from 148.187.12.231 port 46587 ssh2
Jun 19 18:10:01 weisshorn10 sshd[125308]: Accepted publickey for root from 148.187.12.231 port 37159 ssh2
Jun 19 18:10:01 weisshorn13 sshd[128707]: Accepted publickey for root from 148.187.12.231 port 44849 ssh2
Jun 19 18:10:01 weisshorn13 sshd[128707]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 18:10:01 weisshorn11 sshd[128280]: Accepted publickey for root from 148.187.12.231 port 45373 ssh2
Jun 19 18:10:01 weisshorn11 sshd[128280]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 18:10:01 weisshorn12 sshd[125159]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 18:10:01 weisshorn10 sshd[125308]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 18:10:01 weisshorn14 sshd[53932]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 18:10:01 weisshorn14 sshd[53932]: pam_unix(sshd:session): session closed for user root
Jun 19 18:10:01 weisshorn05 sshd[125281]: Accepted publickey for root from 148.187.12.231 port 54882 ssh2
Jun 19 18:10:01 weisshorn08 sshd[124088]: Accepted publickey for root from 148.187.12.231 port 40487 ssh2
Jun 19 18:10:01 weisshorn08 sshd[124088]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 18:10:01 weisshorn03 CROND[125979]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:10:01 weisshorn05 sshd[125281]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 18:10:01 weisshorn09 sshd[125007]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 18:10:01 weisshorn09 sshd[125007]: pam_unix(sshd:session): session closed for user root
Jun 19 18:10:01 weisshorn06 sshd[126179]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 18:10:01 weisshorn06 sshd[126179]: pam_unix(sshd:session): session closed for user root
Jun 19 18:10:01 weisshorn13 CROND[128720]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:10:01 weisshorn04 sshd[124755]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 18:10:01 weisshorn04 sshd[124755]: pam_unix(sshd:session): session closed for user root
Jun 19 18:10:01 weisshorn03 sshd[125965]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 18:10:01 weisshorn03 sshd[125965]: pam_unix(sshd:session): session closed for user root
Jun 19 18:10:01 weisshorn13 sshd[128707]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 18:10:01 weisshorn13 sshd[128707]: pam_unix(sshd:session): session closed for user root
Jun 19 18:10:01 weisshorn10 sshd[125308]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 18:10:01 weisshorn10 sshd[125308]: pam_unix(sshd:session): session closed for user root
Jun 19 18:10:01 weisshorn12 sshd[125159]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 18:10:01 weisshorn12 sshd[125159]: pam_unix(sshd:session): session closed for user root
Jun 19 18:10:01 weisshorn11 CROND[128293]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:10:01 weisshorn07 sshd[125476]: Accepted publickey for root from 148.187.12.231 port 48490 ssh2
Jun 19 18:10:01 weisshorn11 sshd[128280]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 18:10:01 weisshorn11 sshd[128280]: pam_unix(sshd:session): session closed for user root
Jun 19 18:10:01 weisshorn07 sshd[125476]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 18:10:01 weisshorn05 CROND[125298]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:10:01 weisshorn09 CROND[125030]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:10:01 weisshorn08 sshd[124088]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 18:10:01 weisshorn08 sshd[124088]: pam_unix(sshd:session): session closed for user root
Jun 19 18:10:01 weisshorn14 CROND[53955]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:10:01 weisshorn05 sshd[125281]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 18:10:01 weisshorn05 sshd[125281]: pam_unix(sshd:session): session closed for user root
Jun 19 18:10:01 weisshorn07 sshd[125476]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 18:10:01 weisshorn07 sshd[125476]: pam_unix(sshd:session): session closed for user root
Jun 19 18:10:01 weisshorn10 CROND[125331]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:10:01 weisshorn12 CROND[125182]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:10:01 weisshorn01 postfix/pickup[7266]: E44CB92224F: uid=0 from=<root>
Jun 19 18:10:01 weisshorn01 postfix/cleanup[7689]: E44CB92224F: message-id=<20120619161001.E44CB92224F@weisshorn01.admin.cscs.ch>
Jun 19 18:10:01 weisshorn07 CROND[125498]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:10:01 weisshorn01 postfix/qmgr[2710]: E44CB92224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 18:10:02 weisshorn01 postfix/local[7691]: E44CB92224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=0.55, delays=0.5/0.02/0/0.03, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 18:10:02 weisshorn01 postfix/qmgr[2710]: E44CB92224F: removed
Jun 19 18:10:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:10:18 weisshorn10 kernel: Lustre: scratch-OST000a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809f6cac000, cur 1340122218 expire 1340122068 last 1340121991
Jun 19 18:10:18 weisshorn10 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 18:10:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:10:43 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88069dce8c00, cur 1340122243 expire 1340122093 last 1340122016
Jun 19 18:10:43 weisshorn08 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 18:10:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:10:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:11:08 weisshorn05 kernel: Lustre: scratch-OST0027: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880806a05c00, cur 1340122268 expire 1340122118 last 1340122041
Jun 19 18:11:08 weisshorn05 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 18:11:08 weisshorn01 kernel: Lustre: MGS: haven't heard from client 02e8b9ff-fc43-5841-d3bb-815052085829 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ab5300800, cur 1340122268 expire 1340122118 last 1340122041
Jun 19 18:11:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:11:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:11:36 weisshorn02 nslcd[2099]: [65174b] ldap_result() failed: Can't contact LDAP server
Jun 19 18:11:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:11:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:12:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:12:23 weisshorn12 kernel: Lustre: scratch-OST000f: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea0798800, cur 1340122343 expire 1340122193 last 1340122116
Jun 19 18:12:23 weisshorn12 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 18:12:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:12:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:12:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:13:00 weisshorn01 kernel: Lustre: MGS: haven't heard from client aca8aee1-5c22-ca9e-453b-0ad0e5295f34 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806f9503c00, cur 1340122380 expire 1340122230 last 1340122153
Jun 19 18:13:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:13:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:13:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:13:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:14:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:14:23 weisshorn14 rhnsd[53998]: running program /usr/sbin/rhn_check
Jun 19 18:14:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:14:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:14:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:15:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:15:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:15:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:15:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:16:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:16:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:16:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:16:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:17:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:17:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:17:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:17:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:18:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:18:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:18:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:18:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:19:03 weisshorn09 kernel: Lustre: scratch-OST0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88046a22bc00, cur 1340122743 expire 1340122593 last 1340122516
Jun 19 18:19:03 weisshorn09 kernel: Lustre: Skipped 6 previous similar messages
Jun 19 18:19:03 weisshorn06 kernel: Lustre: scratch-OST002a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881012c36000, cur 1340122743 expire 1340122593 last 1340122516
Jun 19 18:19:03 weisshorn06 kernel: Lustre: Skipped 8 previous similar messages
Jun 19 18:19:03 weisshorn03 kernel: Lustre: scratch-OST0014: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880807255400, cur 1340122743 expire 1340122593 last 1340122516
Jun 19 18:19:03 weisshorn03 kernel: Lustre: Skipped 7 previous similar messages
Jun 19 18:19:03 weisshorn01 kernel: Lustre: scratch-MDT0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ac590e400, cur 1340122743 expire 1340122593 last 1340122516
Jun 19 18:19:03 weisshorn01 kernel: Lustre: Skipped 1 previous similar message
Jun 19 18:19:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:19:28 weisshorn07 kernel: Lustre: scratch-OST001f: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88053aa32400, cur 1340122768 expire 1340122618 last 1340122541
Jun 19 18:19:28 weisshorn07 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 18:19:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:19:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:19:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:20:01 weisshorn10 CROND[125362]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:20:01 weisshorn07 CROND[125517]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:20:01 weisshorn02 CROND[46597]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:20:01 weisshorn04 CROND[124811]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:20:01 weisshorn08 CROND[124161]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:20:01 weisshorn06 CROND[126261]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:20:01 weisshorn03 CROND[126052]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:20:01 weisshorn13 CROND[128798]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:20:01 weisshorn11 CROND[128365]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:20:01 weisshorn09 CROND[125084]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:20:01 weisshorn05 CROND[125336]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:20:01 weisshorn14 CROND[54021]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:20:01 weisshorn01 CROND[7768]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:20:02 weisshorn12 CROND[125249]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:20:08 weisshorn01 kernel: Lustre: 4701:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046ee:0x2214:0x0]] [valid 68719476736]
Jun 19 18:20:08 weisshorn01 kernel: Lustre: 4701:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 3 previous similar messages
Jun 19 18:20:14 weisshorn01 kernel: LustreError: 5149:0:(mdt_handler.c:2822:mdt_recovery()) operation 101 on unconnected MDS from 12345-148.187.6.6@o2ib2
Jun 19 18:20:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:20:18 weisshorn04 kernel: Lustre: scratch-OST003d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88015c872400, cur 1340122818 expire 1340122668 last 1340122591
Jun 19 18:20:18 weisshorn04 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 18:20:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:20:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:20:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:21:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:21:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:21:33 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880805885800, cur 1340122893 expire 1340122743 last 1340122666
Jun 19 18:21:33 weisshorn08 kernel: Lustre: Skipped 1 previous similar message
Jun 19 18:21:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:21:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:22:07 weisshorn01 kernel: Lustre: MGS: haven't heard from client 02e8b9ff-fc43-5841-d3bb-815052085829 (at 148.187.6.7@o2ib2) in 229 seconds. I think it's dead, and I am evicting it. exp ffff88069eb2d800, cur 1340122927 expire 1340122777 last 1340122698
Jun 19 18:22:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:22:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:22:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:22:48 weisshorn05 kernel: Lustre: scratch-OST000e: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100e415400, cur 1340122968 expire 1340122818 last 1340122741
Jun 19 18:22:48 weisshorn05 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 18:22:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:23:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:23:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:23:38 weisshorn13 kernel: Lustre: scratch-OST000b: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808083edc00, cur 1340123018 expire 1340122868 last 1340122791
Jun 19 18:23:38 weisshorn13 kernel: Lustre: Skipped 9 previous similar messages
Jun 19 18:23:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:23:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:24:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:24:15 weisshorn12 kernel: Lustre: scratch-OST0040: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88033b884000, cur 1340123055 expire 1340122905 last 1340122828
Jun 19 18:24:15 weisshorn12 kernel: Lustre: Skipped 1 previous similar message
Jun 19 18:24:15 weisshorn10 kernel: Lustre: scratch-OST0016: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810128dd000, cur 1340123055 expire 1340122905 last 1340122828
Jun 19 18:24:15 weisshorn10 kernel: Lustre: Skipped 1 previous similar message
Jun 19 18:24:26 weisshorn01 kernel: LustreError: 5256:0:(mdt_handler.c:2822:mdt_recovery()) operation 400 on unconnected MDS from 12345-148.187.6.7@o2ib2
Jun 19 18:24:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:24:30 weisshorn09 nslcd[3954]: [6f6d8a] ldap_result() failed: Can't contact LDAP server
Jun 19 18:24:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:24:53 weisshorn08 nslcd[3946]: [d601af] ldap_result() failed: Can't contact LDAP server
Jun 19 18:24:54 weisshorn06 nslcd[3953]: [6be6b9] ldap_result() failed: Can't contact LDAP server
Jun 19 18:24:56 weisshorn04 nslcd[3885]: [6f6d8a] ldap_result() failed: Can't contact LDAP server
Jun 19 18:24:57 weisshorn07 nslcd[4297]: [d601af] ldap_result() failed: Can't contact LDAP server
Jun 19 18:24:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:25:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:25:17 weisshorn11 nslcd[3954]: [997556] ldap_result() failed: Can't contact LDAP server
Jun 19 18:25:17 weisshorn12 nslcd[3954]: [d601af] ldap_result() failed: Can't contact LDAP server
Jun 19 18:25:19 weisshorn13 nslcd[4224]: [d601af] ldap_result() failed: Can't contact LDAP server
Jun 19 18:25:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:25:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:25:55 weisshorn11 kernel: Lustre: scratch-OST0008: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808015ee800, cur 1340123155 expire 1340123005 last 1340122928
Jun 19 18:25:55 weisshorn11 kernel: Lustre: Skipped 1 previous similar message
Jun 19 18:25:56 weisshorn05 nslcd[4378]: [c77402] ldap_result() failed: Can't contact LDAP server
Jun 19 18:25:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:26:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:26:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:26:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:26:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:27:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:27:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:27:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:27:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:28:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:28:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:28:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:28:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:29:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:29:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:29:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:29:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:30:01 weisshorn07 CROND[125542]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:30:01 weisshorn01 CROND[7820]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:30:01 weisshorn10 CROND[125423]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:30:01 weisshorn12 CROND[125282]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:30:01 weisshorn02 CROND[46620]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:30:01 weisshorn04 CROND[124867]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:30:01 weisshorn08 CROND[124193]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:30:01 weisshorn06 CROND[126286]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:30:01 weisshorn03 CROND[126076]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:30:01 weisshorn13 CROND[128826]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:30:01 weisshorn11 CROND[128397]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:30:01 weisshorn09 CROND[125114]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:30:01 weisshorn14 CROND[54051]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:30:02 weisshorn05 CROND[125398]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:30:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:30:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:30:35 weisshorn01 kernel: Lustre: 4689:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046e5:0x11094:0x0]] [valid 68719476736]
Jun 19 18:30:35 weisshorn01 kernel: Lustre: 4689:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 6 previous similar messages
Jun 19 18:30:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:31:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:31:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:31:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:31:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:32:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:32:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:32:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:32:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:33:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:33:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:33:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:33:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:33:51 weisshorn13 rhnsd[128839]: running program /usr/sbin/rhn_check
Jun 19 18:34:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:34:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:34:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:34:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:35:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:35:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:35:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:35:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:36:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:36:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:36:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:36:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:37:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:37:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:37:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:37:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:38:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:38:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:38:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:38:46 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:39:01 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:39:03 weisshorn04 kernel: Lustre: scratch-OST000d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810184c0800, cur 1340123943 expire 1340123793 last 1340123716
Jun 19 18:39:03 weisshorn04 kernel: Lustre: Skipped 7 previous similar messages
Jun 19 18:39:03 weisshorn13 kernel: Lustre: scratch-OST0024: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880822111000, cur 1340123943 expire 1340123793 last 1340123716
Jun 19 18:39:03 weisshorn13 kernel: Lustre: Skipped 6 previous similar messages
Jun 19 18:39:03 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080995d800, cur 1340123943 expire 1340123793 last 1340123716
Jun 19 18:39:03 weisshorn08 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 18:39:16 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:39:28 weisshorn06 kernel: Lustre: scratch-OST002a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881013aab400, cur 1340123968 expire 1340123818 last 1340123741
Jun 19 18:39:28 weisshorn06 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 18:39:28 weisshorn12 kernel: Lustre: scratch-OST0028: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bb6a6a400, cur 1340123968 expire 1340123818 last 1340123741
Jun 19 18:39:28 weisshorn12 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 18:39:28 weisshorn03 kernel: Lustre: scratch-OST0014: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88045d569400, cur 1340123968 expire 1340123818 last 1340123741
Jun 19 18:39:28 weisshorn03 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 18:39:28 weisshorn01 kernel: Lustre: scratch-MDT0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88027e642000, cur 1340123968 expire 1340123818 last 1340123741
Jun 19 18:39:28 weisshorn07 kernel: Lustre: scratch-OST0013: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810255c7c00, cur 1340123968 expire 1340123818 last 1340123741
Jun 19 18:39:28 weisshorn07 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 18:39:31 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:39:46 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:40:01 weisshorn14 CROND[54080]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:40:01 weisshorn05 CROND[125428]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:40:01 weisshorn07 CROND[125621]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:40:01 weisshorn10 CROND[125453]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:40:01 weisshorn01 CROND[7918]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:40:01 weisshorn02 CROND[46690]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:40:01 weisshorn04 CROND[124898]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:40:01 weisshorn12 CROND[125307]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:40:01 weisshorn08 CROND[124232]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:40:01 weisshorn06 CROND[126325]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:40:01 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:40:01 weisshorn03 CROND[126110]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:40:01 weisshorn11 CROND[128424]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:40:01 weisshorn09 CROND[125152]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:40:02 weisshorn13 CROND[128858]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:40:16 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:40:31 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:40:46 weisshorn14 nslcd[3437]: [de8af6] ldap_result() failed: Can't contact LDAP server
Jun 19 18:40:46 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:41:01 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:41:16 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:41:31 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:41:46 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:42:01 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:42:16 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:42:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:42:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:43:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:43:11 weisshorn01 kernel: LustreError: 4684:0:(mdt_handler.c:2822:mdt_recovery()) operation 400 on unconnected MDS from 12345-148.187.6.7@o2ib2
Jun 19 18:43:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:43:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:43:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:44:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:44:03 weisshorn09 kernel: Lustre: scratch-OST0034: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881016edf000, cur 1340124243 expire 1340124093 last 1340124016
Jun 19 18:44:03 weisshorn09 kernel: Lustre: Skipped 6 previous similar messages
Jun 19 18:44:03 weisshorn06 kernel: Lustre: scratch-OST001d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805c833b800, cur 1340124243 expire 1340124093 last 1340124016
Jun 19 18:44:03 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 18:44:13 weisshorn01 kernel: Lustre: 5433:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004631:0x178c8:0x0]] [valid 68719476736]
Jun 19 18:44:13 weisshorn01 kernel: Lustre: 5433:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 4 previous similar messages
Jun 19 18:44:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:44:28 weisshorn05 kernel: Lustre: scratch-OST000e: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a6fff4800, cur 1340124268 expire 1340124118 last 1340124041
Jun 19 18:44:28 weisshorn05 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 18:44:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:44:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:45:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:45:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:45:18 weisshorn11 kernel: Lustre: scratch-OST0021: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c05248c00, cur 1340124318 expire 1340124168 last 1340124091
Jun 19 18:45:18 weisshorn11 kernel: Lustre: Skipped 1 previous similar message
Jun 19 18:45:18 weisshorn03 kernel: Lustre: scratch-OST0020: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080c493400, cur 1340124318 expire 1340124168 last 1340124091
Jun 19 18:45:18 weisshorn03 kernel: Lustre: Skipped 1 previous similar message
Jun 19 18:45:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:45:43 weisshorn09 kernel: Lustre: scratch-OST0026: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808b8c6d400, cur 1340124343 expire 1340124193 last 1340124116
Jun 19 18:45:43 weisshorn09 kernel: Lustre: Skipped 1 previous similar message
Jun 19 18:45:43 weisshorn05 kernel: Lustre: scratch-OST0027: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805c815a400, cur 1340124343 expire 1340124193 last 1340124116
Jun 19 18:45:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:45:47 weisshorn10 nslcd[4294]: [c77402] ldap_result() failed: Can't contact LDAP server
Jun 19 18:46:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:46:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:46:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:46:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:47:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:47:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:47:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:47:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:48:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:48:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:48:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:48:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:49:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:49:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:49:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:49:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:50:01 weisshorn09 CROND[125208]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:50:01 weisshorn14 nslcd[3437]: [c32e20] ldap_result() failed: Can't contact LDAP server
Jun 19 18:50:01 weisshorn14 CROND[54147]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:50:01 weisshorn10 CROND[125486]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:50:01 weisshorn07 CROND[125639]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:50:01 weisshorn13 CROND[128927]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:50:01 weisshorn05 CROND[125462]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:50:01 weisshorn01 CROND[7990]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:50:01 weisshorn02 nslcd[2099]: [872c35] ldap_result() failed: Can't contact LDAP server
Jun 19 18:50:01 weisshorn02 CROND[46713]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:50:01 weisshorn04 CROND[124934]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:50:01 weisshorn12 CROND[125373]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:50:01 weisshorn08 CROND[124285]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:50:01 weisshorn06 CROND[126384]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:50:01 weisshorn03 CROND[126174]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:50:01 weisshorn11 CROND[128489]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 18:50:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:50:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:50:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:50:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:51:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:51:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:51:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:51:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:52:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:52:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:52:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:52:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:53:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:53:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:53:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:53:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:54:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:54:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:54:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:54:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:55:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:55:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:55:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:55:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:56:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:56:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:56:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:56:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:57:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:57:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:57:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:57:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:58:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:58:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:58:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:58:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:59:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:59:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:59:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 18:59:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:00:01 weisshorn11 CROND[128519]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:00:01 weisshorn09 CROND[125245]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:00:01 weisshorn14 CROND[54181]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:00:01 weisshorn10 CROND[125550]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:00:01 weisshorn07 CROND[125668]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:00:01 weisshorn05 CROND[125532]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:00:01 weisshorn01 CROND[8048]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:00:01 weisshorn02 CROND[46777]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:00:01 weisshorn04 CROND[124999]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:00:01 weisshorn12 CROND[125403]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:00:01 weisshorn13 CROND[128957]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:00:01 weisshorn06 CROND[126408]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:00:01 weisshorn08 CROND[124320]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:00:01 weisshorn03 CROND[126212]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:00:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:00:16 weisshorn10 nslcd[4294]: [b5b1c6] ldap_result() failed: Can't contact LDAP server
Jun 19 19:00:16 weisshorn06 nslcd[3953]: [234363] ldap_result() failed: Can't contact LDAP server
Jun 19 19:00:16 weisshorn11 nslcd[3954]: [333c89] ldap_result() failed: Can't contact LDAP server
Jun 19 19:00:16 weisshorn05 nslcd[4378]: [b5b1c6] ldap_result() failed: Can't contact LDAP server
Jun 19 19:00:16 weisshorn13 nslcd[4224]: [c77402] ldap_result() failed: Can't contact LDAP server
Jun 19 19:00:16 weisshorn13 nslcd[4224]: [c77402] ldap_abandon() failed to abandon search: Other (e.g., implementation specific) error
Jun 19 19:00:16 weisshorn09 nslcd[3954]: [49361f] ldap_result() failed: Can't contact LDAP server
Jun 19 19:00:16 weisshorn03 nslcd[3944]: [bc6770] ldap_result() failed: Can't contact LDAP server
Jun 19 19:00:16 weisshorn12 nslcd[3954]: [c77402] ldap_result() failed: Can't contact LDAP server
Jun 19 19:00:16 weisshorn07 nslcd[4297]: [c77402] ldap_result() failed: Can't contact LDAP server
Jun 19 19:00:16 weisshorn08 nslcd[3946]: [c77402] ldap_result() failed: Can't contact LDAP server
Jun 19 19:00:16 weisshorn04 nslcd[3885]: [49361f] ldap_result() failed: Can't contact LDAP server
Jun 19 19:00:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:00:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:00:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:01:01 weisshorn11 CROND[128524]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 19:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1285 starting 0anacron
Jun 19 19:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1285 finished 0anacron
Jun 19 19:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1285 starting mcelog.cron
Jun 19 19:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1285 finished mcelog.cron
Jun 19 19:01:01 weisshorn09 CROND[125250]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 19:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1252 starting 0anacron
Jun 19 19:01:01 weisshorn14 CROND[54186]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 19:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5418 starting 0anacron
Jun 19 19:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1252 finished 0anacron
Jun 19 19:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1252 starting mcelog.cron
Jun 19 19:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5419 finished 0anacron
Jun 19 19:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5418 starting mcelog.cron
Jun 19 19:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1252 finished mcelog.cron
Jun 19 19:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5420 finished mcelog.cron
Jun 19 19:01:01 weisshorn07 CROND[125673]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 19:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1256 starting 0anacron
Jun 19 19:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1256 finished 0anacron
Jun 19 19:01:01 weisshorn10 CROND[125555]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 19:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1256 starting mcelog.cron
Jun 19 19:01:01 weisshorn01 CROND[8057]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 19:01:01 weisshorn05 CROND[125537]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 19:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1255 starting 0anacron
Jun 19 19:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1255 finished 0anacron
Jun 19 19:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1255 starting mcelog.cron
Jun 19 19:01:01 weisshorn01 run-parts(/etc/cron.hourly)[8057 starting 0anacron
Jun 19 19:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1255 starting 0anacron
Jun 19 19:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1256 finished mcelog.cron
Jun 19 19:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1255 finished mcelog.cron
Jun 19 19:01:01 weisshorn02 CROND[46781]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 19:01:01 weisshorn04 CROND[125004]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 19:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1250 starting 0anacron
Jun 19 19:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4678 starting 0anacron
Jun 19 19:01:01 weisshorn01 run-parts(/etc/cron.hourly)[8067 finished 0anacron
Jun 19 19:01:01 weisshorn01 run-parts(/etc/cron.hourly)[8057 starting mcelog.cron
Jun 19 19:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1250 finished 0anacron
Jun 19 19:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4679 finished 0anacron
Jun 19 19:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1250 starting mcelog.cron
Jun 19 19:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4678 starting mcelog.cron
Jun 19 19:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1255 finished 0anacron
Jun 19 19:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1255 starting mcelog.cron
Jun 19 19:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1250 finished mcelog.cron
Jun 19 19:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4679 finished mcelog.cron
Jun 19 19:01:01 weisshorn01 run-parts(/etc/cron.hourly)[8074 finished mcelog.cron
Jun 19 19:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1255 finished mcelog.cron
Jun 19 19:01:01 weisshorn12 CROND[125408]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 19:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1254 starting 0anacron
Jun 19 19:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1254 finished 0anacron
Jun 19 19:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1254 starting mcelog.cron
Jun 19 19:01:01 weisshorn08 CROND[124325]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 19:01:01 weisshorn06 CROND[126413]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 19:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1254 finished mcelog.cron
Jun 19 19:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1264 starting 0anacron
Jun 19 19:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1264 finished 0anacron
Jun 19 19:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1264 starting mcelog.cron
Jun 19 19:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1243 starting 0anacron
Jun 19 19:01:01 weisshorn13 CROND[128962]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 19:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1264 finished mcelog.cron
Jun 19 19:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1243 finished 0anacron
Jun 19 19:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1243 starting mcelog.cron
Jun 19 19:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1289 starting 0anacron
Jun 19 19:01:01 weisshorn03 CROND[126217]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 19:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1262 starting 0anacron
Jun 19 19:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1243 finished mcelog.cron
Jun 19 19:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1262 finished 0anacron
Jun 19 19:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1262 starting mcelog.cron
Jun 19 19:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1289 finished 0anacron
Jun 19 19:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1289 starting mcelog.cron
Jun 19 19:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1262 finished mcelog.cron
Jun 19 19:01:02 weisshorn13 run-parts(/etc/cron.hourly)[1289 finished mcelog.cron
Jun 19 19:01:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:01:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:01:25 weisshorn10 rhnsd[125573]: running program /usr/sbin/rhn_check
Jun 19 19:01:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:01:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:02:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:02:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:02:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:02:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:03:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:03:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:03:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:03:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:04:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:04:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:04:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:04:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:05:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:05:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:05:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:05:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:06:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:06:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:06:19 weisshorn11 kernel: Lustre: scratch-OST0015: Client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) reconnecting
Jun 19 19:06:19 weisshorn11 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 19:06:19 weisshorn11 kernel: Lustre: scratch-OST0015: Client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) refused reconnection, still busy with 1 active RPCs
Jun 19 19:06:19 weisshorn11 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:06:21 weisshorn03 nslcd[3944]: [0e6897] ldap_result() failed: Can't contact LDAP server
Jun 19 19:06:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:06:43 weisshorn11 kernel: Lustre: scratch-OST0015: Client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) reconnecting
Jun 19 19:06:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:07:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:07:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:07:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:07:36 weisshorn03 kernel: Lustre: scratch-OST0009: Client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) reconnecting
Jun 19 19:07:36 weisshorn03 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:07:36 weisshorn03 kernel: Lustre: scratch-OST0009: Client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) refused reconnection, still busy with 2 active RPCs
Jun 19 19:07:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:08:01 weisshorn03 kernel: Lustre: scratch-OST0009: Client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) reconnecting
Jun 19 19:08:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:08:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:08:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:08:44 weisshorn11 kernel: Lustre: scratch-OST0015: Client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) reconnecting
Jun 19 19:08:44 weisshorn11 kernel: Lustre: scratch-OST0015: Client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) refused reconnection, still busy with 1 active RPCs
Jun 19 19:08:49 weisshorn05 kernel: Lustre: scratch-OST0027: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100edeb400, cur 1340125729 expire 1340125579 last 1340125502
Jun 19 19:08:49 weisshorn11 kernel: Lustre: scratch-OST0021: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880334a57000, cur 1340125729 expire 1340125579 last 1340125502
Jun 19 19:08:49 weisshorn13 kernel: Lustre: scratch-OST0011: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803e930d000, cur 1340125729 expire 1340125579 last 1340125502
Jun 19 19:08:49 weisshorn13 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 19:08:49 weisshorn10 kernel: Lustre: scratch-OST000a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880808310000, cur 1340125729 expire 1340125579 last 1340125502
Jun 19 19:08:49 weisshorn10 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 19:08:49 weisshorn07 kernel: Lustre: scratch-OST002b: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88014b681000, cur 1340125729 expire 1340125579 last 1340125502
Jun 19 19:08:49 weisshorn07 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:08:49 weisshorn08 kernel: Lustre: scratch-OST000c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080620b400, cur 1340125729 expire 1340125579 last 1340125502
Jun 19 19:08:49 weisshorn08 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 19:08:49 weisshorn04 kernel: Lustre: scratch-OST003d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801c5ea6000, cur 1340125729 expire 1340125579 last 1340125502
Jun 19 19:08:49 weisshorn04 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 19:08:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:08:49 weisshorn09 kernel: Lustre: scratch-OST0004: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808225dc400, cur 1340125729 expire 1340125579 last 1340125502
Jun 19 19:08:49 weisshorn09 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:08:49 weisshorn06 kernel: Lustre: scratch-OST0012: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808050ec000, cur 1340125729 expire 1340125579 last 1340125502
Jun 19 19:08:49 weisshorn06 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:08:49 weisshorn03 kernel: Lustre: scratch-OST002c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881025d88400, cur 1340125729 expire 1340125579 last 1340125502
Jun 19 19:08:49 weisshorn03 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:08:55 weisshorn12 kernel: LustreError: 137-5: scratch-OST0015: Not available for connect from 148.187.6.237@o2ib2 (no target)
Jun 19 19:09:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:09:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:09:20 weisshorn11 kernel: Lustre: scratch-OST0015: Client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) reconnecting
Jun 19 19:09:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:09:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:10:01 weisshorn13 CROND[129041]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:10:01 weisshorn09 CROND[125295]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:10:01 weisshorn14 CROND[54259]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:10:01 weisshorn11 CROND[128577]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:10:01 weisshorn02 CROND[46823]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:10:01 weisshorn01 CROND[8164]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:10:01 weisshorn01 CROND[8166]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 19:10:01 weisshorn04 CROND[125043]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:10:01 weisshorn07 CROND[125761]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:10:01 weisshorn10 CROND[125601]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:10:01 weisshorn05 CROND[125577]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:10:01 weisshorn06 CROND[126473]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:10:01 weisshorn03 CROND[126259]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:10:01 weisshorn06 sshd[126470]: Accepted publickey for root from 148.187.12.231 port 44949 ssh2
Jun 19 19:10:01 weisshorn04 sshd[125045]: Accepted publickey for root from 148.187.12.231 port 44447 ssh2
Jun 19 19:10:01 weisshorn14 sshd[54261]: Accepted publickey for root from 148.187.12.231 port 49810 ssh2
Jun 19 19:10:02 weisshorn06 sshd[126470]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 19:10:02 weisshorn04 sshd[125045]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 19:10:02 weisshorn14 sshd[54261]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 19:10:02 weisshorn09 sshd[125297]: Accepted publickey for root from 148.187.12.231 port 58141 ssh2
Jun 19 19:10:02 weisshorn09 sshd[125297]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 19:10:02 weisshorn08 CROND[124374]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:10:02 weisshorn14 sshd[54261]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 19:10:02 weisshorn14 sshd[54261]: pam_unix(sshd:session): session closed for user root
Jun 19 19:10:02 weisshorn07 sshd[125763]: Accepted publickey for root from 148.187.12.231 port 48504 ssh2
Jun 19 19:10:02 weisshorn08 sshd[124371]: Accepted publickey for root from 148.187.12.231 port 40501 ssh2
Jun 19 19:10:02 weisshorn10 sshd[125603]: Accepted publickey for root from 148.187.12.231 port 37170 ssh2
Jun 19 19:10:02 weisshorn08 sshd[124371]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 19:10:02 weisshorn13 sshd[129043]: Accepted publickey for root from 148.187.12.231 port 44862 ssh2
Jun 19 19:10:02 weisshorn10 sshd[125603]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 19:10:02 weisshorn07 sshd[125763]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 19:10:02 weisshorn04 sshd[125045]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 19:10:02 weisshorn04 sshd[125045]: pam_unix(sshd:session): session closed for user root
Jun 19 19:10:02 weisshorn06 sshd[126470]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 19:10:02 weisshorn06 sshd[126470]: pam_unix(sshd:session): session closed for user root
Jun 19 19:10:02 weisshorn03 sshd[126256]: Accepted publickey for root from 148.187.12.231 port 50693 ssh2
Jun 19 19:10:02 weisshorn11 sshd[128578]: Accepted publickey for root from 148.187.12.231 port 45380 ssh2
Jun 19 19:10:02 weisshorn11 sshd[128578]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 19:10:02 weisshorn03 sshd[126256]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 19:10:02 weisshorn13 sshd[129043]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 19:10:02 weisshorn09 sshd[125297]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 19:10:02 weisshorn09 sshd[125297]: pam_unix(sshd:session): session closed for user root
Jun 19 19:10:02 weisshorn12 CROND[125489]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:10:02 weisshorn05 sshd[125576]: Accepted publickey for root from 148.187.12.231 port 54894 ssh2
Jun 19 19:10:02 weisshorn05 sshd[125576]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 19:10:02 weisshorn12 sshd[125487]: Accepted publickey for root from 148.187.12.231 port 46593 ssh2
Jun 19 19:10:02 weisshorn07 sshd[125763]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 19:10:02 weisshorn07 sshd[125763]: pam_unix(sshd:session): session closed for user root
Jun 19 19:10:02 weisshorn12 sshd[125487]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 19:10:02 weisshorn11 sshd[128578]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 19:10:02 weisshorn11 sshd[128578]: pam_unix(sshd:session): session closed for user root
Jun 19 19:10:02 weisshorn08 sshd[124371]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 19:10:02 weisshorn08 sshd[124371]: pam_unix(sshd:session): session closed for user root
Jun 19 19:10:02 weisshorn03 sshd[126256]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 19:10:02 weisshorn03 sshd[126256]: pam_unix(sshd:session): session closed for user root
Jun 19 19:10:02 weisshorn13 sshd[129043]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 19:10:02 weisshorn13 sshd[129043]: pam_unix(sshd:session): session closed for user root
Jun 19 19:10:02 weisshorn10 sshd[125603]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 19:10:02 weisshorn10 sshd[125603]: pam_unix(sshd:session): session closed for user root
Jun 19 19:10:02 weisshorn05 sshd[125576]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 19:10:02 weisshorn05 sshd[125576]: pam_unix(sshd:session): session closed for user root
Jun 19 19:10:02 weisshorn12 sshd[125487]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 19:10:02 weisshorn12 sshd[125487]: pam_unix(sshd:session): session closed for user root
Jun 19 19:10:02 weisshorn01 postfix/pickup[8117]: A116892224F: uid=0 from=<root>
Jun 19 19:10:02 weisshorn01 postfix/cleanup[8196]: A116892224F: message-id=<20120619171002.A116892224F@weisshorn01.admin.cscs.ch>
Jun 19 19:10:02 weisshorn01 postfix/qmgr[2710]: A116892224F: from=<root@weisshorn01.admin.cscs.ch>, size=752, nrcpt=1 (queue active)
Jun 19 19:10:02 weisshorn01 postfix/local[8198]: A116892224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=0.72, delays=0.64/0.04/0/0.05, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 19:10:02 weisshorn01 postfix/qmgr[2710]: A116892224F: removed
Jun 19 19:10:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:10:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:10:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:10:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:11:03 weisshorn06 rhnsd[126496]: running program /usr/sbin/rhn_check
Jun 19 19:11:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:11:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:11:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:11:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:11:51 weisshorn02 nslcd[2099]: [5ce761] ldap_result() failed: Can't contact LDAP server
Jun 19 19:12:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:12:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:12:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:12:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:13:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:13:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:13:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:13:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:14:04 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:14:19 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:14:28 weisshorn12 kernel: Lustre: scratch-OST0028: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fa7279400, cur 1340126068 expire 1340125918 last 1340125841
Jun 19 19:14:28 weisshorn01 kernel: Lustre: MGS: haven't heard from client 02e8b9ff-fc43-5841-d3bb-815052085829 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aa8535c00, cur 1340126068 expire 1340125918 last 1340125841
Jun 19 19:14:28 weisshorn07 kernel: Lustre: scratch-OST0013: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88033eb3e000, cur 1340126068 expire 1340125918 last 1340125841
Jun 19 19:14:28 weisshorn07 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 19:14:28 weisshorn01 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:14:34 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:14:49 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:14:53 weisshorn06 kernel: Lustre: scratch-OST0007: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808067c5c00, cur 1340126093 expire 1340125943 last 1340125866
Jun 19 19:14:53 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:14:53 weisshorn09 kernel: Lustre: scratch-OST0034: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880613e28800, cur 1340126093 expire 1340125943 last 1340125866
Jun 19 19:14:53 weisshorn09 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 19:15:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:15:18 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a4b8ae400, cur 1340126118 expire 1340125968 last 1340125891
Jun 19 19:15:18 weisshorn04 kernel: Lustre: scratch-OST000d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eecf25800, cur 1340126118 expire 1340125968 last 1340125891
Jun 19 19:15:18 weisshorn04 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:15:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:15:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:15:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:16:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:16:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:16:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:16:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:17:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:17:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:17:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:17:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:18:00 weisshorn01 kernel: Lustre: MGS: haven't heard from client aca8aee1-5c22-ca9e-453b-0ad0e5295f34 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ab4c93400, cur 1340126280 expire 1340126130 last 1340126053
Jun 19 19:18:00 weisshorn03 kernel: Lustre: scratch-OST0043: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804f7a65400, cur 1340126280 expire 1340126130 last 1340126053
Jun 19 19:18:00 weisshorn03 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:18:00 weisshorn09 kernel: Lustre: scratch-OST001a: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810158e9c00, cur 1340126280 expire 1340126130 last 1340126053
Jun 19 19:18:00 weisshorn06 kernel: Lustre: scratch-OST002a: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080baadc00, cur 1340126280 expire 1340126130 last 1340126053
Jun 19 19:18:00 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:18:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:18:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:18:25 weisshorn13 kernel: Lustre: scratch-OST0024: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101399fc00, cur 1340126305 expire 1340126155 last 1340126078
Jun 19 19:18:25 weisshorn13 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:18:25 weisshorn05 kernel: Lustre: scratch-OST003e: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100ffb6800, cur 1340126305 expire 1340126155 last 1340126078
Jun 19 19:18:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:18:50 weisshorn07 kernel: Lustre: scratch-OST001f: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ac83ad000, cur 1340126330 expire 1340126180 last 1340126103
Jun 19 19:18:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:19:03 weisshorn04 kernel: Lustre: scratch-OST0025: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080007b800, cur 1340126343 expire 1340126193 last 1340126116
Jun 19 19:19:03 weisshorn05 kernel: Lustre: scratch-OST000e: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100e73c400, cur 1340126343 expire 1340126193 last 1340126116
Jun 19 19:19:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:19:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:19:28 weisshorn03 kernel: Lustre: scratch-OST0014: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101122b800, cur 1340126368 expire 1340126218 last 1340126141
Jun 19 19:19:28 weisshorn03 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:19:28 weisshorn06 kernel: Lustre: scratch-OST002a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881014c64c00, cur 1340126368 expire 1340126218 last 1340126141
Jun 19 19:19:28 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:19:28 weisshorn01 kernel: Lustre: scratch-MDT0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806de7de000, cur 1340126368 expire 1340126218 last 1340126141
Jun 19 19:19:28 weisshorn09 kernel: Lustre: scratch-OST0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880549245800, cur 1340126368 expire 1340126218 last 1340126141
Jun 19 19:19:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:19:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:19:53 weisshorn10 kernel: Lustre: scratch-OST000a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807116cdc00, cur 1340126393 expire 1340126243 last 1340126166
Jun 19 19:20:01 weisshorn03 CROND[126339]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:20:01 weisshorn06 CROND[126551]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:20:01 weisshorn05 CROND[125630]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:20:01 weisshorn10 CROND[125655]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:20:01 weisshorn08 CROND[124453]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:20:01 weisshorn09 CROND[125378]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:20:01 weisshorn14 CROND[54310]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:20:01 weisshorn12 CROND[125541]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:20:01 weisshorn11 CROND[128656]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:20:01 weisshorn13 CROND[129093]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:20:01 weisshorn01 CROND[8250]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:20:01 weisshorn02 CROND[46847]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:20:01 weisshorn04 CROND[125098]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:20:01 weisshorn07 CROND[125803]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:20:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:20:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:20:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:20:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:21:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:21:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:21:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:21:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:22:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:22:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:22:35 weisshorn11 kernel: Lustre: scratch-OST0039: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805b03ec000, cur 1340126555 expire 1340126405 last 1340126328
Jun 19 19:22:35 weisshorn03 kernel: Lustre: scratch-OST0038: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080800bc00, cur 1340126555 expire 1340126405 last 1340126328
Jun 19 19:22:35 weisshorn03 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:22:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:22:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:23:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:23:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:23:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:23:50 weisshorn13 kernel: Lustre: scratch-OST0003: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100dff5800, cur 1340126630 expire 1340126480 last 1340126403
Jun 19 19:23:50 weisshorn13 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:23:50 weisshorn05 kernel: Lustre: scratch-OST003e: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100d301400, cur 1340126630 expire 1340126480 last 1340126403
Jun 19 19:23:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:24:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:24:15 weisshorn11 kernel: Lustre: scratch-OST0008: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808242cc000, cur 1340126655 expire 1340126505 last 1340126428
Jun 19 19:24:15 weisshorn04 kernel: Lustre: scratch-OST003d: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ec66eb000, cur 1340126655 expire 1340126505 last 1340126428
Jun 19 19:24:15 weisshorn04 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:24:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:24:30 weisshorn10 kernel: Lustre: scratch-OST0022: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ed03d9c00, cur 1340126670 expire 1340126520 last 1340126443
Jun 19 19:24:30 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e599f2000, cur 1340126670 expire 1340126520 last 1340126443
Jun 19 19:24:30 weisshorn07 kernel: Lustre: scratch-OST001f: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810165bd800, cur 1340126670 expire 1340126520 last 1340126443
Jun 19 19:24:30 weisshorn07 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 19:24:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:24:40 weisshorn06 kernel: Lustre: scratch-OST0036: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810143e0c00, cur 1340126680 expire 1340126530 last 1340126453
Jun 19 19:24:45 weisshorn09 nslcd[3954]: [bc6770] ldap_result() failed: Can't contact LDAP server
Jun 19 19:24:50 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:25:05 weisshorn12 kernel: Lustre: scratch-OST0047: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080cb0cc00, cur 1340126705 expire 1340126555 last 1340126478
Jun 19 19:25:05 weisshorn09 kernel: Lustre: scratch-OST0034: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880820c23400, cur 1340126705 expire 1340126555 last 1340126478
Jun 19 19:25:05 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:25:08 weisshorn08 nslcd[3946]: [b5b1c6] ldap_result() failed: Can't contact LDAP server
Jun 19 19:25:09 weisshorn06 nslcd[3953]: [d385c9] ldap_result() failed: Can't contact LDAP server
Jun 19 19:25:11 weisshorn04 nslcd[3885]: [bc6770] ldap_result() failed: Can't contact LDAP server
Jun 19 19:25:12 weisshorn07 nslcd[4297]: [b5b1c6] ldap_result() failed: Can't contact LDAP server
Jun 19 19:25:20 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:25:31 weisshorn03 kernel: Lustre: scratch-OST0009: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100f19fc00, cur 1340126731 expire 1340126581 last 1340126504
Jun 19 19:25:32 weisshorn11 nslcd[3954]: [a9831a] ldap_result() failed: Can't contact LDAP server
Jun 19 19:25:32 weisshorn12 nslcd[3954]: [b5b1c6] ldap_result() failed: Can't contact LDAP server
Jun 19 19:25:34 weisshorn13 nslcd[4224]: [b5b1c6] ldap_result() failed: Can't contact LDAP server
Jun 19 19:25:35 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:25:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:26:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:26:11 weisshorn05 nslcd[4378]: [6be6b9] ldap_result() failed: Can't contact LDAP server
Jun 19 19:26:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:26:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:26:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:27:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:27:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:27:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:27:46 weisshorn11 kernel: ------------[ cut here ]------------
Jun 19 19:27:46 weisshorn11 kernel: kernel BUG at /var/lib/jenkins/workspace/lustre-master/arch/x86_64/build_type/server/distro/el6/ib_stack/inkernel/BUILD/BUILD/lustre-2.2.51/lustre/lvfs/fsfilt-ldiskfs.c:1027!
Jun 19 19:27:46 weisshorn11 kernel: invalid opcode: 0000 [#1] SMP 
Jun 19 19:27:46 weisshorn11 kernel: last sysfs file: /sys/devices/system/cpu/cpu31/cache/index2/shared_cpu_map
Jun 19 19:27:46 weisshorn11 kernel: CPU 8 
Jun 19 19:27:46 weisshorn11 kernel: Modules linked in: lnet_selftest(U) ib_uverbs obdfilter(U) fsfilt_ldiskfs(U) exportfs ost(U) mgc(U) lustre(U) lquota(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) ldiskfs(U) ko2iblnd(U) rdma_cm iw_cm ib_addr obdclass(U) lnet(U) lvfs(U) libcfs(U) autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf dm_round_robin ib_ipoib ib_cm ib_sa ipv6 microcode sg sb_edac edac_core iTCO_wdt iTCO_vendor_support i2c_i801 i2c_core mlx4_ib ib_mad ib_core mlx4_en mlx4_core ioatdma igb dca ext4 mbcache jbd2 sd_mod crc_t10dif qla2xxx scsi_transport_fc scsi_tgt ahci isci libsas scsi_transport_sas wmi dm_multipath dm_mirror dm_region_hash dm_log dm_mod scsi_dh_rdac [last unloaded: scsi_wait_scan]
Jun 19 19:27:46 weisshorn11 kernel:
Jun 19 19:27:46 weisshorn11 kernel: Pid: 7800, comm: ll_ost_io_296 Not tainted 2.6.32-220.7.1.el6_lustre.g9c8f747.x86_64 #1 Intel Corporation S2600GZ/S2600GZ
Jun 19 19:27:46 weisshorn11 kernel: RIP: 0010:[<ffffffffa0d42677>]  [<ffffffffa0d42677>] ldiskfs_ext_new_extent_cb+0x587/0x5f0 [fsfilt_ldiskfs]
Jun 19 19:27:46 weisshorn11 kernel: RSP: 0018:ffff8807bfd7b620  EFLAGS: 00010246
Jun 19 19:27:46 weisshorn11 kernel: RAX: 0000000000000038 RBX: ffff8807bfd7b780 RCX: 0000000000000008
Jun 19 19:27:46 weisshorn11 kernel: RDX: ffff880e9b5bc200 RSI: ffff880d25d06340 RDI: ffff880e9b5bc2f8
Jun 19 19:27:46 weisshorn11 kernel: RBP: ffff8807bfd7b6c0 R08: ffff8807bfd7b780 R09: ffff880d25d06340
Jun 19 19:27:46 weisshorn11 kernel: R10: ffff880f8aab8000 R11: 0000000000000001 R12: ffff8807bfd7b700
Jun 19 19:27:46 weisshorn11 kernel: R13: ffff880e9b5bc2f8 R14: ffff880d25d06340 R15: ffff880e9b5bc2f8
Jun 19 19:27:46 weisshorn11 kernel: FS:  00007f790ee17700(0000) GS:ffff88085c400000(0000) knlGS:0000000000000000
Jun 19 19:27:46 weisshorn11 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jun 19 19:27:46 weisshorn11 kernel: CR2: 00007f98e308d030 CR3: 0000000821784000 CR4: 00000000000406e0
Jun 19 19:27:46 weisshorn11 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 19 19:27:46 weisshorn11 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jun 19 19:27:46 weisshorn11 kernel: Process ll_ost_io_296 (pid: 7800, threadinfo ffff8807bfd7a000, task ffff8807c0d94a80)
Jun 19 19:27:46 weisshorn11 kernel: Stack:
Jun 19 19:27:46 weisshorn11 kernel: ffff880e9b5bc224 ffff880e9b5bc218 0000000000000000 ffffffff8100bdae
Jun 19 19:27:46 weisshorn11 kernel: <0> ffff8807bfd7b6c0 0000000000000001 ffff880f8aab8000 ffff880d25d06340
Jun 19 19:27:46 weisshorn11 kernel: <0> 0000000000000000 ffff880e9b5bc2d8 0000000000000009 0000000000000001
Jun 19 19:27:46 weisshorn11 kernel: Call Trace:
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffff8100bdae>] ? call_function_single_interrupt+0xe/0x20
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa07eb39f>] ldiskfs_ext_walk_space+0x14f/0x340 [ldiskfs]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa0d420f0>] ? ldiskfs_ext_new_extent_cb+0x0/0x5f0 [fsfilt_ldiskfs]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa0d41e53>] fsfilt_map_nblocks+0xa3/0xd0 [fsfilt_ldiskfs]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa0d41fa3>] fsfilt_ldiskfs_map_ext_inode_pages+0x123/0x1e0 [fsfilt_ldiskfs]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa0d420e5>] fsfilt_ldiskfs_map_inode_pages+0x85/0x90 [fsfilt_ldiskfs]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffff81110fb8>] ? add_to_page_cache_lru+0x68/0x80
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa0d7eda6>] filter_do_bio+0xf36/0x1570 [obdfilter]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffff81111bc8>] ? find_or_create_page+0x68/0xb0
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffff81012b59>] ? read_tsc+0x9/0x20
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa0d782f3>] filter_preprw_read+0x673/0xb20 [obdfilter]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa08dfa02>] ? null_alloc_rs+0x172/0x2e0 [ptlrpc]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa08cf5de>] ? sptlrpc_svc_alloc_rs+0x7e/0x2a0 [ptlrpc]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa0d787e0>] filter_preprw+0x40/0x80 [obdfilter]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa0d11508>] obd_preprw+0x128/0x2f0 [ost]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa0d170f4>] ost_brw_read+0xbf4/0x1120 [ost]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa08a86f4>] ? lustre_msg_get_version+0x54/0x90 [ptlrpc]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa08a87a8>] ? lustre_msg_check_version+0x78/0x90 [ptlrpc]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa0d1be00>] ost_handle+0x1da0/0x39e0 [ost]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa08a8af4>] ? lustre_msg_get_transno+0x54/0x90 [ptlrpc]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa08aeb15>] ? ptlrpc_update_export_timer+0x55/0x360 [ptlrpc]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa08b43c1>] ptlrpc_server_handle_request+0x3c1/0xcb0 [ptlrpc]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa03a84ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa03b2ef9>] ? lc_watchdog_touch+0x79/0x110 [libcfs]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa08ae462>] ? ptlrpc_wait_event+0xb2/0x2c0 [ptlrpc]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffff810519c3>] ? __wake_up+0x53/0x70
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa08b53cf>] ptlrpc_main+0x71f/0x1210 [ptlrpc]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 19:27:46 weisshorn11 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Jun 19 19:27:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:27:54 weisshorn02 kernel: Lustre: 2877:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340126867/real 1340126867]  req@ffff88040f744c00 x1403849754836135/t0(0) o400->scratch-OST0021-osc-ffff88081e6aac00@148.187.7.111@o2ib2:28/4 lens 192/192 e 0 to 1 dl 1340126874 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:27:54 weisshorn02 kernel: Lustre: 2877:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 47 previous similar messages
Jun 19 19:27:54 weisshorn02 kernel: Lustre: scratch-OST0021-osc-ffff88081e6aac00: Connection to scratch-OST0021 (at 148.187.7.111@o2ib2) was lost; in progress operations using this service will wait for recovery to complete
Jun 19 19:27:54 weisshorn02 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 19:27:55 weisshorn01 kernel: Lustre: 2961:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340126868/real 1340126868]  req@ffff8807146d7000 x1403850032401297/t0(0) o13->scratch-OST0008-osc-MDT0000@148.187.7.111@o2ib2:7/4 lens 192/336 e 0 to 1 dl 1340126875 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:27:55 weisshorn01 kernel: Lustre: 2961:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
Jun 19 19:27:55 weisshorn01 kernel: Lustre: scratch-OST0008-osc-MDT0000: Connection to scratch-OST0008 (at 148.187.7.111@o2ib2) was lost; in progress operations using this service will wait for recovery to complete
Jun 19 19:27:55 weisshorn01 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 19:27:59 weisshorn14 kernel: Lustre: 14322:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340126872/real 1340126872]  req@ffff881014fb4c00 x1404749288340600/t0(0) o400->scratch-OST0044-osc-ffff8810296e4c00@148.187.7.111@o2ib2:28/4 lens 192/192 e 0 to 1 dl 1340126879 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:27:59 weisshorn14 kernel: Lustre: 14322:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 80 previous similar messages
Jun 19 19:27:59 weisshorn14 kernel: Lustre: scratch-OST0044-osc-ffff8810296e4c00: Connection to scratch-OST0044 (at 148.187.7.111@o2ib2) was lost; in progress operations using this service will wait for recovery to complete
Jun 19 19:27:59 weisshorn14 kernel: Lustre: Skipped 11 previous similar messages
Jun 19 19:28:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:28:18 weisshorn12 kernel: LustreError: 137-5: scratch-OST0008: Not available for connect from 148.187.6.223@o2ib2 (no target)
Jun 19 19:28:18 weisshorn12 kernel: LustreError: 137-5: scratch-OST0015: Not available for connect from 148.187.6.223@o2ib2 (no target)
Jun 19 19:28:18 weisshorn12 kernel: LustreError: Skipped 2 previous similar messages
Jun 19 19:28:18 weisshorn12 kernel: LustreError: 137-5: scratch-OST0008: Not available for connect from 148.187.6.242@o2ib2 (no target)
Jun 19 19:28:18 weisshorn12 kernel: LustreError: Skipped 21 previous similar messages
Jun 19 19:28:19 weisshorn02 kernel: LustreError: 11-0: an error occurred while communicating with 148.187.7.112@o2ib2. The ost_connect operation failed with -19
Jun 19 19:28:19 weisshorn02 kernel: LustreError: Skipped 4 previous similar messages
Jun 19 19:28:20 weisshorn12 kernel: LustreError: 137-5: scratch-OST0008: Not available for connect from 148.187.7.11@o2ib2 (no target)
Jun 19 19:28:20 weisshorn12 kernel: LustreError: Skipped 50 previous similar messages
Jun 19 19:28:20 weisshorn01 kernel: LustreError: 11-0: an error occurred while communicating with 148.187.7.112@o2ib2. The ost_connect operation failed with -19
Jun 19 19:28:20 weisshorn01 kernel: LustreError: Skipped 39 previous similar messages
Jun 19 19:28:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:28:23 weisshorn12 kernel: LustreError: 137-5: scratch-OST0008: Not available for connect from 148.187.7.14@o2ib2 (no target)
Jun 19 19:28:23 weisshorn12 kernel: LustreError: Skipped 21 previous similar messages
Jun 19 19:28:24 weisshorn14 kernel: LustreError: 11-0: an error occurred while communicating with 148.187.7.112@o2ib2. The ost_connect operation failed with -19
Jun 19 19:28:24 weisshorn14 kernel: LustreError: Skipped 8 previous similar messages
Jun 19 19:28:29 weisshorn12 kernel: LustreError: 137-5: scratch-OST0008: Not available for connect from 148.187.6.220@o2ib2 (no target)
Jun 19 19:28:29 weisshorn12 kernel: LustreError: Skipped 33 previous similar messages
Jun 19 19:28:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:28:37 weisshorn12 kernel: LustreError: 137-5: scratch-OST0008: Not available for connect from 148.187.6.237@o2ib2 (no target)
Jun 19 19:28:37 weisshorn12 kernel: LustreError: Skipped 78 previous similar messages
Jun 19 19:28:47 weisshorn02 kernel: LNetError: 2869:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 3 seconds
Jun 19 19:28:47 weisshorn02 kernel: LNetError: 2869:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Timed out RDMA with 148.187.7.111@o2ib2 (53): c: 0, oc: 0, rc: 8
Jun 19 19:28:47 weisshorn02 kernel: Lustre: 2871:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340126924/real 1340126927]  req@ffff88040f650800 x1403849754836301/t0(0) o8->scratch-OST002d-osc-ffff88081e6aac00@148.187.7.111@o2ib2:28/4 lens 368/512 e 0 to 1 dl 1340126935 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:28:47 weisshorn02 kernel: Lustre: 2871:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jun 19 19:28:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:28:51 weisshorn01 kernel: LNetError: 2956:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 5 seconds
Jun 19 19:28:51 weisshorn01 kernel: LNetError: 2956:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Timed out RDMA with 148.187.7.111@o2ib2 (55): c: 0, oc: 0, rc: 8
Jun 19 19:28:52 weisshorn12 kernel: igb: eth4 NIC Link is Down
Jun 19 19:28:58 weisshorn14 kernel: LNetError: 12717:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 8 seconds
Jun 19 19:28:58 weisshorn14 kernel: LNetError: 12717:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Timed out RDMA with 148.187.7.111@o2ib2 (58): c: 0, oc: 0, rc: 8
Jun 19 19:28:58 weisshorn14 kernel: Lustre: 14313:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340126929/real 1340126938]  req@ffff88100461ec00 x1404749288340731/t0(0) o8->scratch-OST002d-osc-ffff8810296e4c00@148.187.7.111@o2ib2:28/4 lens 368/512 e 0 to 1 dl 1340126940 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:28:58 weisshorn14 kernel: Lustre: 14313:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jun 19 19:29:02 weisshorn12 kernel: LustreError: 137-5: scratch-OST0008: Not available for connect from 148.187.6.6@o2ib2 (no target)
Jun 19 19:29:02 weisshorn12 kernel: LustreError: Skipped 39 previous similar messages
Jun 19 19:29:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:29:09 weisshorn02 kernel: LustreError: 11-0: an error occurred while communicating with 148.187.7.112@o2ib2. The ost_connect operation failed with -19
Jun 19 19:29:09 weisshorn02 kernel: LustreError: Skipped 5 previous similar messages
Jun 19 19:29:10 weisshorn01 kernel: LustreError: 11-0: an error occurred while communicating with 148.187.7.112@o2ib2. The ost_connect operation failed with -19
Jun 19 19:29:10 weisshorn01 kernel: LustreError: Skipped 5 previous similar messages
Jun 19 19:29:14 weisshorn14 kernel: LustreError: 11-0: an error occurred while communicating with 148.187.7.112@o2ib2. The ost_connect operation failed with -19
Jun 19 19:29:14 weisshorn14 kernel: LustreError: Skipped 5 previous similar messages
Jun 19 19:29:15 weisshorn12 heartbeat: [5359]: WARN: node weisshorn11.admin.cscs.ch: is dead
Jun 19 19:29:15 weisshorn12 heartbeat: [5359]: WARN: No STONITH device configured.
Jun 19 19:29:15 weisshorn12 heartbeat: [5359]: WARN: Shared disks are not protected.
Jun 19 19:29:15 weisshorn12 heartbeat: [5359]: info: Resources being acquired from weisshorn11.admin.cscs.ch.
Jun 19 19:29:15 weisshorn12 heartbeat: [5359]: info: Link weisshorn11.admin.cscs.ch:eth4 dead.
Jun 19 19:29:15 weisshorn12 heartbeat: [5359]: info: Link weisshorn11.admin.cscs.ch:ib0 dead.
Jun 19 19:29:15 weisshorn12 heartbeat: [125565]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Jun 19 19:29:15 weisshorn12 harc[125565]: info: Running /etc/ha.d//rc.d/status status
Jun 19 19:29:15 weisshorn12 mach_down[125619]: info: Taking over resource group Filesystem::/dev/mapper/Lun00::/mnt/Lun00::lustre
Jun 19 19:29:15 weisshorn12 ResourceManager[125658]: info: Acquiring resource group: weisshorn11.admin.cscs.ch Filesystem::/dev/mapper/Lun00::/mnt/Lun00::lustre Filesystem::/dev/mapper/Lun02::/mnt/Lun02::lustre Filesystem::/dev/mapper/Lun04::/mnt/Lun04::lustre Filesystem::/dev/mapper/Lun06::/mnt/Lun06::lustre Filesystem::/dev/mapper/Lun08::/mnt/Lun08::lustre Filesystem::/dev/mapper/Lun10::/mnt/Lun10::lustre
Jun 19 19:29:15 weisshorn12 Filesystem[125618]: INFO:  Running OK
Jun 19 19:29:15 weisshorn12 heartbeat: [125566]: info: Local Resource acquisition completed.
Jun 19 19:29:15 weisshorn12 heartbeat: [5359]: debug: StartNextRemoteRscReq(): child count 1
Jun 19 19:29:15 weisshorn12 Filesystem[125712]: INFO:  Resource is stopped
Jun 19 19:29:15 weisshorn12 ResourceManager[125658]: info: Running /etc/ha.d/resource.d/Filesystem /dev/mapper/Lun00 /mnt/Lun00 lustre start
Jun 19 19:29:15 weisshorn12 Filesystem[125804]: INFO: Running start for /dev/mapper/Lun00 on /mnt/Lun00
Jun 19 19:29:16 weisshorn12 kernel: LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jun 19 19:29:16 weisshorn12 kernel:
Jun 19 19:29:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:29:35 weisshorn12 kernel: LustreError: 137-5: scratch-OST0008: Not available for connect from 148.187.6.8@o2ib2 (no target)
Jun 19 19:29:35 weisshorn12 kernel: LustreError: Skipped 247 previous similar messages
Jun 19 19:29:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:29:37 weisshorn02 kernel: Lustre: 2871:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340126974/real 1340126977]  req@ffff880412349000 x1403849754836449/t0(0) o8->scratch-OST002d-osc-ffff88081e6aac00@148.187.7.111@o2ib2:28/4 lens 368/512 e 0 to 1 dl 1340126990 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:29:37 weisshorn02 kernel: Lustre: 2871:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Jun 19 19:29:40 weisshorn12 kernel: Lustre: scratch-OST001c: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808a2b41800, cur 1340126980 expire 1340126830 last 1340126753
Jun 19 19:29:40 weisshorn12 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:29:41 weisshorn01 kernel: Lustre: 2958:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340126975/real 1340126981]  req@ffff880312c86400 x1403850032433865/t0(0) o8->scratch-OST002d-osc-MDT0000@148.187.7.111@o2ib2:28/4 lens 368/512 e 0 to 1 dl 1340126991 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:29:41 weisshorn01 kernel: Lustre: 2958:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
Jun 19 19:29:48 weisshorn14 kernel: Lustre: 14313:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340126979/real 1340126988]  req@ffff881004618000 x1404749288340902/t0(0) o8->scratch-OST0044-osc-ffff8810296e4c00@148.187.7.111@o2ib2:28/4 lens 368/512 e 0 to 1 dl 1340126995 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:29:48 weisshorn14 kernel: Lustre: 14313:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Jun 19 19:29:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:29:59 weisshorn02 kernel: LustreError: 11-0: an error occurred while communicating with 148.187.7.112@o2ib2. The ost_connect operation failed with -19
Jun 19 19:29:59 weisshorn02 kernel: LustreError: Skipped 5 previous similar messages
Jun 19 19:30:01 weisshorn03 CROND[126378]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:30:01 weisshorn06 CROND[126588]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:30:01 weisshorn05 CROND[125696]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:30:01 weisshorn10 CROND[125720]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:30:01 weisshorn09 CROND[125416]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:30:01 weisshorn14 CROND[54351]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:30:01 weisshorn08 CROND[124484]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:30:01 weisshorn12 CROND[125867]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:30:01 weisshorn13 CROND[129125]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:30:01 weisshorn01 CROND[8298]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:30:01 weisshorn02 CROND[46916]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:30:01 weisshorn04 CROND[125163]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:30:02 weisshorn07 CROND[125846]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:30:02 weisshorn12 kernel: LDISKFS-fs (dm-0): recovery complete
Jun 19 19:30:02 weisshorn12 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: 
Jun 19 19:30:03 weisshorn12 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: 
Jun 19 19:30:03 weisshorn12 kernel: Lustre: 125936:0:(ldlm_lib.c:2062:target_recovery_init()) RECOVERY: service scratch-OST0008, 1547 recoverable clients, last_transno 42149539
Jun 19 19:30:03 weisshorn12 kernel: Lustre: 125936:0:(ldlm_lib.c:2062:target_recovery_init()) Skipped 1 previous similar message
Jun 19 19:30:03 weisshorn12 kernel: Lustre: scratch-OST0008: Now serving scratch-OST0008 on /dev/mapper/Lun00 with recovery enabled
Jun 19 19:30:03 weisshorn12 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:30:03 weisshorn12 Filesystem[125796]: INFO:  Success
Jun 19 19:30:03 weisshorn12 Filesystem[125956]: INFO:  Resource is stopped
Jun 19 19:30:03 weisshorn12 ResourceManager[125658]: info: Running /etc/ha.d/resource.d/Filesystem /dev/mapper/Lun02 /mnt/Lun02 lustre start
Jun 19 19:30:03 weisshorn12 Filesystem[126035]: INFO: Running start for /dev/mapper/Lun02 on /mnt/Lun02
Jun 19 19:30:03 weisshorn12 kernel: LDISKFS-fs warning (device dm-2): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jun 19 19:30:03 weisshorn12 kernel:
Jun 19 19:30:03 weisshorn12 kernel: Lustre: scratch-OST0008: Will be in recovery for at least 5:00, or until 1547 clients reconnect
Jun 19 19:30:03 weisshorn12 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 19:30:04 weisshorn14 kernel: LustreError: 11-0: an error occurred while communicating with 148.187.7.112@o2ib2. The ost_connect operation failed with -19
Jun 19 19:30:04 weisshorn14 kernel: LustreError: Skipped 5 previous similar messages
Jun 19 19:30:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:30:10 weisshorn03 kernel: Lustre: 5808:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340127003/real 1340127003]  req@ffff881010cc3800 x1403849871423817/t0(0) o400->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 192/192 e 0 to 1 dl 1340127010 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:10 weisshorn03 kernel: Lustre: 5808:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Jun 19 19:30:10 weisshorn03 kernel: LustreError: 166-1: MGC148.187.7.101@o2ib2: Connection to MGS (at 148.187.7.101@o2ib2) was lost; in progress operations using this service will fail
Jun 19 19:30:11 weisshorn04 kernel: Lustre: 5742:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340127004/real 1340127004]  req@ffff881018790000 x1403849861583690/t0(0) o400->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 192/192 e 0 to 1 dl 1340127011 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:11 weisshorn04 kernel: Lustre: 5742:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Jun 19 19:30:11 weisshorn04 kernel: LustreError: 166-1: MGC148.187.7.101@o2ib2: Connection to MGS (at 148.187.7.101@o2ib2) was lost; in progress operations using this service will fail
Jun 19 19:30:11 weisshorn14 kernel: Lustre: scratch-MDT0000-mdc-ffff8810296e4c00: Connection to scratch-MDT0000 (at 148.187.7.101@o2ib2) was lost; in progress operations using this service will wait for recovery to complete
Jun 19 19:30:11 weisshorn14 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 19:30:12 weisshorn09 kernel: Lustre: 5850:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340127005/real 1340127005]  req@ffff8810165d3000 x1403849921463308/t0(0) o400->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 192/192 e 0 to 1 dl 1340127012 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:12 weisshorn09 kernel: Lustre: 5850:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Jun 19 19:30:12 weisshorn09 kernel: LustreError: 166-1: MGC148.187.7.101@o2ib2: Connection to MGS (at 148.187.7.101@o2ib2) was lost; in progress operations using this service will fail
Jun 19 19:30:12 weisshorn09 kernel: LustreError: Skipped 2 previous similar messages
Jun 19 19:30:14 weisshorn10 kernel: Lustre: 6117:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340127007/real 1340127007]  req@ffff88081d39e400 x1403849943172813/t0(0) o400->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 192/192 e 0 to 1 dl 1340127014 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:14 weisshorn10 kernel: Lustre: 6117:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Jun 19 19:30:14 weisshorn10 kernel: LustreError: 166-1: MGC148.187.7.101@o2ib2: Connection to MGS (at 148.187.7.101@o2ib2) was lost; in progress operations using this service will fail
Jun 19 19:30:16 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340127010/real 1340127010]  req@ffff881026e2d400 x1403849871423818/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127016 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:17 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340127011/real 1340127011]  req@ffff880f6582c400 x1403849861583691/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127017 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:17 weisshorn14 kernel: LustreError: 166-1: MGC148.187.7.101@o2ib2: Connection to MGS (at 148.187.7.101@o2ib2) was lost; in progress operations using this service will fail
Jun 19 19:30:18 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127012/real 0]  req@ffff880c95eb7000 x1403849921463309/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127018 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:18 weisshorn02 kernel: LustreError: 166-1: MGC148.187.7.101@o2ib2: Connection to MGS (at 148.187.7.101@o2ib2) was lost; in progress operations using this service will fail
Jun 19 19:30:19 weisshorn13 kernel: Lustre: 6058:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127012/real 0]  req@ffff880f1c72d400 x1403849984281791/t0(0) o400->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 192/192 e 0 to 1 dl 1340127019 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:19 weisshorn13 kernel: Lustre: 6058:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
Jun 19 19:30:19 weisshorn13 kernel: LustreError: 166-1: MGC148.187.7.101@o2ib2: Connection to MGS (at 148.187.7.101@o2ib2) was lost; in progress operations using this service will fail
Jun 19 19:30:20 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127014/real 0]  req@ffff88081d64b000 x1403849943172814/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127020 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:21 weisshorn05 kernel: Lustre: 5267:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127014/real 0]  req@ffff880b6098e400 x1403849636377443/t0(0) o400->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 192/192 e 0 to 1 dl 1340127021 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:21 weisshorn05 kernel: Lustre: 5267:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Jun 19 19:30:21 weisshorn05 kernel: LustreError: 166-1: MGC148.187.7.101@o2ib2: Connection to MGS (at 148.187.7.101@o2ib2) was lost; in progress operations using this service will fail
Jun 19 19:30:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:30:21 weisshorn06 kernel: Lustre: 6006:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127014/real 0]  req@ffff8810152e0400 x1403849897065188/t0(0) o400->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 192/192 e 0 to 1 dl 1340127021 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:21 weisshorn06 kernel: Lustre: 6006:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 19:30:21 weisshorn06 kernel: LustreError: 166-1: MGC148.187.7.101@o2ib2: Connection to MGS (at 148.187.7.101@o2ib2) was lost; in progress operations using this service will fail
Jun 19 19:30:23 weisshorn07 kernel: Lustre: 6140:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127016/real 0]  req@ffff8802e1b87c00 x1403849873211652/t0(0) o400->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 192/192 e 0 to 1 dl 1340127023 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:23 weisshorn07 kernel: Lustre: 6140:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
Jun 19 19:30:23 weisshorn07 kernel: LustreError: 166-1: MGC148.187.7.101@o2ib2: Connection to MGS (at 148.187.7.101@o2ib2) was lost; in progress operations using this service will fail
Jun 19 19:30:25 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127019/real 0]  req@ffff880fc37e6400 x1403849984281792/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127025 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:26 weisshorn08 kernel: Lustre: 5770:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127019/real 0]  req@ffff880c47195000 x1403849822727060/t0(0) o400->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 192/192 e 0 to 1 dl 1340127026 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:26 weisshorn08 kernel: Lustre: 5770:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
Jun 19 19:30:26 weisshorn08 kernel: LustreError: 166-1: MGC148.187.7.101@o2ib2: Connection to MGS (at 148.187.7.101@o2ib2) was lost; in progress operations using this service will fail
Jun 19 19:30:27 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127021/real 0]  req@ffff881010058400 x1403849636377444/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127027 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:27 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127021/real 0]  req@ffff8810137fa400 x1403849897065189/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127027 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:29 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127023/real 0]  req@ffff8805255fe800 x1403849873211653/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127029 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:32 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127026/real 0]  req@ffff880e9f7a1400 x1403849822727061/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127032 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:30:38 weisshorn12 kernel: Lustre: 5763:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127031/real 0]  req@ffff88081c4f0c00 x1403849952681482/t0(0) o400->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 192/192 e 0 to 1 dl 1340127038 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:38 weisshorn12 kernel: Lustre: 5763:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Jun 19 19:30:38 weisshorn12 kernel: LustreError: 166-1: MGC148.187.7.101@o2ib2: Connection to MGS (at 148.187.7.101@o2ib2) was lost; in progress operations using this service will fail
Jun 19 19:30:40 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127039/real 1340127040]  req@ffff88080649e800 x1403849943172815/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127050 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:41 weisshorn12 kernel: LustreError: 137-5: scratch-OST0015: Not available for connect from 148.187.6.201@o2ib2 (no target)
Jun 19 19:30:41 weisshorn12 kernel: LustreError: Skipped 214 previous similar messages
Jun 19 19:30:44 weisshorn12 kernel: Lustre: scratch-OST0008: Denying connection for new client 148.187.6.6@o2ib2 (at 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03), waiting for 4 clients in recovery for 4:19
Jun 19 19:30:44 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127038/real 0]  req@ffff88080be76800 x1403849952681483/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127044 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:46 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127035/real 0]  req@ffff881012b4e000 x1403849871423819/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127046 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:47 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127036/real 0]  req@ffff8809071b9c00 x1403849861583692/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127047 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:48 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127037/real 0]  req@ffff880c95eb7000 x1403849921463310/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127048 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:48 weisshorn12 kernel: LDISKFS-fs (dm-2): recovery complete
Jun 19 19:30:48 weisshorn12 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: 
Jun 19 19:30:49 weisshorn12 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: 
Jun 19 19:30:49 weisshorn12 kernel: Lustre: MGC148.187.7.101@o2ib2: Reactivating import
Jun 19 19:30:49 weisshorn02 kernel: Lustre: 2871:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340127043/real 1340127043]  req@ffff880419830800 x1403849754836553/t0(0) o250->MGC148.187.7.101@o2ib2@0@lo:26/25 lens 368/512 e 0 to 1 dl 1340127049 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:49 weisshorn02 kernel: Lustre: 2871:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 19:30:50 weisshorn02 kernel: Lustre: scratch-MDT0000-mdc-ffff88081e6aac00: Connection to scratch-MDT0000 (at 148.187.7.101@o2ib2) was lost; in progress operations using this service will wait for recovery to complete
Jun 19 19:30:50 weisshorn02 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 19:30:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:30:55 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127044/real 0]  req@ffff8810130cac00 x1403849984281793/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127055 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:55 weisshorn09 kernel: Lustre: scratch-OST001a: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bf6df000, cur 1340127055 expire 1340126905 last 1340126828
Jun 19 19:30:55 weisshorn06 kernel: Lustre: scratch-OST002a: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d68077800, cur 1340127055 expire 1340126905 last 1340126828
Jun 19 19:30:55 weisshorn06 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:30:55 weisshorn05 kernel: Lustre: scratch-OST001b: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802493b6400, cur 1340127055 expire 1340126905 last 1340126828
Jun 19 19:30:55 weisshorn05 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 19:30:55 weisshorn03 kernel: Lustre: scratch-OST002c: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802614a2800, cur 1340127055 expire 1340126905 last 1340126828
Jun 19 19:30:55 weisshorn03 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:30:57 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127046/real 0]  req@ffff88100e7e5c00 x1403849636377445/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127057 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:57 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127049/real 1340127057]  req@ffff8805f4c89400 x1403849952681484/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127060 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:57 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127046/real 0]  req@ffff8810124f0400 x1403849897065190/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127057 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:30:59 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127048/real 0]  req@ffff8804d313e800 x1403849873211654/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127059 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:02 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127061/real 1340127062]  req@ffff880dc5a6c000 x1403849861583693/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:02 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127051/real 0]  req@ffff880e1e525c00 x1403849822727062/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127062 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:02 weisshorn12 kernel: LustreError: 5874:0:(client.c:1031:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88080d5a2c00 x1403849952681485/t0(0) o101->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jun 19 19:31:03 weisshorn14 kernel: Lustre: 14313:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127042/real 0]  req@ffff881014fb6400 x1404749288341006/t0(0) o8->scratch-OST0015-osc-ffff8810296e4c00@148.187.7.111@o2ib2:28/4 lens 368/512 e 0 to 1 dl 1340127063 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:03 weisshorn14 kernel: Lustre: 14313:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jun 19 19:31:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:31:07 weisshorn14 kernel: LustreError: 11-0: an error occurred while communicating with 148.187.7.112@o2ib2. The ost_connect operation failed with -19
Jun 19 19:31:07 weisshorn14 kernel: LustreError: Skipped 4 previous similar messages
Jun 19 19:31:08 weisshorn12 kernel: LustreError: 126092:0:(client.c:1031:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880177870000 x1403849952681486/t0(0) o253->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 4736/4736 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jun 19 19:31:08 weisshorn12 kernel: LustreError: 126092:0:(obd_mount.c:1109:server_register_target()) Cannot talk to the MGS: -5, not fatal
Jun 19 19:31:09 weisshorn12 kernel: Lustre: scratch-OST0008: Denying connection for new client 148.187.6.6@o2ib2 (at 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03), waiting for 4 clients in recovery for 3:54
Jun 19 19:31:14 weisshorn12 kernel: LustreError: 126092:0:(client.c:1031:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff880177870000 x1403849952681488/t0(0) o101->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jun 19 19:31:14 weisshorn12 kernel: LustreError: 126092:0:(client.c:1031:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Jun 19 19:31:14 weisshorn12 kernel: Lustre: 126164:0:(ldlm_lib.c:2062:target_recovery_init()) RECOVERY: service scratch-OST0015, 1547 recoverable clients, last_transno 43053859
Jun 19 19:31:14 weisshorn12 kernel: Lustre: scratch-OST0015: Now serving scratch-OST0015 on /dev/mapper/Lun02 with recovery enabled
Jun 19 19:31:14 weisshorn12 kernel: Lustre: scratch-OST0015: Temporarily refusing client connection from 148.187.7.5@o2ib2
Jun 19 19:31:15 weisshorn12 kernel: Lustre: scratch-OST0015: Temporarily refusing client connection from 148.187.6.207@o2ib2
Jun 19 19:31:15 weisshorn12 kernel: Lustre: scratch-OST0015: Temporarily refusing client connection from 148.187.7.102@o2ib2
Jun 19 19:31:15 weisshorn02 kernel: LustreError: 11-0: an error occurred while communicating with 148.187.7.112@o2ib2. The ost_connect operation failed with -11
Jun 19 19:31:15 weisshorn02 kernel: LustreError: Skipped 5 previous similar messages
Jun 19 19:31:16 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127060/real 0]  req@ffff881012cb6c00 x1403849871423820/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127076 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:17 weisshorn12 kernel: Lustre: scratch-OST0015: Temporarily refusing client connection from 148.187.6.243@o2ib2
Jun 19 19:31:17 weisshorn12 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:31:18 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127062/real 0]  req@ffff880a62fb7800 x1403849921463311/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127078 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:20 weisshorn10 kernel: Lustre: scratch-OST0016: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eb8723c00, cur 1340127080 expire 1340126930 last 1340126853
Jun 19 19:31:20 weisshorn10 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:31:20 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127064/real 0]  req@ffff880355767000 x1403849943172816/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127080 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:20 weisshorn12 kernel: LustreError: 5874:0:(client.c:1031:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88080d5a2c00 x1403849952681490/t0(0) o101->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jun 19 19:31:20 weisshorn12 kernel: LustreError: 5874:0:(client.c:1031:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Jun 19 19:31:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:31:24 weisshorn02 kernel: igb: eth1 NIC Link is Down
Jun 19 19:31:25 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127069/real 0]  req@ffff880e97a7f000 x1403849984281794/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127085 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:25 weisshorn12 kernel: LustreError: 126092:0:(client.c:1031:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88013c424800 x1403849952681491/t0(0) o253->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 4736/4736 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jun 19 19:31:25 weisshorn12 Filesystem[126027]: INFO:  Success
Jun 19 19:31:25 weisshorn12 Filesystem[126184]: INFO:  Resource is stopped
Jun 19 19:31:25 weisshorn12 ResourceManager[125658]: info: Running /etc/ha.d/resource.d/Filesystem /dev/mapper/Lun04 /mnt/Lun04 lustre start
Jun 19 19:31:25 weisshorn12 Filesystem[126263]: INFO: Running start for /dev/mapper/Lun04 on /mnt/Lun04
Jun 19 19:31:26 weisshorn12 kernel: LDISKFS-fs warning (device dm-4): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jun 19 19:31:26 weisshorn12 kernel:
Jun 19 19:31:27 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127071/real 0]  req@ffff88100fe5c800 x1403849636377446/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127087 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:27 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127087/real 1340127087]  req@ffff881019863800 x1403849921463312/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127108 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:27 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127071/real 0]  req@ffff88102b3d2800 x1403849897065191/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127087 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:29 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127073/real 0]  req@ffff880d49924000 x1403849873211655/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127089 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:31 weisshorn12 kernel: Lustre: scratch-OST0015: Will be in recovery for at least 5:00, or until 1547 clients reconnect
Jun 19 19:31:32 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127076/real 0]  req@ffff880930d44000 x1403849822727063/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127092 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:32 weisshorn12 kernel: LustreError: 5874:0:(client.c:1031:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88080d5a2c00 x1403849952681493/t0(0) o101->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jun 19 19:31:32 weisshorn12 kernel: LustreError: 5874:0:(client.c:1031:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Jun 19 19:31:32 weisshorn02 heartbeat: [3934]: WARN: node weisshorn01.admin.cscs.ch: is dead
Jun 19 19:31:32 weisshorn02 heartbeat: [3934]: WARN: No STONITH device configured.
Jun 19 19:31:32 weisshorn02 heartbeat: [3934]: WARN: Shared disks are not protected.
Jun 19 19:31:32 weisshorn02 heartbeat: [3934]: info: Resources being acquired from weisshorn01.admin.cscs.ch.
Jun 19 19:31:32 weisshorn02 heartbeat: [3934]: info: Link weisshorn01.admin.cscs.ch:eth1 dead.
Jun 19 19:31:32 weisshorn02 heartbeat: [3934]: info: Link weisshorn01.admin.cscs.ch:ib0 dead.
Jun 19 19:31:32 weisshorn02 heartbeat: [46919]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Jun 19 19:31:32 weisshorn02 harc[46919]: info: Running /etc/ha.d//rc.d/status status
Jun 19 19:31:32 weisshorn02 heartbeat: [46920]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys weisshorn02.admin.cscs.ch] to acquire.
Jun 19 19:31:32 weisshorn02 heartbeat: [3934]: debug: StartNextRemoteRscReq(): child count 1
Jun 19 19:31:32 weisshorn02 mach_down[46949]: info: Taking over resource group Filesystem::/dev/mapper/mds::/mnt/mdt::lustre
Jun 19 19:31:32 weisshorn02 ResourceManager[46976]: info: Acquiring resource group: weisshorn01.admin.cscs.ch Filesystem::/dev/mapper/mds::/mnt/mdt::lustre
Jun 19 19:31:32 weisshorn02 Filesystem[47004]: INFO:  Resource is stopped
Jun 19 19:31:33 weisshorn02 ResourceManager[46976]: info: Running /etc/ha.d/resource.d/Filesystem /dev/mapper/mds /mnt/mdt lustre start
Jun 19 19:31:33 weisshorn02 Filesystem[47083]: INFO: Running start for /dev/mapper/mds on /mnt/mdt
Jun 19 19:31:33 weisshorn02 kernel: sd 8:0:0:0: [sdb] Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters.
Jun 19 19:31:33 weisshorn02 kernel: LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jun 19 19:31:33 weisshorn02 kernel:
Jun 19 19:31:34 weisshorn12 kernel: Lustre: scratch-OST0008: Denying connection for new client 148.187.6.6@o2ib2 (at 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03), waiting for 5 clients in recovery for 3:29
Jun 19 19:31:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:31:44 weisshorn12 kernel: LustreError: 5874:0:(client.c:1031:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88080d5a2c00 x1403849952681496/t0(0) o101->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jun 19 19:31:44 weisshorn12 kernel: LustreError: 5874:0:(client.c:1031:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Jun 19 19:31:44 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127088/real 0]  req@ffff8801677cf800 x1403849952681494/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127104 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:46 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127085/real 0]  req@ffff881013b22c00 x1403849871423821/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127106 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:47 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127086/real 0]  req@ffff881015cd6000 x1403849861583694/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127107 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:50 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127089/real 0]  req@ffff880626089800 x1403849943172817/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127110 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:31:55 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127094/real 0]  req@ffff880db8585000 x1403849984281795/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127115 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:57 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127096/real 0]  req@ffff8810141e4400 x1403849636377447/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127117 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:57 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127096/real 0]  req@ffff880e9d5d0400 x1403849897065192/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127117 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:31:59 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127098/real 0]  req@ffff881013408000 x1403849873211656/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127119 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:32:02 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127101/real 0]  req@ffff880bd301a800 x1403849822727064/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127122 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:32:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:32:06 weisshorn12 kernel: LustreError: 5874:0:(client.c:1031:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88080d66ac00 x1403849952681499/t0(0) o101->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jun 19 19:32:06 weisshorn12 kernel: LustreError: 5874:0:(client.c:1031:ptlrpc_import_delay_req()) Skipped 1 previous similar message
Jun 19 19:32:10 weisshorn12 kernel: Lustre: scratch-OST0008: Denying connection for new client 148.187.6.6@o2ib2 (at 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03), waiting for 5 clients in recovery for 2:53
Jun 19 19:32:12 weisshorn12 kernel: LDISKFS-fs (dm-4): recovery complete
Jun 19 19:32:12 weisshorn12 kernel: LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: 
Jun 19 19:32:13 weisshorn12 kernel: LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: 
Jun 19 19:32:13 weisshorn12 kernel: Lustre: MGC148.187.7.101@o2ib2: Reactivating import
Jun 19 19:32:16 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127110/real 0]  req@ffff881027b4cc00 x1403849871423822/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127136 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:32:16 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127111/real 1340127136]  req@ffff8810180f2000 x1403849861583695/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127137 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:32:17 weisshorn02 kernel: LDISKFS-fs (dm-0): warning: maximal mount count reached, running e2fsck is recommended
Jun 19 19:32:17 weisshorn02 kernel: LDISKFS-fs (dm-0): recovery complete
Jun 19 19:32:17 weisshorn02 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: 
Jun 19 19:32:17 weisshorn02 kernel: LDISKFS-fs (dm-0): warning: maximal mount count reached, running e2fsck is recommended
Jun 19 19:32:18 weisshorn02 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: 
Jun 19 19:32:18 weisshorn02 kernel: Lustre: MGC148.187.7.102@o2ib2: Reactivating import
Jun 19 19:32:18 weisshorn02 kernel: LustreError: 11-0: an error occurred while communicating with 148.187.7.112@o2ib2. The ost_connect operation failed with -19
Jun 19 19:32:18 weisshorn02 kernel: LustreError: Skipped 4 previous similar messages
Jun 19 19:32:18 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127112/real 0]  req@ffff880ed67d9800 x1403849921463313/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127138 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:32:18 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127133/real 1340127138]  req@ffff8807cc14e800 x1403849952681503/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127159 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:32:19 weisshorn02 kernel: Lustre: mdd_obd-scratch-MDT0000: changelog on
Jun 19 19:32:19 weisshorn02 kernel: Lustre: scratch-MDT0000: used disk, loading
Jun 19 19:32:19 weisshorn02 kernel: Lustre: 47190:0:(ldlm_lib.c:2062:target_recovery_init()) RECOVERY: service scratch-MDT0000, 1547 recoverable clients, last_transno 52555669078
Jun 19 19:32:20 weisshorn02 kernel: LustreError: 137-5: scratch-MDT0000: Not available for connect from 148.187.6.208@o2ib2 (not set up)
Jun 19 19:32:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:32:21 weisshorn02 kernel: LustreError: 137-5: scratch-MDT0000: Not available for connect from 148.187.7.13@o2ib2 (not set up)
Jun 19 19:32:22 weisshorn02 kernel: Lustre: 47190:0:(mds_lov.c:1008:mds_notify()) MDS mdd_obd-scratch-MDT0000: add target scratch-OST0000_UUID
Jun 19 19:32:22 weisshorn02 kernel: Lustre: 2871:0:(mds_lov.c:1028:mds_notify()) MDS mdd_obd-scratch-MDT0000: in recovery, not resetting orphans on scratch-OST0000_UUID
Jun 19 19:32:22 weisshorn09 kernel: Lustre: scratch-OST0000: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn09 kernel: Lustre: 8193:0:(filter.c:2699:filter_connect_internal()) scratch-OST0000: Received MDS connection for group 0
Jun 19 19:32:22 weisshorn09 kernel: Lustre: 8193:0:(filter.c:2699:filter_connect_internal()) Skipped 9 previous similar messages
Jun 19 19:32:22 weisshorn02 kernel: Lustre: 47190:0:(mds_lov.c:1008:mds_notify()) MDS mdd_obd-scratch-MDT0000: add target scratch-OST0001_UUID
Jun 19 19:32:22 weisshorn10 kernel: Lustre: scratch-OST0001: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn10 kernel: Lustre: 8441:0:(filter.c:2699:filter_connect_internal()) scratch-OST0001: Received MDS connection for group 0
Jun 19 19:32:22 weisshorn10 kernel: Lustre: 8441:0:(filter.c:2699:filter_connect_internal()) Skipped 4 previous similar messages
Jun 19 19:32:22 weisshorn02 kernel: Lustre: 2871:0:(mds_lov.c:1028:mds_notify()) MDS mdd_obd-scratch-MDT0000: in recovery, not resetting orphans on scratch-OST0001_UUID
Jun 19 19:32:22 weisshorn08 kernel: Lustre: scratch-OST0002: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn08 kernel: Lustre: 8401:0:(filter.c:2699:filter_connect_internal()) scratch-OST0002: Received MDS connection for group 0
Jun 19 19:32:22 weisshorn08 kernel: Lustre: 8401:0:(filter.c:2699:filter_connect_internal()) Skipped 7 previous similar messages
Jun 19 19:32:22 weisshorn13 kernel: Lustre: scratch-OST0003: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn13 kernel: Lustre: 8350:0:(filter.c:2699:filter_connect_internal()) scratch-OST0003: Received MDS connection for group 0
Jun 19 19:32:22 weisshorn13 kernel: Lustre: 8350:0:(filter.c:2699:filter_connect_internal()) Skipped 8 previous similar messages
Jun 19 19:32:22 weisshorn09 kernel: Lustre: scratch-OST0004: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn07 kernel: Lustre: scratch-OST0006: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn07 kernel: Lustre: 8603:0:(filter.c:2699:filter_connect_internal()) scratch-OST0006: Received MDS connection for group 0
Jun 19 19:32:22 weisshorn07 kernel: Lustre: 8603:0:(filter.c:2699:filter_connect_internal()) Skipped 17 previous similar messages
Jun 19 19:32:22 weisshorn06 kernel: Lustre: scratch-OST0007: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn06 kernel: Lustre: 6215:0:(filter.c:2699:filter_connect_internal()) scratch-OST0007: Received MDS connection for group 0
Jun 19 19:32:22 weisshorn06 kernel: Lustre: 6215:0:(filter.c:2699:filter_connect_internal()) Skipped 5 previous similar messages
Jun 19 19:32:22 weisshorn03 kernel: Lustre: scratch-OST0009: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn03 kernel: Lustre: 7798:0:(filter.c:2699:filter_connect_internal()) scratch-OST0009: Received MDS connection for group 0
Jun 19 19:32:22 weisshorn03 kernel: Lustre: 7798:0:(filter.c:2699:filter_connect_internal()) Skipped 5 previous similar messages
Jun 19 19:32:22 weisshorn10 kernel: Lustre: scratch-OST000a: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn13 kernel: Lustre: scratch-OST000b: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn08 kernel: Lustre: scratch-OST000c: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn04 kernel: Lustre: scratch-OST000d: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn04 kernel: Lustre: 8155:0:(filter.c:2699:filter_connect_internal()) scratch-OST000d: Received MDS connection for group 0
Jun 19 19:32:22 weisshorn04 kernel: Lustre: 8155:0:(filter.c:2699:filter_connect_internal()) Skipped 5 previous similar messages
Jun 19 19:32:22 weisshorn05 kernel: Lustre: scratch-OST000e: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn05 kernel: Lustre: 7759:0:(filter.c:2699:filter_connect_internal()) scratch-OST000e: Received MDS connection for group 0
Jun 19 19:32:22 weisshorn05 kernel: Lustre: 7759:0:(filter.c:2699:filter_connect_internal()) Skipped 5 previous similar messages
Jun 19 19:32:22 weisshorn12 kernel: Lustre: scratch-OST000f: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn12 kernel: Lustre: 8394:0:(filter.c:2699:filter_connect_internal()) scratch-OST000f: Received MDS connection for group 0
Jun 19 19:32:22 weisshorn12 kernel: Lustre: 8394:0:(filter.c:2699:filter_connect_internal()) Skipped 5 previous similar messages
Jun 19 19:32:22 weisshorn06 kernel: Lustre: scratch-OST0012: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn07 kernel: Lustre: scratch-OST0013: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn03 kernel: Lustre: scratch-OST0014: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn04 kernel: Lustre: scratch-OST0019: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn05 kernel: Lustre: scratch-OST001b: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn12 kernel: Lustre: scratch-OST001c: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn02 kernel: Lustre: scratch-MDT0000: Temporarily refusing client connection from 148.187.7.6@o2ib2
Jun 19 19:32:22 weisshorn02 kernel: Lustre: scratch-MDT0000: Temporarily refusing client connection from 148.187.7.114@o2ib2
Jun 19 19:32:22 weisshorn14 kernel: Lustre: Evicted from MGS (at 148.187.7.102@o2ib2) after server handle changed from 0x255b49f9c8d65186 to 0xb92b74c12b1c41af
Jun 19 19:32:22 weisshorn14 kernel: LustreError: 11-0: an error occurred while communicating with 148.187.7.102@o2ib2. The mds_connect operation failed with -11
Jun 19 19:32:22 weisshorn14 kernel: Lustre: MGC148.187.7.101@o2ib2: Reactivating import
Jun 19 19:32:22 weisshorn14 kernel: Lustre: MGC148.187.7.101@o2ib2: Connection restored to MGS (at 148.187.7.102@o2ib2)
Jun 19 19:32:22 weisshorn14 kernel: LustreError: Skipped 6 previous similar messages
Jun 19 19:32:22 weisshorn02 kernel: Lustre: 47190:0:(mds_lov.c:1008:mds_notify()) MDS mdd_obd-scratch-MDT0000: add target scratch-OST002d_UUID
Jun 19 19:32:22 weisshorn02 kernel: Lustre: 47190:0:(mds_lov.c:1008:mds_notify()) Skipped 43 previous similar messages
Jun 19 19:32:22 weisshorn02 kernel: Lustre: 2871:0:(mds_lov.c:1028:mds_notify()) MDS mdd_obd-scratch-MDT0000: in recovery, not resetting orphans on scratch-OST002e_UUID
Jun 19 19:32:22 weisshorn02 kernel: Lustre: 2871:0:(mds_lov.c:1028:mds_notify()) Skipped 36 previous similar messages
Jun 19 19:32:22 weisshorn09 kernel: Lustre: scratch-OST0034: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn09 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:32:22 weisshorn10 kernel: Lustre: scratch-OST003a: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn10 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:32:22 weisshorn08 kernel: Lustre: scratch-OST003b: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn08 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:32:22 weisshorn13 kernel: Lustre: scratch-OST003c: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn13 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:32:22 weisshorn06 kernel: Lustre: scratch-OST0041: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn06 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:32:22 weisshorn07 kernel: Lustre: scratch-OST0042: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn07 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:32:22 weisshorn03 kernel: Lustre: scratch-OST0043: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:22 weisshorn03 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:32:22 weisshorn02 kernel: Lustre: Increasing default stripe size to min 1048576
Jun 19 19:32:22 weisshorn02 Filesystem[47075]: INFO:  Success
Jun 19 19:32:22 weisshorn02 mach_down[46949]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
Jun 19 19:32:23 weisshorn02 mach_down[46949]: info: mach_down takeover complete for node weisshorn01.admin.cscs.ch.
Jun 19 19:32:23 weisshorn02 heartbeat: [3934]: info: mach_down takeover complete.
Jun 19 19:32:23 weisshorn02 kernel: Lustre: scratch-MDT0000: Will be in recovery for at least 5:00, or until 1547 clients reconnect
Jun 19 19:32:23 weisshorn02 kernel: Lustre: 47204:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 52555669079, current: 52024346219, replaying
Jun 19 19:32:23 weisshorn02 nslcd[2099]: [d484d5] ldap_result() failed: Can't contact LDAP server
Jun 19 19:32:23 weisshorn02 nslcd[2099]: [d06255] ldap_result() failed: Can't contact LDAP server
Jun 19 19:32:23 weisshorn02 kernel: LustreError: 47204:0:(mdt_open.c:1293:mdt_reint_open()) @@@ OPEN & CREAT not in open replay.  req@ffff8807f16a3850 x1403838177293726/t0(52024346219) o101->ab7f3497-08b2-1142-1bdd-d3641fdb7ac9@148.187.7.14@o2ib2:0/0 lens 2248/2808 e 0 to 0 dl 1340127149 ref 1 fl Interpret:/4/0 rc 0/0
Jun 19 19:32:23 weisshorn02 kernel: Lustre: 47204:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 52555669079, current: 52024346242, replaying
Jun 19 19:32:23 weisshorn02 kernel: LustreError: 47204:0:(mdt_open.c:1293:mdt_reint_open()) @@@ OPEN & CREAT not in open replay.  req@ffff8807f16a3050 x1403838177293728/t0(52024346242) o101->ab7f3497-08b2-1142-1bdd-d3641fdb7ac9@148.187.7.14@o2ib2:0/0 lens 2248/2808 e 0 to 0 dl 1340127149 ref 1 fl Interpret:/4/0 rc 0/0
Jun 19 19:32:23 weisshorn02 kernel: Lustre: 27709:0:(mgc_request.c:1520:mgc_process_recover_log()) Process recover log scratch-mdtir error -22
Jun 19 19:32:23 weisshorn02 kernel: LustreError: 27709:0:(client.c:1042:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff880c1a977800 x1403849754836944/t0(0) o101->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/0/ffffffff rc 0/-1
Jun 19 19:32:23 weisshorn12 kernel: Lustre: scratch-OST0040: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:23 weisshorn12 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 19:32:24 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127119/real 1340127144]  req@ffff881013757c00 x1403849984281796/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127145 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:32:25 weisshorn02 kernel: Lustre: 47204:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 52555669079, current: 51958866633, replaying
Jun 19 19:32:25 weisshorn02 kernel: Lustre: 47204:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Skipped 46 previous similar messages
Jun 19 19:32:25 weisshorn02 nslcd[2099]: [194ed1] ldap_result() failed: Can't contact LDAP server
Jun 19 19:32:25 weisshorn02 nslcd[2099]: [e2ed6c] ldap_result() failed: Can't contact LDAP server
Jun 19 19:32:25 weisshorn02 nslcd[2099]: [219b33] ldap_result() failed: Can't contact LDAP server
Jun 19 19:32:25 weisshorn02 kernel: LustreError: 47204:0:(mdt_open.c:1293:mdt_reint_open()) @@@ OPEN & CREAT not in open replay.  req@ffff8807f1b5c050 x1403838175658915/t0(52024326098) o101->0e3cd6dc-ddad-3fec-c2bb-679d5749c919@148.187.7.9@o2ib2:0/0 lens 1096/1656 e 0 to 0 dl 1340127187 ref 1 fl Interpret:/4/0 rc 0/0
Jun 19 19:32:25 weisshorn02 kernel: LustreError: 47204:0:(mdt_open.c:1293:mdt_reint_open()) Skipped 22 previous similar messages
Jun 19 19:32:26 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127121/real 1340127146]  req@ffff88100f542c00 x1403849636377448/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127147 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:32:26 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127121/real 1340127146]  req@ffff880bf151b000 x1403849897065193/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127147 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:32:28 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127123/real 1340127148]  req@ffff880e63fdbc00 x1403849873211657/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127149 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:32:28 weisshorn02 kernel: Lustre: 47204:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 52555669079, current: 51733605469, replaying
Jun 19 19:32:28 weisshorn02 kernel: Lustre: 47204:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Skipped 48 previous similar messages
Jun 19 19:32:28 weisshorn12 kernel: LustreError: 126320:0:(obd_mount.c:1109:server_register_target()) Cannot talk to the MGS: -5, not fatal
Jun 19 19:32:29 weisshorn12 kernel: Lustre: 6041:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 42149540, current: 42149548, replaying
Jun 19 19:32:29 weisshorn12 kernel: Lustre: 6041:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 42149540, current: 42149546, replaying
Jun 19 19:32:30 weisshorn02 kernel: Lustre: 47203:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 52555669079, current: 52362547690, replaying
Jun 19 19:32:30 weisshorn02 kernel: Lustre: 47203:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Skipped 37 previous similar messages
Jun 19 19:32:31 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127126/real 1340127151]  req@ffff880d1c82a800 x1403849822727065/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127152 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:32:32 weisshorn02 kernel: LustreError: 27709:0:(client.c:1042:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff880c1d43a800 x1403849754836945/t0(0) o101->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/0/ffffffff rc 0/-1
Jun 19 19:32:34 weisshorn12 kernel: Lustre: 126392:0:(ldlm_lib.c:2062:target_recovery_init()) RECOVERY: service scratch-OST0021, 1548 recoverable clients, last_transno 43816983
Jun 19 19:32:34 weisshorn12 kernel: Lustre: scratch-OST0021: Now serving scratch-OST0021 on /dev/mapper/Lun04 with recovery enabled
Jun 19 19:32:35 weisshorn12 kernel: Lustre: scratch-OST0008: Denying connection for new client 148.187.6.6@o2ib2 (at 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03), waiting for 13 clients in recovery for 2:28
Jun 19 19:32:35 weisshorn12 kernel: Lustre: scratch-OST0021: Temporarily refusing client connection from 1198@gni
Jun 19 19:32:35 weisshorn12 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:32:35 weisshorn02 kernel: Lustre: 47204:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 52555669079, current: 51740533190, replaying
Jun 19 19:32:35 weisshorn02 kernel: Lustre: 47204:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Skipped 3 previous similar messages
Jun 19 19:32:35 weisshorn12 kernel: Lustre: 6041:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 43053860, current: 43053865, replaying
Jun 19 19:32:35 weisshorn12 kernel: Lustre: 6041:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Skipped 9 previous similar messages
Jun 19 19:32:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:32:37 weisshorn12 kernel: Lustre: 6041:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 42149540, current: 42149541, replaying
Jun 19 19:32:39 weisshorn12 kernel: Lustre: scratch-OST0021: Temporarily refusing client connection from 429@gni
Jun 19 19:32:39 weisshorn12 kernel: Lustre: Skipped 1293 previous similar messages
Jun 19 19:32:41 weisshorn12 kernel: Lustre: 6041:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 43053860, current: 43053866, replaying
Jun 19 19:32:41 weisshorn12 kernel: Lustre: 6041:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Skipped 4 previous similar messages
Jun 19 19:32:42 weisshorn12 kernel: LustreError: 5874:0:(client.c:1031:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88080d66ac00 x1403849952681510/t0(0) o101->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jun 19 19:32:42 weisshorn12 kernel: LustreError: 5874:0:(client.c:1031:ptlrpc_import_delay_req()) Skipped 7 previous similar messages
Jun 19 19:32:43 weisshorn02 kernel: Lustre: 47345:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 52555669079, current: 52024326099, replaying
Jun 19 19:32:43 weisshorn02 kernel: Lustre: 47345:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Skipped 59 previous similar messages
Jun 19 19:32:43 weisshorn02 kernel: LustreError: 47345:0:(mdt_open.c:1293:mdt_reint_open()) @@@ OPEN & CREAT not in open replay.  req@ffff8807e4885050 x1403838172803597/t0(52024326099) o101->21a5b6e3-6deb-efbe-614b-a050343fb302@148.187.7.10@o2ib2:0/0 lens 1096/1656 e 0 to 0 dl 1340127205 ref 1 fl Interpret:/4/0 rc 0/0
Jun 19 19:32:43 weisshorn02 kernel: LustreError: 47345:0:(mdt_open.c:1293:mdt_reint_open()) Skipped 23 previous similar messages
Jun 19 19:32:44 weisshorn12 Filesystem[126255]: INFO:  Success
Jun 19 19:32:45 weisshorn12 Filesystem[126412]: INFO:  Resource is stopped
Jun 19 19:32:45 weisshorn12 ResourceManager[125658]: info: Running /etc/ha.d/resource.d/Filesystem /dev/mapper/Lun06 /mnt/Lun06 lustre start
Jun 19 19:32:45 weisshorn12 kernel: Lustre: scratch-OST0021: Will be in recovery for at least 5:00, or until 1548 clients reconnect
Jun 19 19:32:45 weisshorn12 Filesystem[126491]: INFO: Running start for /dev/mapper/Lun06 on /mnt/Lun06
Jun 19 19:32:46 weisshorn02 kernel: LustreError: 47335:0:(mdt_open.c:1293:mdt_reint_open()) @@@ OPEN & CREAT not in open replay.  req@ffff8807e3dc5850 x1403843625538888/t0(52016243316) o101->b20801aa-1a74-3e8a-84a4-a422add23b44@148.187.6.210@o2ib2:0/0 lens 1096/1656 e 0 to 0 dl 1340127208 ref 1 fl Interpret:/4/0 rc 0/0
Jun 19 19:32:46 weisshorn02 kernel: LustreError: 47335:0:(mdt_open.c:1293:mdt_reint_open()) Skipped 71 previous similar messages
Jun 19 19:32:46 weisshorn12 kernel: LDISKFS-fs warning (device dm-7): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jun 19 19:32:46 weisshorn12 kernel:
Jun 19 19:32:47 weisshorn13 kernel: Lustre: scratch-OST0005: Received MDS connection from 148.187.7.102@o2ib2, removing former export from 148.187.7.101@o2ib2
Jun 19 19:32:47 weisshorn02 kernel: Lustre: Evicted from MGS (at 148.187.7.102@o2ib2) after server handle changed from 0x255b49f9c8d3b4e3 to 0xb92b74c12b1c561c
Jun 19 19:32:47 weisshorn02 kernel: Lustre: MGC148.187.7.101@o2ib2: Reactivating import
Jun 19 19:32:47 weisshorn02 kernel: Lustre: MGC148.187.7.101@o2ib2: Connection restored to MGS (at 0@lo)
Jun 19 19:32:47 weisshorn02 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 19:32:47 weisshorn02 kernel: Lustre: 2871:0:(mds_lov.c:1028:mds_notify()) MDS mdd_obd-scratch-MDT0000: in recovery, not resetting orphans on scratch-OST0005_UUID
Jun 19 19:32:47 weisshorn02 kernel: Lustre: 2871:0:(mds_lov.c:1028:mds_notify()) Skipped 21 previous similar messages
Jun 19 19:32:47 weisshorn02 kernel: LustreError: 11-0: an error occurred while communicating with 148.187.7.112@o2ib2. The ost_connect operation failed with -19
Jun 19 19:32:47 weisshorn02 kernel: LustreError: Skipped 3 previous similar messages
Jun 19 19:32:50 weisshorn12 kernel: Lustre: 6041:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 43053864, current: 43053871, replaying
Jun 19 19:32:50 weisshorn12 kernel: Lustre: 6041:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Skipped 5 previous similar messages
Jun 19 19:32:50 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127139/real 0]  req@ffff880805e71000 x1403849943172819/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127170 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:32:50 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:32:51 weisshorn14 kernel: Lustre: scratch-OST0008-osc-ffff8810296e4c00: Connection restored to scratch-OST0008 (at 148.187.7.112@o2ib2)
Jun 19 19:32:51 weisshorn02 kernel: Lustre: scratch-OST0008-osc-ffff88081e6aac00: Connection restored to scratch-OST0008 (at 148.187.7.112@o2ib2)
Jun 19 19:32:51 weisshorn02 kernel: Lustre: 2871:0:(mds_lov.c:1028:mds_notify()) MDS mdd_obd-scratch-MDT0000: in recovery, not resetting orphans on scratch-OST0008_UUID
Jun 19 19:32:51 weisshorn02 kernel: Lustre: 2871:0:(mds_lov.c:1028:mds_notify()) Skipped 5 previous similar messages
Jun 19 19:32:51 weisshorn12 kernel: Lustre: scratch-OST0008: Recovery over after 2:48, of 1547 clients 1547 recovered and 0 were evicted.
Jun 19 19:32:51 weisshorn12 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 19:32:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:33:00 weisshorn12 kernel: Lustre: scratch-OST0021: Denying connection for new client 148.187.6.7@o2ib2 (at 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7), waiting for 0 clients in recovery for 4:45
Jun 19 19:33:03 weisshorn02 kernel: Lustre: 2871:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127167/real 0]  req@ffff880412376000 x1403849754836947/t0(0) o38->scratch-MDT0000-mdc-ffff88081e6aac00@148.187.7.101@o2ib2:12/10 lens 368/512 e 0 to 1 dl 1340127183 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:33:03 weisshorn02 kernel: Lustre: 2871:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 30 previous similar messages
Jun 19 19:33:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:33:08 weisshorn02 kernel: Lustre: scratch-MDT0000: Denying connection for new client 148.187.6.7@o2ib2 (at 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7), waiting for 12 clients in recovery for 4:14
Jun 19 19:33:11 weisshorn12 kernel: LustreError: 137-5: scratch-OST0044: Not available for connect from 148.187.6.6@o2ib2 (no target)
Jun 19 19:33:11 weisshorn12 kernel: LustreError: Skipped 5072 previous similar messages
Jun 19 19:33:11 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127160/real 0]  req@ffff8810102be000 x1403849871423823/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127191 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:33:12 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127161/real 0]  req@ffff880d0f287400 x1403849861583696/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127192 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:33:13 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127162/real 0]  req@ffff8809f3e78400 x1403849921463314/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127193 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:33:14 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127163/real 0]  req@ffff88026cb3dc00 x1403849952681513/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127194 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:33:18 weisshorn14 kernel: Lustre: 14313:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127167/real 0]  req@ffff880fff194000 x1404749288341412/t0(0) o8->scratch-OST0039-osc-ffff8810296e4c00@148.187.7.111@o2ib2:28/4 lens 368/512 e 0 to 1 dl 1340127198 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:33:18 weisshorn14 kernel: Lustre: 14313:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 17 previous similar messages
Jun 19 19:33:20 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127169/real 0]  req@ffff880eb1708400 x1403849984281797/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127200 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:33:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:33:22 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127171/real 0]  req@ffff88100d601400 x1403849636377608/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127202 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:33:22 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127171/real 0]  req@ffff880bf60b2800 x1403849897065219/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127202 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:33:24 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127173/real 0]  req@ffff881018160000 x1403849873211775/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127204 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:33:25 weisshorn12 kernel: Lustre: scratch-OST0021: Denying connection for new client 148.187.6.7@o2ib2 (at 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7), waiting for 2 clients in recovery for 4:20
Jun 19 19:33:27 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127176/real 0]  req@ffff880a712ed400 x1403849822727308/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127207 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:33:28 weisshorn02 kernel: LustreError: 11-0: an error occurred while communicating with 148.187.7.112@o2ib2. The ost_connect operation failed with -19
Jun 19 19:33:28 weisshorn02 kernel: LustreError: Skipped 2 previous similar messages
Jun 19 19:33:32 weisshorn12 kernel: LDISKFS-fs (dm-7): recovery complete
Jun 19 19:33:32 weisshorn12 kernel: LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts: 
Jun 19 19:33:33 weisshorn12 kernel: LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts: 
Jun 19 19:33:33 weisshorn12 kernel: LustreError: 126548:0:(ldlm_resource.c:759:ldlm_resource_complain()) Namespace MGC148.187.7.101@o2ib2 resource refcount nonzero (1) after lock cleanup; forcing cleanup.
Jun 19 19:33:33 weisshorn12 kernel: LustreError: 5874:0:(client.c:1042:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff880809f16c00 x1403849952681911/t0(0) o101->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/0/ffffffff rc 0/-1
Jun 19 19:33:33 weisshorn12 kernel: LustreError: 5874:0:(client.c:1042:ptlrpc_import_delay_req()) Skipped 5 previous similar messages
Jun 19 19:33:33 weisshorn12 kernel: LustreError: 126548:0:(ldlm_resource.c:765:ldlm_resource_complain()) Resource: ffff88025d3d16c0 (29382749080150899/0/0/0) (rc: 0)
Jun 19 19:33:33 weisshorn12 kernel: Lustre: MGC148.187.7.101@o2ib2: Reactivating import
Jun 19 19:33:33 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127213/real 1340127213]  req@ffff880369ecd800 x1403849952681913/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127249 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:33:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:33:41 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127221/real 1340127221]  req@ffff88101249a800 x1403849897065252/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127257 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:33:43 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127223/real 1340127223]  req@ffff880e11af7800 x1403849873211780/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127259 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:33:44 weisshorn12 kernel: LustreError: 126548:0:(obd_mount.c:1109:server_register_target()) Cannot talk to the MGS: -5, not fatal
Jun 19 19:33:45 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127189/real 0]  req@ffff88073ea88800 x1403849943173020/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127225 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:33:46 weisshorn12 kernel: LustreError: 5874:0:(client.c:1031:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff88011f5a0c00 x1403849952681915/t0(0) o101->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jun 19 19:33:46 weisshorn12 kernel: LustreError: 5874:0:(client.c:1031:ptlrpc_import_delay_req()) Skipped 9 previous similar messages
Jun 19 19:33:47 weisshorn13 kernel: Lustre: scratch-OST0005: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 208 seconds. I think it's dead, and I am evicting it. exp ffff880b84e17800, cur 1340127227 expire 1340127077 last 1340127019
Jun 19 19:33:47 weisshorn13 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 19:33:50 weisshorn02 kernel: Lustre: scratch-MDT0000: Denying connection for new client 148.187.6.7@o2ib2 (at 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7), waiting for 12 clients in recovery for 3:32
Jun 19 19:33:50 weisshorn12 kernel: Lustre: scratch-OST0021: Denying connection for new client 148.187.6.7@o2ib2 (at 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7), waiting for 8 clients in recovery for 3:55
Jun 19 19:33:50 weisshorn12 kernel: Lustre: 126625:0:(ldlm_lib.c:2062:target_recovery_init()) RECOVERY: service scratch-OST002d, 1547 recoverable clients, last_transno 43665873
Jun 19 19:33:51 weisshorn12 kernel: Lustre: scratch-OST002d: Now serving scratch-OST002d on /dev/mapper/Lun06 with recovery enabled
Jun 19 19:33:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:33:52 weisshorn12 kernel: Lustre: scratch-OST002d: Temporarily refusing client connection from 148.187.6.201@o2ib2
Jun 19 19:33:52 weisshorn12 kernel: Lustre: Skipped 20 previous similar messages
Jun 19 19:34:02 weisshorn12 Filesystem[126483]: INFO:  Success
Jun 19 19:34:02 weisshorn12 Filesystem[126645]: INFO:  Resource is stopped
Jun 19 19:34:02 weisshorn12 ResourceManager[125658]: info: Running /etc/ha.d/resource.d/Filesystem /dev/mapper/Lun08 /mnt/Lun08 lustre start
Jun 19 19:34:02 weisshorn12 Filesystem[126724]: INFO: Running start for /dev/mapper/Lun08 on /mnt/Lun08
Jun 19 19:34:02 weisshorn12 kernel: LDISKFS-fs warning (device dm-8): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jun 19 19:34:02 weisshorn12 kernel:
Jun 19 19:34:02 weisshorn12 kernel: Lustre: scratch-OST002d: Will be in recovery for at least 5:00, or until 1547 clients reconnect
Jun 19 19:34:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:34:06 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127210/real 0]  req@ffff88100de13800 x1403849871423941/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127246 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:34:07 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127211/real 0]  req@ffff880863474400 x1403849861583756/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127247 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:34:08 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127212/real 0]  req@ffff8810167fb400 x1403849921463315/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127248 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:34:15 weisshorn02 kernel: Lustre: scratch-MDT0000: Denying connection for new client 148.187.6.7@o2ib2 (at 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7), waiting for 12 clients in recovery for 3:07
Jun 19 19:34:17 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127221/real 0]  req@ffff88100dd32400 x1403849636377717/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127257 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:34:18 weisshorn12 kernel: Lustre: 8218:0:(filter.c:2699:filter_connect_internal()) scratch-OST002d: Received MDS connection for group 0
Jun 19 19:34:18 weisshorn12 kernel: Lustre: 8218:0:(filter.c:2699:filter_connect_internal()) Skipped 8 previous similar messages
Jun 19 19:34:18 weisshorn02 kernel: LustreError: 11-0: an error occurred while communicating with 148.187.7.112@o2ib2. The ost_connect operation failed with -19
Jun 19 19:34:18 weisshorn02 kernel: LustreError: Skipped 2 previous similar messages
Jun 19 19:34:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:34:22 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127226/real 0]  req@ffff880e1d187400 x1403849822727309/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127262 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:34:31 weisshorn04 kernel: Lustre: scratch-OST0025: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804c9c68000, cur 1340127271 expire 1340127121 last 1340127044
Jun 19 19:34:31 weisshorn04 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 19:34:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:34:38 weisshorn12 kernel: Lustre: 6112:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 43816984, current: 43816998, replaying
Jun 19 19:34:39 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127238/real 0]  req@ffff880789980000 x1403849952681921/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127279 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:34:39 weisshorn02 kernel: Lustre: 47330:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 52555669079, current: 52554536767, replaying
Jun 19 19:34:39 weisshorn02 kernel: Lustre: 47330:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Skipped 280 previous similar messages
Jun 19 19:34:40 weisshorn02 kernel: Lustre: scratch-MDT0000: Denying connection for new client 148.187.6.7@o2ib2 (at 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7), waiting for 12 clients in recovery for 2:42
Jun 19 19:34:40 weisshorn12 kernel: Lustre: scratch-OST0021: Denying connection for new client 148.187.6.7@o2ib2 (at 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7), waiting for 434 clients in recovery for 3:05
Jun 19 19:34:40 weisshorn12 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:34:40 weisshorn02 kernel: LustreError: 47204:0:(mdt_open.c:1293:mdt_reint_open()) @@@ OPEN & CREAT not in open replay.  req@ffff8807df808850 x1403846432659357/t0(52472594894) o101->e4afd43d-a6ec-05ca-628d-52e5a5951b41@1292@gni:0/0 lens 2048/1656 e 0 to 0 dl 1340127372 ref 1 fl Interpret:/4/0 rc 0/0
Jun 19 19:34:40 weisshorn02 kernel: LustreError: 47204:0:(mdt_open.c:1293:mdt_reint_open()) Skipped 10 previous similar messages
Jun 19 19:34:47 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127246/real 0]  req@ffff881012308000 x1403849897065253/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127287 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:34:49 weisshorn12 kernel: LDISKFS-fs (dm-8): recovery complete
Jun 19 19:34:49 weisshorn12 kernel: LDISKFS-fs (dm-8): mounted filesystem with ordered data mode. Opts: 
Jun 19 19:34:49 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127248/real 0]  req@ffff8809391af800 x1403849873211781/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127289 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:34:49 weisshorn12 kernel: LDISKFS-fs (dm-8): mounted filesystem with ordered data mode. Opts: 
Jun 19 19:34:49 weisshorn12 kernel: Lustre: MGC148.187.7.101@o2ib2: Reactivating import
Jun 19 19:34:50 weisshorn02 kernel: LustreError: 47467:0:(mdt_open.c:1293:mdt_reint_open()) @@@ OPEN & CREAT not in open replay.  req@ffff8807e37a7850 x1403846436213727/t0(52472524348) o101->83c2ed0a-e514-2f83-8ad8-b03b850d683b@1335@gni:0/0 lens 2048/1656 e 0 to 0 dl 1340127373 ref 1 fl Interpret:/4/0 rc 0/0
Jun 19 19:34:50 weisshorn02 kernel: LustreError: 47467:0:(mdt_open.c:1293:mdt_reint_open()) Skipped 153 previous similar messages
Jun 19 19:34:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:34:52 weisshorn14 kernel: LustreError: 11-0: an error occurred while communicating with 148.187.7.112@o2ib2. The ost_connect operation failed with -19
Jun 19 19:34:52 weisshorn14 kernel: LustreError: Skipped 5 previous similar messages
Jun 19 19:34:54 weisshorn02 kernel: Lustre: 47380:0:(service.c:1563:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 7s  req@ffff88079524f050 x1403846436035468/t0(0) o101->1dc70294-bdee-0bba-99d4-6e362d21de01@680@gni:0/0 lens 296/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1
Jun 19 19:34:54 weisshorn02 kernel: Lustre: 47380:0:(service.c:1563:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 7s  req@ffff8803d8939050 x1403846437474499/t0(0) o101->1c7446d4-d2e6-7539-5dad-cdbb5b19e64f@1497@gni:0/0 lens 296/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1
Jun 19 19:34:54 weisshorn02 kernel: Lustre: 47380:0:(service.c:1563:ptlrpc_server_handle_req_in()) Skipped 54 previous similar messages
Jun 19 19:35:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:35:09 weisshorn02 kernel: Lustre: 47399:0:(service.c:1563:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 15s  req@ffff880ff02d3850 x1403846437442487/t0(0) o101->e2ebefa1-13c3-f5a1-4b16-e474a306c3e8@1490@gni:0/0 lens 296/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1
Jun 19 19:35:09 weisshorn02 kernel: Lustre: 47399:0:(service.c:1563:ptlrpc_server_handle_req_in()) Skipped 1960 previous similar messages
Jun 19 19:35:09 weisshorn02 kernel: Lustre: mdt: This server is not able to keep up with request traffic (cpu-bound).
Jun 19 19:35:09 weisshorn02 kernel: Lustre: 47399:0:(service.c:1199:ptlrpc_at_check_timed()) earlyQ=57 reqQ=6787 recA=0, svcEst=69, delay=13461(jiff)
Jun 19 19:35:09 weisshorn02 kernel: Lustre: 47399:0:(service.c:996:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)?  req@ffff8807e3ab5050 x1403843719462156/t0(0) o101->910dea22-9def-d2fb-5cd8-2ccd284230ba@148.187.6.160@o2ib2:0/0 lens 296/0 e 4 to 0 dl 1340127308 ref 2 fl Complete:/40/ffffffff rc 0/-1
Jun 19 19:35:09 weisshorn02 kernel: LustreError: 11-0: an error occurred while communicating with 148.187.7.112@o2ib2. The ost_connect operation failed with -19
Jun 19 19:35:09 weisshorn02 kernel: LustreError: Skipped 3 previous similar messages
Jun 19 19:35:09 weisshorn02 kernel: LustreError: 47317:0:(mdt_open.c:1293:mdt_reint_open()) @@@ OPEN & CREAT not in open replay.  req@ffff88100dcc0050 x1403846435750188/t0(52472524287) o101->9b07e84a-efa3-e49c-e618-d424d0898c6b@1289@gni:0/0 lens 2048/1656 e 0 to 0 dl 1340127373 ref 1 fl Interpret:/4/0 rc 0/0
Jun 19 19:35:10 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127269/real 0]  req@ffff880a51ccac00 x1403849984282309/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127310 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:35:10 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:35:17 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127276/real 0]  req@ffff880e297c4000 x1403849822727310/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127317 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:35:18 weisshorn07 kernel: Lustre: scratch-OST0042: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 163 seconds. I think it's dead, and I am evicting it. exp ffff8801aadae800, cur 1340127318 expire 1340127168 last 1340127155
Jun 19 19:35:18 weisshorn07 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:35:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:35:21 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bd301ac00, cur 1340127321 expire 1340127171 last 1340127094
Jun 19 19:35:21 weisshorn08 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:35:31 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127285/real 0]  req@ffff88102c0f1000 x1403849871423943/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127331 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:35:31 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:35:32 weisshorn12 kernel: Lustre: scratch-OST0040: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88061eb55400, cur 1340127332 expire 1340127182 last 1340127105
Jun 19 19:35:34 weisshorn02 kernel: Lustre: 47465:0:(service.c:1563:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 25s  req@ffff880fe3626050 x1403846441899245/t0(0) o101->a8e5122f-f7eb-1e43-bc1f-6bfd1088ee4b@494@gni:0/0 lens 296/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1
Jun 19 19:35:34 weisshorn02 kernel: Lustre: 47465:0:(service.c:1563:ptlrpc_server_handle_req_in()) Skipped 5757 previous similar messages
Jun 19 19:35:35 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127289/real 0]  req@ffff880805e10c00 x1403849943173022/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127335 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:35:35 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:35:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:35:49 weisshorn02 kernel: Lustre: 47466:0:(service.c:1563:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 14s  req@ffff880bd4127050 x1403846442489691/t0(0) o101->fb875ba0-9a12-0383-00fd-96cd217a39e9@235@gni:0/0 lens 296/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1
Jun 19 19:35:49 weisshorn02 kernel: Lustre: 47466:0:(service.c:1563:ptlrpc_server_handle_req_in()) Skipped 6358 previous similar messages
Jun 19 19:35:49 weisshorn02 kernel: Lustre: 47400:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 52555669079, current: 52555665344, replaying
Jun 19 19:35:49 weisshorn02 kernel: Lustre: 47400:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Skipped 4074 previous similar messages
Jun 19 19:35:49 weisshorn12 kernel: LustreError: 126781:0:(obd_mount.c:1109:server_register_target()) Cannot talk to the MGS: -5, not fatal
Jun 19 19:35:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:35:55 weisshorn12 kernel: Lustre: scratch-OST0021: Denying connection for new client 148.187.6.7@o2ib2 (at 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7), waiting for 16 clients in recovery for 1:50
Jun 19 19:35:55 weisshorn12 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 19:35:55 weisshorn12 kernel: LustreError: 126781:0:(client.c:1031:ptlrpc_import_delay_req()) @@@ send limit expired   req@ffff8802c0a8d800 x1403849952681941/t0(0) o101->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Jun 19 19:35:55 weisshorn12 kernel: LustreError: 126781:0:(client.c:1031:ptlrpc_import_delay_req()) Skipped 21 previous similar messages
Jun 19 19:35:55 weisshorn12 kernel: Lustre: 126853:0:(ldlm_lib.c:2062:target_recovery_init()) RECOVERY: service scratch-OST0039, 1548 recoverable clients, last_transno 43006730
Jun 19 19:35:56 weisshorn12 kernel: Lustre: scratch-OST0039: Now serving scratch-OST0039 on /dev/mapper/Lun08 with recovery enabled
Jun 19 19:35:57 weisshorn05 kernel: Lustre: scratch-OST000e: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100ed38000, cur 1340127357 expire 1340127207 last 1340127130
Jun 19 19:35:57 weisshorn05 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 19:35:57 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127311/real 0]  req@ffff88101a27d400 x1403849861583758/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127357 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:35:57 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:35:58 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127312/real 0]  req@ffff8808a927ec00 x1403849921463317/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127358 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:35:58 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:35:58 weisshorn12 kernel: Lustre: scratch-OST0039: Temporarily refusing client connection from 148.187.6.8@o2ib2
Jun 19 19:35:58 weisshorn12 kernel: Lustre: Skipped 14 previous similar messages
Jun 19 19:36:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:36:07 weisshorn12 Filesystem[126716]: INFO:  Success
Jun 19 19:36:07 weisshorn12 Filesystem[126910]: INFO:  Resource is stopped
Jun 19 19:36:07 weisshorn12 ResourceManager[125658]: info: Running /etc/ha.d/resource.d/Filesystem /dev/mapper/Lun10 /mnt/Lun10 lustre start
Jun 19 19:36:07 weisshorn12 Filesystem[126989]: INFO: Running start for /dev/mapper/Lun10 on /mnt/Lun10
Jun 19 19:36:07 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127321/real 0]  req@ffff8809ef5d2800 x1403849636377719/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127367 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:36:07 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:36:07 weisshorn12 kernel: LDISKFS-fs warning (device dm-9): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jun 19 19:36:07 weisshorn12 kernel:
Jun 19 19:36:07 weisshorn12 kernel: Lustre: scratch-OST0039: Will be in recovery for at least 5:00, or until 1548 clients reconnect
Jun 19 19:36:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:36:24 weisshorn12 kernel: Lustre: 8343:0:(filter.c:2699:filter_connect_internal()) scratch-OST0039: Received MDS connection for group 0
Jun 19 19:36:25 weisshorn12 kernel: Lustre: 6112:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 43665874, current: 43665875, replaying
Jun 19 19:36:25 weisshorn12 kernel: Lustre: 6112:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Skipped 7 previous similar messages
Jun 19 19:36:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:36:36 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127346/real 1340127396]  req@ffff8810142a3c00 x1403849897065255/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127397 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:36:36 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:36:38 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127348/real 1340127398]  req@ffff881027e8fc00 x1403849873211783/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127399 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:36:38 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:36:41 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127351/real 1340127401]  req@ffff880dd722e400 x1403849822727312/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127402 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:36:41 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:36:47 weisshorn02 kernel: Lustre: 47420:0:(service.c:1563:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 58s  req@ffff8803ae419050 x1403846435836261/t0(0) o101->bee5ad94-9116-26d0-ae46-02f606feec04@920@gni:0/0 lens 296/0 e 0 to 0 dl 0 ref 1 fl New:/0/ffffffff rc 0/-1
Jun 19 19:36:47 weisshorn02 kernel: Lustre: 47420:0:(service.c:1563:ptlrpc_server_handle_req_in()) Skipped 2975 previous similar messages
Jun 19 19:36:47 weisshorn02 kernel: Lustre: mdt: This server is not able to keep up with request traffic (cpu-bound).
Jun 19 19:36:47 weisshorn02 kernel: Lustre: 47420:0:(service.c:1199:ptlrpc_at_check_timed()) earlyQ=28238 reqQ=9811 recA=0, svcEst=69, delay=56161(jiff)
Jun 19 19:36:47 weisshorn02 kernel: Lustre: 47420:0:(service.c:996:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)?  req@ffff880bdc732050 x1403846441900541/t0(0) o101->a8e5122f-f7eb-1e43-bc1f-6bfd1088ee4b@494@gni:0/0 lens 296/0 e 0 to 0 dl 1340127406 ref 2 fl New:/0/ffffffff rc 0/-1
Jun 19 19:36:47 weisshorn02 kernel: Lustre: 47420:0:(service.c:996:ptlrpc_at_send_early_reply()) Skipped 55 previous similar messages
Jun 19 19:36:47 weisshorn02 kernel: LustreError: 47418:0:(service.c:1704:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-644@gni: deadline 100:25s ago
Jun 19 19:36:47 weisshorn02 kernel:  req@ffff8803eba36850 x1403846437833771/t0(0) o38-><?>@<?>:0/0 lens 368/0 e 0 to 0 dl 1340127382 ref 1 fl Interpret:/0/ffffffff rc 0/-1
Jun 19 19:36:47 weisshorn02 kernel: Lustre: 47418:0:(service.c:1739:ptlrpc_server_handle_request()) @@@ Request x1403846437833771 took longer than estimated (100:25s); client may timeout.  req@ffff8803eba36850 x1403846437833771/t0(0) o38-><?>@<?>:0/0 lens 368/0 e 0 to 0 dl 1340127382 ref 1 fl Interpret:/0/ffffffff rc 0/-1
Jun 19 19:36:47 weisshorn02 kernel: LustreError: 47418:0:(service.c:1704:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-186@gni: deadline 92:33s ago
Jun 19 19:36:47 weisshorn02 kernel:  req@ffff8807a7367850 x1403846438351913/t0(0) o101->0c2b61b4-e1b4-4853-b43f-39d6c0d6e847@186@gni:0/0 lens 296/0 e 0 to 0 dl 1340127374 ref 2 fl Interpret:/0/ffffffff rc 0/-1
Jun 19 19:36:47 weisshorn02 kernel: Lustre: 47418:0:(service.c:1739:ptlrpc_server_handle_request()) @@@ Request x1403846438351913 took longer than estimated (92:33s); client may timeout.  req@ffff8807a7367850 x1403846438351913/t0(0) o101->0c2b61b4-e1b4-4853-b43f-39d6c0d6e847@186@gni:0/0 lens 296/0 e 0 to 0 dl 1340127374 ref 2 fl Interpret:/0/ffffffff rc 0/-1
Jun 19 19:36:47 weisshorn02 kernel: Lustre: scratch-MDT0000: Client eb4e6430-8b9a-0c5f-2468-03a6547b065e (at 148.187.6.181@o2ib2) reconnecting, waiting for 1547 clients in recovery for 0:35
Jun 19 19:36:47 weisshorn02 kernel: Lustre: scratch-MDT0000: Client eb4e6430-8b9a-0c5f-2468-03a6547b065e (at 148.187.6.181@o2ib2) refused reconnection, still busy with 1 active RPCs
Jun 19 19:36:47 weisshorn02 kernel: Lustre: scratch-MDT0000: Client 8fc01ddf-0133-fca5-60a0-a7dafc75d7a2 (at 148.187.6.221@o2ib2) reconnecting, waiting for 1547 clients in recovery for 0:35
Jun 19 19:36:47 weisshorn02 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:36:47 weisshorn02 kernel: Lustre: scratch-MDT0000: Client 8fc01ddf-0133-fca5-60a0-a7dafc75d7a2 (at 148.187.6.221@o2ib2) refused reconnection, still busy with 1 active RPCs
Jun 19 19:36:47 weisshorn02 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:36:47 weisshorn02 kernel: LustreError: 47420:0:(mdt_open.c:1293:mdt_reint_open()) @@@ OPEN & CREAT not in open replay.  req@ffff8803b5940050 x1403846442198990/t0(52472524296) o101->9b2bb080-5619-1e0e-6c9e-8d1099aee1c4@308@gni:0/0 lens 2048/1656 e 0 to 0 dl 1340127426 ref 1 fl Interpret:/4/0 rc 0/0
Jun 19 19:36:48 weisshorn02 kernel: Lustre: scratch-MDT0000: Denying connection for new client 148.187.6.7@o2ib2 (at 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7), waiting for 621 clients in recovery for 0:35
Jun 19 19:36:49 weisshorn02 kernel: LustreError: 11-0: an error occurred while communicating with 148.187.7.112@o2ib2. The ost_connect operation failed with -19
Jun 19 19:36:49 weisshorn02 kernel: LustreError: Skipped 3 previous similar messages
Jun 19 19:36:50 weisshorn03 rhnsd[126392]: running program /usr/sbin/rhn_check
Jun 19 19:36:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:36:51 weisshorn02 kernel: Lustre: scratch-MDT0000: Client c1660554-5b39-26ad-1e36-75243baca90e (at 148.187.6.204@o2ib2) reconnecting, waiting for 1547 clients in recovery for 0:31
Jun 19 19:36:51 weisshorn02 kernel: Lustre: Skipped 47 previous similar messages
Jun 19 19:36:51 weisshorn02 kernel: Lustre: scratch-MDT0000: Client c1660554-5b39-26ad-1e36-75243baca90e (at 148.187.6.204@o2ib2) refused reconnection, still busy with 1 active RPCs
Jun 19 19:36:51 weisshorn02 kernel: Lustre: Skipped 47 previous similar messages
Jun 19 19:36:53 weisshorn02 kernel: Lustre: 47317:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 52555669079, current: 52409646671, replaying
Jun 19 19:36:53 weisshorn02 kernel: Lustre: 47317:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Skipped 26533 previous similar messages
Jun 19 19:36:53 weisshorn02 kernel: Lustre: scratch-MDT0000: Client 7eefd778-9608-3e05-4cbf-22a2da6f8087 (at 148.187.6.242@o2ib2) reconnecting, waiting for 1547 clients in recovery for 0:29
Jun 19 19:36:53 weisshorn02 kernel: Lustre: scratch-MDT0000: Client 7eefd778-9608-3e05-4cbf-22a2da6f8087 (at 148.187.6.242@o2ib2) refused reconnection, still busy with 1 active RPCs
Jun 19 19:36:53 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127363/real 1340127413]  req@ffff88060e616000 x1403849952681945/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127419 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:36:53 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:36:54 weisshorn14 kernel: Lustre: scratch-OST0021-osc-ffff8810296e4c00: Connection restored to scratch-OST0021 (at 148.187.7.112@o2ib2)
Jun 19 19:36:54 weisshorn14 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:36:54 weisshorn02 kernel: Lustre: 2871:0:(mds_lov.c:1028:mds_notify()) MDS mdd_obd-scratch-MDT0000: in recovery, not resetting orphans on scratch-OST0021_UUID
Jun 19 19:36:54 weisshorn02 kernel: Lustre: 2871:0:(mds_lov.c:1028:mds_notify()) Skipped 1 previous similar message
Jun 19 19:36:54 weisshorn02 kernel: Lustre: scratch-OST0021-osc-MDT0000: Connection restored to scratch-OST0021 (at 148.187.7.112@o2ib2)
Jun 19 19:36:54 weisshorn02 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:36:54 weisshorn12 kernel: Lustre: scratch-OST0021: Recovery over after 4:09, of 1548 clients 1548 recovered and 0 were evicted.
Jun 19 19:36:54 weisshorn12 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:36:55 weisshorn12 kernel: Lustre: scratch-OST002d: Recovery over after 2:53, of 1547 clients 1547 recovered and 0 were evicted.
Jun 19 19:36:55 weisshorn02 kernel: Lustre: scratch-MDT0000: Client e16602a1-c1af-d220-e874-6fd0602c7df1 (at 148.187.6.205@o2ib2) reconnecting, waiting for 1547 clients in recovery for 0:27
Jun 19 19:36:55 weisshorn02 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 19:36:55 weisshorn02 kernel: Lustre: scratch-MDT0000: Client e16602a1-c1af-d220-e874-6fd0602c7df1 (at 148.187.6.205@o2ib2) refused reconnection, still busy with 1 active RPCs
Jun 19 19:36:55 weisshorn02 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 19:36:56 weisshorn12 kernel: LDISKFS-fs (dm-9): recovery complete
Jun 19 19:36:56 weisshorn12 kernel: LDISKFS-fs (dm-9): mounted filesystem with ordered data mode. Opts: 
Jun 19 19:36:56 weisshorn12 kernel: LDISKFS-fs (dm-9): mounted filesystem with ordered data mode. Opts: 
Jun 19 19:36:56 weisshorn12 kernel: Lustre: MGC148.187.7.101@o2ib2: Reactivating import
Jun 19 19:37:00 weisshorn12 kernel: Lustre: 8004:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 43006736, current: 43006737, replaying
Jun 19 19:37:00 weisshorn12 kernel: Lustre: 8004:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Skipped 125 previous similar messages
Jun 19 19:37:01 weisshorn14 kernel: Lustre: scratch-OST0039-osc-ffff8810296e4c00: Connection restored to scratch-OST0039 (at 148.187.7.112@o2ib2)
Jun 19 19:37:01 weisshorn14 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:37:01 weisshorn02 kernel: Lustre: scratch-OST0039-osc-ffff88081e6aac00: Connection restored to scratch-OST0039 (at 148.187.7.112@o2ib2)
Jun 19 19:37:01 weisshorn02 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:37:01 weisshorn12 kernel: Lustre: scratch-OST0039: Recovery over after 0:54, of 1548 clients 1548 recovered and 0 were evicted.
Jun 19 19:37:03 weisshorn02 kernel: Lustre: scratch-MDT0000: Client aff73de9-4107-e3bb-98a6-7863d59cf2c2 (at 148.187.7.11@o2ib2) reconnecting, waiting for 1547 clients in recovery for 0:19
Jun 19 19:37:03 weisshorn02 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 19:37:03 weisshorn02 kernel: Lustre: scratch-MDT0000: Client aff73de9-4107-e3bb-98a6-7863d59cf2c2 (at 148.187.7.11@o2ib2) refused reconnection, still busy with 2 active RPCs
Jun 19 19:37:03 weisshorn02 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 19:37:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:37:10 weisshorn02 kernel: Lustre: scratch-MDT0000: Denying connection for new client 148.187.6.7@o2ib2 (at 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7), waiting for 442 clients in recovery for 0:12
Jun 19 19:37:10 weisshorn02 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:37:12 weisshorn02 kernel: Lustre: scratch-MDT0000: Client 20a82831-c862-76dd-c58f-f636b44d13aa (at 148.187.6.201@o2ib2) reconnecting, waiting for 1547 clients in recovery for 0:10
Jun 19 19:37:12 weisshorn02 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 19:37:12 weisshorn02 kernel: Lustre: scratch-MDT0000: Client 20a82831-c862-76dd-c58f-f636b44d13aa (at 148.187.6.201@o2ib2) refused reconnection, still busy with 1 active RPCs
Jun 19 19:37:12 weisshorn02 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 19:37:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:37:23 weisshorn02 kernel: Lustre: 47202:0:(ldlm_lib.c:1698:target_recovery_overseer()) recovery is timed out, evict stale exports
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client afab74a2-bc3f-7e16-4c45-f29c8a41539c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 842cfdc7-3b80-a53b-229e-6cef0720a2e9@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 420e47bb-31a8-1219-080f-41980ab1ff1a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6e3dc35b-ab1a-b678-4072-1f5479e7d34c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7529b79a-bc32-0317-fadb-300cb099a64a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6440b7c5-c649-8d5a-6bfa-310e7cf2da20@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d63eab3e-265f-edfe-4459-1ab7d2b82ede@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c8c56c1a-1362-38d7-7567-856a2afc48b8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1e2f72b0-60cc-d72b-20be-f7da05c18ffd@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d65f8ceb-af5a-b754-4a7f-33a1c6c019ec@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1861887b-5973-b0d5-dbea-3b7776aa718c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 23ca167f-da52-9ca1-5612-2cac0837e1a1@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 943962d1-dbac-5d17-5053-5c19ddf5af47@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 77a80782-bfc9-e835-4d25-336b507fd0d9@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ae2c631d-891c-e046-1b27-de3aae8223e4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4ddceebb-4857-7e8e-c4f6-e2ce63fa231e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0339d481-a2fc-75ab-5088-df529d9e6392@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ba5ee80d-1689-2674-b550-a0c884dda83a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 61c2f42e-21d6-c534-4149-dc26b833723a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f455405f-ca5b-7942-3444-23fa76f7586d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 084ee072-f62b-2ed0-359f-f0e27eb5dd5f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9458a2fd-45ae-0f3a-8f0f-9654a74b32af@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a27cab32-0892-aaba-06ba-9688c4673649@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9a92fad6-88e3-bedb-6215-d034d833327c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 5bda7575-e0c7-4de0-ef8a-9320d615f57e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d7aa1fbf-3e06-72b4-de26-f965bda5b19a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a3d4b482-20d2-70a2-24d7-8b1021f27b20@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 597cb6d3-4c07-81c7-0e30-30d46235d7bb@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 01c5f19e-567e-fecb-8f56-c846b4e7377c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client dc269f87-6b14-9ea4-e54a-aa777767b10d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 59bedbd0-bbd6-fc61-a643-60debf73a1c0@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0e6e55ed-a8ac-770b-437e-ccc14d6a1f4f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 60307849-7866-22b7-e469-eefca09c022b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3eb42151-7403-d0dc-41b1-c9aa2ef1f209@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 19bafad0-85d4-5118-79b5-fff6a8d909ab@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e941194d-4752-b572-c355-cf825e131b53@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e14889a3-6683-33e5-07eb-02aa955e7768@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 11dcfeef-ebc6-2942-8121-f833d422dc45@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client affba05e-a493-ff13-bc7f-aff424e1b714@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9f4ffe3b-cc9a-7e45-159e-99e64749bc80@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3b5971bd-4770-a1e5-406a-151fcb09a397@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 45e5fafa-4302-74fc-917e-3cae46aad878@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a0e6f50b-8c73-00eb-e677-be10f3a403c5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 309c769b-6717-a028-2229-5574668ed3cc@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c621bfbf-35f0-f8f0-6235-ea9a7320be16@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 434798f5-a307-3199-d663-b025e98f1d0c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 32e77249-7792-b3e3-ac0c-d5225cc7597a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 01a3be2e-0406-970e-fcae-21d696992fab@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client da25b55b-59ea-06d7-5c36-efc8b148ff1d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1b6c6125-d5df-a23c-86fb-06d943273b9b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4ec05da2-66da-4221-2868-740fb226c271@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 81c9c1be-ce7e-2255-4455-ab0bde7227c4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 83cff5cc-1050-1188-97ff-89252360c8ef@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4d2bb1c9-af30-2afd-fe73-8710ebd04783@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 50cedd98-ee20-b46c-a939-028978de8a84@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e98629d7-d8d6-058f-2cb0-835e3d81c535@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8895df2e-52d7-e720-9cea-4abd8fd63ac0@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 51d92860-ab8f-46c8-1a97-cd0827b29857@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 78ff7e0c-c597-5418-73ae-e0aae3126e45@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9a94a72b-c43e-f807-e260-a5fbb8a19f4d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1ff88d6d-7d0b-26e1-5efe-a5d01019cfe2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 97f457bb-7d62-248d-4994-2daa16437c16@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b0a9413f-b5f5-bda2-8754-c3e03554fc64@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ac077425-aed6-3996-e7ba-b75ce25d3d26@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 947ec250-02ab-a208-174b-bb01ae573da2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c4f37f65-8e04-c3f4-9dd8-7ef054993bc3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 695cfdc2-1806-4861-2eda-75bb99cbc27c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7d5fa0d0-10bd-e801-0475-836a82adaaf7@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 16b0cdbb-a191-f147-9d3e-cad146171181@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 54fe54e7-c915-1714-866d-ac984d5c72dd@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6d812ff0-003d-360f-3b2b-684519dae220@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b9b1f98b-b01e-b544-645b-b4432a2f1571@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c0b4e1ee-dacd-ac71-7b5d-8987be43b9ea@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 5cba5147-0197-1397-0843-3408531bbe8a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7ebc9d02-25e9-c2eb-24e9-7d8092b387b3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0e926cba-6326-1ca2-9e60-514c008d442b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 576fafae-c67a-2342-737b-19cc6d896c47@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7b1001f4-60b7-2817-f68b-551397b65b61@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 48f65d45-ced3-6c48-c1ae-08af6d4c9778@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client bb531f8f-a208-2746-a646-8268f2fb20d5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f0f3e74e-e648-f2c6-acb9-e9453e191eb7@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a553ea6b-1ef9-ecd7-ebf2-872f05923ef8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7bdf41c1-6c5b-6f85-df5e-4114faa76b4a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6a257b35-fbce-d2c7-b2bd-7340d5f78586@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c478ca52-a1e4-0407-c295-8c4da817c206@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 80d7a675-0b15-5541-c36e-a1eb4fefcff1@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 88abb234-b6a7-5a0b-c91e-ca48eef82464@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 46009ad4-a6e8-62c9-6caa-d183e98662f1@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d729929e-b76f-bb4b-7a7a-cd83a2feadcd@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6c8cde5e-96a5-f341-818d-a20071cfe63d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 07567467-5e01-a34f-f8f4-3a7b3ed9ceb6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e12fc583-2204-9681-5bac-3e7e120c9d72@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 54f9e32c-fd79-2c76-a923-7d15effd8f17@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client bd1cf10e-1655-5709-398f-084084092282@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 226bcd46-f089-5935-bdc5-f5ac9e9acaf0@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c1876933-2940-b030-65a7-a336d76ac95c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 851529a0-57de-f52e-495a-bdb3c2548382@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a5fa6dcf-5f57-a182-e02b-5ed8da11533d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 65cea9d1-878f-64cd-40d4-16ccf93624bd@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 36524cd4-3eaa-1789-1fcd-8c29de48f6a2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client fb9d99c1-a590-c507-e0ab-5a52f848d00b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 65e16b56-4953-19e4-b839-a97c7061fbf2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 11067cfc-9182-86eb-65a5-6b98010d0295@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7a05940b-fae1-e7be-7e15-9bee7a5590ea@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client fb71ab0a-4916-25e8-46a8-e05e7f29e475@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 194bf11c-1731-0643-2084-bcee2d03d440@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e9e72289-a3c7-197d-cd2c-fd628ed93783@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ce2d913f-f56f-3e9c-8b67-51d1c3645cd1@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3d8d7fb9-4e5b-1763-14bb-37500a3455ad@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c225b447-618f-6fab-a2c2-59c8886f4b1e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6d348874-ac2f-8d09-e34d-9a76ae2229f0@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d0abbd22-8259-6f69-5471-23a369de00f2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client bccba43f-b8ed-867c-7f12-53b6a6e2a90d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 69689c30-37d3-7b50-1dd5-26195afa7287@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9d539b3b-33af-1959-b09f-cb751cc32261@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client af978390-0478-ec16-abf3-e3c34ce08acc@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a44a9dcf-17b0-ee74-299d-4adc2480f2b8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client df8ce658-6b85-e3ae-fbb9-0f3defa0ef27@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client fba74bd7-9b0c-268d-ccb9-94104d8b9f66@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client dc7d6c0a-b61e-7070-fb7d-94696a4fac96@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4d631296-c499-f446-1df5-b1580c21ad94@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6a60d370-52ce-1834-d88b-c38eb71768bb@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d078bb1c-5c77-5690-d971-d5f20c7bd417@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6a687dcf-24d6-a517-14e2-88face199ffa@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7a58ce71-48e6-8fb6-87e1-33e11f7a9e5a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2e573837-4015-e6d3-6743-604bfcacc108@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 52d4c6bc-d410-1e5f-8728-5075ae31a077@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 026c1366-f430-f165-509e-d9431cb859bd@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e83f432d-2e6f-7e60-e545-b12b8569dc76@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ed74c81c-8a8c-0086-9f31-397f2e5874f3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 674fe0fd-fab2-388a-6ed4-b6836cc7154c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9870125c-6c5d-cf66-3b00-e23e0f4917a4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f81a4dc5-97c1-f5bc-183e-f4de47a4e053@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 83f14f0b-a514-4b59-f90f-bf688830c96f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3e641ab0-76d6-5274-9d86-e3cef17db1d3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d828f342-627e-51bc-09fe-64000f9ccacc@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9a4dc421-f4cb-7e13-a861-6d44f104fdac@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 142b9b77-5334-9f80-9979-152feb92fc80@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 87d07ff8-9643-95a3-a131-dc10c5e8095c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4b5b5d42-19d6-2dd7-8d37-14558c7f26dd@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 82aba7d0-8094-0ccf-c2fc-67a2f0c38676@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 596ac6d4-c389-751f-5060-1e827e81f4ae@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e8cfcf06-b3df-4ebb-d5b2-a0587c947c6f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8a27c787-d7d9-f50b-6ccd-afd9e391bf75@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1c5cd71e-d3ee-60c7-2942-63836740b20c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client cc711d3a-79d3-6acc-e221-b681040b2c9e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 798164d4-f48b-36e8-c44b-dadbdf7de0aa@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 07fabf10-7d21-36c7-5687-628efa87f508@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 5e28801f-3724-0ccb-10cd-772c5450022d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d8858f28-6bad-c3db-5c95-5a86a69f8403@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client aff48c3a-68e1-40b3-3317-26a7af528b1a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 5a18ee38-cac2-81db-5cf0-cd5fa59f7113@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 29b1b10b-d5e8-3058-fce6-e5bf9630cbb4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b8b196f5-7494-30f8-6cc8-74dac6238ebe@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a3b2bb31-61c4-1689-5a20-dc0f9f701b1e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1d27856b-2655-2648-c16a-e73bf9b28940@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 95c45cd3-2a10-e4d6-4e7a-919797ae466a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client bbf334d1-b447-cfd9-949c-5e3a6b437ece@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a04dc3db-0023-49c5-ee67-3a10b0e47c51@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e1e5d02a-6559-984e-7274-2288a4328d42@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a1fe5f9e-bb80-0185-4f15-42efb4130098@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 78a23a0a-4e0d-4212-1f9c-68e4c35f94f6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 03f39dd2-d561-fc2d-7651-e23ba2e8e248@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c846342c-e1a4-f02e-d6bf-1c711c62ecb4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8ed92a5d-8074-1e5d-813f-65004f67e32e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8f8046e4-fc0b-0a73-cc2e-58ccad13dd7d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7f91ba79-7fc2-abc8-5bb6-e4080b90bb67@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6f76a95f-de38-2839-1568-4b3247dbd27e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client bc177661-115d-7442-9dc1-bdcf9c7f7e12@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 496761c5-eb7d-63bf-2b13-a84b8a2f7403@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2cb1f97f-dd48-ff45-4da8-81afa668b3ee@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client cc536d56-6435-96e6-acb4-0c25676cf895@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7e6485bc-8edd-6013-e00e-f4e16ccc508d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a7201f30-f20b-4264-7e56-feb9a31f688a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 05b97a1f-cbfb-4222-bbf5-8fc4e349f801@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 781d8d0d-c77b-fac5-d4b8-e3f2f4e525f9@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client dcc22dbd-8e77-db96-9b03-cc0391a2f0b8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9651d548-a80e-d292-affe-7762ce1d9539@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d91a3d24-7d1a-c6db-7e06-d1e3e31625ee@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d7a54e4b-8d82-dbde-2deb-f800873e8abe@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 992f38ac-3f4c-511c-c04f-39c1d4dd75ae@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client dfee91e1-98fe-acfc-c844-2fd7593b1d4f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f7001a3a-9b41-db3c-497d-47852e647e89@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c158a996-3292-6a04-fa46-ab616116b5e8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client eeba71d9-2b95-3496-93fe-7ddd96dc0949@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ba256758-e35b-b112-0070-beae56f7f09f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d35729ea-9adb-a8e9-1f41-c125bca20631@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4a3a64b4-89de-a7d8-c7dd-1c5ae60f6d50@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 19fe9c01-058d-749f-a027-e754e0d17574@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3366c43b-1646-f5c6-7c0d-96b10fc0ac0b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 25e626bd-33b7-902d-e993-3ea66c861cab@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3a12bfee-fa9c-045b-363b-45c7b3c3a0c8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 69725735-9e12-ad5b-ee21-624c53ab1da3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1da4fa32-fd01-40f6-1811-c0504d1769a6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client cb7498e4-5a9a-939d-148a-1d05d8f081e5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 89705642-d364-550a-7d9f-5fc3bb0c9f8b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 52d8350c-8861-f781-9279-537ea19f44e8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 578e8d7d-aba6-e270-74b8-2a7aa05ef4eb@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 46af108f-a02b-eec4-c5fc-768c945e8337@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b581d865-6f99-4aab-77b5-dd8d127350f8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f57e7037-75ee-c9a4-d692-738a98ed2278@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2c2513d5-9edb-33a9-e7f7-eb67825e9cd6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client bdb9538e-5eea-38c2-d8f0-d7b64005414b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8dea362f-a048-c8a6-22fb-b21e9711ed92@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2f124f7e-2ccd-a847-8e1c-72f8c3e34204@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client cf9163fd-8196-3e0b-e748-2b120c8428d8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 34faa76e-8d63-f124-ad8a-5319c58a1d94@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 696d5965-ac25-446e-e1eb-ef1d06cb4483@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0cf051cc-bec8-7eeb-be5d-2407ee35b7e5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ef374cd2-63a3-b6bc-3832-22d4df77ac5f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f62acc3b-7b10-9ea7-537e-f9acadb197a6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8df53526-51cc-ef17-d6e8-cdba5f03f4da@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client bce16a5f-372a-a247-9c55-e4154f96c72d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0be42f69-5b09-0d63-ff4c-acab48d23b9f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 707ff06b-abde-9b46-42c5-b7ae2db7cee8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c501b987-0227-4ff7-5ca7-a98b805375ee@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7491684b-0e5b-7a7a-7c55-55e2a292a784@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a180306f-3637-655e-659d-0f73530247f0@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0dd0a644-d979-d90c-9084-45bc81496a92@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c5aa3abb-aa29-c316-ee28-d671201c27ea@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 60b471e7-271a-95ff-9cf0-49df0d441ac4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0624a163-4c0d-82d2-5796-c8616a8c0fce@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d5e7c064-db2e-002d-d11b-dcf6437e3548@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8ca816d5-dd31-cc7c-33e5-7846b730086d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7a45fa32-a971-83cf-2ee7-948a5a17e403@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 59134caf-031f-13bf-586d-e8f53fcc579e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client db56e109-5313-19ee-7fe1-85e0f6413a52@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 00145824-541f-71cd-42c2-2f0e01112095@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client fd5ac564-d35c-0329-fff2-a14c4225f160@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 25090019-4d61-95d0-b5c5-20f848c18035@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1fa43584-30ac-5da4-4d47-e8e4ef1101ac@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f8002a58-096c-a834-26ee-1e055b3beba6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 171b0170-a299-c233-8f19-f80cd7996daf@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c09a5a59-c9d9-ccb5-30fe-cb53bce4fcea@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6ffb5160-83e3-9a1f-ed34-29eff502af2d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 990476a8-a800-9b39-29f7-fc26ba96aecd@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 36b97dfc-a31d-787d-32ec-31ea379f7fe4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e6678c67-a8c2-3e58-e54f-b6e741d7da44@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4127745f-eba2-bd7c-2538-e61972ad23d8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4928354b-7102-c9a0-b99c-0287ecdb8898@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8ba9ba38-674e-2e0f-4981-1317e317a884@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 091c392c-61ee-230f-16a3-c390c8cab5c9@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client fe336703-3c09-dfaf-4fe5-592eef5e826d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2a8e8ca9-8876-68f0-4423-0dd09364de9c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 766b0aed-8107-b694-b0bf-7a3403af7502@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9b0701dd-ccf3-95df-e44f-6acfa0233c9c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e460d8f4-3c3b-4e41-136c-6584e9056859@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 21cc4d06-1f39-a4be-1947-c9e1f81b579e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 74f80e5c-b8a5-99c7-06b5-0e3a6024b0f0@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ffb4f64e-b549-d031-8967-1d6a71a6db60@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e6fce092-6d86-d329-1347-b0d9005e6a92@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c55dc0c3-57bc-11b8-d26d-19416be75c60@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 88388cdd-ee34-4a0a-c826-8b9f6e0ea191@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4a924c18-40f9-7a9e-784a-6d4fcd8dadcf@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0eba9c71-c58b-19c7-d24f-9408c952f536@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a7bdebec-0b99-c220-c664-4bdfa9dc1958@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 72ace812-cbf3-61a7-47a8-4e7b7203bd18@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3cbebb62-29be-7674-08cb-f52006994cfc@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4c73590b-3be5-b5e9-799e-5662b6f2e3c3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client af9b413b-f361-40e0-eda0-34ac39ad7e87@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 77a7d008-24ed-ca23-dcda-827519c9a6d5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0c8e16f6-d468-48f1-4498-0bf831e37d0d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2e8c4e0d-f46e-03e7-5401-a9da631d1886@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0d27a402-5276-2694-b555-70a55e16c627@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4648d9e4-1df2-be0a-f75f-02b056ef8fbe@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7d803232-e576-30d4-1e9f-0bb32234dcac@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f82bd1d2-c009-717c-0eff-dc3035236ca4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a589fb1e-9e4f-8040-8841-361bd566d764@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 172bbb70-ae3e-61e1-2a64-7407099cb72c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 50ae1bc3-26c6-a145-f4b2-e8ab60648815@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client cae9bb10-c269-7bab-9dc5-b9dd47c31e01@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 87d33852-134f-d1d4-7191-f74c8af42a0b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 127dd107-828e-de4c-13da-6980f524e4ac@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 14d7af86-41d8-b291-f1f2-c31b4e1efc70@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8205a75f-c0f5-eda7-dd01-63760bb65525@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9100db3b-9a7b-b48c-05f0-b749c7e1d1d8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8130abb0-7324-77a6-fe68-d504aca4fcad@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 90d9ae54-335e-164f-241a-f33ffe3e7dd6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 171c786d-e48c-a1ac-4482-a5422d398ce2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 45fa1234-789f-728d-53ef-80f6bb947504@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7e30aed6-6033-4bde-d2a0-fafaf2922244@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 49c8f334-4036-b1d5-8a83-45559a44aa49@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2c304868-c5de-cdf9-ca61-a5806510b17e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4ae13090-1b7a-5937-4f9b-c286d0901142@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8a485eb5-d2c5-2f68-7fe0-c9169816977f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 998b2410-0516-3627-8702-42751ec0357e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2bba5faa-4f25-c803-5593-c63799932ed4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2da91fdc-ee25-41ba-2b69-51dcb2de17de@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 36373e1a-1d45-b488-e947-70b106210202@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2b599255-ea91-c5e5-ac9b-e6f1c11634bb@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client abb87e76-4701-4f0d-949a-9ed157c19b7b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9a86d126-2c04-03ad-1c3d-3c578621d5f5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 287ba419-1d37-6a55-e27c-f21dc4068124@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client eeac1eec-60d6-f3e8-8563-30f9392abe76@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e027c3a9-2fc7-24b3-efc7-2c9fe872f50a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ce6cd93d-28d5-1345-8871-092e920bfd2b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3dab725f-bd17-3715-939d-cbcff63df45b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 21e77e23-69fe-35df-13b6-56c4510e471f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 80736d53-c221-f31f-d4fd-14d039f469b2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 222d2627-f2a4-b179-9db5-568522888eb6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f437f593-81f6-16e1-f382-138faa28fc00@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4b344115-a2c4-2063-2a80-ec82ea44efed@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b1994354-8c7a-d103-31a5-faba4b880c2d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f7dd47bd-b6d5-e0b3-a243-4888a5968491@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 80862306-0d7b-0c6a-8dff-9a1b020e42bb@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 295ea80f-4514-fff5-6d07-f66e48949d9d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d90ba72e-088a-bd41-b717-85af567bb4bd@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 15310a62-a153-f013-782f-60d7de2eadbe@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 281ead21-3c41-db78-a402-eeaa5361ecb7@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 69ecc449-a15b-df6b-e2c2-7e859ea7e418@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d55facf8-cc62-e6f0-5bf2-6583bd363836@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 54d75793-b94d-ed5d-0e54-c133f2115b83@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client da49d7b1-fb24-f15e-703a-0f08c6a252ca@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1ec47c75-5fb0-76dd-60ba-a44a03410ed5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b3ff386b-35ad-c6b2-3421-673f1fb73108@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 45fd5fe2-ba9d-7f85-c623-70a33d5ca298@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1700ef1d-0672-68db-3cc9-78d5e4cddfb8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d1f9bc51-17ac-3fab-abe7-e070a6b392a5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7eaf9105-5218-9b16-e0bc-ef185c6459c9@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 06a25698-7700-083b-cbd7-072c38c1e2ac@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7356d635-2546-6a26-bd26-8821ae731836@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7a96bc84-864b-e0b6-317a-8c3d43d7404e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 26f08c9a-d3ad-ffd7-61e9-fb1f2bb7b8a2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 08c2f75b-3a78-8e13-6316-a38873c671b7@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client fc7e1a83-60ef-3b4d-2984-7ea439ad8bf9@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b3e7620f-cc6c-ed90-f99c-3c80316ffe70@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client fa86192a-893c-5a5b-acbb-9fab7f21c839@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1414c939-f596-9c3f-01b0-618593dda133@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8fcffd1c-d399-ad21-4d92-d78781653551@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6a779876-bac7-a5f2-0c33-d102b3d7438a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 350ef216-b142-4cce-c71c-27784cc01916@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 55255aa2-9623-f65f-0861-f187fe8cc103@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 37523b14-96e6-a207-f19d-fa04ea6f7cde@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1324e175-46b0-89c2-a6df-908e2a6eb0ec@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 56f9dd4e-3ecd-5da4-3cc2-24b326483ee3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ed389224-213c-6cf3-e663-55c9ca580195@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1788286e-5c8d-503d-777c-1d2f84433e5f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client aae81c7a-7e78-e401-809f-ec98c3937414@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 5d0ef91a-9b5c-d7e3-5852-461a8412bf85@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 35b360c5-f222-dc68-f04c-4e418f36e130@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b3c1a046-19bb-7866-627f-85853a5a0f26@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7f31a137-9c4d-e7ee-a2c3-579db9876e2a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 826212b4-a623-9ce3-a213-06738432bfca@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 774e614f-a46e-8c44-83f8-18abe7274516@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 12f00a6e-583b-91da-9725-304474663bcf@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8cf99302-263e-a864-0f59-28c7c2533c80@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8a17f955-b5c3-ec41-00ec-fc11836ecf93@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8893a121-49ec-0499-5fb5-b7fd10bfd97d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9533b5a0-2076-97a3-e8f8-1359965738a2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7070d1f3-e659-7a5d-6f6c-5eda304387b7@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ffd1469d-c4fc-148b-6171-55308a176d8b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c0add243-94bd-143d-53b2-f5d4d3a6d382@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 02f0d615-6397-d422-3e35-4512fb48e406@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client db8d2f31-2794-0805-b4ad-f5b3feda7624@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 77ba8b28-d5ec-7e98-7adb-0d53996b0b22@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 897dc92f-73c7-b96f-792d-f1efb05ddfc7@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8b66a96d-c902-875f-3d7a-11aac862ab5f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c5caf603-28d2-ad32-c4ef-f30037792185@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d72c750d-102a-6666-3079-ee92158c1db2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 06b64fd1-fc95-b7b3-7bba-efcc5d93510f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8d5a7b64-d6ba-d2d9-340d-f3b50bc8ff61@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client cb91cacf-c9d6-775e-47c7-e491230cfa3d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1bc162d3-bf2d-9efb-76b9-2cbd2569599e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b0f083d5-2b13-be34-659d-d2ad8d065a30@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f5c559cb-9642-3b4f-8dbc-404e97eaba79@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7713745e-6a6e-9c1a-a877-1e4c4c05f32f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1eb57c28-06f3-f5d2-f097-0a1e4d784b89@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6ccf3fe0-db8b-5aa0-0c14-c47b9eb662e2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 05ee4567-5958-c667-3f38-8685b9d6aa93@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 77e0ab97-3c72-f61d-49ed-46baaf01be11@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4bf82344-dde9-954e-7071-65333ba33d56@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 883bface-9ebc-9e60-e513-297c76bc07b5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1597fe78-86d5-ba41-d5cc-4cd0fa330db0@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 5b2842c4-07d4-b79c-b302-eae132974360@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6159090e-38f0-bbcc-897a-15798166fae6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6f1c0539-1895-e11f-d382-a74113ef3d19@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c1a82e0b-4de1-c8c1-66a0-0cf073eb9f7e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 28f32e99-67c7-efe8-5f0d-f7e4fae245d4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f7d9c695-3fa9-8a86-951d-5eee02dd253f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 602ab572-3fe3-af38-b5cb-86da74e2ccaa@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b529a80b-d84f-ae66-8831-a2cec96d8944@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6deec890-f2a1-99da-3328-5a918b56a324@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8943b602-4f81-926c-be74-3827386bc8ef@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4993e69d-a735-2e35-11fc-e0a8bab8b461@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 23cf5339-a0cf-71ed-bdc9-a8491bf29daa@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c6bdb967-cc94-182a-20ff-26c5378070ed@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a6adbb60-9e9b-d14f-7db2-c3b251e68762@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3a78afc7-2de3-13df-e448-d4633027bee6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d8c641d2-5e0c-f8a1-2d0f-3307aaf143d0@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 54d4ee8c-3acb-75e0-03ea-c2994cc62090@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1e151156-1e3c-adcb-6708-1b18c79c6dfb@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 604bfcaf-65f9-9787-e68f-586a714026af@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a388e424-1673-f6ee-1542-fe064eda4d25@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 23429cb1-0daa-1b38-6910-56301821d8d3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2cbb2884-fa9b-2807-f85e-a3cf11de818d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 23406a9e-2dac-47d6-ec6b-2d829a18ebe1@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 776654da-ffe8-f0e6-1f3e-7285b049e58b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b7b072bd-c45c-87f9-84d7-b3b7e938345e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d1d33885-16a4-5bf7-292e-1b942d4b8dff@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a26c8275-4edf-3661-18a5-94b099eb23c3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4e650e1a-08a3-444a-1ed7-f5b6f92ac32d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client cec228da-c44e-3487-cd3e-ac1ad4747f6c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4f59294d-ac74-aa89-6732-4d24527c79a8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8d733b54-d7df-4b01-e045-ceff0699d111@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client edcd550b-e1b5-cc9f-f036-0784e858316e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client cffe551d-bc93-2b69-d502-10a3b413a4e9@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c2868068-3814-8093-0517-ec8683676285@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 59befb8f-fd78-1255-3dea-8fefaadc6b08@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 83be8527-03d0-2478-2382-7dbb1b08fbf4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 62616ea5-646b-614f-10eb-8e1043287355@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client fc44dd5b-c0d8-d511-bbf5-a760c6a5b91f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0ac03be9-9ea5-7c09-fe45-64ccf9553c73@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client cb59ca9d-3808-725a-0d00-a4057cc9f785@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f424e949-0080-bbf3-a908-00b4323ae57f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6434bf51-2b3a-84ed-4ca8-374266fd1687@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 571dd69d-843c-21c3-9d22-90379bb38ba6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client de7e777b-b831-8fd3-93cd-c2018477f9ef@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b5f59d8e-bf64-18b9-ac8c-b556f93a71a2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f3ac13ef-2cfa-c960-608b-d00340717c55@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 06b98fd2-fc9b-fdd5-f0f7-43824ba49c1c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client cacc824f-1b99-462e-8e86-c5c037593762@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 290dfd6d-10b1-4caf-5763-63594fa2b9ec@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4291b9f4-f3cd-eae2-a183-8706631be15c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e6730de9-5443-805b-c780-c5f94eb54d20@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client eb9826ac-2f28-79bb-ca47-2423f7db9e75@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e6aa4269-94a7-a13f-cbbc-10067879625d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 22954588-97a0-8398-37b7-32eb19c6fbcd@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 390f156b-7b68-84c2-13e8-0501a3176cfd@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2cf9e25f-53d7-b7f6-4ab3-f98495f9403e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 98247635-4c91-3d9b-747d-efd5c750c7b3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d64c6fb9-dcd7-fed0-1d94-ad1e663897d5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9a1d71bd-e0ec-47f0-91f0-3aad37219d44@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2871c44f-0773-e569-026c-b08a33874c33@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 84d92100-d904-d9cb-8d0c-37554b3450fd@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1c7ecd1c-b68c-0098-4216-896d13f582ba@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 65f11b2d-bacf-779c-b90c-43aec2cf0b39@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 84bba3dd-1ccd-2b7e-4a7a-f8fe93784ad7@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 5698bb7b-ecfb-e78a-cedb-5c6d7a564b3e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 104e9097-f1d3-f44a-b86d-6f62e036708b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 19c4b771-c044-21c5-bc4d-b8cb60f6128e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1ca943cd-2eef-a3c1-9d0f-6f1c8751d5bf@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a6af589a-066e-542d-33c8-8e81445c880f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 15ccd154-8c62-194a-6331-cb6d61b3a661@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3de38765-0c5c-0bb5-b28b-21f191a6d428@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9ff8f01e-f8d6-adfc-9cde-a5ca343298ed@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9b09c17c-2b68-66b8-6a8c-b15e2ca67805@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client bc2d5476-bad1-47ee-aa44-aea8ab017b12@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7499f328-5811-d335-73ce-fc3c9c183a8f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 49f0e0b6-8333-f883-78c4-667338bf9bf5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b6cdc261-0679-9285-78d7-60a4bd3fabf3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 549d2388-0af0-afaa-cef0-0d7c77813a01@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b833abff-db2f-e67b-6243-195f4cfc6c82@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 547db71b-8881-9313-3bbb-2cc881cec2d9@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7dd4b290-5b5b-d6d7-b176-221db05ae207@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 951808f9-cd47-45da-48a7-4b9ff440fd57@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 637d250b-1b7f-4c47-75c5-413bc80faae9@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ea1c135a-f470-30e0-5589-5d3ae0d9d355@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a43bcdcf-2b06-39af-4fc5-2c9dee886c96@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e598f4bd-70a2-da22-b52a-2e931f7eab29@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ebb1ce6b-8551-807d-3ab6-34fbf574c6a8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9ea4ce22-901f-914a-9920-c05edd6c5daa@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 32cc30ea-44a1-b998-c13e-f742099258f0@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a841589e-2e1f-20ee-89e7-b0eab1d33a2a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 68f40a5f-79a8-5593-0be9-fd4f01afdcbe@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 15237f5b-cbb8-f9c8-6739-4d9376f4c39f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client eaa1a6a6-2735-1709-12c5-91e634aabe45@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client fc9e21f4-217a-eed0-a825-0459f3ae1691@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c951a008-f2b5-be24-7e5b-8855d6015ebb@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 22f6d92c-be19-0ea5-8879-937680a7c1c5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a6c50cfa-6507-2c72-e2bd-e62911f4938e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d2a81bdd-9d7f-6a39-dccd-214e1989b58b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ef726bec-9ce3-c44e-0dc8-7b5fe4d58a05@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7825dffd-fa61-73bb-dbbf-c18b07ab6674@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 274fb16f-4fd8-78ab-0500-9c40a9a72623@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 05a98856-2e4f-e1bb-312a-ab6ce75cabc4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 04c92050-680d-55e0-23ba-5e680bb8d676@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3c33d35e-04ca-5412-2227-716794eaf28e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 23974d00-53c8-4166-8e85-7b746884d39d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 5ea56004-54fc-2610-a6ea-c629cbfaacb3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7c6ddb6a-3d70-679b-a22e-999c9223e2df@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 21367c7b-544c-9588-9a21-aaa28bebc72f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ad3f7787-cef7-13b8-9cdc-8a2a5e0fc8e9@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client cd408d7c-2144-80c4-1075-b1360e396b29@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f5c79b5a-8a92-236c-7349-b427d92f12fc@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e4799591-9b68-9767-851a-9a22d7ac2ccd@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 95d35c03-1d42-c757-367f-542695ea4311@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 32edfbff-e5a1-3b86-9435-0c55854cd313@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6082f6fa-6d25-8fe6-ce10-269b5f5634cd@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6ff4b49c-44ac-31cb-6a59-d68e5a26cfc3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b8ab4eba-f656-aca4-bcbb-927ba8a63499@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d46567b9-3cab-7482-468d-ca01657c3a21@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a1358b8c-c253-d515-e165-0229922b1c0a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0e0766ec-f7c9-3ddb-3f16-0351520c009d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 622be696-049d-d3b9-e592-39cefb4edf01@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 468e5cd3-fc63-29b4-6ed8-63951fc76907@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client de764435-d0d1-68e0-90a0-ca6ee61c7f1b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 450699fa-d0c9-be3e-a1d0-79c64682376b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c65316ce-0763-b760-239e-305206285c1b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3bff48e0-adb6-5298-c6d7-73298028156b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3744556b-bfb0-7403-4718-a037e1791a73@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 369b07f9-1604-b1ac-464f-9190ff427a64@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7eddfab8-9b3a-ab17-2d4d-4ac5a468de2e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d62e522d-5c33-cb21-56bd-41a74c167e57@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9e85b752-3ec7-5614-47d2-3b0ddea10042@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ffbbdf1b-f2bd-a721-a581-d303c459565e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7a4fd98a-0bff-f532-297d-a7ff036df38a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client dda4e8fa-15e9-de21-9224-ebd66e27ebfb@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 5e17906b-c8e6-f25b-045d-9dadba5278ab@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client bcdf711c-c1f1-fb1d-d46e-e340aeefd29f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 36ff436b-3892-cc0d-69b8-06d38928c113@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 91f98932-8405-e453-9690-06493857be3b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1d6602b6-ee26-87c6-bc96-5cfae7b8f34b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client dc889893-7773-fe1d-1a04-0432fdc1cfcb@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4314e5ad-6562-771c-9e35-f2815f2feda4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b20897a7-fec9-3490-fb9a-0516f452a91d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 597aed28-e413-0c23-27f7-89fd6a69ac87@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8dd66202-efdf-03ef-0a33-6262e5e47743@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client dc82ada3-8757-e024-e956-304fb313dd9b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d20caffc-94a7-6e39-5236-3bf3ce3ca360@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 69737dd0-da3c-f223-5248-e669d2cb1fd6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7e9ac313-b47c-0f73-0de0-032e2188dd98@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4463ec08-7d6a-5780-7311-500be14f8b6c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 52585ef2-e395-d567-7ecf-5657ad5bfc92@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c118991a-07ab-7b61-c5ef-edef548aaf9b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a7baa74f-2b6c-d2bc-d5b7-ea361d8df295@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 741fe5b9-9a54-029c-71c9-6588e76a19d1@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 667d0501-5a0c-9ae2-4434-90598c6180dd@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6c6ddd16-29eb-ad4c-f748-62eb8f7e9370@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b71e1252-7d5f-ccd5-89da-daa17176f400@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ca40c4f8-b5bc-c75d-9a7b-43560729093b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 241a90d6-ed27-6f70-3172-e989f5b7d24b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ea7392d8-d4c1-d767-de00-24ad4d45593d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 99830da2-d5bc-70df-ebb9-bf02f0f57c5f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 69b50995-2420-a4b6-14f6-bdd10806f20a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8f0729c1-b4df-b801-334f-ce8d814202fe@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 51f39b17-6922-1510-5adf-ac12c98b96d4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 50dcf529-0f90-1d43-123b-95c758bf63de@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ba166416-386f-0bb4-051a-b1ff0e4a7915@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 567d4aaf-191f-37e4-97da-4507e711946d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 275366c6-4c46-3b92-59fd-ea163f1f2d8d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 47cde4d5-6dc2-09bc-eee2-27da23db008f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 332b1769-44f3-c993-d938-6c9f5cbf4c78@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4716c234-9aad-416a-d7ff-fd2ec233601c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 71eb74d9-62a3-4795-a2d1-2bf3a052ea0e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3bd5ed23-b149-5d63-29ea-18a81d7b5540@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6a1a9167-9f15-ee7b-811c-590a36dcd1ed@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 901f929a-6ce9-d6a3-2f0d-c62423d7cfad@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e009680a-7ab1-888c-98a9-350409edd0d4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 091534ac-d2b4-0579-add3-d526e63426a3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a5d4256a-d5ca-8e79-4ed8-b1252e8b623a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 25554230-0403-9cf7-b2c4-2e87ba979d68@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 36257ceb-9f9c-0c1b-6408-481f19b46d40@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6adb6b69-b2cc-dc2c-4edb-ae3ad875f53a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0ee4abb4-bd21-db48-5eea-128e0564d516@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client cff060cb-e478-a38a-8eb9-ba3bee659951@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 65aadb03-4d83-4676-de70-aec804a6560f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c6c2e6c2-1527-52d5-c4ba-e830998c5cbc@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4cfcee73-4a4b-c579-7ea2-d8dd7793ad95@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 64a52d7c-fde9-b67d-e479-715775957272@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client eb710d61-2f10-103d-c1f7-d53689f5654b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 163a37d5-918a-80f1-dc41-ed684a635a95@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a6bdf7f3-0661-95e7-06c8-8e77aac17681@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6f09ba5d-f0ac-d4e2-205f-2e8ff3837989@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3095626b-1f38-d37e-acdd-18fd9ed2367f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 82aa8393-8762-169a-ca08-0f8e85b21b94@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9b12f00c-f31f-ad79-89ac-4c97fae8e223@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b42a4573-a01a-4056-18fb-c13decdaad6d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client aafef3dd-14df-a6d8-4263-07eb98625f0d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0d757fe5-f5e4-c463-74c6-cc628d8117ff@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 50dd891f-52c7-a234-5f24-b4fe327db9f2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 713b204a-748a-620c-6412-ce2419e738ff@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 64938409-b7e1-6a01-28f4-3fc10e7d6d25@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2bf2ac13-4e13-e78d-de56-b6bf0fb2c424@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 51444237-a90c-f19a-a8b6-702057399605@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3c286de4-4b66-1d8f-435d-a81f348982be@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 42945726-841b-43a6-4052-e48e6e147a13@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f92e218e-180c-bae1-5803-14f26ff422c4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client eeed0c37-dba7-6ae5-32d7-2eec9307f530@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 68f4b084-234b-7e93-d11d-285e5e995fd3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 49ab23c8-ffa9-1dc7-508f-4838c847f4d4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 94343e24-9903-767a-b655-2005d86885a7@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 5f350f36-dcd7-aafd-b3e0-78ee679b82ad@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 16850fe0-6497-916b-b4c2-793faa42551d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3604368a-3616-95d5-f081-de1741c0a6f7@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f95ca840-3817-fa50-8653-e8f94c098cf6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 590a06a9-28d9-2d90-c41c-4b6f268c2f04@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f4200005-f4b7-2919-975f-1685f7429387@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2bf7d21d-60f8-95cc-8f2f-2bd4bb12c4e2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 79d70bfc-16ef-dd1e-dc10-92310b3d6955@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 586c717f-15cd-f304-9ff7-25147584816e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1654e873-dcc5-6434-b4d3-026d4fd86b5c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 879039d0-fee5-b4a1-bdbf-3fe6c1d00066@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9e01de4b-a9fb-d311-ef1f-f6b5ee7417ed@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d9378974-70b7-2ede-8d65-72335644be99@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3513207b-a334-ecc6-1f3e-3380f9efb21b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 40234ddf-6e5b-e136-78e1-ffbe6e81125b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ddfa9570-c39f-5cd1-13a0-4ac6f0a896c9@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 5229d562-f632-c6ea-808b-abf99b01f879@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 76faffd9-f0ca-d77e-a484-c7392801f44b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 604bcf0b-b056-34fb-de77-ab26f9543cf6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c933c770-1434-21f6-4be9-677583a81193@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 410b1b04-0724-bde5-ef6b-3f2ebdc148a8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 53f5a45c-a737-6763-5565-84b06b420a77@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 94295441-330b-d340-ebee-834790bed219@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8c32b16b-bc70-16de-9595-b8a98c0c59cc@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ef029e67-b70d-e657-17cd-e1b92c0ca6d0@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ae74b9da-8f0f-df0f-5246-8b67f11c3b03@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7905d8bd-e168-491b-370b-3833785c47db@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client da6b70ec-42a2-26fe-3ab0-1e51023b41af@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client daf7ef0a-0d17-f34a-e09f-8e62b4e576e2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 36087590-e887-d640-4f25-414735590e09@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7f28ce06-46e4-cec8-7192-b1934a805605@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6485e1a1-a69d-7f61-5de2-9db20735f956@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client dd654e01-772e-5115-1730-edcbcaa50d99@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a2798b3b-8db4-a8fd-504e-7946baa161ae@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e84d8e50-9bb8-014c-c023-560f46ff7b6b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 958d77e7-5997-0259-54b8-43c555b18580@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9f05e4bc-0abe-77d0-2394-309ef4fdf2b6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c2b66cbb-2d5c-b764-9ba3-90bbc282eab1@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 76ab8284-561d-c2b4-8609-a21e7ea581b1@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client cbb7f9ab-1408-8928-11c3-c94f7d6d73aa@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8fb2126c-04c9-68ec-d168-4a821a781a19@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9464f439-0c7b-8fee-0dba-cddf4fe637ad@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1b5b1e20-8b2f-39ce-25bc-f1102f085bd2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e4f11331-0dda-9f36-1d56-a4af69114220@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client fa72975b-8c32-7ea3-7fe8-8102a4a16918@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a3eec7cc-ee9d-9993-4bfa-bca0d31cde53@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6ce496e4-445e-ea98-0933-70b7363787c0@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0f59cf06-bf73-f8f2-462f-90e863f0a06f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 319b2026-feb1-a808-4dd2-e983beb87488@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9f845346-f760-5078-d58a-0c9cc4953de3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0f497ce5-87c5-355d-b3bf-d01daefe762a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7544bef2-44ba-c076-5048-3e7783f72fff@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6743d8ad-0efa-9d6a-da3c-79e9545fc66b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b521b3ca-6d73-3e8f-cc35-22c16e765352@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3180f692-1961-4fbd-63cc-658636918043@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 5c1c4780-f5f0-aa15-b252-40a28b8f3bf4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9d7343fe-2b44-a545-5666-074e163fea80@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e8e7e04e-0e77-8854-c333-23dc7bd75b7d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 35b9bb90-effc-7be5-8928-7e723c8e356b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6a111db0-e4d9-504a-462f-2065586245ed@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 50dff5db-7e7e-735a-6382-f87ed47ac23b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2babdc1b-36ca-54cc-abe6-baefa4fa1b40@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ad43341e-2027-a504-b35c-c3ef3bde032c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e304b693-4f4d-796f-ae29-5e2a0d908de5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client dc430577-7bfe-8f84-2d14-a38c37aef00d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1ee55392-edb1-4d0a-8514-87f9a6f8a198@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e3457082-0c5f-2b36-49bf-7920bb6b3281@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b790ece0-f5f1-1338-fb7a-a1bf81e996e7@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e581357d-9334-685c-68c7-de428333fc59@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ec487824-4a0a-fdd2-c40a-8230df3e2edf@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 146a949f-1dce-d59a-a4a3-4a846a4b8217@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client da3a94b7-b03b-a0ef-d4f5-9c57d3d3771e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0651de6f-0d68-2739-95c8-0621f88ffe06@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3c1c5991-142b-77e9-e391-c0277db72a36@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2fa8a836-65fd-77e0-527b-4e0cc4b508c9@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 59017456-1ece-a3bd-1f9e-2270cc4eb9f6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 01546893-5fea-d833-d437-075bba818278@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b1957b2c-7de8-4659-a83b-6c8a391b6d38@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 41ef827d-f7a2-26a9-414b-6fb894421d52@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 359e62ba-9ca8-ad01-ab20-57565bfa7dcd@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 29acfd01-d2e0-a51f-6cfb-559b2271f88f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f6dda00d-599a-bc5d-df71-f80474d2ccb2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0926f33b-12ec-bf9f-fb2e-ea00c50c547b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 76909175-20d4-6fd3-3296-6088494201df@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e0f2389f-51fd-54d7-a018-e6f9e7207f5c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 5fdfba95-8192-e761-3f1b-966d1cfd33c7@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8c1c3f5b-eb2c-8a72-b07c-e84918eeed33@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 15a0577e-7f14-ae49-a192-d1e776a742ab@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2991870b-abe2-7eae-e8a6-0c984b929645@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 995ca97c-c9a0-9798-86dd-84a3e32bfa5d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 208c38fb-e798-3809-be03-c5c51cd84a84@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f3798cea-73bd-c11c-e53c-55c8656d406c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 86ee2fd0-9122-da97-e5b1-101027e45750@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 64ca8da4-b9ca-0195-b169-cf2a27afef2d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 55737aa3-881d-5f14-b776-fa1470d45b66@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e0bea96b-63f1-e5fc-d343-343da357af01@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 614db09f-a2dd-05b6-0fc0-c1ad575cf781@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 28b89817-4f5d-6ed1-eb9a-99b1d9b0c979@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ae578ba0-5f7a-787e-3556-27b3595dece0@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client dbd8800f-575d-1c5a-6527-eeb13b2f05ae@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client cc00f9a6-8577-6ea1-2d8a-1ed932b241b7@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6aa58209-22ba-6a0f-e22e-69898a591494@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7a1fcfa8-699b-b5d9-2268-ea2476a483f1@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0db7123e-4730-8f51-5632-2406e7f554fe@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d25295f0-3dfe-f235-f41b-b290ef6d2481@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client cbd8baf4-e549-6e55-9d6a-35a35400cc52@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7b396daa-8675-b7cc-8a3a-8a8fd59d5dd2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 30ebc150-b996-882a-f183-2b42a746d33d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 741db80f-3685-f365-c767-55ffc91ea3a1@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 50b95eaa-6dcb-4a19-4333-4bf1e5d32aa5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f2346c11-db33-a8d6-9d4e-f1893ac1e4a1@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 923d9434-489f-c295-3475-586d46caacb0@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8f116ac0-a53d-7be8-2601-43795d5ee81a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0731df65-ec6f-21e2-6cb8-a291d11390f5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 33624adb-6975-4d8f-357c-0f2d43334486@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client af717fce-b184-2fbc-799b-8a8570e58785@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 00234988-c8bd-e231-1166-1ffb66090e7d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 5d177ba5-1192-6156-0cb1-98af014381d7@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ee57894a-e473-0217-63f4-9ab0c4f8d517@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client adc25ef1-8f13-a258-ed73-d11045dd1cff@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 532a3ca0-7941-f8a3-ae19-9d2ae030d649@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 67fd6e53-8bfd-5ac5-bf42-7f46e4afbfb3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6cfeef27-2f46-a00b-bd65-0304631c14a5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ed89d05f-fc10-928b-690f-3c135c710caa@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client beeb4f66-70d0-0428-9314-6bf2d769c1ef@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6f8b5445-9861-c1bf-f2c7-d3cf6432e210@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e88f84c3-c324-adf1-089c-c07fff82ea49@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9ac5e8eb-c1d4-dcdd-41f5-9e25428e5220@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f2207497-6455-da6d-77c9-a905efa3a691@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client fc0d5134-6531-ca19-762d-24b65813e7dd@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 304376a9-064b-5a79-ce40-15339a19a5d7@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a0d36340-da5f-c4ec-6d6a-c3e1417c15b6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 98c31b4d-799f-56d6-db54-77da63900519@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client da75cc50-17ee-b38c-7c0e-b5523e5dbb0a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 86ab88a4-9dce-c8ed-f5ff-dd64e1cc4b86@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 59f62034-d0c7-87f9-807b-d243f79ce5c0@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b391a90a-b703-5331-3fba-1e572b7fd161@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 77ab4990-2c3a-d2f7-ef60-e29655655d03@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client aef2a235-2e73-885e-c94a-79c2a6dc25a5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c53c1574-d31b-748c-5da0-877621ac1fac@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3d7606aa-9043-6898-3ddd-353f17fd38f4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7a190433-208a-d4dc-ee5c-b2a8b73ab95f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 4e944137-662b-330f-881f-21d7c707669d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 073b4181-c375-1e27-3f04-f2632b06891a@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7fbe65e4-881a-ec9a-616d-9c0969ef0349@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client aaec8a77-e643-f626-a8ae-69cc02d2b3ab@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2c86ebac-f6ec-b946-aae9-6b890fb95979@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2f00d1ce-19cb-f545-c024-39b225b59907@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client dff623e3-861a-18e7-997d-506a212ac0c0@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 74b8e161-fafe-7840-b4ef-405019827616@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a39d4c1e-f4b7-68f1-d513-7ece6f75aa6e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8946984e-b39a-f26c-c714-17b903aff2cd@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client d57a5af2-8115-3793-5cf1-b1f0989702ac@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c64fa198-6e93-3b4e-0aba-c01793f8afeb@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e68ba2a3-b133-4eed-ba82-6a32baec946d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7ad97c44-1a66-290b-e6ce-ffe96849c930@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 08298070-23e0-2be3-722e-cafefbd0ef95@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9e550f6a-e482-ff97-6ef2-a5e823ac7f4e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 93b52b54-0256-e67c-64da-aedb9a3bf35d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9222d697-6d73-aa85-1d59-21b3c8ed4da2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ea630820-0edf-b9e4-41d9-856ca187bba1@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2c34f444-bb5b-8d98-4493-4d52cf297c9c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client aa1a8ac5-4e02-e8a1-0b6c-b2d246ce0687@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a2a00694-3e59-380a-7abf-4846b2776eaf@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 667efec8-f63e-ed19-21f1-bcab036651d4@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 5bf0ee60-7588-17b0-91e5-652f8e5b6b8f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 700a382f-1986-9a70-cd8d-72a0d47f79a9@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8dc459e1-7a2a-f1d1-5f90-1a43bf55749c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c439058d-d3d3-3c71-47f4-3ac900e04bb1@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b9012b42-8725-51be-30e5-b41fb446844d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 94576235-c737-11cc-93a2-b5f61a552fc5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 592770f7-4a4a-18da-afeb-dd496341d20b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 52bbc532-b3dc-b176-2ac2-e8b2be022f4b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 442a2692-da7a-4ecb-1c75-940611ff08d9@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 967377a5-14a1-a7a0-7649-2fab6e0cbaf7@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 92cf985b-1238-1c09-0088-238240f3f773@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ed2111e3-8509-004a-ce0a-083cda628f4f@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 502ea153-7ccd-ef8f-6e20-ac3daa48e3d6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f2449ac5-9c64-d991-f3b9-261a16f50510@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7b315b7d-d620-c91e-0c5c-4caab869e19e@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f48956cf-75bd-73d3-1518-a26e97e5ecec@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 49ac670c-196d-62d5-e4c8-573dc770be8d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client b18a7c09-367e-71a2-b0cd-6f387973cabd@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 033c81f8-fc9c-ae77-1644-4126ec576d31@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client dd12266f-13da-3e65-2a58-1a6424b143c8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9c382b79-4e19-9e46-61df-9080d515d650@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9cf979d5-3d9c-dfde-2c09-8ab264777cfa@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 82282cb7-e874-550d-f19f-22673c6c8cf8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2201a2da-6cf2-a56d-d526-7d33f3c84813@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 90be7796-42e9-f265-ec4d-96eaaa97ded5@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client dafb27c1-0bbe-5f9c-34f9-0382b937f4f6@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1ee7c339-8577-96af-0d0c-488226c09416@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client bc6b2de9-dd2f-d283-b222-65ffa14c3251@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1802c0ab-aaba-7022-f308-37bd620a1793@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 66172775-8192-587a-c97f-bce2b0756d7b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 11e01173-7c5c-a818-3140-2426cc1a46e8@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 6b1113ec-0cd2-fa63-682c-3b424519d59b@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 19e84776-4333-2a4e-efba-89ec03fb248c@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 16b91969-4ab8-8d28-13fb-7b33064370ea@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client e9cb0075-df3d-cb7e-a2aa-e779d9c6aa23@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 9685f575-2998-7d5d-0200-ca1d7c7d7750@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 8eb919b0-9a67-edf0-1efd-4cbe6e413712@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a1105a35-8056-47a4-296e-67fd1e6b55e2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 2d75fb99-dcff-16ea-9e9c-d5bae4ab8796@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client dab25d1d-766e-05d1-3903-f016d1e2fc0d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 048694b0-445a-2492-f7ad-6f2fb20ce208@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 68773d9f-1b5b-568e-08bf-2428b46a588d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 14830027-99b3-f7f8-4850-57aa0d2af187@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 132ef548-5a22-4c6b-830a-6c7f38c106ed@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c978df86-db65-555c-3e2b-82729ceaeddb@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3ed061f0-0cab-c857-2c58-33ad12557e65@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 11ba3db6-3a2f-d28b-4831-98256226c2a2@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a682435e-aa86-ab3d-f39c-8ca71dec14a3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 1d02a2df-14db-96ec-37e6-5a95f1da64a9@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 86666313-8977-b835-cadc-064bec3e4fe0@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client abbab4cb-98d9-56d4-ba26-60fc419f5454@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 0fede281-4a46-f44a-e6b8-24c1c9407fdf@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 73625501-7e64-f84c-aa9a-643623993f88@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 28e4df64-0056-4f9f-cfa2-ab1ddc567e21@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 85cda0e3-5210-9ce7-f4af-8a1efa254706@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client c7e81819-1ba2-1bcb-7abf-137ea6db6874@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client f001e40d-8cb2-874a-c198-21376cff7d49@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 058673a1-8369-b4e2-f4f9-6e202e812dc1@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client a2d36907-d332-9d6c-9a78-5d1cfe2c0783@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 57c515fd-8058-c424-af2e-4c66956bbbd9@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client ab5e0bb8-b13b-a260-6f2f-43f5140a8936@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 3d0c82c4-b23d-b612-d545-241f6cb757c3@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7fa8c599-dd15-96c4-d885-3c3e06ba5b08@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 7239fb20-3cfb-8764-45b0-b6f643b05794@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 5f376652-fea1-874f-2cb3-d18279dd914d@<unknown>
Jun 19 19:37:23 weisshorn02 kernel: LustreError: 47202:0:(genops.c:1310:class_disconnect_stale_exports()) scratch-MDT0000: disconnect stale client 65051446-0d37-b90f-bdbd-a97ef025c85e@<unknown>
Jun 19 19:37:27 weisshorn12 kernel: LustreError: 137-5: scratch-OST0044: Not available for connect from 148.187.6.241@o2ib2 (no target)
Jun 19 19:37:27 weisshorn12 kernel: LustreError: Skipped 1767 previous similar messages
Jun 19 19:37:28 weisshorn02 kernel: Lustre: scratch-MDT0000: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting, waiting for 1547 clients in recovery for 0:54
Jun 19 19:37:28 weisshorn02 kernel: Lustre: Skipped 152 previous similar messages
Jun 19 19:37:28 weisshorn02 kernel: Lustre: scratch-MDT0000: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) refused reconnection, still busy with 16 active RPCs
Jun 19 19:37:28 weisshorn02 kernel: Lustre: Skipped 84 previous similar messages
Jun 19 19:37:35 weisshorn02 kernel: Lustre: scratch-MDT0000: Denying connection for new client 148.187.6.7@o2ib2 (at 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7), waiting for 299 clients in recovery for 0:47
Jun 19 19:37:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:37:40 weisshorn02 kernel: Lustre: 2871:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for sent delay: [sent 1340127434/real 0]  req@ffff88039b0c0000 x1403849754838280/t0(0) o8->scratch-OST0044-osc-MDT0000@148.187.7.111@o2ib2:28/4 lens 368/512 e 0 to 1 dl 1340127460 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:37:40 weisshorn02 kernel: Lustre: 2871:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 22 previous similar messages
Jun 19 19:37:40 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127410/real 1340127460]  req@ffff88101114c000 x1403849871423945/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127466 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:37:40 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:37:41 weisshorn04 rhnsd[125184]: running program /usr/sbin/rhn_check
Jun 19 19:37:44 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127414/real 1340127464]  req@ffff880a650e6400 x1403849943173096/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127470 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:37:44 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:37:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:37:51 weisshorn02 kernel: LustreError: 47466:0:(mdt_open.c:1293:mdt_reint_open()) @@@ OPEN & CREAT not in open replay.  req@ffff880770895050 x1403846434859166/t0(52472524841) o101->268553ff-85a4-ef84-c18a-ee1ecc0782b7@309@gni:0/0 lens 2048/1656 e 0 to 0 dl 1340127628 ref 1 fl Interpret:/4/0 rc 0/0
Jun 19 19:37:51 weisshorn02 kernel: LustreError: 47466:0:(mdt_open.c:1293:mdt_reint_open()) Skipped 2042 previous similar messages
Jun 19 19:37:54 weisshorn12 kernel: LustreError: 127046:0:(obd_mount.c:1109:server_register_target()) Cannot talk to the MGS: -5, not fatal
Jun 19 19:38:00 weisshorn02 kernel: Lustre: scratch-MDT0000: Denying connection for new client 148.187.6.7@o2ib2 (at 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7), waiting for 11 clients in recovery for 2:31
Jun 19 19:38:00 weisshorn12 kernel: Lustre: 127118:0:(ldlm_lib.c:2062:target_recovery_init()) RECOVERY: service scratch-OST0044, 1548 recoverable clients, last_transno 46151594
Jun 19 19:38:00 weisshorn12 kernel: Lustre: scratch-OST0044: Now serving scratch-OST0044 on /dev/mapper/Lun10 with recovery enabled
Jun 19 19:38:02 weisshorn02 kernel: Lustre: scratch-MDT0000: Client 20a82831-c862-76dd-c58f-f636b44d13aa (at 148.187.6.201@o2ib2) reconnecting, waiting for 1547 clients in recovery for 2:30
Jun 19 19:38:02 weisshorn02 kernel: Lustre: Skipped 404 previous similar messages
Jun 19 19:38:02 weisshorn02 kernel: Lustre: scratch-MDT0000: Client 20a82831-c862-76dd-c58f-f636b44d13aa (at 148.187.6.201@o2ib2) refused reconnection, still busy with 1 active RPCs
Jun 19 19:38:02 weisshorn02 kernel: Lustre: Skipped 42 previous similar messages
Jun 19 19:38:04 weisshorn12 kernel: Lustre: scratch-OST0044: Temporarily refusing client connection from 148.187.7.102@o2ib2
Jun 19 19:38:04 weisshorn12 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 19:38:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:38:06 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127436/real 1340127486]  req@ffff88102827f800 x1403849861583808/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127492 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:38:06 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:38:07 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127437/real 1340127487]  req@ffff881025643000 x1403849921463331/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127493 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:38:07 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:38:12 weisshorn12 Filesystem[126981]: INFO:  Success
Jun 19 19:38:12 weisshorn12 mach_down[125619]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
Jun 19 19:38:12 weisshorn12 mach_down[125619]: info: mach_down takeover complete for node weisshorn11.admin.cscs.ch.
Jun 19 19:38:12 weisshorn12 heartbeat: [5359]: info: mach_down takeover complete.
Jun 19 19:38:14 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127444/real 1340127494]  req@ffff88101382a000 x1403849984283311/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127500 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:38:14 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Jun 19 19:38:14 weisshorn12 kernel: Lustre: scratch-OST0044: Will be in recovery for at least 5:00, or until 1548 clients reconnect
Jun 19 19:38:16 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127446/real 1340127496]  req@ffff880931122c00 x1403849636378183/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127502 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:38:16 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:38:19 weisshorn02 kernel: Lustre: scratch-MDT0000: Recovery over after 5:56, of 1547 clients 735 recovered and 812 were evicted.
Jun 19 19:38:20 weisshorn02 kernel: Lustre: 47202:0:(mds_lov.c:1028:mds_notify()) MDS mdd_obd-scratch-MDT0000: in recovery, not resetting orphans on scratch-OST0000_UUID
Jun 19 19:38:20 weisshorn02 kernel: Lustre: 47202:0:(mds_lov.c:1028:mds_notify()) Skipped 2 previous similar messages
Jun 19 19:38:20 weisshorn09 kernel: Lustre: scratch-OST0000: received MDS connection from 148.187.7.102@o2ib2
Jun 19 19:38:20 weisshorn09 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 19:38:20 weisshorn09 kernel: Lustre: 8407:0:(lustre_log.h:473:llog_group_set_export()) scratch-OST0000: export for group 0 is changed: 0xffff8810189b5c00 -> 0xffff880688ac0800
Jun 19 19:38:20 weisshorn09 kernel: Lustre: 8407:0:(lustre_log.h:473:llog_group_set_export()) Skipped 3 previous similar messages
Jun 19 19:38:20 weisshorn09 kernel: Lustre: 8407:0:(llog_net.c:168:llog_receptor_accept()) changing the import ffff880a83273800 - ffff8801cd129000
Jun 19 19:38:20 weisshorn09 kernel: Lustre: 8407:0:(llog_net.c:168:llog_receptor_accept()) Skipped 11 previous similar messages
Jun 19 19:38:20 weisshorn13 kernel: Lustre: scratch-OST0005: received MDS connection from 148.187.7.102@o2ib2
Jun 19 19:38:20 weisshorn13 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 19:38:20 weisshorn13 kernel: Lustre: 8465:0:(lustre_log.h:473:llog_group_set_export()) scratch-OST0005: export for group 0 is changed: 0xffff88080e5e8c00 -> 0xffff881010fba400
Jun 19 19:38:20 weisshorn13 kernel: Lustre: 8465:0:(lustre_log.h:473:llog_group_set_export()) Skipped 5 previous similar messages
Jun 19 19:38:20 weisshorn13 kernel: Lustre: 8465:0:(llog_net.c:168:llog_receptor_accept()) changing the import ffff8802935e5000 - ffff880c2743c000
Jun 19 19:38:20 weisshorn13 kernel: Lustre: 8465:0:(llog_net.c:168:llog_receptor_accept()) Skipped 11 previous similar messages
Jun 19 19:38:20 weisshorn12 kernel: Lustre: scratch-OST0008: received MDS connection from 148.187.7.102@o2ib2
Jun 19 19:38:20 weisshorn12 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 19:38:20 weisshorn12 kernel: Lustre: 8419:0:(lustre_log.h:473:llog_group_set_export()) scratch-OST000f: export for group 0 is changed: 0xffff881024af6000 -> 0xffff880b1607fc00
Jun 19 19:38:20 weisshorn12 kernel: Lustre: 8419:0:(lustre_log.h:473:llog_group_set_export()) Skipped 3 previous similar messages
Jun 19 19:38:20 weisshorn12 kernel: Lustre: 8419:0:(llog_net.c:168:llog_receptor_accept()) changing the import ffff880c052fc000 - ffff880d5f210000
Jun 19 19:38:20 weisshorn12 kernel: Lustre: 8419:0:(llog_net.c:168:llog_receptor_accept()) Skipped 11 previous similar messages
Jun 19 19:38:20 weisshorn10 kernel: Lustre: scratch-OST0001: received MDS connection from 148.187.7.102@o2ib2
Jun 19 19:38:20 weisshorn10 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:38:20 weisshorn03 kernel: Lustre: scratch-OST0009: received MDS connection from 148.187.7.102@o2ib2
Jun 19 19:38:20 weisshorn03 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 19:38:20 weisshorn03 kernel: Lustre: 7751:0:(lustre_log.h:473:llog_group_set_export()) scratch-OST0009: export for group 0 is changed: 0xffff880809e54c00 -> 0xffff881013bb6000
Jun 19 19:38:20 weisshorn10 kernel: Lustre: 8733:0:(lustre_log.h:473:llog_group_set_export()) scratch-OST0001: export for group 0 is changed: 0xffff880e6ef5c800 -> 0xffff880daf89c000
Jun 19 19:38:20 weisshorn10 kernel: Lustre: 8733:0:(lustre_log.h:473:llog_group_set_export()) Skipped 1 previous similar message
Jun 19 19:38:20 weisshorn10 kernel: Lustre: 8733:0:(llog_net.c:168:llog_receptor_accept()) changing the import ffff880bab2b9800 - ffff880a1e65c800
Jun 19 19:38:20 weisshorn03 kernel: Lustre: 7751:0:(llog_net.c:168:llog_receptor_accept()) changing the import ffff880860ef4000 - ffff880c63fd0800
Jun 19 19:38:20 weisshorn03 kernel: Lustre: 7751:0:(llog_net.c:168:llog_receptor_accept()) Skipped 11 previous similar messages
Jun 19 19:38:20 weisshorn10 kernel: Lustre: 8733:0:(llog_net.c:168:llog_receptor_accept()) Skipped 6 previous similar messages
Jun 19 19:38:20 weisshorn04 kernel: Lustre: scratch-OST000d: received MDS connection from 148.187.7.102@o2ib2
Jun 19 19:38:20 weisshorn07 kernel: Lustre: scratch-OST0006: received MDS connection from 148.187.7.102@o2ib2
Jun 19 19:38:20 weisshorn04 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 19:38:20 weisshorn04 kernel: Lustre: 5914:0:(lustre_log.h:473:llog_group_set_export()) scratch-OST000d: export for group 0 is changed: 0xffff881027d1c000 -> 0xffff880d753f0000
Jun 19 19:38:20 weisshorn07 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 19:38:20 weisshorn07 kernel: Lustre: 6346:0:(lustre_log.h:473:llog_group_set_export()) scratch-OST0006: export for group 0 is changed: 0xffff881006591000 -> 0xffff8801cc04a400
Jun 19 19:38:20 weisshorn07 kernel: Lustre: 6346:0:(llog_net.c:168:llog_receptor_accept()) changing the import ffff8800b3a0c000 - ffff88011ef88800
Jun 19 19:38:20 weisshorn07 kernel: Lustre: 6346:0:(llog_net.c:168:llog_receptor_accept()) Skipped 11 previous similar messages
Jun 19 19:38:20 weisshorn08 kernel: Lustre: scratch-OST0002: received MDS connection from 148.187.7.102@o2ib2
Jun 19 19:38:20 weisshorn08 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:38:20 weisshorn08 kernel: Lustre: 8276:0:(lustre_log.h:473:llog_group_set_export()) scratch-OST0002: export for group 0 is changed: 0xffff8810141a3800 -> 0xffff880c11f6d400
Jun 19 19:38:20 weisshorn08 kernel: Lustre: 8276:0:(lustre_log.h:473:llog_group_set_export()) Skipped 3 previous similar messages
Jun 19 19:38:20 weisshorn08 kernel: Lustre: 8276:0:(llog_net.c:168:llog_receptor_accept()) changing the import ffff880aa6be4000 - ffff8809d3814800
Jun 19 19:38:20 weisshorn08 kernel: Lustre: 8276:0:(llog_net.c:168:llog_receptor_accept()) Skipped 3 previous similar messages
Jun 19 19:38:20 weisshorn05 kernel: Lustre: scratch-OST001b: received MDS connection from 148.187.7.102@o2ib2
Jun 19 19:38:20 weisshorn05 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 19:38:20 weisshorn05 kernel: Lustre: 7819:0:(lustre_log.h:473:llog_group_set_export()) scratch-OST001b: export for group 0 is changed: 0xffff881012070c00 -> 0xffff881014e7dc00
Jun 19 19:38:20 weisshorn05 kernel: Lustre: 7819:0:(llog_net.c:168:llog_receptor_accept()) changing the import ffff880626613800 - ffff880e6efeb800
Jun 19 19:38:20 weisshorn05 kernel: Lustre: 7819:0:(llog_net.c:168:llog_receptor_accept()) Skipped 5 previous similar messages
Jun 19 19:38:20 weisshorn04 kernel: Lustre: 5914:0:(llog_net.c:168:llog_receptor_accept()) changing the import ffff880c955a0000 - ffff880d76e6c000
Jun 19 19:38:20 weisshorn04 kernel: Lustre: 5914:0:(llog_net.c:168:llog_receptor_accept()) Skipped 11 previous similar messages
Jun 19 19:38:20 weisshorn06 kernel: Lustre: scratch-OST0007: received MDS connection from 148.187.7.102@o2ib2
Jun 19 19:38:20 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:38:20 weisshorn06 kernel: Lustre: 7814:0:(lustre_log.h:473:llog_group_set_export()) scratch-OST0007: export for group 0 is changed: 0xffff88080dbc5800 -> 0xffff8808173cdc00
Jun 19 19:38:20 weisshorn06 kernel: Lustre: 7814:0:(lustre_log.h:473:llog_group_set_export()) Skipped 1 previous similar message
Jun 19 19:38:20 weisshorn06 kernel: Lustre: 7814:0:(llog_net.c:168:llog_receptor_accept()) changing the import ffff880b31651000 - ffff8804717bc000
Jun 19 19:38:20 weisshorn06 kernel: Lustre: 7814:0:(llog_net.c:168:llog_receptor_accept()) Skipped 3 previous similar messages
Jun 19 19:38:20 weisshorn03 kernel: Lustre: 7751:0:(lustre_log.h:473:llog_group_set_export()) scratch-OST0020: export for group 0 is changed: 0xffff88101728e800 -> 0xffff881025b7d000
Jun 19 19:38:20 weisshorn03 kernel: Lustre: 7751:0:(lustre_log.h:473:llog_group_set_export()) Skipped 1 previous similar message
Jun 19 19:38:20 weisshorn09 kernel: Lustre: 8407:0:(lustre_log.h:473:llog_group_set_export()) scratch-OST0034: export for group 0 is changed: 0xffff8810272ea800 -> 0xffff880abe063400
Jun 19 19:38:20 weisshorn09 kernel: Lustre: 8407:0:(lustre_log.h:473:llog_group_set_export()) Skipped 3 previous similar messages
Jun 19 19:38:20 weisshorn13 kernel: Lustre: 8480:0:(lustre_log.h:473:llog_group_set_export()) scratch-OST0035: export for group 0 is changed: 0xffff88080c052400 -> 0xffff880b01048000
Jun 19 19:38:20 weisshorn12 kernel: Lustre: 8419:0:(lustre_log.h:473:llog_group_set_export()) scratch-OST0032: export for group 0 is changed: 0xffff88100be3e000 -> 0xffff88050b6fb000
Jun 19 19:38:20 weisshorn13 kernel: Lustre: 8480:0:(lustre_log.h:473:llog_group_set_export()) Skipped 13 previous similar messages
Jun 19 19:38:20 weisshorn12 kernel: Lustre: 8419:0:(lustre_log.h:473:llog_group_set_export()) Skipped 3 previous similar messages
Jun 19 19:38:20 weisshorn04 kernel: Lustre: 5897:0:(lustre_log.h:473:llog_group_set_export()) scratch-OST0031: export for group 0 is changed: 0xffff8810232fb800 -> 0xffff88101b404800
Jun 19 19:38:20 weisshorn06 kernel: Lustre: 7659:0:(lustre_log.h:473:llog_group_set_export()) scratch-OST0036: export for group 0 is changed: 0xffff880819e21400 -> 0xffff8810110b8400
Jun 19 19:38:20 weisshorn04 kernel: Lustre: 5897:0:(lustre_log.h:473:llog_group_set_export()) Skipped 3 previous similar messages
Jun 19 19:38:20 weisshorn06 kernel: Lustre: 7659:0:(lustre_log.h:473:llog_group_set_export()) Skipped 7 previous similar messages
Jun 19 19:38:20 weisshorn05 kernel: Lustre: 7768:0:(lustre_log.h:473:llog_group_set_export()) scratch-OST0027: export for group 0 is changed: 0xffff88101275f400 -> 0xffff880b2f975000
Jun 19 19:38:20 weisshorn05 kernel: Lustre: 7768:0:(lustre_log.h:473:llog_group_set_export()) Skipped 3 previous similar messages
Jun 19 19:38:20 weisshorn08 kernel: Lustre: 8276:0:(filter.c:2555:filter_llog_connect()) scratch-OST0002: Recovery from log 0x1400004/0x0:b3e57b05
Jun 19 19:38:20 weisshorn08 kernel: Lustre: 8276:0:(filter.c:2555:filter_llog_connect()) Skipped 3 previous similar messages
Jun 19 19:38:20 weisshorn09 kernel: Lustre: 8362:0:(filter.c:2555:filter_llog_connect()) scratch-OST0000: Recovery from log 0x1400002/0x0:b3e57b03
Jun 19 19:38:20 weisshorn09 kernel: Lustre: 8362:0:(filter.c:2555:filter_llog_connect()) Skipped 1 previous similar message
Jun 19 19:38:20 weisshorn07 kernel: Lustre: 6346:0:(filter.c:2555:filter_llog_connect()) scratch-OST0006: Recovery from log 0x1400008/0x0:b3e57b09
Jun 19 19:38:20 weisshorn07 kernel: Lustre: 6346:0:(filter.c:2555:filter_llog_connect()) Skipped 2 previous similar messages
Jun 19 19:38:20 weisshorn04 kernel: Lustre: 8312:0:(filter.c:2555:filter_llog_connect()) scratch-OST000d: Recovery from log 0x140000f/0x0:b3e57b10
Jun 19 19:38:20 weisshorn04 kernel: Lustre: 8312:0:(filter.c:2555:filter_llog_connect()) Skipped 3 previous similar messages
Jun 19 19:38:20 weisshorn10 kernel: Lustre: 8733:0:(filter.c:2555:filter_llog_connect()) scratch-OST0016: Recovery from log 0x1400018/0x0:b3e57b19
Jun 19 19:38:20 weisshorn03 kernel: Lustre: 7751:0:(filter.c:2555:filter_llog_connect()) scratch-OST0020: Recovery from log 0x1400022/0x0:b3e57b23
Jun 19 19:38:20 weisshorn10 kernel: Lustre: 8733:0:(filter.c:2555:filter_llog_connect()) Skipped 4 previous similar messages
Jun 19 19:38:20 weisshorn03 kernel: Lustre: 7751:0:(filter.c:2555:filter_llog_connect()) Skipped 1 previous similar message
Jun 19 19:38:20 weisshorn02 kernel: Lustre: MDS mdd_obd-scratch-MDT0000: scratch-OST0002_UUID now active, resetting orphans
Jun 19 19:38:20 weisshorn06 kernel: Lustre: 7814:0:(filter.c:2555:filter_llog_connect()) scratch-OST0007: Recovery from log 0x1400009/0x0:b3e57b0a
Jun 19 19:38:20 weisshorn06 kernel: Lustre: 7814:0:(filter.c:2555:filter_llog_connect()) Skipped 3 previous similar messages
Jun 19 19:38:20 weisshorn12 kernel: Lustre: 8419:0:(filter.c:2555:filter_llog_connect()) scratch-OST0008: Recovery from log 0x140000a/0x0:b3e57b0b
Jun 19 19:38:20 weisshorn12 kernel: Lustre: 8419:0:(filter.c:2555:filter_llog_connect()) Skipped 5 previous similar messages
Jun 19 19:38:20 weisshorn13 kernel: Lustre: 6266:0:(filter.c:2555:filter_llog_connect()) scratch-OST0011: Recovery from log 0x1400013/0x0:b3e57b14
Jun 19 19:38:20 weisshorn05 kernel: Lustre: 7768:0:(filter.c:2555:filter_llog_connect()) scratch-OST0033: Recovery from log 0x1400035/0x0:b3e57b36
Jun 19 19:38:20 weisshorn13 kernel: Lustre: 6266:0:(filter.c:2555:filter_llog_connect()) Skipped 5 previous similar messages
Jun 19 19:38:20 weisshorn05 kernel: Lustre: 7768:0:(filter.c:2555:filter_llog_connect()) Skipped 5 previous similar messages
Jun 19 19:38:20 weisshorn02 kernel: Lustre: MDS mdd_obd-scratch-MDT0000: scratch-OST0008_UUID now active, resetting orphans
Jun 19 19:38:20 weisshorn02 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 19:38:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:38:27 weisshorn02 kernel: Lustre: scratch-MDT0000: Client 20a82831-c862-76dd-c58f-f636b44d13aa (at 148.187.6.201@o2ib2) reconnecting
Jun 19 19:38:28 weisshorn02 kernel: Lustre: scratch-MDT0000: Client 86930980-35a2-c223-d318-db1f36e1b760 (at 148.187.6.222@o2ib2) reconnecting
Jun 19 19:38:28 weisshorn02 kernel: Lustre: scratch-MDT0000: Client 77ed2fd6-7ad2-3010-3442-924b80b07058 (at 148.187.6.8@o2ib2) reconnecting
Jun 19 19:38:29 weisshorn02 kernel: Lustre: scratch-MDT0000-mdc-ffff88081e6aac00: Connection restored to scratch-MDT0000 (at 0@lo)
Jun 19 19:38:29 weisshorn02 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:38:33 weisshorn02 kernel: Lustre: scratch-MDT0000: Client 7eefd778-9608-3e05-4cbf-22a2da6f8087 (at 148.187.6.242@o2ib2) reconnecting
Jun 19 19:38:33 weisshorn02 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:38:35 weisshorn02 kernel: Lustre: scratch-MDT0000: Client e16602a1-c1af-d220-e874-6fd0602c7df1 (at 148.187.6.205@o2ib2) reconnecting
Jun 19 19:38:35 weisshorn02 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:38:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:38:37 weisshorn14 kernel: Lustre: scratch-MDT0000-mdc-ffff8810296e4c00: Connection restored to scratch-MDT0000 (at 148.187.7.102@o2ib2)
Jun 19 19:38:37 weisshorn14 kernel: Lustre: 14313:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127467/real 1340127517]  req@ffff881014871800 x1404749288342281/t0(0) o8->scratch-OST0044-osc-ffff8810296e4c00@148.187.7.111@o2ib2:28/4 lens 368/512 e 0 to 1 dl 1340127518 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:38:37 weisshorn14 kernel: Lustre: 14313:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
Jun 19 19:38:38 weisshorn02 kernel: Lustre: MGS: haven't heard from client 02e8b9ff-fc43-5841-d3bb-815052085829 (at 148.187.6.7@o2ib2) in 213 seconds. I think it's dead, and I am evicting it. exp ffff8807a9884400, cur 1340127518 expire 1340127368 last 1340127305
Jun 19 19:38:43 weisshorn02 kernel: Lustre: scratch-MDT0000: Client c479bd2d-64fa-98c9-8762-0b4c0acb04b0 (at 148.187.6.202@o2ib2) reconnecting
Jun 19 19:38:43 weisshorn02 kernel: Lustre: Skipped 9 previous similar messages
Jun 19 19:38:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:39:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:39:06 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127496/real 1340127546]  req@ffff880dc6950400 x1403849897065438/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127552 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:39:06 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:39:08 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127498/real 1340127548]  req@ffff88092caabc00 x1403849873212090/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127554 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:39:08 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:39:11 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127501/real 1340127551]  req@ffff880d67d71800 x1403849822728789/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127557 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:39:11 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Jun 19 19:39:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:39:22 weisshorn02 kernel: Lustre: scratch-MDT0000: Client 6782dca5-a693-8435-8f30-85ccc6fb0c14 (at 313@gni) reconnecting
Jun 19 19:39:22 weisshorn02 kernel: Lustre: Skipped 1 previous similar message
Jun 19 19:39:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:39:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:39:52 weisshorn02 kernel: Lustre: scratch-MDT0000: Client 2ef4c6d2-6d28-279f-97af-9079fa3fa2bb (at 687@gni) reconnecting
Jun 19 19:39:52 weisshorn02 kernel: Lustre: Skipped 59 previous similar messages
Jun 19 19:40:01 weisshorn07 CROND[126010]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:40:01 weisshorn03 CROND[126515]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:40:01 weisshorn05 CROND[125820]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:40:01 weisshorn06 CROND[126754]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:40:01 weisshorn10 CROND[125845]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:40:01 weisshorn09 CROND[125539]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:40:01 weisshorn14 CROND[54411]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:40:01 weisshorn08 CROND[124598]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:40:01 weisshorn12 CROND[127363]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:40:01 weisshorn13 CROND[129433]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:40:01 weisshorn04 CROND[125283]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:40:01 weisshorn02 CROND[47729]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:40:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:40:13 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127563/real 1340127613]  req@ffff8805efa26000 x1403849952685240/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127619 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:40:13 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Jun 19 19:40:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:40:29 weisshorn12 kernel: Lustre: 7990:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Next recovery transno: 46151595, current: 46151607, replaying
Jun 19 19:40:29 weisshorn12 kernel: Lustre: 7990:0:(ldlm_lib.c:2183:target_queue_recovery_request()) Skipped 5 previous similar messages
Jun 19 19:40:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:40:42 weisshorn04 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 334@gni  ns: filter-scratch-OST003d_UUID lock: ffff8808c0ab2000/0x6607ebb0f2f057f6 lrc: 3/0,0 mode: PW/PW res: 5728756/0 rrc: 36 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 remote: 0x58a1d8c789bc2723 expref: 4 pid: 8155 timeout 5607249296
Jun 19 19:40:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:40:51 weisshorn02 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 371s: evicting client at 290@gni  ns: mdt-ffff8807fe19e000 lock: ffff880c0bedb240/0xb92b74c12b1c73f8 lrc: 3/0,0 mode: PR/PR res: 8589951862/23897 bits 0x3 rrc: 2 type: IBT flags: 0x20 remote: 0xc32353092937bc7f expref: 15 pid: 47422 timeout: 5606916025
Jun 19 19:40:52 weisshorn02 kernel: Lustre: scratch-OST0044-osc-MDT0000: Connection restored to scratch-OST0044 (at 148.187.7.112@o2ib2)
Jun 19 19:40:52 weisshorn12 kernel: Lustre: scratch-OST0044: received MDS connection from 148.187.7.102@o2ib2
Jun 19 19:40:52 weisshorn12 kernel: Lustre: Skipped 10 previous similar messages
Jun 19 19:40:52 weisshorn12 kernel: Lustre: 8109:0:(filter.c:2555:filter_llog_connect()) scratch-OST0044: Recovery from log 0x1400046/0x0:b3e57b47
Jun 19 19:40:52 weisshorn12 kernel: Lustre: 8109:0:(filter.c:2555:filter_llog_connect()) Skipped 10 previous similar messages
Jun 19 19:40:52 weisshorn02 kernel: Lustre: MDS mdd_obd-scratch-MDT0000: scratch-OST0044_UUID now active, resetting orphans
Jun 19 19:40:52 weisshorn02 kernel: Lustre: Skipped 64 previous similar messages
Jun 19 19:40:52 weisshorn14 kernel: Lustre: scratch-OST0044-osc-ffff8810296e4c00: Connection restored to scratch-OST0044 (at 148.187.7.112@o2ib2)
Jun 19 19:40:52 weisshorn12 kernel: Lustre: scratch-OST0044: Recovery over after 2:38, of 1548 clients 1548 recovered and 0 were evicted.
Jun 19 19:40:58 weisshorn02 kernel: LustreError: 47725:0:(mdt_handler.c:2822:mdt_recovery()) operation 101 on unconnected MDS from 12345-290@gni
Jun 19 19:41:01 weisshorn14 nslcd[3437]: [c0e823] ldap_result() failed: Can't contact LDAP server
Jun 19 19:41:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:41:12 weisshorn02 kernel: Lustre: 47721:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d8:0xbe7b:0x0]] [valid 68719476736]
Jun 19 19:41:12 weisshorn02 kernel: Lustre: 47501:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d8:0xbe7c:0x0]] [valid 68719476736]
Jun 19 19:41:13 weisshorn02 kernel: Lustre: 47538:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x2000046d8:0xbe7d:0x0]] [valid 68719476736]
Jun 19 19:41:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:41:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:41:47 weisshorn03 kernel: Lustre: scratch-OST002c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810150d8c00, cur 1340127707 expire 1340127557 last 1340127480
Jun 19 19:41:47 weisshorn03 kernel: Lustre: Skipped 7 previous similar messages
Jun 19 19:41:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:42:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:42:19 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127689/real 1340127739]  req@ffff880807f3f400 x1403849943194732/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127745 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:42:19 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Jun 19 19:42:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:42:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:42:37 weisshorn06 kernel: Lustre: scratch-OST0012: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809a8964c00, cur 1340127757 expire 1340127607 last 1340127530
Jun 19 19:42:37 weisshorn06 kernel: Lustre: Skipped 11 previous similar messages
Jun 19 19:42:40 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127710/real 1340127760]  req@ffff8810123eac00 x1403849871446551/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127766 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:42:40 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Jun 19 19:42:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:43:05 weisshorn02 kernel: igb: eth1 NIC Link is Up 10 Mbps Full Duplex, Flow Control: RX/TX
Jun 19 19:43:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:43:06 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127736/real 1340127786]  req@ffff88101869ac00 x1403849861612140/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127792 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:43:06 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Jun 19 19:43:07 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127737/real 1340127787]  req@ffff880f3b569000 x1403849921491903/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127793 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:43:07 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Jun 19 19:43:13 weisshorn10 kernel: Lustre: scratch-OST0022: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880432004000, cur 1340127793 expire 1340127643 last 1340127566
Jun 19 19:43:13 weisshorn10 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 19:43:14 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127744/real 1340127794]  req@ffff88102a1e9c00 x1403849984361731/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127800 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:43:14 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Jun 19 19:43:16 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127746/real 1340127796]  req@ffff88081a177400 x1403849636419514/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127802 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:43:16 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Jun 19 19:43:18 weisshorn02 kernel: igb: eth1 NIC Link is Down
Jun 19 19:43:20 weisshorn02 kernel: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jun 19 19:43:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:43:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:43:38 weisshorn09 kernel: Lustre: scratch-OST0034: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88081886e400, cur 1340127818 expire 1340127668 last 1340127591
Jun 19 19:43:38 weisshorn09 kernel: Lustre: Skipped 12 previous similar messages
Jun 19 19:43:41 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127771/real 1340127821]  req@ffff880817334800 x1403849897115113/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:43:41 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Jun 19 19:43:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:43:52 weisshorn02 kernel: Lustre: scratch-MDT0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88040e83d400, cur 1340127832 expire 1340127682 last 1340127605
Jun 19 19:44:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:44:08 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127798/real 1340127848]  req@ffff881026cddc00 x1403849873274927/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127854 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:44:08 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Jun 19 19:44:11 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127801/real 1340127851]  req@ffff880daef06c00 x1403849822779770/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127857 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:44:11 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Jun 19 19:44:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:44:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:44:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:45:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:45:13 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has failed due to network error: [sent 1340127863/real 1340127913]  req@ffff880239f51800 x1403849952848779/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340127919 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:45:13 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Jun 19 19:45:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:45:26 weisshorn02 kernel: igb: eth1 NIC Link is Down
Jun 19 19:45:32 weisshorn04 kernel: Lustre: scratch-OST0025: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101ba23000, cur 1340127932 expire 1340127782 last 1340127705
Jun 19 19:45:32 weisshorn04 kernel: Lustre: Skipped 13 previous similar messages
Jun 19 19:45:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:45:36 weisshorn02 kernel: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0x97] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0x98] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x99] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x9a] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x9b] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x9c] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x9d] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x9e] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0x9f] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0xa0] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0xa1] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0xa2] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0xa3] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0xa4] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0xa5] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0xa6] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0xa7] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0xa8] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xa9] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xaa] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xab] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xac] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xad] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xae] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0xaf] disabled)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Jun 19 18:45:42 weisshorn01 kernel: ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
Jun 19 18:45:42 weisshorn01 kernel: ACPI: Skipping IOAPIC probe due to 'noapic' option.
Jun 19 18:45:42 weisshorn01 kernel: Using ACPI for processor (LAPIC) configuration information
Jun 19 18:45:42 weisshorn01 kernel: ACPI: HPET id: 0x8300 base: 0xfed00000
Jun 19 18:45:42 weisshorn01 kernel: Intel MultiProcessor Specification v1.4
Jun 19 18:45:42 weisshorn01 kernel: MPTABLE: OEM ID: AMD     
Jun 19 18:45:42 weisshorn01 kernel: MPTABLE: Product ID: RD890       
Jun 19 18:45:42 weisshorn01 kernel: MPTABLE: APIC at: 0xFEE00000
Jun 19 18:45:42 weisshorn01 kernel: I/O APIC #0 Version 33 at 0xFEC00000.
Jun 19 18:45:42 weisshorn01 kernel: I/O APIC #1 Version 33 at 0xFEC20000.
Jun 19 18:45:42 weisshorn01 kernel: Processors: 16
Jun 19 18:45:42 weisshorn01 kernel: SMP: Allowing 48 CPUs, 32 hotplug CPUs
Jun 19 18:45:42 weisshorn01 kernel: nr_irqs_gsi: 56
Jun 19 18:45:42 weisshorn01 kernel: PM: Registered nosave memory: 0000000000099000 - 000000000009a000
Jun 19 18:45:42 weisshorn01 kernel: PM: Registered nosave memory: 000000000009a000 - 00000000000a0000
Jun 19 18:45:42 weisshorn01 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000e6000
Jun 19 18:45:42 weisshorn01 kernel: PM: Registered nosave memory: 00000000000e6000 - 0000000000100000
Jun 19 18:45:42 weisshorn01 kernel: PM: Registered nosave memory: 00000000d7e60000 - 00000000d7e6e000
Jun 19 18:45:42 weisshorn01 kernel: PM: Registered nosave memory: 00000000d7e6e000 - 00000000d7e70000
Jun 19 18:45:42 weisshorn01 kernel: PM: Registered nosave memory: 00000000d7e70000 - 00000000d7e88000
Jun 19 18:45:42 weisshorn01 kernel: PM: Registered nosave memory: 00000000d7e88000 - 00000000d7ec0000
Jun 19 18:45:42 weisshorn01 kernel: PM: Registered nosave memory: 00000000d7ec0000 - 00000000d7ee0000
Jun 19 18:45:42 weisshorn01 kernel: PM: Registered nosave memory: 00000000d7ee0000 - 00000000d7eed000
Jun 19 18:45:42 weisshorn01 kernel: PM: Registered nosave memory: 00000000d7eed000 - 00000000d8000000
Jun 19 18:45:42 weisshorn01 kernel: PM: Registered nosave memory: 00000000d8000000 - 00000000ffe00000
Jun 19 18:45:42 weisshorn01 kernel: PM: Registered nosave memory: 00000000ffe00000 - 0000000100000000
Jun 19 18:45:42 weisshorn01 kernel: Allocating PCI resources starting at d8000000 (gap: d8000000:27e00000)
Jun 19 18:45:42 weisshorn01 kernel: Booting paravirtualized kernel on bare hardware
Jun 19 18:45:42 weisshorn01 kernel: NR_CPUS:4096 nr_cpumask_bits:48 nr_cpu_ids:48 nr_node_ids:4
Jun 19 18:45:42 weisshorn01 kernel: PERCPU: Embedded 30 pages/cpu @ffff880028200000 s92696 r8192 d21992 u131072
Jun 19 18:45:42 weisshorn01 kernel: pcpu-alloc: s92696 r8192 d21992 u131072 alloc=1*2097152
Jun 19 18:45:42 weisshorn01 kernel: pcpu-alloc: [0] 00 01 02 03 16 20 24 28 32 36 40 44 -- -- -- -- 
Jun 19 18:45:42 weisshorn01 kernel: pcpu-alloc: [1] 04 05 06 07 17 21 25 29 33 37 41 45 -- -- -- -- 
Jun 19 18:45:42 weisshorn01 kernel: pcpu-alloc: [2] 08 09 10 11 18 22 26 30 34 38 42 46 -- -- -- -- 
Jun 19 18:45:42 weisshorn01 kernel: pcpu-alloc: [3] 12 13 14 15 19 23 27 31 35 39 43 47 -- -- -- -- 
Jun 19 18:45:42 weisshorn01 kernel: Built 4 zonelists in Zone order, mobility grouping on.  Total pages: 15510687
Jun 19 18:45:42 weisshorn01 kernel: Policy zone: Normal
Jun 19 18:45:42 weisshorn01 kernel: Kernel command line: ro root=UUID=006606c4-be64-49f9-8c6b-39011b641e68 rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 noapic crashkernel=auto rhgb quiet rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb crashkernel=132M@0M rd_NO_LVM rd_NO_DM rdloaddriver=scsi_dh_rdac
Jun 19 18:45:42 weisshorn01 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Jun 19 18:45:42 weisshorn01 kernel: Checking aperture...
Jun 19 18:45:42 weisshorn01 kernel: No AGP bridge found
Jun 19 18:45:42 weisshorn01 kernel: Node 0: aperture @ 20000000 size 32 MB
Jun 19 18:45:42 weisshorn01 kernel: Aperture pointing to e820 RAM. Ignoring.
Jun 19 18:45:42 weisshorn01 kernel: Your BIOS doesn't leave a aperture memory hole
Jun 19 18:45:42 weisshorn01 kernel: Please enable the IOMMU option in the BIOS setup
Jun 19 18:45:42 weisshorn01 kernel: This costs you 64 MB of RAM
Jun 19 18:45:42 weisshorn01 kernel: Mapping aperture over 65536 KB of RAM @ 20000000
Jun 19 18:45:42 weisshorn01 kernel: PM: Registered nosave memory: 0000000020000000 - 0000000024000000
Jun 19 18:45:42 weisshorn01 kernel: AMD-Vi disabled by default: pass amd_iommu=on to enable
Jun 19 18:45:42 weisshorn01 kernel: Memory: 61766408k/63569920k available (5085k kernel code, 657500k absent, 1146012k reserved, 7228k data, 1244k init)
Jun 19 18:45:42 weisshorn01 kernel: Hierarchical RCU implementation.
Jun 19 18:45:42 weisshorn01 kernel: NR_IRQS:33024 nr_irqs:1336
Jun 19 18:45:42 weisshorn01 kernel: Extended CMOS year: 2000
Jun 19 18:45:42 weisshorn01 kernel: Console: colour VGA+ 80x25
Jun 19 18:45:42 weisshorn01 kernel: console [tty0] enabled
Jun 19 18:45:42 weisshorn01 kernel: allocated 503316480 bytes of page_cgroup
Jun 19 18:45:42 weisshorn01 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups
Jun 19 18:45:42 weisshorn01 kernel: hpet clockevent registered
Jun 19 18:45:42 weisshorn01 kernel: Fast TSC calibration using PIT
Jun 19 18:45:42 weisshorn01 kernel: Detected 1999.951 MHz processor.
Jun 19 18:45:42 weisshorn01 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3999.90 BogoMIPS (lpj=1999951)
Jun 19 18:45:42 weisshorn01 kernel: pid_max: default: 49152 minimum: 384
Jun 19 18:45:42 weisshorn01 kernel: Security Framework initialized
Jun 19 18:45:42 weisshorn01 kernel: SELinux:  Initializing.
Jun 19 18:45:42 weisshorn01 kernel: SELinux:  Starting in permissive mode
Jun 19 18:45:42 weisshorn01 kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Jun 19 18:45:42 weisshorn01 kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Jun 19 18:45:42 weisshorn01 kernel: Mount-cache hash table entries: 256
Jun 19 18:45:42 weisshorn01 kernel: Initializing cgroup subsys ns
Jun 19 18:45:42 weisshorn01 kernel: Initializing cgroup subsys cpuacct
Jun 19 18:45:42 weisshorn01 kernel: Initializing cgroup subsys memory
Jun 19 18:45:42 weisshorn01 kernel: Initializing cgroup subsys devices
Jun 19 18:45:42 weisshorn01 kernel: Initializing cgroup subsys freezer
Jun 19 18:45:42 weisshorn01 kernel: Initializing cgroup subsys net_cls
Jun 19 18:45:42 weisshorn01 kernel: Initializing cgroup subsys blkio
Jun 19 18:45:42 weisshorn01 kernel: Initializing cgroup subsys perf_event
Jun 19 18:45:42 weisshorn01 kernel: tseg: 00d7f00000
Jun 19 18:45:42 weisshorn01 kernel: CPU: Physical Processor ID: 0
Jun 19 18:45:42 weisshorn01 kernel: CPU: Processor Core ID: 0
Jun 19 18:45:42 weisshorn01 kernel: mce: CPU supports 6 MCE banks
Jun 19 18:45:42 weisshorn01 kernel: ACPI: Core revision 20090903
Jun 19 18:45:42 weisshorn01 kernel: ACPI: setting ELCR to 0200 (from 0c80)
Jun 19 18:45:42 weisshorn01 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00
Jun 19 18:45:42 weisshorn01 kernel: ftrace: allocating 20786 entries in 82 pages
Jun 19 18:45:42 weisshorn01 kernel: Setting APIC routing to physical flat
Jun 19 18:45:42 weisshorn01 kernel: CPU0: AMD Opteron(tm) Processor 6128 stepping 01
Jun 19 18:45:42 weisshorn01 kernel: Performance Events: AMD PMU driver.
Jun 19 18:45:42 weisshorn01 kernel: ... version:                0
Jun 19 18:45:42 weisshorn01 kernel: ... bit width:              48
Jun 19 18:45:42 weisshorn01 kernel: ... generic registers:      4
Jun 19 18:45:42 weisshorn01 kernel: ... value mask:             0000ffffffffffff
Jun 19 18:45:42 weisshorn01 kernel: ... max period:             00007fffffffffff
Jun 19 18:45:42 weisshorn01 kernel: ... fixed-purpose events:   0
Jun 19 18:45:42 weisshorn01 kernel: ... event mask:             000000000000000f
Jun 19 18:45:42 weisshorn01 kernel: NMI watchdog enabled, takes one hw-pmu counter.
Jun 19 18:45:42 weisshorn01 kernel: Booting Node   0, Processors  #1 #2 #3 Ok.
Jun 19 18:45:42 weisshorn01 kernel: Booting Node   1, Processors  #4 #5 #6 #7 Ok.
Jun 19 18:45:42 weisshorn01 kernel: Booting Node   2, Processors  #8 #9 #10 #11 Ok.
Jun 19 18:45:42 weisshorn01 kernel: Booting Node   3, Processors  #12 #13 #14 #15
Jun 19 18:45:42 weisshorn01 kernel: Brought up 16 CPUs
Jun 19 18:45:42 weisshorn01 kernel: Total of 16 processors activated (64000.57 BogoMIPS).
Jun 19 18:45:42 weisshorn01 kernel: sizeof(vma)=200 bytes
Jun 19 18:45:42 weisshorn01 kernel: sizeof(page)=56 bytes
Jun 19 18:45:42 weisshorn01 kernel: sizeof(inode)=592 bytes
Jun 19 18:45:42 weisshorn01 kernel: sizeof(dentry)=192 bytes
Jun 19 18:45:42 weisshorn01 kernel: sizeof(ext3inode)=800 bytes
Jun 19 18:45:42 weisshorn01 kernel: sizeof(buffer_head)=104 bytes
Jun 19 18:45:42 weisshorn01 kernel: sizeof(skbuff)=232 bytes
Jun 19 18:45:42 weisshorn01 kernel: sizeof(task_struct)=2616 bytes
Jun 19 18:45:42 weisshorn01 kernel: devtmpfs: initialized
Jun 19 18:45:42 weisshorn01 kernel: regulator: core version 0.5
Jun 19 18:45:42 weisshorn01 kernel: NET: Registered protocol family 16
Jun 19 18:45:42 weisshorn01 kernel: node 0 link 3: io port [7000, dfff]
Jun 19 18:45:42 weisshorn01 kernel: node 2 link 3: io port [e000, efff]
Jun 19 18:45:42 weisshorn01 kernel: TOM: 00000000d8000000 aka 3456M
Jun 19 18:45:42 weisshorn01 kernel: Fam 10h mmconf [e0000000, efffffff]
Jun 19 18:45:42 weisshorn01 kernel: node 0 link 3: mmio [db000000, dfffffff]
Jun 19 18:45:42 weisshorn01 kernel: node 2 link 3: mmio [d8000000, daffffff]
Jun 19 18:45:42 weisshorn01 kernel: node 0 link 3: mmio [a0000, bffff]
Jun 19 18:45:42 weisshorn01 kernel: node 0 link 3: mmio [f0000000, ffffffff]
Jun 19 18:45:42 weisshorn01 kernel: TOM2: 0000000f28000000 aka 62080M
Jun 19 18:45:42 weisshorn01 kernel: bus: [00,04] on node 0 link 3
Jun 19 18:45:42 weisshorn01 kernel: bus: 00 index 0 io port: [0, dfff]
Jun 19 18:45:42 weisshorn01 kernel: bus: 00 index 1 io port: [f000, ffff]
Jun 19 18:45:42 weisshorn01 kernel: bus: 00 index 2 mmio: [db000000, dfffffff]
Jun 19 18:45:42 weisshorn01 kernel: bus: 00 index 3 mmio: [a0000, bffff]
Jun 19 18:45:42 weisshorn01 kernel: bus: 00 index 4 mmio: [f0000000, ffffffff]
Jun 19 18:45:42 weisshorn01 kernel: bus: 00 index 5 mmio: [f28000000, fcffffffff]
Jun 19 18:45:42 weisshorn01 kernel: bus: [40,41] on node 2 link 3
Jun 19 18:45:42 weisshorn01 kernel: bus: 40 index 0 io port: [e000, efff]
Jun 19 18:45:42 weisshorn01 kernel: bus: 40 index 1 mmio: [d8000000, daffffff]
Jun 19 18:45:42 weisshorn01 kernel: ACPI: bus type pci registered
Jun 19 18:45:42 weisshorn01 kernel: PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
Jun 19 18:45:42 weisshorn01 kernel: PCI: Not using MMCONFIG.
Jun 19 18:45:42 weisshorn01 kernel: PCI: Using configuration type 1 for base access
Jun 19 18:45:42 weisshorn01 kernel: PCI: Using configuration type 1 for extended access
Jun 19 18:45:42 weisshorn01 kernel: bio: create slab <bio-0> at 0
Jun 19 18:45:42 weisshorn01 kernel: ACPI: EC: Look up EC in DSDT
Jun 19 18:45:42 weisshorn01 kernel: ACPI Warning for \_SB_._OSC: Return type mismatch - found Integer, expected Buffer (20090903/nspredef-1006)
Jun 19 18:45:42 weisshorn01 kernel: \_SB_:_OSC evaluation returned wrong type
Jun 19 18:45:42 weisshorn01 kernel: _OSC request data:1 17 
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psargs-0359): [ECEN] Namespace lookup failure, AE_NOT_FOUND
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\] (Node ffffffff81fcd7b0), AE_NOT_FOUND
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (dswload-0659): [PCI0] Namespace lookup failure, AE_NOT_FOUND
Jun 19 18:45:42 weisshorn01 kernel: ACPI Exception: AE_NOT_FOUND, During name lookup/catalog (20090903/psloop-230)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\] (Node ffffffff81fcd7b0), AE_NOT_FOUND
Jun 19 18:45:42 weisshorn01 kernel: ACPI: Executed 5 blocks of module-level executable AML code
Jun 19 18:45:42 weisshorn01 kernel: ACPI: Interpreter enabled
Jun 19 18:45:42 weisshorn01 kernel: ACPI: (supports S0 S1 S4 S5)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: Using PIC for interrupt routing
Jun 19 18:45:42 weisshorn01 kernel: PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\PRID.P_D0._STA] (Node ffff880320782c18), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (uteval-0250): Method execution failed [\PRID.P_D0._STA] (Node ffff880320782c18), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\PRID.P_D1._STA] (Node ffff880b1e52cda8), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (uteval-0250): Method execution failed [\PRID.P_D1._STA] (Node ffff880b1e52cda8), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\SECD.S_D0._STA] (Node ffff880b1e52cec0), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (uteval-0250): Method execution failed [\SECD.S_D0._STA] (Node ffff880b1e52cec0), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\SECD.S_D1._STA] (Node ffff88031fe909c0), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (uteval-0250): Method execution failed [\SECD.S_D1._STA] (Node ffff88031fe909c0), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: PCI: MCFG area at e0000000 reserved in ACPI motherboard resources
Jun 19 18:45:42 weisshorn01 kernel: PCI: Using MMCONFIG at e0000000 - efffffff
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\PRID.P_D0._STA] (Node ffff880320782c18), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\PRID.P_D1._STA] (Node ffff880b1e52cda8), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\SECD.S_D0._STA] (Node ffff880b1e52cec0), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\SECD.S_D1._STA] (Node ffff88031fe909c0), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI: No dock devices found.
Jun 19 18:45:42 weisshorn01 kernel: HEST: Table parsing has been initialized.
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00)
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:02.0: PME# disabled
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:04.0: PME# supported from D0 D3hot D3cold
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:04.0: PME# disabled
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:0d.0: PME# supported from D0 D3hot D3cold
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:0d.0: PME# disabled
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:11.0: reg 10 io port: [0xb000-0xb007]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:11.0: reg 14 io port: [0xa000-0xa003]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:11.0: reg 18 io port: [0x9000-0x9007]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:11.0: reg 1c io port: [0x8000-0x8003]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:11.0: reg 20 io port: [0x7000-0x700f]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:11.0: reg 24 32bit mmio: [0xdeffe400-0xdeffe7ff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:11.0: set SATA to AHCI mode
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:12.0: reg 10 32bit mmio: [0xdeffa000-0xdeffafff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:12.1: reg 10 32bit mmio: [0xdeffb000-0xdeffbfff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:12.2: reg 10 32bit mmio: [0xdeffe800-0xdeffe8ff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:12.2: supports D1 D2
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:12.2: PME# supported from D0 D1 D2 D3hot
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:12.2: PME# disabled
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:13.0: reg 10 32bit mmio: [0xdeffc000-0xdeffcfff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:13.1: reg 10 32bit mmio: [0xdeffd000-0xdeffdfff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:13.2: reg 10 32bit mmio: [0xdeffec00-0xdeffecff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:13.2: supports D1 D2
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:13.2: PME# disabled
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:14.1: reg 10 io port: [0x00-0x07]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:14.1: reg 14 io port: [0x00-0x03]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:14.1: reg 18 io port: [0x00-0x07]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:14.1: reg 1c io port: [0x00-0x03]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:14.1: reg 20 io port: [0xff00-0xff0f]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:14.5: reg 10 32bit mmio: [0xdefff000-0xdeffffff]
Jun 19 18:45:42 weisshorn01 kernel: PCI: peer root bus 00 res updated from pci conf
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:04:00.0: reg 10 64bit mmio: [0xdff00000-0xdfffffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:04:00.0: reg 18 64bit mmio pref: [0xdc000000-0xdc7fffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:02.0: bridge 32bit mmio: [0xdff00000-0xdfffffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:02.0: bridge 64bit mmio pref: [0xdc000000-0xdc7fffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:03:00.0: reg 10 io port: [0xd000-0xd0ff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:03:00.0: reg 14 64bit mmio: [0xdfe3c000-0xdfe3ffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:03:00.0: reg 1c 64bit mmio: [0xdfe40000-0xdfe7ffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:03:00.0: reg 30 32bit mmio pref: [0xdfe80000-0xdfefffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:03:00.0: supports D1 D2
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:04.0: bridge io port: [0xd000-0xdfff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:04.0: bridge 32bit mmio: [0xdfe00000-0xdfefffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: reg 10 32bit mmio: [0xdfd60000-0xdfd7ffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: reg 14 32bit mmio: [0xdfd40000-0xdfd5ffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: reg 18 io port: [0xc400-0xc41f]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: reg 1c 32bit mmio: [0xdfd98000-0xdfd9bfff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: reg 30 32bit mmio pref: [0xdfd20000-0xdfd3ffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: PME# disabled
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: reg 184 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: reg 190 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.1: reg 10 32bit mmio: [0xdfde0000-0xdfdfffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.1: reg 14 32bit mmio: [0xdfdc0000-0xdfddffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.1: reg 18 io port: [0xc800-0xc81f]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.1: reg 1c 32bit mmio: [0xdfd9c000-0xdfd9ffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.1: reg 30 32bit mmio pref: [0xdfda0000-0xdfdbffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.1: PME# disabled
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.1: reg 184 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.1: reg 190 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:0d.0: bridge io port: [0xc000-0xcfff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:0d.0: bridge 32bit mmio: [0xdfd00000-0xdfdfffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:01:04.0: reg 10 32bit mmio pref: [0xdb000000-0xdbffffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:01:04.0: reg 14 32bit mmio: [0xdfcfc000-0xdfcfffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:01:04.0: reg 18 32bit mmio: [0xdf000000-0xdf7fffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:14.4: transparent bridge
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:14.4: bridge 32bit mmio: [0xdf000000-0xdfcfffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:14.4: bridge 32bit mmio pref: [0xdb000000-0xdbffffff]
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PC02._PRT]
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PC04._PRT]
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PC0D._PRT]
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0PC._PRT]
Jun 19 18:45:42 weisshorn01 kernel: pci0000:00: Requesting ACPI _OSC control (0x1d)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Root Bridge [PC40] (0000:40)
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:40:02.0: PME# supported from D0 D3hot D3cold
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:40:02.0: PME# disabled
Jun 19 18:45:42 weisshorn01 kernel: PCI: peer root bus 40 res updated from pci conf
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:41:00.0: reg 10 io port: [0xe400-0xe4ff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:41:00.0: reg 14 64bit mmio: [0xdaeb8000-0xdaebbfff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:41:00.0: reg 1c 64bit mmio: [0xdad00000-0xdadfffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:41:00.0: reg 30 32bit mmio pref: [0xdae40000-0xdae7ffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:41:00.1: reg 10 io port: [0xe800-0xe8ff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:41:00.1: reg 14 64bit mmio: [0xdaebc000-0xdaebffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:41:00.1: reg 1c 64bit mmio: [0xdaf00000-0xdaffffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:41:00.1: reg 30 32bit mmio pref: [0xdaec0000-0xdaefffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:40:02.0: bridge io port: [0xe000-0xefff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:40:02.0: bridge 32bit mmio: [0xdad00000-0xdaffffff]
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PC40._PRT]
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PC40.PC12._PRT]
Jun 19 18:45:42 weisshorn01 kernel: pci0000:40: Requesting ACPI _OSC control (0x1d)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 4 7 *10 11 12 14 15)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 4 7 10 *11 12 14 15)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 4 *7 10 11 12 14 15)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 4 7 *10 11 12 14 15)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 4 7 10 *11 12 14 15)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 4 7 10 11 12 14 15) *0, disabled.
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 4 7 *10 11 12 14 15)
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 4 7 10 11 12 14 15) *0, disabled.
Jun 19 18:45:42 weisshorn01 kernel: vgaarb: device added: PCI:0000:01:04.0,decodes=io+mem,owns=io+mem,locks=none
Jun 19 18:45:42 weisshorn01 kernel: vgaarb: loaded
Jun 19 18:45:42 weisshorn01 kernel: vgaarb: bridge control possible 0000:01:04.0
Jun 19 18:45:42 weisshorn01 kernel: SCSI subsystem initialized
Jun 19 18:45:42 weisshorn01 kernel: libata version 3.00 loaded.
Jun 19 18:45:42 weisshorn01 kernel: usbcore: registered new interface driver usbfs
Jun 19 18:45:42 weisshorn01 kernel: usbcore: registered new interface driver hub
Jun 19 18:45:42 weisshorn01 kernel: usbcore: registered new device driver usb
Jun 19 18:45:42 weisshorn01 kernel: PCI: Using ACPI for IRQ routing
Jun 19 18:45:42 weisshorn01 kernel: PCI: pci_cache_line_size set to 64 bytes
Jun 19 18:45:42 weisshorn01 kernel: NetLabel: Initializing
Jun 19 18:45:42 weisshorn01 kernel: NetLabel:  domain hash size = 128
Jun 19 18:45:42 weisshorn01 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Jun 19 18:45:42 weisshorn01 kernel: NetLabel:  unlabeled traffic allowed by default
Jun 19 18:45:42 weisshorn01 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0
Jun 19 18:45:42 weisshorn01 kernel: hpet0: 4 comparators, 32-bit 14.318180 MHz counter
Jun 19 18:45:42 weisshorn01 kernel: Switching to clocksource hpet
Jun 19 18:45:42 weisshorn01 kernel: spurious 8259A interrupt: IRQ7.
Jun 19 18:45:42 weisshorn01 kernel: pnp: PnP ACPI init
Jun 19 18:45:42 weisshorn01 kernel: ACPI: bus type pnp registered
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\PRID.P_D0._STA] (Node ffff880320782c18), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (uteval-0250): Method execution failed [\PRID.P_D0._STA] (Node ffff880320782c18), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\PRID.P_D1._STA] (Node ffff880b1e52cda8), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (uteval-0250): Method execution failed [\PRID.P_D1._STA] (Node ffff880b1e52cda8), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\SECD.S_D0._STA] (Node ffff880b1e52cec0), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (uteval-0250): Method execution failed [\SECD.S_D0._STA] (Node ffff880b1e52cec0), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\SECD.S_D1._STA] (Node ffff88031fe909c0), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (uteval-0250): Method execution failed [\SECD.S_D1._STA] (Node ffff88031fe909c0), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: pnp: PnP ACPI: found 20 devices
Jun 19 18:45:42 weisshorn01 kernel: ACPI: ACPI bus type pnp unregistered
Jun 19 18:45:42 weisshorn01 kernel: system 00:01: iomem range 0xfec20000-0xfec200ff has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:02: iomem range 0xffffff00-0x100003eff could not be reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0a: ioport range 0xf40-0xfbf has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0a: ioport range 0x295-0x296 has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0c: iomem range 0xfec00000-0xfec00fff has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0c: iomem range 0xfee00000-0xfee00fff has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0x4d0-0x4d1 has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0x40b-0x40b has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0x4d6-0x4d6 has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0xc00-0xc01 has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0xc14-0xc14 has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0xc50-0xc51 has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0xc52-0xc52 has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0xc6c-0xc6c has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0xc6f-0xc6f has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0xcd0-0xcd1 has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0xcd2-0xcd3 has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0xcd4-0xcd5 has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0xcd6-0xcd7 has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0xcd8-0xcdf has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0x800-0x89f has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0xb00-0xb0f has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0xb20-0xb3f has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0x900-0x90f has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0x910-0x91f has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: ioport range 0xfe00-0xfefe has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: iomem range 0xffb80000-0xffbfffff has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0d: iomem range 0xfec10000-0xfec1001f has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0e: ioport range 0xca2-0xca3 has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:0f: iomem range 0xe0000000-0xefffffff has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:10: iomem range 0x0-0x9ffff could not be reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:10: iomem range 0xc0000-0xcffff could not be reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:10: iomem range 0xe0000-0xfffff could not be reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:10: iomem range 0xfec00000-0xffffffff could not be reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:12: iomem range 0xd8f00000-0xd8f000ff has been reserved
Jun 19 18:45:42 weisshorn01 kernel: system 00:13: iomem range 0xffffff00-0x100003eff could not be reserved
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:02.0: PCI bridge to [bus 04-04]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:02.0: PCI bridge, secondary bus 0000:04
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:02.0:   bridge window [io  disabled]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:02.0:   bridge window [0xdff00000-0xdfffffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:02.0:   bridge window [0xdc000000-0xdc7fffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:04.0: PCI bridge to [bus 03-03]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:04.0: PCI bridge, secondary bus 0000:03
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:04.0:   bridge window [0xd000-0xdfff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:04.0:   bridge window [0xdfe00000-0xdfefffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:04.0:   bridge window [mem pref disabled]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: reg 184 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: reg 190 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: reg 184 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.1: reg 184 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: reg 184 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: reg 190 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.1: reg 190 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: reg 184 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: reg 190 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.1: reg 184 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: reg 184 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: reg 184 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: reg 190 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: reg 190 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.0: BAR 10: can't allocate mem resource [0x000000-0x01ffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.1: reg 184 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.1: reg 184 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.1: BAR 7: can't allocate mem resource [0x000000-0x01ffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.1: reg 190 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.1: reg 190 64bit mmio: [0x000000-0x003fff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:02:00.1: BAR 10: can't allocate mem resource [0x000000-0x01ffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:0d.0: PCI bridge to [bus 02-02]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:0d.0: PCI bridge, secondary bus 0000:02
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:0d.0:   bridge window [0xc000-0xcfff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:0d.0:   bridge window [0xdfd00000-0xdfdfffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:0d.0:   bridge window [mem pref disabled]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:14.4: PCI bridge to [bus 01-01]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:14.4: PCI bridge, secondary bus 0000:01
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:14.4:   bridge window [io  disabled]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:14.4:   bridge window [0xdf000000-0xdfcfffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:14.4:   bridge window [0xdb000000-0xdbffffff]
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Link [LNKE] BIOS reported IRQ 15, using IRQ 11
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Link [LNKE] enabled at IRQ 11
Jun 19 18:45:42 weisshorn01 kernel: PCI: setting IRQ 11 as level-triggered
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:02.0: PCI INT A -> Link[LNKE] -> GSI 11 (level, low) -> IRQ 11
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:02.0: setting latency timer to 64
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:04.0: PCI INT A -> Link[LNKE] -> GSI 11 (level, low) -> IRQ 11
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:04.0: setting latency timer to 64
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Link [LNKG] enabled at IRQ 10
Jun 19 18:45:42 weisshorn01 kernel: PCI: setting IRQ 10 as level-triggered
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:0d.0: PCI INT A -> Link[LNKG] -> GSI 10 (level, low) -> IRQ 10
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:00:0d.0: setting latency timer to 64
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:40:02.0: PCI bridge to [bus 41-41]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:40:02.0: PCI bridge, secondary bus 0000:41
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:40:02.0:   bridge window [0xe000-0xefff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:40:02.0:   bridge window [0xdad00000-0xdaffffff]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:40:02.0:   bridge window [mem pref disabled]
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:40:02.0: PCI INT A -> Link[LNKE] -> GSI 11 (level, low) -> IRQ 11
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:40:02.0: setting latency timer to 64
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:00: resource 0 io:  [0x00-0xdfff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:00: resource 1 io:  [0xf000-0xffff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:00: resource 2 mem: [0xdb000000-0xdfffffff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:00: resource 3 mem: [0x0a0000-0x0bffff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:00: resource 4 mem: [0xf0000000-0xffffffff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:00: resource 5 mem: [0xf28000000-0xfcffffffff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:04: resource 1 mem: [0xdff00000-0xdfffffff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:04: resource 2 pref mem [0xdc000000-0xdc7fffff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:03: resource 0 io:  [0xd000-0xdfff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:03: resource 1 mem: [0xdfe00000-0xdfefffff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:02: resource 0 io:  [0xc000-0xcfff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:02: resource 1 mem: [0xdfd00000-0xdfdfffff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:01: resource 1 mem: [0xdf000000-0xdfcfffff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:01: resource 2 pref mem [0xdb000000-0xdbffffff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:01: resource 3 io:  [0x00-0xdfff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:01: resource 4 io:  [0xf000-0xffff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:01: resource 5 mem: [0xdb000000-0xdfffffff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:01: resource 6 mem: [0x0a0000-0x0bffff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:01: resource 7 mem: [0xf0000000-0xffffffff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:01: resource 8 mem: [0xf28000000-0xfcffffffff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:40: resource 0 io:  [0xe000-0xefff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:40: resource 1 mem: [0xd8000000-0xdaffffff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:41: resource 0 io:  [0xe000-0xefff]
Jun 19 18:45:42 weisshorn01 kernel: pci_bus 0000:41: resource 1 mem: [0xdad00000-0xdaffffff]
Jun 19 18:45:42 weisshorn01 kernel: NET: Registered protocol family 2
Jun 19 18:45:42 weisshorn01 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
Jun 19 18:45:42 weisshorn01 kernel: TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
Jun 19 18:45:42 weisshorn01 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Jun 19 18:45:42 weisshorn01 kernel: TCP: Hash tables configured (established 524288 bind 65536)
Jun 19 18:45:42 weisshorn01 kernel: TCP reno registered
Jun 19 18:45:42 weisshorn01 kernel: NET: Registered protocol family 1
Jun 19 18:45:42 weisshorn01 kernel: pci 0000:01:04.0: Boot video device
Jun 19 18:45:42 weisshorn01 kernel: Trying to unpack rootfs image as initramfs...
Jun 19 18:45:42 weisshorn01 kernel: Freeing initrd memory: 59211k freed
Jun 19 18:45:42 weisshorn01 kernel: PCI-DMA: Disabling AGP.
Jun 19 18:45:42 weisshorn01 kernel: PCI-DMA: aperture base @ 20000000 size 65536 KB
Jun 19 18:45:42 weisshorn01 kernel: PCI-DMA: using GART IOMMU.
Jun 19 18:45:42 weisshorn01 kernel: PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
Jun 19 18:45:42 weisshorn01 kernel: audit: initializing netlink socket (disabled)
Jun 19 18:45:42 weisshorn01 kernel: type=2000 audit(1340124307.382:1): initialized
Jun 19 18:45:42 weisshorn01 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages
Jun 19 18:45:42 weisshorn01 kernel: VFS: Disk quotas dquot_6.5.2
Jun 19 18:45:42 weisshorn01 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Jun 19 18:45:42 weisshorn01 kernel: msgmni has been set to 32768
Jun 19 18:45:42 weisshorn01 kernel: SELinux:  Registering netfilter hooks
Jun 19 18:45:42 weisshorn01 kernel: alg: No test for stdrng (krng)
Jun 19 18:45:42 weisshorn01 kernel: ksign: Installing public key data
Jun 19 18:45:42 weisshorn01 kernel: Loading keyring
Jun 19 18:45:42 weisshorn01 kernel: - Added public key 974DEFD04714FA08
Jun 19 18:45:42 weisshorn01 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)
Jun 19 18:45:42 weisshorn01 kernel: - Added public key D4A26C9CCD09BEDA
Jun 19 18:45:42 weisshorn01 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Jun 19 18:45:42 weisshorn01 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
Jun 19 18:45:42 weisshorn01 kernel: io scheduler noop registered
Jun 19 18:45:42 weisshorn01 kernel: io scheduler anticipatory registered
Jun 19 18:45:42 weisshorn01 kernel: io scheduler deadline registered (default)
Jun 19 18:45:42 weisshorn01 kernel: io scheduler cfq registered
Jun 19 18:45:42 weisshorn01 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Jun 19 18:45:42 weisshorn01 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Jun 19 18:45:42 weisshorn01 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Jun 19 18:45:42 weisshorn01 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0
Jun 19 18:45:42 weisshorn01 kernel: ACPI: Power Button [PWRB]
Jun 19 18:45:42 weisshorn01 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
Jun 19 18:45:42 weisshorn01 kernel: ACPI: Power Button [PWRF]
Jun 19 18:45:42 weisshorn01 kernel: ACPI: acpi_idle registered with cpuidle
Jun 19 18:45:42 weisshorn01 kernel: processor LNXCPU:00: registered as cooling_device0
Jun 19 18:45:42 weisshorn01 kernel: processor LNXCPU:01: registered as cooling_device1
Jun 19 18:45:42 weisshorn01 kernel: processor LNXCPU:02: registered as cooling_device2
Jun 19 18:45:42 weisshorn01 kernel: processor LNXCPU:03: registered as cooling_device3
Jun 19 18:45:42 weisshorn01 kernel: processor LNXCPU:04: registered as cooling_device4
Jun 19 18:45:42 weisshorn01 kernel: processor LNXCPU:05: registered as cooling_device5
Jun 19 18:45:42 weisshorn01 kernel: processor LNXCPU:06: registered as cooling_device6
Jun 19 18:45:42 weisshorn01 kernel: processor LNXCPU:07: registered as cooling_device7
Jun 19 18:45:42 weisshorn01 kernel: processor LNXCPU:08: registered as cooling_device8
Jun 19 18:45:42 weisshorn01 kernel: processor LNXCPU:09: registered as cooling_device9
Jun 19 18:45:42 weisshorn01 kernel: processor LNXCPU:0a: registered as cooling_device10
Jun 19 18:45:42 weisshorn01 kernel: processor LNXCPU:0b: registered as cooling_device11
Jun 19 18:45:42 weisshorn01 kernel: processor LNXCPU:0c: registered as cooling_device12
Jun 19 18:45:42 weisshorn01 kernel: processor LNXCPU:0d: registered as cooling_device13
Jun 19 18:45:42 weisshorn01 kernel: processor LNXCPU:0e: registered as cooling_device14
Jun 19 18:45:42 weisshorn01 kernel: processor LNXCPU:0f: registered as cooling_device15
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\PRID.P_D0._STA] (Node ffff880320782c18), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (uteval-0250): Method execution failed [\PRID.P_D0._STA] (Node ffff880320782c18), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\PRID.P_D1._STA] (Node ffff880b1e52cda8), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (uteval-0250): Method execution failed [\PRID.P_D1._STA] (Node ffff880b1e52cda8), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\SECD.S_D0._STA] (Node ffff880b1e52cec0), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (uteval-0250): Method execution failed [\SECD.S_D0._STA] (Node ffff880b1e52cec0), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\SECD.S_D1._STA] (Node ffff88031fe909c0), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (uteval-0250): Method execution failed [\SECD.S_D1._STA] (Node ffff88031fe909c0), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\PRID.P_D0._STA] (Node ffff880320782c18), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (uteval-0250): Method execution failed [\PRID.P_D0._STA] (Node ffff880320782c18), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\PRID.P_D1._STA] (Node ffff880b1e52cda8), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (uteval-0250): Method execution failed [\PRID.P_D1._STA] (Node ffff880b1e52cda8), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\SECD.S_D0._STA] (Node ffff880b1e52cec0), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (uteval-0250): Method execution failed [\SECD.S_D0._STA] (Node ffff880b1e52cec0), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: No handler for Region [SACS] (ffff880f1e7776f0) [PCI_Config] (20090903/evregion-319)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error: Region PCI_Config(2) has no handler (20090903/exfldio-295)
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (psparse-0537): Method parse/execution failed [\SECD.S_D1._STA] (Node ffff88031fe909c0), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: ACPI Error (uteval-0250): Method execution failed [\SECD.S_D1._STA] (Node ffff88031fe909c0), AE_NOT_EXIST
Jun 19 18:45:42 weisshorn01 kernel: APEI: Can not request iomem region <00000000d7e9c13a-00000000d7e9c13c> for GARs.
Jun 19 18:45:42 weisshorn01 kernel: [Firmware Warn]: GHES: Poll interval is 0 for generic hardware error source: 1, disabled.
Jun 19 18:45:42 weisshorn01 kernel: GHES: APEI firmware first mode is enabled by WHEA _OSC.
Jun 19 18:45:42 weisshorn01 kernel: Non-volatile memory driver v1.3
Jun 19 18:45:42 weisshorn01 kernel: Linux agpgart interface v0.103
Jun 19 18:45:42 weisshorn01 kernel: crash memory driver: version 1.1
Jun 19 18:45:42 weisshorn01 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Jun 19 18:45:42 weisshorn01 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 0) is a 16550A
Jun 19 18:45:42 weisshorn01 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 0) is a 16550A
Jun 19 18:45:42 weisshorn01 kernel: serial8250: ttyS2 at I/O 0x3e8 (irq = 0) is a 16550A
Jun 19 18:45:42 weisshorn01 kernel: 00:07: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Jun 19 18:45:42 weisshorn01 kernel: 00:08: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Jun 19 18:45:42 weisshorn01 kernel: brd: module loaded
Jun 19 18:45:42 weisshorn01 kernel: loop: module loaded
Jun 19 18:45:42 weisshorn01 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input2
Jun 19 18:45:42 weisshorn01 kernel: Fixed MDIO Bus: probed
Jun 19 18:45:42 weisshorn01 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
Jun 19 18:45:42 weisshorn01 kernel: ehci_hcd 0000:00:12.2: PCI INT B -> Link[LNKB] -> GSI 11 (level, low) -> IRQ 11
Jun 19 18:45:42 weisshorn01 kernel: ehci_hcd 0000:00:12.2: EHCI Host Controller
Jun 19 18:45:42 weisshorn01 kernel: ehci_hcd 0000:00:12.2: new USB bus registered, assigned bus number 1
Jun 19 18:45:42 weisshorn01 kernel: ehci_hcd 0000:00:12.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
Jun 19 18:45:42 weisshorn01 kernel: ehci_hcd 0000:00:12.2: debug port 1
Jun 19 18:45:42 weisshorn01 kernel: ehci_hcd 0000:00:12.2: irq 11, io mem 0xdeffe800
Jun 19 18:45:42 weisshorn01 kernel: ehci_hcd 0000:00:12.2: USB 2.0 started, EHCI 1.00
Jun 19 18:45:42 weisshorn01 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
Jun 19 18:45:42 weisshorn01 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jun 19 18:45:42 weisshorn01 kernel: usb usb1: Product: EHCI Host Controller
Jun 19 18:45:42 weisshorn01 kernel: usb usb1: Manufacturer: Linux 2.6.32-220.7.1.el6_lustre.g9c8f747.x86_64 ehci_hcd
Jun 19 18:45:42 weisshorn01 kernel: usb usb1: SerialNumber: 0000:00:12.2
Jun 19 18:45:42 weisshorn01 kernel: usb usb1: configuration #1 chosen from 1 choice
Jun 19 18:45:42 weisshorn01 kernel: hub 1-0:1.0: USB hub found
Jun 19 18:45:42 weisshorn01 kernel: hub 1-0:1.0: 6 ports detected
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10
Jun 19 18:45:42 weisshorn01 kernel: ehci_hcd 0000:00:13.2: PCI INT B -> Link[LNKD] -> GSI 10 (level, low) -> IRQ 10
Jun 19 18:45:42 weisshorn01 kernel: ehci_hcd 0000:00:13.2: EHCI Host Controller
Jun 19 18:45:42 weisshorn01 kernel: ehci_hcd 0000:00:13.2: new USB bus registered, assigned bus number 2
Jun 19 18:45:42 weisshorn01 kernel: ehci_hcd 0000:00:13.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
Jun 19 18:45:42 weisshorn01 kernel: ehci_hcd 0000:00:13.2: debug port 1
Jun 19 18:45:42 weisshorn01 kernel: ehci_hcd 0000:00:13.2: irq 10, io mem 0xdeffec00
Jun 19 18:45:42 weisshorn01 kernel: ehci_hcd 0000:00:13.2: USB 2.0 started, EHCI 1.00
Jun 19 18:45:42 weisshorn01 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
Jun 19 18:45:42 weisshorn01 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jun 19 18:45:42 weisshorn01 kernel: usb usb2: Product: EHCI Host Controller
Jun 19 18:45:42 weisshorn01 kernel: usb usb2: Manufacturer: Linux 2.6.32-220.7.1.el6_lustre.g9c8f747.x86_64 ehci_hcd
Jun 19 18:45:42 weisshorn01 kernel: usb usb2: SerialNumber: 0000:00:13.2
Jun 19 18:45:42 weisshorn01 kernel: usb usb2: configuration #1 chosen from 1 choice
Jun 19 18:45:42 weisshorn01 kernel: hub 2-0:1.0: USB hub found
Jun 19 18:45:42 weisshorn01 kernel: hub 2-0:1.0: 6 ports detected
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:12.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, low) -> IRQ 10
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:12.0: OHCI Host Controller
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:12.0: new USB bus registered, assigned bus number 3
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:12.0: irq 10, io mem 0xdeffa000
Jun 19 18:45:42 weisshorn01 kernel: usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
Jun 19 18:45:42 weisshorn01 kernel: usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jun 19 18:45:42 weisshorn01 kernel: usb usb3: Product: OHCI Host Controller
Jun 19 18:45:42 weisshorn01 kernel: usb usb3: Manufacturer: Linux 2.6.32-220.7.1.el6_lustre.g9c8f747.x86_64 ohci_hcd
Jun 19 18:45:42 weisshorn01 kernel: usb usb3: SerialNumber: 0000:00:12.0
Jun 19 18:45:42 weisshorn01 kernel: usb usb3: configuration #1 chosen from 1 choice
Jun 19 18:45:42 weisshorn01 kernel: hub 3-0:1.0: USB hub found
Jun 19 18:45:42 weisshorn01 kernel: hub 3-0:1.0: 3 ports detected
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:12.1: PCI INT A -> Link[LNKA] -> GSI 10 (level, low) -> IRQ 10
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:12.1: OHCI Host Controller
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:12.1: new USB bus registered, assigned bus number 4
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:12.1: irq 10, io mem 0xdeffb000
Jun 19 18:45:42 weisshorn01 kernel: usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
Jun 19 18:45:42 weisshorn01 kernel: usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jun 19 18:45:42 weisshorn01 kernel: usb usb4: Product: OHCI Host Controller
Jun 19 18:45:42 weisshorn01 kernel: usb usb4: Manufacturer: Linux 2.6.32-220.7.1.el6_lustre.g9c8f747.x86_64 ohci_hcd
Jun 19 18:45:42 weisshorn01 kernel: usb usb4: SerialNumber: 0000:00:12.1
Jun 19 18:45:42 weisshorn01 kernel: usb usb4: configuration #1 chosen from 1 choice
Jun 19 18:45:42 weisshorn01 kernel: hub 4-0:1.0: USB hub found
Jun 19 18:45:42 weisshorn01 kernel: hub 4-0:1.0: 3 ports detected
Jun 19 18:45:42 weisshorn01 kernel: ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 7
Jun 19 18:45:42 weisshorn01 kernel: PCI: setting IRQ 7 as level-triggered
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:13.0: PCI INT A -> Link[LNKC] -> GSI 7 (level, low) -> IRQ 7
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:13.0: OHCI Host Controller
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:13.0: new USB bus registered, assigned bus number 5
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:13.0: irq 7, io mem 0xdeffc000
Jun 19 18:45:42 weisshorn01 kernel: usb usb5: New USB device found, idVendor=1d6b, idProduct=0001
Jun 19 18:45:42 weisshorn01 kernel: usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jun 19 18:45:42 weisshorn01 kernel: usb usb5: Product: OHCI Host Controller
Jun 19 18:45:42 weisshorn01 kernel: usb usb5: Manufacturer: Linux 2.6.32-220.7.1.el6_lustre.g9c8f747.x86_64 ohci_hcd
Jun 19 18:45:42 weisshorn01 kernel: usb usb5: SerialNumber: 0000:00:13.0
Jun 19 18:45:42 weisshorn01 kernel: usb usb5: configuration #1 chosen from 1 choice
Jun 19 18:45:42 weisshorn01 kernel: hub 5-0:1.0: USB hub found
Jun 19 18:45:42 weisshorn01 kernel: hub 5-0:1.0: 3 ports detected
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:13.1: PCI INT A -> Link[LNKC] -> GSI 7 (level, low) -> IRQ 7
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:13.1: OHCI Host Controller
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:13.1: new USB bus registered, assigned bus number 6
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:13.1: irq 7, io mem 0xdeffd000
Jun 19 18:45:42 weisshorn01 kernel: Refined TSC clocksource calibration: 2000.023 MHz.
Jun 19 18:45:42 weisshorn01 kernel: Switching to clocksource tsc
Jun 19 18:45:42 weisshorn01 kernel: usb usb6: New USB device found, idVendor=1d6b, idProduct=0001
Jun 19 18:45:42 weisshorn01 kernel: usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jun 19 18:45:42 weisshorn01 kernel: usb usb6: Product: OHCI Host Controller
Jun 19 18:45:42 weisshorn01 kernel: usb usb6: Manufacturer: Linux 2.6.32-220.7.1.el6_lustre.g9c8f747.x86_64 ohci_hcd
Jun 19 18:45:42 weisshorn01 kernel: usb usb6: SerialNumber: 0000:00:13.1
Jun 19 18:45:42 weisshorn01 kernel: usb usb6: configuration #1 chosen from 1 choice
Jun 19 18:45:42 weisshorn01 kernel: hub 6-0:1.0: USB hub found
Jun 19 18:45:42 weisshorn01 kernel: hub 6-0:1.0: 3 ports detected
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:14.5: PCI INT C -> Link[LNKC] -> GSI 7 (level, low) -> IRQ 7
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:14.5: OHCI Host Controller
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:14.5: new USB bus registered, assigned bus number 7
Jun 19 18:45:42 weisshorn01 kernel: ohci_hcd 0000:00:14.5: irq 7, io mem 0xdefff000
Jun 19 18:45:42 weisshorn01 kernel: usb usb7: New USB device found, idVendor=1d6b, idProduct=0001
Jun 19 18:45:42 weisshorn01 kernel: usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jun 19 18:45:42 weisshorn01 kernel: usb usb7: Product: OHCI Host Controller
Jun 19 18:45:42 weisshorn01 kernel: usb usb7: Manufacturer: Linux 2.6.32-220.7.1.el6_lustre.g9c8f747.x86_64 ohci_hcd
Jun 19 18:45:42 weisshorn01 kernel: usb usb7: SerialNumber: 0000:00:14.5
Jun 19 18:45:42 weisshorn01 kernel: usb usb7: configuration #1 chosen from 1 choice
Jun 19 18:45:42 weisshorn01 kernel: hub 7-0:1.0: USB hub found
Jun 19 18:45:42 weisshorn01 kernel: hub 7-0:1.0: 2 ports detected
Jun 19 18:45:42 weisshorn01 kernel: uhci_hcd: USB Universal Host Controller Interface driver
Jun 19 18:45:42 weisshorn01 kernel: PNP: No PS/2 controller found. Probing ports directly.
Jun 19 18:45:42 weisshorn01 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1
Jun 19 18:45:42 weisshorn01 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12
Jun 19 18:45:42 weisshorn01 kernel: mice: PS/2 mouse device common for all mice
Jun 19 18:45:42 weisshorn01 kernel: rtc_cmos 00:04: RTC can wake from S4
Jun 19 18:45:42 weisshorn01 kernel: rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0
Jun 19 18:45:42 weisshorn01 kernel: rtc0: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
Jun 19 18:45:42 weisshorn01 kernel: cpuidle: using governor ladder
Jun 19 18:45:42 weisshorn01 kernel: cpuidle: using governor menu
Jun 19 18:45:42 weisshorn01 kernel: usbcore: registered new interface driver hiddev
Jun 19 18:45:42 weisshorn01 kernel: usbcore: registered new interface driver usbhid
Jun 19 18:45:42 weisshorn01 kernel: usbhid: v2.6:USB HID core driver
Jun 19 18:45:42 weisshorn01 kernel: TCP cubic registered
Jun 19 18:45:42 weisshorn01 kernel: Initializing XFRM netlink socket
Jun 19 18:45:42 weisshorn01 kernel: NET: Registered protocol family 17
Jun 19 18:45:42 weisshorn01 kernel: registered taskstats version 1
Jun 19 18:45:42 weisshorn01 kernel: rtc_cmos 00:04: setting system clock to 2012-06-19 16:45:09 UTC (1340124309)
Jun 19 18:45:42 weisshorn01 kernel: Initalizing network drop monitor service
Jun 19 18:45:42 weisshorn01 kernel: Freeing unused kernel memory: 1244k freed
Jun 19 18:45:42 weisshorn01 kernel: Write protecting the kernel read-only data: 10240k
Jun 19 18:45:42 weisshorn01 kernel: Freeing unused kernel memory: 1040k freed
Jun 19 18:45:42 weisshorn01 kernel: Freeing unused kernel memory: 1756k freed
Jun 19 18:45:42 weisshorn01 kernel: usb 1-5: new high speed USB device using ehci_hcd and address 2
Jun 19 18:45:42 weisshorn01 kernel: rdac: device handler registered
Jun 19 18:45:42 weisshorn01 kernel: dracut: dracut-004-256.el6_2.1
Jun 19 18:45:42 weisshorn01 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation
Jun 19 18:45:42 weisshorn01 kernel: dracut: rd_NO_LVM: removing LVM activation
Jun 19 18:45:42 weisshorn01 kernel: device-mapper: uevent: version 1.0.3
Jun 19 18:45:42 weisshorn01 kernel: device-mapper: ioctl: 4.22.6-ioctl (2011-10-19) initialised: dm-devel@redhat.com
Jun 19 18:45:42 weisshorn01 kernel: udev: starting version 147
Jun 19 18:45:42 weisshorn01 kernel: usb 1-5: New USB device found, idVendor=05e3, idProduct=0719
Jun 19 18:45:42 weisshorn01 kernel: usb 1-5: New USB device strings: Mfr=0, Product=1, SerialNumber=2
Jun 19 18:45:42 weisshorn01 kernel: usb 1-5: Product: USB Storage
Jun 19 18:45:42 weisshorn01 kernel: usb 1-5: SerialNumber: 000000000033
Jun 19 18:45:42 weisshorn01 kernel: usb 1-5: configuration #1 chosen from 1 choice
Jun 19 18:45:42 weisshorn01 kernel: Initializing USB Mass Storage driver...
Jun 19 18:45:42 weisshorn01 kernel: scsi0 : SCSI emulation for USB Mass Storage devices
Jun 19 18:45:42 weisshorn01 kernel: usb-storage: device found at 2
Jun 19 18:45:42 weisshorn01 kernel: usb-storage: waiting for device to settle before scanning
Jun 19 18:45:42 weisshorn01 kernel: usbcore: registered new interface driver usb-storage
Jun 19 18:45:42 weisshorn01 kernel: USB Mass Storage support registered.
Jun 19 18:45:42 weisshorn01 kernel: dracut: Starting plymouth daemon
Jun 19 18:45:42 weisshorn01 kernel: dracut: rd_NO_DM: removing DM RAID activation
Jun 19 18:45:42 weisshorn01 kernel: dracut: rd_NO_MD: removing MD RAID activation
Jun 19 18:45:42 weisshorn01 kernel: ahci 0000:00:11.0: version 3.0
Jun 19 18:45:42 weisshorn01 kernel: ahci 0000:00:11.0: PCI INT A -> Link[LNKG] -> GSI 10 (level, low) -> IRQ 10
Jun 19 18:45:42 weisshorn01 kernel: ahci 0000:00:11.0: AHCI 0001.0100 32 slots 4 ports 3 Gbps 0xf impl SATA mode
Jun 19 18:45:42 weisshorn01 kernel: ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part ccc 
Jun 19 18:45:42 weisshorn01 kernel: scsi1 : ahci
Jun 19 18:45:42 weisshorn01 kernel: scsi2 : ahci
Jun 19 18:45:42 weisshorn01 kernel: scsi3 : ahci
Jun 19 18:45:42 weisshorn01 kernel: scsi4 : ahci
Jun 19 18:45:42 weisshorn01 kernel: ata1: SATA max UDMA/133 abar m1024@0xdeffe400 port 0xdeffe500 irq 10
Jun 19 18:45:42 weisshorn01 kernel: ata2: SATA max UDMA/133 abar m1024@0xdeffe400 port 0xdeffe580 irq 10
Jun 19 18:45:42 weisshorn01 kernel: ata3: SATA max UDMA/133 abar m1024@0xdeffe400 port 0xdeffe600 irq 10
Jun 19 18:45:42 weisshorn01 kernel: ata4: SATA max UDMA/133 abar m1024@0xdeffe400 port 0xdeffe680 irq 10
Jun 19 18:45:42 weisshorn01 kernel: usb 7-1: new full speed USB device using ohci_hcd and address 2
Jun 19 18:45:42 weisshorn01 kernel: usb 7-1: New USB device found, idVendor=046b, idProduct=ff10
Jun 19 18:45:42 weisshorn01 kernel: usb 7-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jun 19 18:45:42 weisshorn01 kernel: usb 7-1: Product: Virtual Keyboard and Mouse
Jun 19 18:45:42 weisshorn01 kernel: usb 7-1: Manufacturer: American Megatrends Inc.
Jun 19 18:45:42 weisshorn01 kernel: usb 7-1: SerialNumber: serial
Jun 19 18:45:42 weisshorn01 kernel: usb 7-1: configuration #1 chosen from 1 choice
Jun 19 18:45:42 weisshorn01 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:14.5/usb7/7-1/7-1:1.0/input/input3
Jun 19 18:45:42 weisshorn01 kernel: generic-usb 0003:046B:FF10.0001: input,hidraw0: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:14.5-1/input0
Jun 19 18:45:42 weisshorn01 kernel: input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:14.5/usb7/7-1/7-1:1.1/input/input4
Jun 19 18:45:42 weisshorn01 kernel: generic-usb 0003:046B:FF10.0002: input,hidraw1: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:14.5-1/input1
Jun 19 18:45:42 weisshorn01 kernel: ata2: SATA link down (SStatus 0 SControl 300)
Jun 19 18:45:42 weisshorn01 kernel: ata3: SATA link down (SStatus 0 SControl 300)
Jun 19 18:45:42 weisshorn01 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Jun 19 18:45:42 weisshorn01 kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Jun 19 18:45:42 weisshorn01 kernel: ata1.00: ATA-7: HUA721010KLA330      59Y5306 59Y1795IBM, GKAOABDA, max UDMA/133
Jun 19 18:45:42 weisshorn01 kernel: ata1.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
Jun 19 18:45:42 weisshorn01 kernel: ata1.00: configured for UDMA/133
Jun 19 18:45:42 weisshorn01 kernel: scsi 1:0:0:0: Direct-Access     ATA      HUA721010KLA330  GKAO PQ: 0 ANSI: 5
Jun 19 18:45:42 weisshorn01 kernel: pata_atiixp 0000:00:14.1: PCI INT A -> Link[LNKA] -> GSI 10 (level, low) -> IRQ 10
Jun 19 18:45:42 weisshorn01 kernel: pata_atiixp 0000:00:14.1: setting latency timer to 64
Jun 19 18:45:42 weisshorn01 kernel: scsi5 : pata_atiixp
Jun 19 18:45:42 weisshorn01 kernel: scsi6 : pata_atiixp
Jun 19 18:45:42 weisshorn01 kernel: ata5: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0xff00 irq 14
Jun 19 18:45:42 weisshorn01 kernel: ata6: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0xff08 irq 15
Jun 19 18:45:42 weisshorn01 kernel: usb-storage: device scan complete
Jun 19 18:45:42 weisshorn01 kernel: scsi 0:0:0:0: CD-ROM            TEAC     DV-28S-W         1.2A PQ: 0 ANSI: 0
Jun 19 18:45:42 weisshorn01 kernel: mpt2sas version 09.101.00.00 loaded
Jun 19 18:45:42 weisshorn01 kernel: scsi7 : Fusion MPT SAS Host
Jun 19 18:45:42 weisshorn01 kernel: mpt2sas 0000:03:00.0: PCI INT A -> Link[LNKE] -> GSI 11 (level, low) -> IRQ 11
Jun 19 18:45:42 weisshorn01 kernel: mpt2sas 0000:03:00.0: setting latency timer to 64
Jun 19 18:45:42 weisshorn01 kernel: mpt2sas0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (61829660 kB)
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 56 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mpt2sas 0000:03:00.0: irq 56 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel: mpt2sas0: PCI-MSI-X enabled: IRQ 56
Jun 19 18:45:42 weisshorn01 kernel: mpt2sas0: iomem(0x00000000dfe3c000), mapped(0xffffc90016e38000), size(16384)
Jun 19 18:45:42 weisshorn01 kernel: mpt2sas0: ioport(0x000000000000d000), size(256)
Jun 19 18:45:42 weisshorn01 kernel: mpt2sas0: sending diag reset !!
Jun 19 18:45:42 weisshorn01 kernel: mpt2sas0: diag reset: SUCCESS
Jun 19 18:45:42 weisshorn01 kernel: mpt2sas0: Allocated physical memory: size(4506 kB)
Jun 19 18:45:42 weisshorn01 kernel: mpt2sas0: Current Controller Queue Depth(1978), Max Controller Queue Depth(2239)
Jun 19 18:45:42 weisshorn01 kernel: mpt2sas0: Scatter Gather Elements per IO(128)
Jun 19 18:45:42 weisshorn01 kernel: mpt2sas0: LSISAS2008: FWVersion(07.00.00.00), ChipRevision(0x02), BiosVersion(07.11.00.00)
Jun 19 18:45:42 weisshorn01 kernel: mpt2sas0: Protocol=(Initiator,Target), Capabilities=(Raid,TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
Jun 19 18:45:42 weisshorn01 kernel: mpt2sas0: sending port enable !!
Jun 19 18:45:42 weisshorn01 kernel: mpt2sas0: host_add: handle(0x0001), sas_addr(0x50030480003a5d90), phys(8)
Jun 19 18:45:42 weisshorn01 kernel: mpt2sas0: port enable: SUCCESS
Jun 19 18:45:42 weisshorn01 kernel: sd 1:0:0:0: [sda] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
Jun 19 18:45:42 weisshorn01 kernel: sd 1:0:0:0: [sda] Write Protect is off
Jun 19 18:45:42 weisshorn01 kernel: sd 1:0:0:0: [sda] Mode Sense: 00 3a 00 00
Jun 19 18:45:42 weisshorn01 kernel: sd 1:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jun 19 18:45:42 weisshorn01 kernel: sda: sda1 sda2 sda3
Jun 19 18:45:42 weisshorn01 kernel: sd 1:0:0:0: [sda] Attached SCSI disk
Jun 19 18:45:42 weisshorn01 kernel: QLogic Fibre Channel HBA Driver: 8.03.07.05.06.2-k
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, low) -> IRQ 10
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.0: Found an ISP2532, irq 10, iobase 0xffffc90016e18000
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 57 on node 2
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 2
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.0: irq 57 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 58 on node 2
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 2
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.0: irq 58 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.0: Configuring PCI space...
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.0: setting latency timer to 64
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.0: Configure NVRAM parameters...
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.0: Verifying loaded RISC code...
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.0: firmware: requesting ql2500_fw.bin
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.0: FW: Loading via request-firmware...
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.0: Allocated (64 KB) for FCE...
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.0: Allocated (64 KB) for EFT...
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.0: Allocated (1350 KB) for firmware dump...
Jun 19 18:45:42 weisshorn01 kernel: scsi8 : qla2xxx
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.0: 
Jun 19 18:45:42 weisshorn01 kernel: QLogic Fibre Channel HBA Driver: 8.03.07.05.06.2-k
Jun 19 18:45:42 weisshorn01 kernel:  QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA
Jun 19 18:45:42 weisshorn01 kernel:  ISP2532: PCIe (5.0GT/s x4) @ 0000:41:00.0 hdma+, host#=8, fw=5.06.02 (90d5)
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.1: PCI INT B -> Link[LNKB] -> GSI 11 (level, low) -> IRQ 11
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.1: Found an ISP2532, irq 11, iobase 0xffffc90016de8000
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 59 on node 2
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 2
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.1: irq 59 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 60 on node 2
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 2
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.1: irq 60 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.1: Configuring PCI space...
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.1: setting latency timer to 64
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.1: Configure NVRAM parameters...
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.1: Verifying loaded RISC code...
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.1: FW: Loading via request-firmware...
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.1: Allocated (64 KB) for FCE...
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.1: Allocated (64 KB) for EFT...
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.1: Allocated (1350 KB) for firmware dump...
Jun 19 18:45:42 weisshorn01 kernel: scsi9 : qla2xxx
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.1: 
Jun 19 18:45:42 weisshorn01 kernel: QLogic Fibre Channel HBA Driver: 8.03.07.05.06.2-k
Jun 19 18:45:42 weisshorn01 kernel:  QLogic QLE2562 - PCI-Express Dual Channel 8Gb Fibre Channel HBA
Jun 19 18:45:42 weisshorn01 kernel:  ISP2532: PCIe (5.0GT/s x4) @ 0000:41:00.1 hdma+, host#=9, fw=5.06.02 (90d5)
Jun 19 18:45:42 weisshorn01 kernel: sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Jun 19 18:45:42 weisshorn01 kernel: Uniform CD-ROM driver Revision: 3.20
Jun 19 18:45:42 weisshorn01 kernel: sr 0:0:0:0: Attached scsi CD-ROM sr0
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.1: LIP reset occurred (f7f7).
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.0: LIP reset occurred (f7f7).
Jun 19 18:45:42 weisshorn01 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: 
Jun 19 18:45:42 weisshorn01 kernel: dracut: Mounted root filesystem /dev/sda2
Jun 19 18:45:42 weisshorn01 kernel: SELinux:  Disabled at runtime.
Jun 19 18:45:42 weisshorn01 kernel: SELinux:  Unregistering netfilter hooks
Jun 19 18:45:42 weisshorn01 kernel: type=1404 audit(1340124320.497:2): selinux=0 auid=4294967295 ses=4294967295
Jun 19 18:45:42 weisshorn01 kernel: dracut: 
Jun 19 18:45:42 weisshorn01 kernel: dracut: Switching root
Jun 19 18:45:42 weisshorn01 kernel: udev: starting version 147
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.0: LIP occurred (f7f7).
Jun 19 18:45:42 weisshorn01 kernel: qla2xxx 0000:41:00.0: LOOP UP detected (8 Gbps).
Jun 19 18:45:42 weisshorn01 kernel: scsi 8:0:0:0: Direct-Access     LSI      INF-01-00        0780 PQ: 0 ANSI: 5
Jun 19 18:45:42 weisshorn01 kernel: scsi 8:0:0:0: rdac: LUN 0 (RDAC) (owned)
Jun 19 18:45:42 weisshorn01 kernel: sd 8:0:0:0: [sdb] 779345723 512-byte logical blocks: (399 GB/371 GiB)
Jun 19 18:45:42 weisshorn01 kernel: sd 8:0:0:0: [sdb] Write Protect is off
Jun 19 18:45:42 weisshorn01 kernel: sd 8:0:0:0: [sdb] Mode Sense: 83 00 10 08
Jun 19 18:45:42 weisshorn01 kernel: sd 8:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
Jun 19 18:45:42 weisshorn01 kernel: sdb: unknown partition table
Jun 19 18:45:42 weisshorn01 kernel: scsi 9:0:0:0: rdac: LUN 0 (RDAC) (unowned)
Jun 19 18:45:42 weisshorn01 kernel: sd 9:0:0:0: [sdc] 779345723 512-byte logical blocks: (399 GB/371 GiB)
Jun 19 18:45:42 weisshorn01 kernel: sd 9:0:0:0: [sdc] Write Protect is off
Jun 19 18:45:42 weisshorn01 kernel: sd 9:0:0:0: [sdc] Mode Sense: 83 00 10 08
Jun 19 18:45:42 weisshorn01 kernel: sd 9:0:0:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA
Jun 19 18:45:42 weisshorn01 kernel: sdc:Dev sdc: unable to read RDB block 0
Jun 19 18:45:42 weisshorn01 kernel: unable to read partition table
Jun 19 18:45:42 weisshorn01 kernel: sd 9:0:0:0: [sdc] Attached SCSI disk
Jun 19 18:45:42 weisshorn01 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
Jun 19 18:45:42 weisshorn01 kernel: piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00, revision 0
Jun 19 18:45:42 weisshorn01 kernel: MCE: In-kernel MCE decoding enabled.
Jun 19 18:45:42 weisshorn01 kernel: EDAC MC: Ver: 2.1.0 Apr 24 2012
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
Jun 19 18:45:42 weisshorn01 kernel: Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
Jun 19 18:45:42 weisshorn01 kernel: (Note that use of the override may cause unknown side effects.)
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: DRAM ECC disabled.
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
Jun 19 18:45:42 weisshorn01 kernel: Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
Jun 19 18:45:42 weisshorn01 kernel: (Note that use of the override may cause unknown side effects.)
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: DRAM ECC disabled.
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
Jun 19 18:45:42 weisshorn01 kernel: Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
Jun 19 18:45:42 weisshorn01 kernel: (Note that use of the override may cause unknown side effects.)
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: DRAM ECC disabled.
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
Jun 19 18:45:42 weisshorn01 kernel: Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
Jun 19 18:45:42 weisshorn01 kernel: (Note that use of the override may cause unknown side effects.)
Jun 19 18:45:42 weisshorn01 kernel: AMD64 EDAC driver v3.4.0
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: DRAM ECC disabled.
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
Jun 19 18:45:42 weisshorn01 kernel: Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
Jun 19 18:45:42 weisshorn01 kernel: (Note that use of the override may cause unknown side effects.)
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: DRAM ECC disabled.
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
Jun 19 18:45:42 weisshorn01 kernel: Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
Jun 19 18:45:42 weisshorn01 kernel: (Note that use of the override may cause unknown side effects.)
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: DRAM ECC disabled.
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
Jun 19 18:45:42 weisshorn01 kernel: Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
Jun 19 18:45:42 weisshorn01 kernel: (Note that use of the override may cause unknown side effects.)
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: DRAM ECC disabled.
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
Jun 19 18:45:42 weisshorn01 kernel: Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
Jun 19 18:45:42 weisshorn01 kernel: (Note that use of the override may cause unknown side effects.)
Jun 19 18:45:42 weisshorn01 kernel: AMD64 EDAC driver v3.4.0
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: DRAM ECC disabled.
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
Jun 19 18:45:42 weisshorn01 kernel: Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
Jun 19 18:45:42 weisshorn01 kernel: (Note that use of the override may cause unknown side effects.)
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: DRAM ECC disabled.
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
Jun 19 18:45:42 weisshorn01 kernel: Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
Jun 19 18:45:42 weisshorn01 kernel: (Note that use of the override may cause unknown side effects.)
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: DRAM ECC disabled.
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
Jun 19 18:45:42 weisshorn01 kernel: Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
Jun 19 18:45:42 weisshorn01 kernel: (Note that use of the override may cause unknown side effects.)
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: DRAM ECC disabled.
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
Jun 19 18:45:42 weisshorn01 kernel: Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
Jun 19 18:45:42 weisshorn01 kernel: (Note that use of the override may cause unknown side effects.)
Jun 19 18:45:42 weisshorn01 kernel: AMD64 EDAC driver v3.4.0
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: DRAM ECC disabled.
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
Jun 19 18:45:42 weisshorn01 kernel: Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
Jun 19 18:45:42 weisshorn01 kernel: (Note that use of the override may cause unknown side effects.)
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: DRAM ECC disabled.
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
Jun 19 18:45:42 weisshorn01 kernel: Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
Jun 19 18:45:42 weisshorn01 kernel: (Note that use of the override may cause unknown side effects.)
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: DRAM ECC disabled.
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
Jun 19 18:45:42 weisshorn01 kernel: Either enable ECC checking or force module loading by setting 'ecc_enable_override'.
Jun 19 18:45:42 weisshorn01 kernel: (Note that use of the override may cause unknown side effects.)
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: DRAM ECC disabled.
Jun 19 18:45:42 weisshorn01 kernel: EDAC amd64: ECC disabled in the BIOS or no ECC capability, module will not load.
Jun 19 18:45:42 weisshorn01 kernel: sd 8:0:0:0: Attached scsi generic sg2 type 0
Jun 19 18:45:42 weisshorn01 kernel: sd 9:0:0:0: Attached scsi generic sg3 type 0
Jun 19 18:45:42 weisshorn01 kernel: platform microcode: firmware: requesting amd-ucode/microcode_amd.bin
Jun 19 18:45:42 weisshorn01 kernel: microcode: CPU0: patch_level=0x10000c4
Jun 19 18:45:42 weisshorn01 kernel: microcode: CPU1: patch_level=0x10000c4
Jun 19 18:45:42 weisshorn01 kernel: microcode: CPU2: patch_level=0x10000c4
Jun 19 18:45:42 weisshorn01 kernel: microcode: CPU3: patch_level=0x10000c4
Jun 19 18:45:42 weisshorn01 kernel: microcode: CPU4: patch_level=0x10000c4
Jun 19 18:45:42 weisshorn01 kernel: microcode: CPU5: patch_level=0x10000c4
Jun 19 18:45:42 weisshorn01 kernel: microcode: CPU6: patch_level=0x10000c4
Jun 19 18:45:42 weisshorn01 kernel: microcode: CPU7: patch_level=0x10000c4
Jun 19 18:45:42 weisshorn01 kernel: microcode: CPU8: patch_level=0x10000c4
Jun 19 18:45:42 weisshorn01 kernel: microcode: CPU9: patch_level=0x10000c4
Jun 19 18:45:42 weisshorn01 kernel: microcode: CPU10: patch_level=0x10000c4
Jun 19 18:45:42 weisshorn01 kernel: microcode: CPU11: patch_level=0x10000c4
Jun 19 18:45:42 weisshorn01 kernel: microcode: CPU12: patch_level=0x10000c4
Jun 19 18:45:42 weisshorn01 kernel: microcode: CPU13: patch_level=0x10000c4
Jun 19 18:45:42 weisshorn01 kernel: microcode: CPU14: patch_level=0x10000c4
Jun 19 18:45:42 weisshorn01 kernel: microcode: CPU15: patch_level=0x10000c4
Jun 19 18:45:42 weisshorn01 kernel: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Jun 19 18:45:42 weisshorn01 kernel: microcode: AMD microcode update via /dev/cpu/microcode not supported
Jun 19 18:45:42 weisshorn01 kernel: microcode: AMD microcode update via /dev/cpu/microcode not supported
Jun 19 18:45:42 weisshorn01 kernel: microcode: AMD microcode update via /dev/cpu/microcode not supported
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core: Mellanox ConnectX core driver v0.01 (May 1, 2007)
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core: Initializing 0000:04:00.0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, low) -> IRQ 10
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: setting latency timer to 64
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 61 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 61 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 62 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 62 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 63 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 63 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 64 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 64 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 65 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 74 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 75 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 75 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 76 on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 81 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 82 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 82 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 83 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 83 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 84 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 84 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 85 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 85 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 86 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 86 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 87 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 87 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 88 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 88 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 89 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 89 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 90 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 90 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 91 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 91 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 92 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 92 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 93 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 93 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 94 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 94 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 95 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 95 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 96 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 96 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 97 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 97 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 98 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: mlx4_core 0000:04:00.0: irq 98 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.5.4.1 (March 2011)
Jun 19 18:45:42 weisshorn01 kernel: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (April 4, 2008)
Jun 19 18:45:42 weisshorn01 kernel: dca service started, version 1.12.1
Jun 19 18:45:42 weisshorn01 kernel: Intel(R) Gigabit Ethernet Network Driver - version 3.0.6-k
Jun 19 18:45:42 weisshorn01 kernel: Copyright (c) 2007-2011 Intel Corporation.
Jun 19 18:45:42 weisshorn01 kernel: igb 0000:02:00.0: power state changed by ACPI to D0
Jun 19 18:45:42 weisshorn01 kernel: igb 0000:02:00.0: power state changed by ACPI to D0
Jun 19 18:45:42 weisshorn01 kernel: igb 0000:02:00.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, low) -> IRQ 10
Jun 19 18:45:42 weisshorn01 kernel: igb 0000:02:00.0: setting latency timer to 64
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 99 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: igb 0000:02:00.0: irq 99 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 100 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: igb 0000:02:00.0: irq 100 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 101 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: igb 0000:02:00.0: irq 101 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 102 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: igb 0000:02:00.0: irq 102 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 103 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: igb 0000:02:00.0: irq 103 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 104 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: igb 0000:02:00.0: irq 104 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 105 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: igb 0000:02:00.0: irq 105 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 106 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: igb 0000:02:00.0: irq 106 for MSI/MSI-X
Jun 19 18:45:42 weisshorn01 kernel:  alloc irq_desc for 107 on node 0
Jun 19 18:45:42 weisshorn01 kernel:  alloc kstat_irqs on node 0
Jun 19 18:45:42 weisshorn01 kernel: Adding 26836984k swap on /dev/sda3.  Priority:-1 extents:1 across:26836984k 
Jun 19 18:45:42 weisshorn01 kernel: igb 0000:02:00.0: changing MTU from 1500 to 9000
Jun 19 18:45:42 weisshorn01 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jun 19 18:45:42 weisshorn01 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jun 19 18:45:42 weisshorn01 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jun 19 18:45:42 weisshorn01 kernel: ADDRCONF(NETDEV_UP): eth1: link is not ready
Jun 19 18:45:42 weisshorn01 kernel: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jun 19 18:45:42 weisshorn01 kernel: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Jun 19 18:45:42 weisshorn01 kernel: ib0: enabling connected mode will cause multicast packet drops
Jun 19 18:45:42 weisshorn01 kernel: ib0: mtu > 2044 will cause multicast packet drops.
Jun 19 18:45:42 weisshorn01 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
Jun 19 18:45:42 weisshorn01 kernel: eth0: no IPv6 routers present
Jun 19 18:45:42 weisshorn01 kernel: powernow-k8: Found 4 AMD Opteron(tm) Processor 6128 (16 cpu cores) (version 2.20.00)
Jun 19 18:45:42 weisshorn01 kernel: powernow-k8:    0 : pstate 0 (2000 MHz)
Jun 19 18:45:42 weisshorn01 kernel: powernow-k8:    1 : pstate 1 (1500 MHz)
Jun 19 18:45:42 weisshorn01 kernel: powernow-k8:    2 : pstate 2 (1200 MHz)
Jun 19 18:45:42 weisshorn01 kernel: powernow-k8:    3 : pstate 3 (1000 MHz)
Jun 19 18:45:42 weisshorn01 kernel: powernow-k8:    4 : pstate 4 (800 MHz)
Jun 19 18:45:42 weisshorn01 kernel: powernow-k8:    0 : pstate 0 (2000 MHz)
Jun 19 18:45:42 weisshorn01 kernel: powernow-k8:    1 : pstate 1 (1500 MHz)
Jun 19 18:45:42 weisshorn01 kernel: powernow-k8:    2 : pstate 2 (1200 MHz)
Jun 19 18:45:42 weisshorn01 kernel: powernow-k8:    3 : pstate 3 (1000 MHz)
Jun 19 18:45:42 weisshorn01 kernel: powernow-k8:    4 : pstate 4 (800 MHz)
Jun 19 18:45:42 weisshorn01 cpuspeed: Enabling ondemand cpu frequency scaling governor
Jun 19 18:45:42 weisshorn01 rpc.statd[2269]: Version 1.2.3 starting
Jun 19 18:45:42 weisshorn01 sm-notify[2270]: Version 1.2.3 starting
Jun 19 18:45:42 weisshorn01 kernel: RPC: Registered udp transport module.
Jun 19 18:45:42 weisshorn01 kernel: RPC: Registered tcp transport module.
Jun 19 18:45:42 weisshorn01 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Jun 19 18:45:43 weisshorn01 kdump: kexec: loaded kdump kernel
Jun 19 18:45:43 weisshorn01 kdump: started up
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: Found user 'avahi' (UID 70) and group 'avahi' (GID 70).
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: Successfully dropped root privileges.
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: avahi-daemon 0.6.25 starting up.
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: Successfully called chroot().
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: Successfully dropped remaining capabilities.
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: Loading service file /services/ssh.service.
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: Joining mDNS multicast group on interface ib0.IPv4 with address 148.187.7.101.
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: New relevant interface ib0.IPv4 for mDNS.
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: Joining mDNS multicast group on interface eth1.IPv4 with address 192.168.0.101.
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: New relevant interface eth1.IPv4 for mDNS.
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: Joining mDNS multicast group on interface eth0.IPv4 with address 148.187.12.231.
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: New relevant interface eth0.IPv4 for mDNS.
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: Network interface enumeration completed.
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: Registering new address record for 148.187.7.101 on ib0.IPv4.
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: Registering new address record for fe80::230:48ff:fefe:ccdd on eth1.*.
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: Registering new address record for 192.168.0.101 on eth1.IPv4.
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: Registering new address record for fe80::230:48ff:fefe:ccdc on eth0.*.
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: Registering new address record for 148.187.12.231 on eth0.IPv4.
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: Registering HINFO record with values 'X86_64'/'LINUX'.
Jun 19 18:45:44 weisshorn01 acpid: starting up
Jun 19 18:45:44 weisshorn01 acpid: 1 rule loaded
Jun 19 18:45:44 weisshorn01 acpid: waiting for events: event logging is off
Jun 19 18:45:44 weisshorn01 avahi-daemon[2421]: Server startup complete. Host name is weisshorn01.local. Local service cookie is 2631207856.
Jun 19 18:45:45 weisshorn01 acpid: client connected from 2513[68:68]
Jun 19 18:45:45 weisshorn01 acpid: 1 client rule loaded
Jun 19 18:45:45 weisshorn01 avahi-daemon[2421]: Service "weisshorn01" (/services/ssh.service) successfully established.
Jun 19 18:45:46 weisshorn01 kernel: eth1: no IPv6 routers present
Jun 19 18:45:46 weisshorn01 mcelog: failed to prefill DIMM database from DMI data
Jun 19 18:45:46 weisshorn01 mcelog: mcelog read: No such device
Jun 19 18:45:47 weisshorn01 sshd[2576]: Server listening on 0.0.0.0 port 22.
Jun 19 18:45:47 weisshorn01 sshd[2576]: Server listening on :: port 22.
Jun 19 18:45:47 weisshorn01 xinetd[2584]: Reading included configuration file: /etc/xinetd.d/chargen-dgram [file=/etc/xinetd.conf] [line=49]
Jun 19 18:45:47 weisshorn01 xinetd[2584]: Reading included configuration file: /etc/xinetd.d/chargen-stream [file=/etc/xinetd.d/chargen-stream] [line=67]
Jun 19 18:45:47 weisshorn01 xinetd[2584]: Reading included configuration file: /etc/xinetd.d/cvs [file=/etc/xinetd.d/cvs] [line=67]
Jun 19 18:45:47 weisshorn01 xinetd[2584]: Reading included configuration file: /etc/xinetd.d/daytime-dgram [file=/etc/xinetd.d/daytime-dgram] [line=19]
Jun 19 18:45:47 weisshorn01 xinetd[2584]: Reading included configuration file: /etc/xinetd.d/daytime-stream [file=/etc/xinetd.d/daytime-stream] [line=67]
Jun 19 18:45:47 weisshorn01 xinetd[2584]: Reading included configuration file: /etc/xinetd.d/discard-dgram [file=/etc/xinetd.d/discard-dgram] [line=67]
Jun 19 18:45:47 weisshorn01 xinetd[2584]: Reading included configuration file: /etc/xinetd.d/discard-stream [file=/etc/xinetd.d/discard-stream] [line=67]
Jun 19 18:45:47 weisshorn01 xinetd[2584]: Reading included configuration file: /etc/xinetd.d/echo-dgram [file=/etc/xinetd.d/echo-dgram] [line=67]
Jun 19 18:45:47 weisshorn01 xinetd[2584]: Reading included configuration file: /etc/xinetd.d/echo-stream [file=/etc/xinetd.d/echo-stream] [line=67]
Jun 19 18:45:47 weisshorn01 xinetd[2584]: Reading included configuration file: /etc/xinetd.d/rsync [file=/etc/xinetd.d/rsync] [line=67]
Jun 19 18:45:47 weisshorn01 xinetd[2584]: Reading included configuration file: /etc/xinetd.d/tcpmux-server [file=/etc/xinetd.d/tcpmux-server] [line=14]
Jun 19 18:45:47 weisshorn01 xinetd[2584]: Reading included configuration file: /etc/xinetd.d/time-dgram [file=/etc/xinetd.d/time-dgram] [line=68]
Jun 19 18:45:47 weisshorn01 xinetd[2584]: Reading included configuration file: /etc/xinetd.d/time-stream [file=/etc/xinetd.d/time-stream] [line=67]
Jun 19 18:45:47 weisshorn01 xinetd[2584]: removing chargen
Jun 19 18:45:47 weisshorn01 xinetd[2584]: removing chargen
Jun 19 18:45:47 weisshorn01 xinetd[2584]: removing cvspserver
Jun 19 18:45:47 weisshorn01 xinetd[2584]: removing daytime
Jun 19 18:45:47 weisshorn01 xinetd[2584]: removing daytime
Jun 19 18:45:47 weisshorn01 xinetd[2584]: removing discard
Jun 19 18:45:47 weisshorn01 xinetd[2584]: removing discard
Jun 19 18:45:47 weisshorn01 xinetd[2584]: removing echo
Jun 19 18:45:47 weisshorn01 xinetd[2584]: removing echo
Jun 19 18:45:47 weisshorn01 xinetd[2584]: removing rsync
Jun 19 18:45:47 weisshorn01 xinetd[2584]: removing tcpmux
Jun 19 18:45:47 weisshorn01 xinetd[2584]: removing time
Jun 19 18:45:47 weisshorn01 xinetd[2584]: removing time
Jun 19 18:45:47 weisshorn01 xinetd[2584]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in.
Jun 19 18:45:47 weisshorn01 xinetd[2584]: Started working: 0 available services
Jun 19 19:45:47 weisshorn01 ntpdate[2598]: step time server 148.187.18.145 offset 3599.965335 sec
Jun 19 19:45:47 weisshorn01 ntpd[2608]: ntpd 4.2.4p8@1.1612-o Thu May 13 14:38:25 UTC 2010 (1)
Jun 19 19:45:47 weisshorn01 ntpd[2609]: precision = 0.345 usec
Jun 19 19:45:47 weisshorn01 ntpd[2609]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
Jun 19 19:45:47 weisshorn01 ntpd[2609]: Listening on interface #0 wildcard, 0.0.0.0#123 Disabled
Jun 19 19:45:47 weisshorn01 ntpd[2609]: Listening on interface #1 wildcard, ::#123 Disabled
Jun 19 19:45:47 weisshorn01 ntpd[2609]: Listening on interface #2 lo, ::1#123 Enabled
Jun 19 19:45:47 weisshorn01 ntpd[2609]: Listening on interface #3 eth0, fe80::230:48ff:fefe:ccdc#123 Enabled
Jun 19 19:45:47 weisshorn01 ntpd[2609]: Listening on interface #4 eth1, fe80::230:48ff:fefe:ccdd#123 Enabled
Jun 19 19:45:47 weisshorn01 ntpd[2609]: Listening on interface #5 lo, 127.0.0.1#123 Enabled
Jun 19 19:45:47 weisshorn01 ntpd[2609]: Listening on interface #6 eth0, 148.187.12.231#123 Enabled
Jun 19 19:45:47 weisshorn01 ntpd[2609]: Listening on interface #7 eth1, 192.168.0.101#123 Enabled
Jun 19 19:45:47 weisshorn01 ntpd[2609]: Listening on interface #8 ib0, 148.187.7.101#123 Enabled
Jun 19 19:45:47 weisshorn01 ntpd[2609]: Listening on routing socket on fd #25 for interface updates
Jun 19 19:45:47 weisshorn01 ntpd[2609]: kernel time sync status 2040
Jun 19 19:45:47 weisshorn01 ntpd[2609]: frequency initialized -27.756 PPM from /var/lib/ntp/drift
Jun 19 19:45:48 weisshorn01 nrpe[2625]: Starting up daemon
Jun 19 19:45:48 weisshorn01 nrpe[2625]: Listening for connections on port 5666
Jun 19 19:45:48 weisshorn01 nrpe[2625]: Allowing connections from: 127.0.0.1,148.187.12.21,148.187.3.23
Jun 19 19:45:48 weisshorn01 postfix/postfix-script[2700]: starting the Postfix mail system
Jun 19 19:45:48 weisshorn01 postfix/master[2701]: daemon started -- version 2.6.6, configuration /etc/postfix
Jun 19 19:45:49 weisshorn01 abrtd: Init complete, entering main loop
Jun 19 19:45:49 weisshorn01 runuser: pam_unix(runuser:session): session opened for user qpidd by (uid=0)
Jun 19 19:45:49 weisshorn01 runuser: pam_unix(runuser:session): session closed for user qpidd
Jun 19 19:45:49 weisshorn01 runuser: pam_unix(runuser:session): session opened for user qpidd by (uid=0)
Jun 19 19:45:49 weisshorn01 qpidd[2744]: 2012-06-19 19:45:49 notice Listening on TCP/TCP6 port 5672
Jun 19 19:45:49 weisshorn01 qpidd[2744]: 2012-06-19 19:45:49 notice SSL plugin not enabled, you must set --ssl-cert-db to enable it.
Jun 19 19:45:49 weisshorn01 qpidd[2744]: 2012-06-19 19:45:49 notice Broker running
Jun 19 19:45:49 weisshorn01 runuser: pam_unix(runuser:session): session closed for user qpidd
Jun 19 19:45:49 weisshorn01 runuser: pam_unix(runuser-l:session): session opened for user qpidd by (uid=0)
Jun 19 19:45:49 weisshorn01 runuser: pam_unix(runuser-l:session): session closed for user qpidd
Jun 19 19:45:50 weisshorn01 crond[2801]: (CRON) STARTUP (1.4.4)
Jun 19 19:45:50 weisshorn01 crond[2801]: (CRON) INFO (running with inotify support)
Jun 19 19:45:50 weisshorn01 rhnsd[2832]: Red Hat Network Services Daemon starting up, check in interval 240 minutes.
Jun 19 19:45:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:45:52 weisshorn01 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
Jun 19 19:45:52 weisshorn01 kernel: Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel-498-gd2c1a39-PRISTINE-2.6.32-220.7.1.el6_lustre.g9c8f747.x86_64
Jun 19 19:45:53 weisshorn01 avahi-daemon[2421]: Registering new address record for fe80::202:c903:30:2771 on ib0.*.
Jun 19 19:45:54 weisshorn01 kernel: LNet: Added LNI 148.187.7.101@o2ib2 [8/64/0/180]
Jun 19 19:45:54 weisshorn01 ntpd[2609]: Listening on interface #9 ib0, fe80::202:c903:30:2771#123 Enabled
Jun 19 19:45:54 weisshorn01 ntpd[2609]: new interface(s) found: waking up resolver
Jun 19 19:45:54 weisshorn01 ntpd[2609]: synchronized to 148.187.18.145, stratum 2
Jun 19 19:45:54 weisshorn01 ntpd[2609]: kernel time sync status change 2001
Jun 19 19:46:02 weisshorn10 nslcd[4294]: [6be6b9] ldap_result() failed: Can't contact LDAP server
Jun 19 19:46:02 weisshorn01 kernel: ib0: no IPv6 routers present
Jun 19 19:46:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:46:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:46:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:46:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:47:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:47:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:47:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:47:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:48:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:48:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:48:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:48:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:49:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:49:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:49:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:49:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:50:01 weisshorn02 CROND[47864]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:50:01 weisshorn07 CROND[126029]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:50:01 weisshorn01 CROND[2986]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:50:01 weisshorn03 CROND[126580]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:50:01 weisshorn10 CROND[125914]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:50:01 weisshorn05 CROND[125888]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:50:01 weisshorn06 CROND[126814]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:50:01 weisshorn14 CROND[54448]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:50:01 weisshorn09 CROND[125604]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:50:01 weisshorn08 CROND[124662]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:50:01 weisshorn13 CROND[129470]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:50:01 weisshorn12 CROND[127431]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:50:02 weisshorn04 CROND[125352]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 19:50:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:50:16 weisshorn14 nslcd[3437]: [d53685] ldap_result() failed: Can't contact LDAP server
Jun 19 19:50:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:50:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:50:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:51:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:51:10 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340128214/real 1340128214]  req@ffff8806add77400 x1403849943319252/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340128270 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:51:10 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Jun 19 19:51:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:51:31 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340128235/real 1340128235]  req@ffff88100de7a000 x1403849871564781/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340128291 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:51:31 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Jun 19 19:51:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:51:36 weisshorn12 rhnsd[127435]: running program /usr/sbin/rhn_check
Jun 19 19:51:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:51:57 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340128261/real 1340128261]  req@ffff880b15016800 x1403849861694441/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340128317 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:51:57 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Jun 19 19:52:05 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340128269/real 1340128269]  req@ffff880a39981c00 x1403849984584460/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340128325 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:52:05 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Jun 19 19:52:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:52:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:52:32 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340128296/real 1340128296]  req@ffff880f47b78400 x1403849897227551/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340128352 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:52:32 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Jun 19 19:52:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:52:48 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340128312/real 1340128312]  req@ffff880911672800 x1403849921630087/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340128368 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:52:48 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 19:52:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:52:57 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340128321/real 1340128321]  req@ffff880809d2ac00 x1403849636569542/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340128377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:52:57 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 19:52:59 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340128323/real 1340128323]  req@ffff88081a42dc00 x1403849873376172/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340128379 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:52:59 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Jun 19 19:53:02 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340128326/real 1340128326]  req@ffff880a5b4ae400 x1403849822815123/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340128382 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:53:02 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Jun 19 19:53:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:53:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:53:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:53:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:54:04 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340128388/real 1340128388]  req@ffff8805016e6c00 x1403849953048127/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340128444 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 19:54:04 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Jun 19 19:54:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:54:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:54:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:54:42 weisshorn03 kernel: Lustre: scratch-OST002c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101008e000, cur 1340128482 expire 1340128332 last 1340128255
Jun 19 19:54:42 weisshorn03 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 19:54:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:55:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:55:07 weisshorn08 kernel: Lustre: scratch-OST000c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d52230c00, cur 1340128507 expire 1340128357 last 1340128280
Jun 19 19:55:07 weisshorn08 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 19:55:07 weisshorn13 kernel: Lustre: scratch-OST000b: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802f1942800, cur 1340128507 expire 1340128357 last 1340128280
Jun 19 19:55:07 weisshorn13 kernel: Lustre: Skipped 20 previous similar messages
Jun 19 19:55:07 weisshorn12 kernel: Lustre: scratch-OST0021: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aeb0bd400, cur 1340128507 expire 1340128357 last 1340128280
Jun 19 19:55:07 weisshorn12 kernel: Lustre: Skipped 8 previous similar messages
Jun 19 19:55:07 weisshorn07 kernel: Lustre: scratch-OST0042: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d8b65f800, cur 1340128507 expire 1340128357 last 1340128280
Jun 19 19:55:07 weisshorn07 kernel: Lustre: Skipped 8 previous similar messages
Jun 19 19:55:07 weisshorn06 kernel: Lustre: scratch-OST0041: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881015f9a000, cur 1340128507 expire 1340128357 last 1340128280
Jun 19 19:55:07 weisshorn06 kernel: Lustre: Skipped 6 previous similar messages
Jun 19 19:55:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:55:32 weisshorn05 kernel: Lustre: scratch-OST0027: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e63449c00, cur 1340128532 expire 1340128382 last 1340128305
Jun 19 19:55:32 weisshorn05 kernel: Lustre: Skipped 6 previous similar messages
Jun 19 19:55:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:55:39 weisshorn02 kernel: Lustre: 47681:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004fb4:0x5:0x0]] [valid 68719476736]
Jun 19 19:55:39 weisshorn02 kernel: Lustre: 47681:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 1 previous similar message
Jun 19 19:55:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:56:02 weisshorn01 nslcd[2168]: [6afb66] ldap_result() failed: Can't contact LDAP server
Jun 19 19:56:02 weisshorn01 nslcd[2168]: [6afb66] ldap_abandon() failed to abandon search: Other (e.g., implementation specific) error
Jun 19 19:56:02 weisshorn01 nslcd[2168]: [e45d32] ldap_result() failed: Can't contact LDAP server
Jun 19 19:56:02 weisshorn01 nslcd[2168]: [9b500d] ldap_result() failed: Can't contact LDAP server
Jun 19 19:56:02 weisshorn01 nslcd[2168]: [1bd7b7] ldap_result() failed: Can't contact LDAP server
Jun 19 19:56:02 weisshorn01 nslcd[2168]: [2dba31] ldap_result() failed: Can't contact LDAP server
Jun 19 19:56:04 weisshorn02 kernel: Lustre: 47680:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004fb4:0xa:0x0]] [valid 68719476736]
Jun 19 19:56:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:56:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:56:34 weisshorn02 kernel: Lustre: 47824:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004fb4:0xf:0x0]] [valid 68719476736]
Jun 19 19:56:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:56:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:57:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:57:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:57:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:57:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:58:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:58:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:58:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:58:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:59:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:59:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:59:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 19:59:40 weisshorn02 kernel: LustreError: 47685:0:(mdt_handler.c:2822:mdt_recovery()) operation 400 on unconnected MDS from 12345-148.187.6.7@o2ib2
Jun 19 19:59:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:00:01 weisshorn02 CROND[47968]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:00:01 weisshorn12 CROND[127454]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:00:01 weisshorn04 CROND[125383]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:00:01 weisshorn07 CROND[126073]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:00:01 weisshorn01 CROND[3052]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:00:01 weisshorn03 CROND[126615]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:00:01 weisshorn10 CROND[125945]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:00:01 weisshorn05 CROND[125920]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:00:01 weisshorn06 CROND[126848]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:00:01 weisshorn14 CROND[54482]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:00:01 weisshorn08 CROND[124689]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:00:01 weisshorn09 CROND[125638]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:00:02 weisshorn13 CROND[129496]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:00:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:00:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:00:31 weisshorn10 nslcd[4294]: [234363] ldap_result() failed: Can't contact LDAP server
Jun 19 20:00:31 weisshorn06 nslcd[3953]: [6f6d8a] ldap_result() failed: Can't contact LDAP server
Jun 19 20:00:31 weisshorn05 nslcd[4378]: [234363] ldap_result() failed: Can't contact LDAP server
Jun 19 20:00:31 weisshorn13 nslcd[4224]: [6be6b9] ldap_result() failed: Can't contact LDAP server
Jun 19 20:00:31 weisshorn09 nslcd[3954]: [0e6897] ldap_result() failed: Can't contact LDAP server
Jun 19 20:00:31 weisshorn03 nslcd[3944]: [5138de] ldap_result() failed: Can't contact LDAP server
Jun 19 20:00:31 weisshorn12 nslcd[3954]: [6be6b9] ldap_result() failed: Can't contact LDAP server
Jun 19 20:00:31 weisshorn07 nslcd[4297]: [6be6b9] ldap_result() failed: Can't contact LDAP server
Jun 19 20:00:31 weisshorn08 nslcd[3946]: [6be6b9] ldap_result() failed: Can't contact LDAP server
Jun 19 20:00:31 weisshorn04 nslcd[3885]: [0e6897] ldap_result() failed: Can't contact LDAP server
Jun 19 20:00:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:00:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:01:01 weisshorn13 CROND[129506]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 20:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1295 starting 0anacron
Jun 19 20:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1295 finished 0anacron
Jun 19 20:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1295 starting mcelog.cron
Jun 19 20:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1295 finished mcelog.cron
Jun 19 20:01:01 weisshorn02 CROND[47985]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 20:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4798 starting 0anacron
Jun 19 20:01:01 weisshorn04 CROND[125388]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 20:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4799 finished 0anacron
Jun 19 20:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4798 starting mcelog.cron
Jun 19 20:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1253 starting 0anacron
Jun 19 20:01:01 weisshorn12 CROND[127464]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 20:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4800 finished mcelog.cron
Jun 19 20:01:01 weisshorn07 CROND[126113]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 20:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1253 finished 0anacron
Jun 19 20:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1253 starting mcelog.cron
Jun 19 20:01:01 weisshorn01 CROND[3057]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 20:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1261 starting 0anacron
Jun 19 20:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3057 starting 0anacron
Jun 19 20:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1254 finished mcelog.cron
Jun 19 20:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1274 starting 0anacron
Jun 19 20:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1261 finished 0anacron
Jun 19 20:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1261 starting mcelog.cron
Jun 19 20:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3067 finished 0anacron
Jun 19 20:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3057 starting mcelog.cron
Jun 19 20:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1274 finished 0anacron
Jun 19 20:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1274 starting mcelog.cron
Jun 19 20:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3074 finished mcelog.cron
Jun 19 20:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1261 finished mcelog.cron
Jun 19 20:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1274 finished mcelog.cron
Jun 19 20:01:01 weisshorn03 CROND[126620]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 20:01:01 weisshorn10 CROND[125950]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 20:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1266 starting 0anacron
Jun 19 20:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1259 starting 0anacron
Jun 19 20:01:01 weisshorn05 CROND[125925]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 20:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1259 finished 0anacron
Jun 19 20:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1259 starting mcelog.cron
Jun 19 20:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1266 finished 0anacron
Jun 19 20:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1266 starting mcelog.cron
Jun 19 20:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1259 finished mcelog.cron
Jun 19 20:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1259 starting 0anacron
Jun 19 20:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1266 finished mcelog.cron
Jun 19 20:01:01 weisshorn14 CROND[54487]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 20:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5448 starting 0anacron
Jun 19 20:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1259 finished 0anacron
Jun 19 20:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1259 starting mcelog.cron
Jun 19 20:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5449 finished 0anacron
Jun 19 20:01:01 weisshorn06 CROND[126853]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 20:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5448 starting mcelog.cron
Jun 19 20:01:01 weisshorn08 CROND[124699]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 20:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5450 finished mcelog.cron
Jun 19 20:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1268 starting 0anacron
Jun 19 20:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1259 finished mcelog.cron
Jun 19 20:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1246 starting 0anacron
Jun 19 20:01:01 weisshorn09 CROND[125648]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 20:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1256 starting 0anacron
Jun 19 20:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1268 finished 0anacron
Jun 19 20:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1268 starting mcelog.cron
Jun 19 20:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1256 finished 0anacron
Jun 19 20:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1256 starting mcelog.cron
Jun 19 20:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1247 finished 0anacron
Jun 19 20:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1246 starting mcelog.cron
Jun 19 20:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1268 finished mcelog.cron
Jun 19 20:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1256 finished mcelog.cron
Jun 19 20:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1247 finished mcelog.cron
Jun 19 20:01:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:01:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:01:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:01:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:02:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:02:12 weisshorn04 kernel: Lustre: scratch-OST0025: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c682f5800, cur 1340128932 expire 1340128782 last 1340128705
Jun 19 20:02:12 weisshorn04 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 20:02:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:02:25 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340128889/real 1340128889]  req@ffff880e3bb36c00 x1403849943428643/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340128945 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:02:25 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:02:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:02:37 weisshorn09 kernel: Lustre: scratch-OST0004: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881024aab800, cur 1340128957 expire 1340128807 last 1340128730
Jun 19 20:02:37 weisshorn09 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 20:02:46 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340128910/real 1340128910]  req@ffff8809de7af800 x1403849871677785/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340128966 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:02:46 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:02:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:03:02 weisshorn13 kernel: Lustre: scratch-OST0024: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880467b6a400, cur 1340128982 expire 1340128832 last 1340128755
Jun 19 20:03:02 weisshorn13 kernel: Lustre: Skipped 1 previous similar message
Jun 19 20:03:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:03:12 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340128936/real 1340128936]  req@ffff8810195da000 x1403849861769736/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340128992 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:03:12 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:03:20 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340128944/real 1340128944]  req@ffff880f272df000 x1403849984811873/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340129000 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:03:20 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:03:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:03:27 weisshorn02 kernel: Lustre: scratch-MDT0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bb4e9a800, cur 1340129007 expire 1340128857 last 1340128780
Jun 19 20:03:27 weisshorn02 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 20:03:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:03:47 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340128971/real 1340128971]  req@ffff880620182c00 x1403849897352469/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340129027 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:03:47 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:03:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:03:52 weisshorn10 kernel: Lustre: scratch-OST000a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ef9b1dc00, cur 1340129032 expire 1340128882 last 1340128805
Jun 19 20:03:52 weisshorn10 kernel: Lustre: Skipped 1 previous similar message
Jun 19 20:04:03 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340128987/real 1340128987]  req@ffff88080511e400 x1403849921769460/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340129043 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:04:03 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:04:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:04:12 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340128996/real 1340128996]  req@ffff8810239b6000 x1403849636710777/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340129052 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:04:12 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:04:14 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340128998/real 1340128998]  req@ffff8803c4f37000 x1403849873501202/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340129054 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:04:14 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:04:17 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340129001/real 1340129001]  req@ffff880c41779000 x1403849822858218/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340129057 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:04:17 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:04:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:04:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:04:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:05:02 weisshorn02 kernel: LustreError: 47715:0:(mdt_handler.c:2822:mdt_recovery()) operation 41 on unconnected MDS from 12345-148.187.6.7@o2ib2
Jun 19 20:05:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:05:19 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340129063/real 1340129063]  req@ffff880e85c41800 x1403849953289734/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340129119 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:05:19 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:05:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:05:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:05:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:06:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:06:17 weisshorn01 nslcd[2168]: [1da317] ldap_result() failed: Can't contact LDAP server
Jun 19 20:06:17 weisshorn01 nslcd[2168]: [43a858] ldap_result() failed: Can't contact LDAP server
Jun 19 20:06:17 weisshorn01 nslcd[2168]: [1d5ae9] ldap_result() failed: Can't contact LDAP server
Jun 19 20:06:17 weisshorn01 nslcd[2168]: [63845e] ldap_result() failed: Can't contact LDAP server
Jun 19 20:06:18 weisshorn01 nslcd[2168]: [a2a8d4] ldap_result() failed: Can't contact LDAP server
Jun 19 20:06:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:06:22 weisshorn06 kernel: Lustre: scratch-OST001d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881013c05c00, cur 1340129182 expire 1340129032 last 1340128955
Jun 19 20:06:22 weisshorn06 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 20:06:22 weisshorn13 kernel: Lustre: scratch-OST0005: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c668a4400, cur 1340129182 expire 1340129032 last 1340128955
Jun 19 20:06:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:06:36 weisshorn03 nslcd[3944]: [22ad0c] ldap_result() failed: Can't contact LDAP server
Jun 19 20:06:47 weisshorn08 kernel: Lustre: scratch-OST000c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c2dcc5800, cur 1340129207 expire 1340129057 last 1340128980
Jun 19 20:06:47 weisshorn12 kernel: Lustre: scratch-OST0021: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d9feae400, cur 1340129207 expire 1340129057 last 1340128980
Jun 19 20:06:47 weisshorn07 kernel: Lustre: scratch-OST0042: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807debfcc00, cur 1340129207 expire 1340129057 last 1340128980
Jun 19 20:06:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:07:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:07:12 weisshorn02 kernel: Lustre: MGS: haven't heard from client 02e8b9ff-fc43-5841-d3bb-815052085829 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bb51ba400, cur 1340129232 expire 1340129082 last 1340129005
Jun 19 20:07:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:07:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:07:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:08:02 weisshorn05 kernel: Lustre: scratch-OST0027: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080bbb6400, cur 1340129282 expire 1340129132 last 1340129055
Jun 19 20:08:02 weisshorn05 kernel: Lustre: Skipped 1 previous similar message
Jun 19 20:08:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:08:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:08:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:08:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:09:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:09:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:09:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:09:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:10:01 weisshorn13 CROND[129581]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:10:01 weisshorn02 CROND[48038]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:10:01 weisshorn04 CROND[125428]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:10:01 weisshorn01 CROND[3105]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:10:01 weisshorn01 CROND[3107]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 20:10:01 weisshorn07 CROND[126154]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:10:01 weisshorn12 CROND[127550]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:10:01 weisshorn03 CROND[126660]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:10:01 weisshorn10 CROND[125990]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:10:01 weisshorn14 CROND[54555]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:10:01 weisshorn05 CROND[125966]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:10:01 weisshorn06 CROND[126901]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:10:01 weisshorn08 CROND[124744]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:10:01 weisshorn05 sshd[125965]: Accepted publickey for root from 148.187.12.231 port 46585 ssh2
Jun 19 20:10:01 weisshorn14 sshd[54557]: Accepted publickey for root from 148.187.12.231 port 47163 ssh2
Jun 19 20:10:01 weisshorn14 sshd[54557]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 20:10:01 weisshorn05 sshd[125965]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 20:10:01 weisshorn04 sshd[125430]: Accepted publickey for root from 148.187.12.231 port 60149 ssh2
Jun 19 20:10:01 weisshorn04 sshd[125430]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 20:10:01 weisshorn07 sshd[126156]: Accepted publickey for root from 148.187.12.231 port 57362 ssh2
Jun 19 20:10:01 weisshorn07 sshd[126156]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 20:10:01 weisshorn12 sshd[127552]: Accepted publickey for root from 148.187.12.231 port 42312 ssh2
Jun 19 20:10:01 weisshorn14 sshd[54557]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 20:10:01 weisshorn14 sshd[54557]: pam_unix(sshd:session): session closed for user root
Jun 19 20:10:01 weisshorn10 sshd[125992]: Accepted publickey for root from 148.187.12.231 port 42185 ssh2
Jun 19 20:10:01 weisshorn12 sshd[127552]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 20:10:01 weisshorn10 sshd[125992]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 20:10:01 weisshorn03 sshd[126662]: Accepted publickey for root from 148.187.12.231 port 38199 ssh2
Jun 19 20:10:01 weisshorn06 sshd[126898]: Accepted publickey for root from 148.187.12.231 port 54700 ssh2
Jun 19 20:10:01 weisshorn03 sshd[126662]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 20:10:01 weisshorn09 CROND[125685]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:10:01 weisshorn08 sshd[124741]: Accepted publickey for root from 148.187.12.231 port 39093 ssh2
Jun 19 20:10:01 weisshorn06 sshd[126898]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 20:10:02 weisshorn08 sshd[124741]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 20:10:02 weisshorn04 sshd[125430]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 20:10:02 weisshorn04 sshd[125430]: pam_unix(sshd:session): session closed for user root
Jun 19 20:10:02 weisshorn05 sshd[125965]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 20:10:02 weisshorn05 sshd[125965]: pam_unix(sshd:session): session closed for user root
Jun 19 20:10:02 weisshorn07 sshd[126156]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 20:10:02 weisshorn07 sshd[126156]: pam_unix(sshd:session): session closed for user root
Jun 19 20:10:02 weisshorn13 sshd[129583]: Accepted publickey for root from 148.187.12.231 port 40119 ssh2
Jun 19 20:10:02 weisshorn13 sshd[129583]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 20:10:02 weisshorn09 sshd[125682]: Accepted publickey for root from 148.187.12.231 port 56743 ssh2
Jun 19 20:10:02 weisshorn09 sshd[125682]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 20:10:02 weisshorn03 sshd[126662]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 20:10:02 weisshorn03 sshd[126662]: pam_unix(sshd:session): session closed for user root
Jun 19 20:10:02 weisshorn08 sshd[124741]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 20:10:02 weisshorn08 sshd[124741]: pam_unix(sshd:session): session closed for user root
Jun 19 20:10:02 weisshorn06 sshd[126898]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 20:10:02 weisshorn06 sshd[126898]: pam_unix(sshd:session): session closed for user root
Jun 19 20:10:02 weisshorn12 sshd[127552]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 20:10:02 weisshorn12 sshd[127552]: pam_unix(sshd:session): session closed for user root
Jun 19 20:10:02 weisshorn10 sshd[125992]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 20:10:02 weisshorn10 sshd[125992]: pam_unix(sshd:session): session closed for user root
Jun 19 20:10:02 weisshorn13 sshd[129583]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 20:10:02 weisshorn13 sshd[129583]: pam_unix(sshd:session): session closed for user root
Jun 19 20:10:02 weisshorn09 sshd[125682]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 20:10:02 weisshorn09 sshd[125682]: pam_unix(sshd:session): session closed for user root
Jun 19 20:10:04 weisshorn01 postfix/pickup[2716]: CB63E92224F: uid=0 from=<root>
Jun 19 20:10:04 weisshorn01 postfix/cleanup[3137]: CB63E92224F: message-id=<20120619181004.CB63E92224F@weisshorn01.admin.cscs.ch>
Jun 19 20:10:04 weisshorn01 postfix/qmgr[2717]: CB63E92224F: from=<root@weisshorn01.admin.cscs.ch>, size=1128, nrcpt=1 (queue active)
Jun 19 20:10:05 weisshorn01 postfix/local[3139]: CB63E92224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=3.3, delays=3.2/0.07/0/0.07, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 20:10:05 weisshorn01 postfix/qmgr[2717]: CB63E92224F: removed
Jun 19 20:10:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:10:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:10:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:10:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:10:58 weisshorn02 kernel: LustreError: 48116:0:(mdt_recovery.c:813:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 55836153037, new: 55836152975 replay: 0. see LU-617.
Jun 19 20:10:58 weisshorn02 kernel: LustreError: 48113:0:(mdt_recovery.c:813:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 55836153176, new: 55836153147 replay: 0. see LU-617.
Jun 19 20:11:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:11:08 weisshorn03 kernel: Lustre: scratch-OST0020: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881011b2a000, cur 1340129468 expire 1340129318 last 1340129241
Jun 19 20:11:08 weisshorn03 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 20:11:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:11:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:11:39 weisshorn02 kernel: Lustre: scratch-MDT0000: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080ea8f000, cur 1340129499 expire 1340129349 last 1340129272
Jun 19 20:11:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:12:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:12:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:12:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:12:46 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340129510/real 1340129510]  req@ffff8809e33d6000 x1403849871785438/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340129566 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:12:46 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 20:12:51 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:13:06 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:13:20 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340129544/real 1340129544]  req@ffff8802bd3f1400 x1403849984991388/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340129600 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:13:20 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 20:13:21 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:13:36 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:13:40 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340129564/real 1340129564]  req@ffff88100deac000 x1403849943533902/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340129620 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:13:40 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:13:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:14:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:14:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:14:27 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340129611/real 1340129611]  req@ffff8810195a4c00 x1403849861853706/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340129667 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:14:27 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:14:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:14:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:15:02 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340129646/real 1340129646]  req@ffff8810132bec00 x1403849897472446/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340129702 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:15:02 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:15:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:15:18 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340129662/real 1340129662]  req@ffff8808020a8400 x1403849921908328/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340129718 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:15:18 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:15:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:15:27 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340129671/real 1340129671]  req@ffff8806634eb800 x1403849636826778/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340129727 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:15:27 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:15:29 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340129673/real 1340129673]  req@ffff88102c702800 x1403849873619439/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340129729 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:15:29 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:15:32 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340129676/real 1340129676]  req@ffff8809bdff3000 x1403849822908276/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340129732 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:15:32 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:15:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:15:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:16:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:16:18 weisshorn01 nslcd[2168]: [838cb2] ldap_result() failed: Can't contact LDAP server
Jun 19 20:16:18 weisshorn01 nslcd[2168]: [53d0cd] ldap_result() failed: Can't contact LDAP server
Jun 19 20:16:18 weisshorn01 nslcd[2168]: [53d0cd] ldap_abandon() failed to abandon search: Other (e.g., implementation specific) error
Jun 19 20:16:22 weisshorn13 kernel: Lustre: scratch-OST0011: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101225c800, cur 1340129782 expire 1340129632 last 1340129555
Jun 19 20:16:22 weisshorn13 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 20:16:22 weisshorn09 kernel: Lustre: scratch-OST0026: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8d201c00, cur 1340129782 expire 1340129632 last 1340129555
Jun 19 20:16:22 weisshorn09 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 20:16:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:16:33 weisshorn01 nslcd[2168]: [03e0c6] ldap_result() failed: Can't contact LDAP server
Jun 19 20:16:33 weisshorn01 nslcd[2168]: [9a769b] ldap_result() failed: Can't contact LDAP server
Jun 19 20:16:34 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340129738/real 1340129738]  req@ffff880440592000 x1403849953503985/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340129794 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:16:34 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:16:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:16:47 weisshorn06 kernel: Lustre: scratch-OST0007: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809ab437800, cur 1340129807 expire 1340129657 last 1340129580
Jun 19 20:16:47 weisshorn06 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 20:16:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:17:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:17:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:17:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:17:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:18:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:18:11 weisshorn02 kernel: LustreError: 47420:0:(mdt_handler.c:2822:mdt_recovery()) operation 400 on unconnected MDS from 12345-148.187.6.6@o2ib2
Jun 19 20:18:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:18:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:18:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:18:56 weisshorn12 kernel: LNet: Service thread pid 8332 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Jun 19 20:18:56 weisshorn12 kernel: LNet: Skipped 3 previous similar messages
Jun 19 20:18:56 weisshorn12 kernel: Pid: 8332, comm: ll_ost_416
Jun 19 20:18:56 weisshorn12 kernel:
Jun 19 20:18:56 weisshorn12 kernel: Call Trace:
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa08c21c0>] ? at_measured+0xd0/0x310 [ptlrpc]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffff8107bf8c>] ? lock_timer_base+0x3c/0x70
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffff814edc52>] schedule_timeout+0x192/0x2e0
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffff8107c0a0>] ? process_timeout+0x0/0x10
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa03a85c1>] cfs_waitq_timedwait+0x11/0x20 [libcfs]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa089f6e8>] ptlrpc_set_wait+0x288/0x700 [ptlrpc]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffff8105e7f0>] ? default_wake_function+0x0/0x20
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa089fbe6>] ptlrpc_queue_wait+0x86/0x1c0 [ptlrpc]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa0888d05>] ldlm_server_glimpse_ast+0x115/0x340 [ptlrpc]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa0d64aea>] filter_intent_policy+0x38a/0x6e0 [obdfilter]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa086be69>] ldlm_lock_enqueue+0x2f9/0x830 [ptlrpc]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa088d197>] ldlm_handle_enqueue0+0x427/0xda0 [ptlrpc]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa088db76>] ldlm_handle_enqueue+0x66/0x70 [ptlrpc]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa088db80>] ? ldlm_server_completion_ast+0x0/0x5b0 [ptlrpc]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa0d14a80>] ? ost_blocking_ast+0x0/0xb00 [ost]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa0888bf0>] ? ldlm_server_glimpse_ast+0x0/0x340 [ptlrpc]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa0d1b9b2>] ost_handle+0x1952/0x39e0 [ost]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa08a8af4>] ? lustre_msg_get_transno+0x54/0x90 [ptlrpc]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa08aeb15>] ? ptlrpc_update_export_timer+0x55/0x360 [ptlrpc]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa08b43c1>] ptlrpc_server_handle_request+0x3c1/0xcb0 [ptlrpc]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa03a84ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa03b2ef9>] ? lc_watchdog_touch+0x79/0x110 [libcfs]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa08ae462>] ? ptlrpc_wait_event+0xb2/0x2c0 [ptlrpc]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffff810519c3>] ? __wake_up+0x53/0x70
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa08b53cf>] ptlrpc_main+0x71f/0x1210 [ptlrpc]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 20:18:56 weisshorn12 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Jun 19 20:18:56 weisshorn12 kernel:
Jun 19 20:18:56 weisshorn12 kernel: LustreError: dumping log to /tmp/lustre-log.1340129936.8332
Jun 19 20:19:03 weisshorn04 kernel: Lustre: scratch-OST0031: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88042338fc00, cur 1340129943 expire 1340129793 last 1340129716
Jun 19 20:19:03 weisshorn04 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 20:19:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:19:22 weisshorn08 rhnsd[124827]: running program /usr/sbin/rhn_check
Jun 19 20:19:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:19:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:19:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:20:01 weisshorn09 CROND[125770]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:20:01 weisshorn02 CROND[48297]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:20:01 weisshorn04 CROND[125518]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:20:01 weisshorn13 CROND[129640]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:20:01 weisshorn07 CROND[126195]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:20:01 weisshorn12 CROND[127605]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:20:01 weisshorn03 CROND[126747]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:20:01 weisshorn14 CROND[54612]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:20:01 weisshorn01 CROND[3168]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:20:01 weisshorn05 CROND[126054]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:20:01 weisshorn10 CROND[126079]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:20:01 weisshorn06 CROND[126981]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:20:02 weisshorn08 CROND[124830]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:20:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:20:18 weisshorn12 kernel: Lustre: scratch-OST0008: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88092a309000, cur 1340130018 expire 1340129868 last 1340129791
Jun 19 20:20:18 weisshorn12 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 20:20:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:20:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:20:43 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ffe97f400, cur 1340130043 expire 1340129893 last 1340129816
Jun 19 20:20:43 weisshorn08 kernel: Lustre: Skipped 1 previous similar message
Jun 19 20:20:43 weisshorn05 kernel: Lustre: scratch-OST003e: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100db01000, cur 1340130043 expire 1340129893 last 1340129816
Jun 19 20:20:43 weisshorn05 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 20:20:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:21:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:21:22 weisshorn03 kernel: Lustre: scratch-OST0014: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881012ed8800, cur 1340130082 expire 1340129932 last 1340129855
Jun 19 20:21:22 weisshorn03 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 20:21:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:21:22 weisshorn07 kernel: Lustre: scratch-OST002b: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b33380000, cur 1340130082 expire 1340129932 last 1340129855
Jun 19 20:21:22 weisshorn07 kernel: Lustre: Skipped 1 previous similar message
Jun 19 20:21:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:21:47 weisshorn10 kernel: Lustre: scratch-OST000a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802cb456000, cur 1340130107 expire 1340129957 last 1340129880
Jun 19 20:21:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:21:58 weisshorn02 kernel: Lustre: scratch-MDT0000: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fb5da2000, cur 1340130118 expire 1340129968 last 1340129891
Jun 19 20:22:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:22:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:22:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:22:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:23:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:23:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:23:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:23:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:24:01 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340130185/real 1340130185]  req@ffff880ac971f400 x1403849871912926/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340130241 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:24:01 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:24:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:24:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:24:35 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340130219/real 1340130219]  req@ffff88100dc30c00 x1403849985208786/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340130275 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:24:35 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:24:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:24:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:24:55 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340130239/real 1340130239]  req@ffff8806ca776000 x1403849943645415/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340130295 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:24:55 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:25:00 weisshorn09 nslcd[3954]: [5138de] ldap_result() failed: Can't contact LDAP server
Jun 19 20:25:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:25:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:25:23 weisshorn08 nslcd[3946]: [234363] ldap_result() failed: Can't contact LDAP server
Jun 19 20:25:24 weisshorn06 nslcd[3953]: [49361f] ldap_result() failed: Can't contact LDAP server
Jun 19 20:25:26 weisshorn04 nslcd[3885]: [5138de] ldap_result() failed: Can't contact LDAP server
Jun 19 20:25:27 weisshorn07 nslcd[4297]: [234363] ldap_result() failed: Can't contact LDAP server
Jun 19 20:25:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:25:42 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340130286/real 1340130286]  req@ffff88101b16c000 x1403849861970168/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340130342 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:25:42 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:25:47 weisshorn12 nslcd[3954]: [234363] ldap_result() failed: Can't contact LDAP server
Jun 19 20:25:49 weisshorn13 nslcd[4224]: [234363] ldap_result() failed: Can't contact LDAP server
Jun 19 20:25:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:26:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:26:17 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340130321/real 1340130321]  req@ffff880e6a5a4400 x1403849897601440/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340130377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:26:17 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:26:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:26:26 weisshorn05 nslcd[4378]: [d385c9] ldap_result() failed: Can't contact LDAP server
Jun 19 20:26:33 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340130337/real 1340130337]  req@ffff880801ffa000 x1403849922039114/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340130393 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:26:33 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:26:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:26:42 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340130346/real 1340130346]  req@ffff88080b23e000 x1403849636916575/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340130402 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:26:42 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:26:44 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340130348/real 1340130348]  req@ffff881017bb2800 x1403849873718432/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340130404 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:26:44 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:26:47 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340130351/real 1340130351]  req@ffff8804eed49400 x1403849823002223/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340130407 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:26:47 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:26:48 weisshorn01 nslcd[2168]: [e49eb4] ldap_result() failed: Can't contact LDAP server
Jun 19 20:26:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:27:05 weisshorn02 rhnsd[48368]: running program /usr/sbin/rhn_check
Jun 19 20:27:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:27:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:27:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:27:49 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340130413/real 1340130413]  req@ffff880b10d8bc00 x1403849953687819/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340130469 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:27:49 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:27:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:28:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:28:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:28:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:28:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:29:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:29:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:29:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:29:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:30:01 weisshorn06 CROND[127014]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:30:01 weisshorn09 CROND[125810]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:30:01 weisshorn08 CROND[124861]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:30:01 weisshorn02 CROND[48382]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:30:01 weisshorn04 CROND[125548]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:30:01 weisshorn13 CROND[129671]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:30:01 weisshorn07 CROND[126275]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:30:01 weisshorn14 CROND[54647]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:30:01 weisshorn01 CROND[3237]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:30:01 weisshorn05 CROND[126085]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:30:01 weisshorn12 CROND[127642]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:30:01 weisshorn03 CROND[126782]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:30:01 weisshorn10 CROND[126111]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:30:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:30:07 weisshorn12 kernel: Lustre: 8137:0:(service.c:1034:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-271), not sending early reply
Jun 19 20:30:07 weisshorn12 kernel:  req@ffff8809aedd1400 x1403846449906439/t0(0) o101->422f3e68-7034-6fb9-875c-aa3ea002d267@149@gni:0/0 lens 296/352 e 3 to 0 dl 1340130612 ref 2 fl Interpret:/0/0 rc 0/0
Jun 19 20:30:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:30:26 weisshorn05 rhnsd[126088]: running program /usr/sbin/rhn_check
Jun 19 20:30:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:30:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:31:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:31:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:31:33 weisshorn04 kernel: Lustre: scratch-OST0045: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88028b2a5800, cur 1340130693 expire 1340130543 last 1340130466
Jun 19 20:31:33 weisshorn04 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 20:31:33 weisshorn12 kernel: Lustre: scratch-OST001c: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c9bdc4800, cur 1340130693 expire 1340130543 last 1340130466
Jun 19 20:31:33 weisshorn12 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 20:31:33 weisshorn09 kernel: Lustre: scratch-OST0000: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810161f8c00, cur 1340130693 expire 1340130543 last 1340130466
Jun 19 20:31:33 weisshorn09 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 20:31:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:31:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:31:58 weisshorn03 kernel: Lustre: scratch-OST0043: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100f012c00, cur 1340130718 expire 1340130568 last 1340130491
Jun 19 20:31:58 weisshorn03 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 20:31:58 weisshorn06 kernel: Lustre: scratch-OST002a: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880afe98a400, cur 1340130718 expire 1340130568 last 1340130491
Jun 19 20:31:58 weisshorn06 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 20:32:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:32:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:32:23 weisshorn10 kernel: Lustre: scratch-OST0016: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880874175400, cur 1340130743 expire 1340130593 last 1340130516
Jun 19 20:32:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:32:48 weisshorn13 kernel: Lustre: scratch-OST003f: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881010dbb400, cur 1340130768 expire 1340130618 last 1340130541
Jun 19 20:32:48 weisshorn13 kernel: Lustre: Skipped 8 previous similar messages
Jun 19 20:32:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:33:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:33:13 weisshorn07 kernel: Lustre: scratch-OST0006: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881013f46800, cur 1340130793 expire 1340130643 last 1340130566
Jun 19 20:33:13 weisshorn07 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 20:33:13 weisshorn05 kernel: Lustre: scratch-OST003e: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808088ae400, cur 1340130793 expire 1340130643 last 1340130566
Jun 19 20:33:13 weisshorn05 kernel: Lustre: Skipped 1 previous similar message
Jun 19 20:33:20 weisshorn12 kernel: Lustre: scratch-OST0044: Client 422f3e68-7034-6fb9-875c-aa3ea002d267 (at 149@gni) reconnecting
Jun 19 20:33:20 weisshorn12 kernel: Lustre: scratch-OST0044: Client 422f3e68-7034-6fb9-875c-aa3ea002d267 (at 149@gni) refused reconnection, still busy with 1 active RPCs
Jun 19 20:33:20 weisshorn12 kernel: Lustre: Skipped 15 previous similar messages
Jun 19 20:33:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:33:31 weisshorn02 kernel: Lustre: 47380:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004fb4:0x126:0x0]] [valid 68719476736]
Jun 19 20:33:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:33:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:34:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:34:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:34:35 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340130819/real 1340130819]  req@ffff881011ca0400 x1403849985291937/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340130875 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:34:35 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 20:34:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:34:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:35:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:35:16 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340130860/real 1340130860]  req@ffff880da92ce800 x1403849871996112/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340130916 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:35:16 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:35:21 weisshorn12 kernel: Lustre: scratch-OST0044: Client 422f3e68-7034-6fb9-875c-aa3ea002d267 (at 149@gni) reconnecting
Jun 19 20:35:21 weisshorn12 kernel: Lustre: scratch-OST0044: Client 422f3e68-7034-6fb9-875c-aa3ea002d267 (at 149@gni) refused reconnection, still busy with 1 active RPCs
Jun 19 20:35:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:35:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:35:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:36:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:36:10 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340130914/real 1340130914]  req@ffff880deaa6dc00 x1403849943687152/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340130970 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:36:10 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:36:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:36:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:36:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:36:57 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340130961/real 1340130961]  req@ffff8810252c7800 x1403849862020271/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340131017 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:36:57 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:37:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:37:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:37:22 weisshorn12 kernel: Lustre: scratch-OST0044: Client 422f3e68-7034-6fb9-875c-aa3ea002d267 (at 149@gni) reconnecting
Jun 19 20:37:22 weisshorn12 kernel: Lustre: scratch-OST0044: Client 422f3e68-7034-6fb9-875c-aa3ea002d267 (at 149@gni) refused reconnection, still busy with 1 active RPCs
Jun 19 20:37:32 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340130996/real 1340130996]  req@ffff881013f70000 x1403849897658317/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340131052 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:37:32 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:37:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:37:48 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340131012/real 1340131012]  req@ffff880801973800 x1403849922119713/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340131068 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:37:48 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:37:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:37:57 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340131021/real 1340131021]  req@ffff88080a46a400 x1403849636982410/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340131077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:37:57 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:37:59 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340131023/real 1340131023]  req@ffff880da8390800 x1403849873778839/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340131079 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:37:59 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:38:02 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340131026/real 1340131026]  req@ffff88024b541800 x1403849823064285/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340131082 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:38:02 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:38:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:38:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:38:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:38:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:39:04 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340131088/real 1340131088]  req@ffff8807b077b000 x1403849953784519/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340131144 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:39:04 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 9 previous similar messages
Jun 19 20:39:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:39:07 weisshorn12 kernel: LNet: Service thread pid 5928 was inactive for 906.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Jun 19 20:39:07 weisshorn12 kernel: Pid: 5928, comm: ll_ost_51
Jun 19 20:39:07 weisshorn12 kernel:
Jun 19 20:39:07 weisshorn12 kernel: Call Trace:
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8107bf8c>] ? lock_timer_base+0x3c/0x70
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff814edc52>] schedule_timeout+0x192/0x2e0
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8107c0a0>] ? process_timeout+0x0/0x10
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa03a85c1>] cfs_waitq_timedwait+0x11/0x20 [libcfs]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa089f6e8>] ptlrpc_set_wait+0x288/0x700 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8105e7f0>] ? default_wake_function+0x0/0x20
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa089fbe6>] ptlrpc_queue_wait+0x86/0x1c0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0888d05>] ldlm_server_glimpse_ast+0x115/0x340 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0d64aea>] filter_intent_policy+0x38a/0x6e0 [obdfilter]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa086be69>] ldlm_lock_enqueue+0x2f9/0x830 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa088d197>] ldlm_handle_enqueue0+0x427/0xda0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa088db76>] ldlm_handle_enqueue+0x66/0x70 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa088db80>] ? ldlm_server_completion_ast+0x0/0x5b0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0d14a80>] ? ost_blocking_ast+0x0/0xb00 [ost]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0888bf0>] ? ldlm_server_glimpse_ast+0x0/0x340 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0d1b9b2>] ost_handle+0x1952/0x39e0 [ost]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8105236e>] ? account_entity_enqueue+0x7e/0x90
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08a8af4>] ? lustre_msg_get_transno+0x54/0x90 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08aeb15>] ? ptlrpc_update_export_timer+0x55/0x360 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b43c1>] ptlrpc_server_handle_request+0x3c1/0xcb0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa03a84ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa03b2ef9>] ? lc_watchdog_touch+0x79/0x110 [libcfs]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08ae462>] ? ptlrpc_wait_event+0xb2/0x2c0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff810519c3>] ? __wake_up+0x53/0x70
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b53cf>] ptlrpc_main+0x71f/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Jun 19 20:39:07 weisshorn12 kernel:
Jun 19 20:39:07 weisshorn12 kernel: LustreError: dumping log to /tmp/lustre-log.1340131147.5928
Jun 19 20:39:07 weisshorn12 kernel: LNet: Service thread pid 8111 was inactive for 906.03s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Jun 19 20:39:07 weisshorn12 kernel: Pid: 8111, comm: ll_ost_195
Jun 19 20:39:07 weisshorn12 kernel:
Jun 19 20:39:07 weisshorn12 kernel: Call Trace:
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8107bf8c>] ? lock_timer_base+0x3c/0x70
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff814edc52>] schedule_timeout+0x192/0x2e0
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8107c0a0>] ? process_timeout+0x0/0x10
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa03a85c1>] cfs_waitq_timedwait+0x11/0x20 [libcfs]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa089f6e8>] ptlrpc_set_wait+0x288/0x700 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8105e7f0>] ? default_wake_function+0x0/0x20
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa089fbe6>] ptlrpc_queue_wait+0x86/0x1c0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0888d05>] ldlm_server_glimpse_ast+0x115/0x340 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0d64aea>] filter_intent_policy+0x38a/0x6e0 [obdfilter]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa086be69>] ldlm_lock_enqueue+0x2f9/0x830 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa088d197>] ldlm_handle_enqueue0+0x427/0xda0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa088db76>] ldlm_handle_enqueue+0x66/0x70 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa088db80>] ? ldlm_server_completion_ast+0x0/0x5b0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0d14a80>] ? ost_blocking_ast+0x0/0xb00 [ost]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0888bf0>] ? ldlm_server_glimpse_ast+0x0/0x340 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0d1b9b2>] ost_handle+0x1952/0x39e0 [ost]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8105236e>] ? account_entity_enqueue+0x7e/0x90
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08a8af4>] ? lustre_msg_get_transno+0x54/0x90 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08aeb15>] ? ptlrpc_update_export_timer+0x55/0x360 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b43c1>] ptlrpc_server_handle_request+0x3c1/0xcb0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa03a84ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa03b2ef9>] ? lc_watchdog_touch+0x79/0x110 [libcfs]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08ae462>] ? ptlrpc_wait_event+0xb2/0x2c0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff810519c3>] ? __wake_up+0x53/0x70
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b53cf>] ptlrpc_main+0x71f/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Jun 19 20:39:07 weisshorn12 kernel:
Jun 19 20:39:07 weisshorn12 kernel: Pid: 8025, comm: ll_ost_128
Jun 19 20:39:07 weisshorn12 kernel:
Jun 19 20:39:07 weisshorn12 kernel: Call Trace:
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8107bf8c>] ? lock_timer_base+0x3c/0x70
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff814edc52>] schedule_timeout+0x192/0x2e0
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8107c0a0>] ? process_timeout+0x0/0x10
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa03a85c1>] cfs_waitq_timedwait+0x11/0x20 [libcfs]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa089f6e8>] ptlrpc_set_wait+0x288/0x700 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8105e7f0>] ? default_wake_function+0x0/0x20
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa089fbe6>] ptlrpc_queue_wait+0x86/0x1c0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0888d05>] ldlm_server_glimpse_ast+0x115/0x340 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0d64aea>] filter_intent_policy+0x38a/0x6e0 [obdfilter]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa086be69>] ldlm_lock_enqueue+0x2f9/0x830 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa088d197>] ldlm_handle_enqueue0+0x427/0xda0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa088db76>] ldlm_handle_enqueue+0x66/0x70 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa088db80>] ? ldlm_server_completion_ast+0x0/0x5b0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0d14a80>] ? ost_blocking_ast+0x0/0xb00 [ost]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0888bf0>] ? ldlm_server_glimpse_ast+0x0/0x340 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0d1b9b2>] ost_handle+0x1952/0x39e0 [ost]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08a8af4>] ? lustre_msg_get_transno+0x54/0x90 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08aeb15>] ? ptlrpc_update_export_timer+0x55/0x360 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b43c1>] ptlrpc_server_handle_request+0x3c1/0xcb0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa03a84ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa03b2ef9>] ? lc_watchdog_touch+0x79/0x110 [libcfs]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08ae462>] ? ptlrpc_wait_event+0xb2/0x2c0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff810519c3>] ? __wake_up+0x53/0x70
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b53cf>] ptlrpc_main+0x71f/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Jun 19 20:39:07 weisshorn12 kernel:
Jun 19 20:39:07 weisshorn12 kernel: Pid: 8351, comm: ll_ost_435
Jun 19 20:39:07 weisshorn12 kernel:
Jun 19 20:39:07 weisshorn12 kernel: Call Trace:
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8107bf8c>] ? lock_timer_base+0x3c/0x70
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff814edc52>] schedule_timeout+0x192/0x2e0
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8107c0a0>] ? process_timeout+0x0/0x10
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa03a85c1>] cfs_waitq_timedwait+0x11/0x20 [libcfs]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa089f6e8>] ptlrpc_set_wait+0x288/0x700 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8105e7f0>] ? default_wake_function+0x0/0x20
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa089fbe6>] ptlrpc_queue_wait+0x86/0x1c0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0888d05>] ldlm_server_glimpse_ast+0x115/0x340 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0d64aea>] filter_intent_policy+0x38a/0x6e0 [obdfilter]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa086be69>] ldlm_lock_enqueue+0x2f9/0x830 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa088d197>] ldlm_handle_enqueue0+0x427/0xda0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa088db76>] ldlm_handle_enqueue+0x66/0x70 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa088db80>] ? ldlm_server_completion_ast+0x0/0x5b0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0d14a80>] ? ost_blocking_ast+0x0/0xb00 [ost]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0888bf0>] ? ldlm_server_glimpse_ast+0x0/0x340 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0d1b9b2>] ost_handle+0x1952/0x39e0 [ost]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08a8af4>] ? lustre_msg_get_transno+0x54/0x90 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08aeb15>] ? ptlrpc_update_export_timer+0x55/0x360 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b43c1>] ptlrpc_server_handle_request+0x3c1/0xcb0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa03a84ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa03b2ef9>] ? lc_watchdog_touch+0x79/0x110 [libcfs]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08ae462>] ? ptlrpc_wait_event+0xb2/0x2c0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8105e7f0>] ? default_wake_function+0x0/0x20
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b53cf>] ptlrpc_main+0x71f/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Jun 19 20:39:07 weisshorn12 kernel:
Jun 19 20:39:07 weisshorn12 kernel: Pid: 8437, comm: ll_ost_511
Jun 19 20:39:07 weisshorn12 kernel:
Jun 19 20:39:07 weisshorn12 kernel: Call Trace:
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8107bf8c>] ? lock_timer_base+0x3c/0x70
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff814edc52>] schedule_timeout+0x192/0x2e0
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8107c0a0>] ? process_timeout+0x0/0x10
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa03a85c1>] cfs_waitq_timedwait+0x11/0x20 [libcfs]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa089f6e8>] ptlrpc_set_wait+0x288/0x700 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8105e7f0>] ? default_wake_function+0x0/0x20
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa089fbe6>] ptlrpc_queue_wait+0x86/0x1c0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0888d05>] ldlm_server_glimpse_ast+0x115/0x340 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0d64aea>] filter_intent_policy+0x38a/0x6e0 [obdfilter]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa086be69>] ldlm_lock_enqueue+0x2f9/0x830 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa088d197>] ldlm_handle_enqueue0+0x427/0xda0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa088db76>] ldlm_handle_enqueue+0x66/0x70 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa088db80>] ? ldlm_server_completion_ast+0x0/0x5b0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0d14a80>] ? ost_blocking_ast+0x0/0xb00 [ost]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0888bf0>] ? ldlm_server_glimpse_ast+0x0/0x340 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa0d1b9b2>] ost_handle+0x1952/0x39e0 [ost]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08a8af4>] ? lustre_msg_get_transno+0x54/0x90 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08aeb15>] ? ptlrpc_update_export_timer+0x55/0x360 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b43c1>] ptlrpc_server_handle_request+0x3c1/0xcb0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa03a84ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa03b2ef9>] ? lc_watchdog_touch+0x79/0x110 [libcfs]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08ae462>] ? ptlrpc_wait_event+0xb2/0x2c0 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff810519c3>] ? __wake_up+0x53/0x70
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b53cf>] ptlrpc_main+0x71f/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffffa08b4cb0>] ? ptlrpc_main+0x0/0x1210 [ptlrpc]
Jun 19 20:39:07 weisshorn12 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Jun 19 20:39:07 weisshorn12 kernel:
Jun 19 20:39:07 weisshorn12 kernel: LNet: Service thread pid 5982 was inactive for 906.15s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Jun 19 20:39:07 weisshorn12 kernel: LNet: Skipped 3 previous similar messages
Jun 19 20:39:07 weisshorn12 kernel: LustreError: dumping log to /tmp/lustre-log.1340131147.8297
Jun 19 20:39:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:39:23 weisshorn12 kernel: Lustre: scratch-OST0044: Client 422f3e68-7034-6fb9-875c-aa3ea002d267 (at 149@gni) reconnecting
Jun 19 20:39:23 weisshorn12 kernel: Lustre: scratch-OST0044: Client 422f3e68-7034-6fb9-875c-aa3ea002d267 (at 149@gni) refused reconnection, still busy with 1 active RPCs
Jun 19 20:39:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:39:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:40:01 weisshorn10 CROND[126145]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:40:01 weisshorn12 CROND[127710]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:40:01 weisshorn03 CROND[126843]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:40:01 weisshorn06 CROND[127046]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:40:01 weisshorn09 CROND[125866]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:40:01 weisshorn08 CROND[124924]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:40:01 weisshorn04 CROND[125582]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:40:01 weisshorn02 CROND[48444]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:40:01 weisshorn13 CROND[129732]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:40:01 weisshorn07 CROND[126302]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:40:01 weisshorn14 CROND[54701]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:40:01 weisshorn01 CROND[3271]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:40:02 weisshorn05 CROND[126120]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:40:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:40:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:40:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:40:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:41:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:41:16 weisshorn14 nslcd[3437]: [0f856c] ldap_result() failed: Can't contact LDAP server
Jun 19 20:41:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:41:24 weisshorn12 kernel: Lustre: scratch-OST0044: Client 422f3e68-7034-6fb9-875c-aa3ea002d267 (at 149@gni) reconnecting
Jun 19 20:41:24 weisshorn12 kernel: Lustre: scratch-OST0044: Client 422f3e68-7034-6fb9-875c-aa3ea002d267 (at 149@gni) refused reconnection, still busy with 1 active RPCs
Jun 19 20:41:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:41:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:42:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:42:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:42:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:42:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:43:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:43:22 weisshorn12 kernel: Lustre: 5952:0:(service.c:1034:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-561), not sending early reply
Jun 19 20:43:22 weisshorn12 kernel:  req@ffff880f5c262800 x1403846435595126/t0(0) o101->502ea153-7ccd-ef8f-6e20-ac3daa48e3d6@396@gni:0/0 lens 296/352 e 1 to 0 dl 1340131407 ref 2 fl Interpret:/0/0 rc 0/0
Jun 19 20:43:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:43:25 weisshorn12 kernel: Lustre: scratch-OST0044: Client 422f3e68-7034-6fb9-875c-aa3ea002d267 (at 149@gni) reconnecting
Jun 19 20:43:25 weisshorn12 kernel: Lustre: scratch-OST0044: Client 422f3e68-7034-6fb9-875c-aa3ea002d267 (at 149@gni) refused reconnection, still busy with 1 active RPCs
Jun 19 20:43:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:43:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:44:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:44:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:44:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:44:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:45:05 weisshorn02 kernel: LustreError: 47679:0:(mdt_handler.c:2822:mdt_recovery()) operation 101 on unconnected MDS from 12345-148.187.6.6@o2ib2
Jun 19 20:45:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:45:07 weisshorn04 kernel: Lustre: scratch-OST0019: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880735949c00, cur 1340131507 expire 1340131357 last 1340131280
Jun 19 20:45:07 weisshorn04 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 20:45:07 weisshorn03 kernel: Lustre: scratch-OST002c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101391d800, cur 1340131507 expire 1340131357 last 1340131280
Jun 19 20:45:07 weisshorn03 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 20:45:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:45:26 weisshorn12 kernel: Lustre: scratch-OST0044: Client 422f3e68-7034-6fb9-875c-aa3ea002d267 (at 149@gni) refused reconnection, still busy with 1 active RPCs
Jun 19 20:45:32 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880570392c00, cur 1340131532 expire 1340131382 last 1340131305
Jun 19 20:45:32 weisshorn08 kernel: Lustre: Skipped 1 previous similar message
Jun 19 20:45:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:45:50 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340131494/real 1340131494]  req@ffff880bfbd9e000 x1403849985375269/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340131550 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:45:50 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:45:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:46:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:46:17 weisshorn10 nslcd[4294]: [d385c9] ldap_result() failed: Can't contact LDAP server
Jun 19 20:46:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:46:31 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340131535/real 1340131535]  req@ffff88080767e400 x1403849872061838/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340131591 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:46:31 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:46:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:46:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:47:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:47:12 weisshorn02 kernel: Lustre: scratch-MDT0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880751ef5000, cur 1340131632 expire 1340131482 last 1340131405
Jun 19 20:47:12 weisshorn07 kernel: Lustre: scratch-OST002b: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b31c4cc00, cur 1340131632 expire 1340131482 last 1340131405
Jun 19 20:47:12 weisshorn13 kernel: Lustre: scratch-OST0018: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880366855c00, cur 1340131632 expire 1340131482 last 1340131405
Jun 19 20:47:12 weisshorn13 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 20:47:12 weisshorn06 kernel: Lustre: scratch-OST002a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88021d701800, cur 1340131632 expire 1340131482 last 1340131405
Jun 19 20:47:12 weisshorn12 kernel: Lustre: scratch-OST0044: Client 502ea153-7ccd-ef8f-6e20-ac3daa48e3d6 (at 396@gni) reconnecting
Jun 19 20:47:12 weisshorn12 kernel: Lustre: Skipped 1 previous similar message
Jun 19 20:47:12 weisshorn12 kernel: Lustre: scratch-OST0044: Client 502ea153-7ccd-ef8f-6e20-ac3daa48e3d6 (at 396@gni) refused reconnection, still busy with 1 active RPCs
Jun 19 20:47:17 weisshorn01 nslcd[2168]: [f32454] ldap_result() failed: Can't contact LDAP server
Jun 19 20:47:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:47:25 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340131589/real 1340131589]  req@ffff880bc2e99000 x1403849943719576/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340131645 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:47:25 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:47:37 weisshorn10 kernel: Lustre: scratch-OST000a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805723f6c00, cur 1340131657 expire 1340131507 last 1340131430
Jun 19 20:47:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:47:48 weisshorn12 kernel: Lustre: scratch-OST0047: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805c2e2c400, cur 1340131668 expire 1340131518 last 1340131441
Jun 19 20:47:48 weisshorn12 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 20:47:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:47:57 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340131621/real 1340131621]  req@ffff88100e427000 x1403849637018037/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340131677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:47:57 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 20:48:02 weisshorn09 kernel: Lustre: scratch-OST0026: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809806a2800, cur 1340131682 expire 1340131532 last 1340131455
Jun 19 20:48:02 weisshorn09 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 20:48:02 weisshorn05 kernel: Lustre: scratch-OST0027: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805b591b400, cur 1340131682 expire 1340131532 last 1340131455
Jun 19 20:48:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:48:12 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340131636/real 1340131636]  req@ffff88101838bc00 x1403849862084234/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340131692 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:48:12 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:48:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:48:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:48:38 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880dfd248c00, cur 1340131718 expire 1340131568 last 1340131491
Jun 19 20:48:38 weisshorn08 kernel: Lustre: Skipped 1 previous similar message
Jun 19 20:48:47 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340131671/real 1340131671]  req@ffff880ed8a93800 x1403849897732791/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340131727 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:48:47 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:48:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:49:03 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340131687/real 1340131687]  req@ffff8810172c0c00 x1403849922172433/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340131743 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:49:03 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:49:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:49:14 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340131698/real 1340131698]  req@ffff88101552b000 x1403849873831317/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340131754 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:49:14 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:49:17 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340131701/real 1340131701]  req@ffff8809d13b7400 x1403849823107523/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340131757 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:49:17 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:49:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:49:28 weisshorn12 kernel: Lustre: scratch-OST0044: Client 422f3e68-7034-6fb9-875c-aa3ea002d267 (at 149@gni) refused reconnection, still busy with 1 active RPCs
Jun 19 20:49:28 weisshorn12 kernel: Lustre: Skipped 14 previous similar messages
Jun 19 20:49:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:49:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:50:01 weisshorn14 CROND[54738]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:50:01 weisshorn01 CROND[3299]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:50:01 weisshorn10 CROND[126206]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:50:01 weisshorn12 CROND[127744]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:50:01 weisshorn03 CROND[126874]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:50:01 weisshorn05 CROND[126180]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:50:01 weisshorn06 CROND[127107]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:50:01 weisshorn09 CROND[125897]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:50:01 weisshorn08 CROND[124955]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:50:01 weisshorn04 CROND[125643]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:50:01 weisshorn02 CROND[48503]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:50:01 weisshorn13 CROND[129770]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:50:01 weisshorn07 CROND[126323]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 20:50:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:50:19 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340131763/real 1340131763]  req@ffff880d1edbd000 x1403849953890103/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340131819 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:50:19 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
Jun 19 20:50:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:50:31 weisshorn14 nslcd[3437]: [aa85fb] ldap_result() failed: Can't contact LDAP server
Jun 19 20:50:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:50:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:51:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:51:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:51:29 weisshorn12 kernel: Lustre: scratch-OST0044: Client 422f3e68-7034-6fb9-875c-aa3ea002d267 (at 149@gni) reconnecting
Jun 19 20:51:29 weisshorn12 kernel: Lustre: Skipped 22 previous similar messages
Jun 19 20:51:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:51:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:52:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:52:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:52:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:52:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:53:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:53:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:53:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:53:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:54:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:54:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:54:28 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881015a58800, cur 1340132068 expire 1340131918 last 1340131841
Jun 19 20:54:28 weisshorn10 kernel: Lustre: scratch-OST0022: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080661c000, cur 1340132068 expire 1340131918 last 1340131841
Jun 19 20:54:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:54:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:54:53 weisshorn10 kernel: Lustre: scratch-OST0001: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881013ab9400, cur 1340132093 expire 1340131943 last 1340131866
Jun 19 20:55:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:55:16 weisshorn12 kernel: Lustre: scratch-OST0044: Client 502ea153-7ccd-ef8f-6e20-ac3daa48e3d6 (at 396@gni) refused reconnection, still busy with 1 active RPCs
Jun 19 20:55:16 weisshorn12 kernel: Lustre: Skipped 16 previous similar messages
Jun 19 20:55:18 weisshorn04 kernel: Lustre: scratch-OST0019: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c21c19400, cur 1340132118 expire 1340131968 last 1340131891
Jun 19 20:55:18 weisshorn04 kernel: Lustre: Skipped 6 previous similar messages
Jun 19 20:55:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:55:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:55:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:56:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:56:17 weisshorn05 kernel: Lustre: scratch-OST0027: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 20:56:17 weisshorn05 kernel: Lustre: Skipped 388 previous similar messages
Jun 19 20:56:17 weisshorn05 kernel: Lustre: scratch-OST0027: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) refused reconnection, still busy with 1 active RPCs
Jun 19 20:56:17 weisshorn05 kernel: Lustre: Skipped 1 previous similar message
Jun 19 20:56:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:56:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:56:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:56:59 weisshorn09 rhnsd[125924]: running program /usr/sbin/rhn_check
Jun 19 20:57:05 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340132169/real 1340132169]  req@ffff8805a444b800 x1403849985503629/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340132225 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:57:05 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:57:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:57:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:57:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:57:46 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340132210/real 1340132210]  req@ffff880bf78ca400 x1403849872140596/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340132266 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:57:46 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:57:48 weisshorn12 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 2067s: evicting client at 402@gni  ns: filter-scratch-OST0044_UUID lock: ffff88026ac5b240/0xbc58479c9fa22a08 lrc: 3/0,0 mode: PR/PR res: 9619263/0 rrc: 9 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0x1ba3b5635408b8d2 expref: 24 pid: 8305 timeout 5611853367
Jun 19 20:57:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:58:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:58:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:58:22 weisshorn05 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 125s: evicting client at 148.187.6.236@o2ib2  ns: filter-scratch-OST0027_UUID lock: ffff880659f90900/0xf8449fdfde4820fa lrc: 3/0,0 mode: PW/PW res: 9526322/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 remote: 0xfc0c4f7b63966df7 expref: 5535 pid: 8019 timeout 5611889537
Jun 19 20:58:27 weisshorn09 kernel: Lustre: scratch-OST0034: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88075b506400, cur 1340132307 expire 1340132157 last 1340132080
Jun 19 20:58:27 weisshorn09 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 20:58:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:58:40 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340132264/real 1340132264]  req@ffff880ebd3c5800 x1403849943773725/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340132320 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:58:40 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:58:47 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340132271/real 1340132271]  req@ffff881029a0c800 x1403849897797846/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340132327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:58:47 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 20:58:52 weisshorn13 kernel: Lustre: scratch-OST0024: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881021958000, cur 1340132332 expire 1340132182 last 1340132105
Jun 19 20:58:52 weisshorn13 kernel: Lustre: Skipped 12 previous similar messages
Jun 19 20:58:52 weisshorn05 kernel: Lustre: scratch-OST000e: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880809c95400, cur 1340132332 expire 1340132182 last 1340132105
Jun 19 20:58:52 weisshorn05 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 20:58:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:59:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:59:12 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340132296/real 1340132296]  req@ffff88080e707800 x1403849637087708/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340132352 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:59:12 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:59:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:59:27 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340132311/real 1340132311]  req@ffff88060d4b4000 x1403849862152545/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340132367 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 20:59:27 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 20:59:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:59:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 20:59:55 weisshorn08 kernel: Lustre: scratch-OST000c: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 20:59:55 weisshorn08 kernel: Lustre: Skipped 33 previous similar messages
Jun 19 20:59:55 weisshorn08 kernel: Lustre: scratch-OST000c: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) refused reconnection, still busy with 1 active RPCs
Jun 19 21:00:01 weisshorn01 CROND[3369]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:00:01 weisshorn14 CROND[54807]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:00:01 weisshorn10 CROND[126243]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:00:01 weisshorn03 CROND[126908]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:00:01 weisshorn06 CROND[127151]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:00:01 weisshorn12 CROND[127816]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:00:01 weisshorn05 CROND[126217]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:00:01 weisshorn09 CROND[125937]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:00:01 weisshorn08 CROND[124994]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:00:01 weisshorn04 CROND[125680]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:00:01 weisshorn02 CROND[48604]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:00:01 weisshorn13 CROND[129841]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:00:02 weisshorn07 CROND[126405]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:00:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:00:14 weisshorn08 kernel: Lustre: scratch-OST000c: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 21:00:18 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340132362/real 1340132362]  req@ffff880c97cab000 x1403849922245871/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340132418 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:00:18 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:00:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:00:29 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340132373/real 1340132373]  req@ffff8810283ca400 x1403849873891476/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340132429 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:00:29 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:00:32 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340132376/real 1340132376]  req@ffff880c84378400 x1403849823159119/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340132432 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:00:32 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:00:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:00:46 weisshorn10 nslcd[4294]: [6f6d8a] ldap_result() failed: Can't contact LDAP server
Jun 19 21:00:46 weisshorn06 nslcd[3953]: [bc6770] ldap_result() failed: Can't contact LDAP server
Jun 19 21:00:46 weisshorn05 nslcd[4378]: [6f6d8a] ldap_result() failed: Can't contact LDAP server
Jun 19 21:00:46 weisshorn13 nslcd[4224]: [d385c9] ldap_result() failed: Can't contact LDAP server
Jun 19 21:00:46 weisshorn09 nslcd[3954]: [22ad0c] ldap_result() failed: Can't contact LDAP server
Jun 19 21:00:46 weisshorn03 nslcd[3944]: [997556] ldap_result() failed: Can't contact LDAP server
Jun 19 21:00:46 weisshorn12 nslcd[3954]: [d385c9] ldap_result() failed: Can't contact LDAP server
Jun 19 21:00:46 weisshorn07 nslcd[4297]: [d385c9] ldap_result() failed: Can't contact LDAP server
Jun 19 21:00:46 weisshorn08 nslcd[3946]: [d385c9] ldap_result() failed: Can't contact LDAP server
Jun 19 21:00:46 weisshorn01 nslcd[2168]: [a88611] ldap_result() failed: Can't contact LDAP server
Jun 19 21:00:46 weisshorn04 nslcd[3885]: [22ad0c] ldap_result() failed: Can't contact LDAP server
Jun 19 21:00:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:01:01 weisshorn13 CROND[129846]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 21:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1298 starting 0anacron
Jun 19 21:01:01 weisshorn07 CROND[126410]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 21:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1298 finished 0anacron
Jun 19 21:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1298 starting mcelog.cron
Jun 19 21:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1264 starting 0anacron
Jun 19 21:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1298 finished mcelog.cron
Jun 19 21:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1264 finished 0anacron
Jun 19 21:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1264 starting mcelog.cron
Jun 19 21:01:01 weisshorn14 CROND[54812]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 21:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5481 starting 0anacron
Jun 19 21:01:01 weisshorn01 CROND[3374]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 21:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5482 finished 0anacron
Jun 19 21:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5481 starting mcelog.cron
Jun 19 21:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3374 starting 0anacron
Jun 19 21:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5482 finished mcelog.cron
Jun 19 21:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3383 finished 0anacron
Jun 19 21:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1264 finished mcelog.cron
Jun 19 21:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3374 starting mcelog.cron
Jun 19 21:01:01 weisshorn10 CROND[126248]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 21:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3390 finished mcelog.cron
Jun 19 21:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1262 starting 0anacron
Jun 19 21:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1262 finished 0anacron
Jun 19 21:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1262 starting mcelog.cron
Jun 19 21:01:01 weisshorn03 CROND[126913]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 21:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1262 finished mcelog.cron
Jun 19 21:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1269 starting 0anacron
Jun 19 21:01:01 weisshorn06 CROND[127156]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 21:01:01 weisshorn12 CROND[127821]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 21:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1269 finished 0anacron
Jun 19 21:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1269 starting mcelog.cron
Jun 19 21:01:01 weisshorn05 CROND[126221]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 21:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1271 starting 0anacron
Jun 19 21:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1278 starting 0anacron
Jun 19 21:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1262 starting 0anacron
Jun 19 21:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1271 finished 0anacron
Jun 19 21:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1271 starting mcelog.cron
Jun 19 21:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1269 finished mcelog.cron
Jun 19 21:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1271 finished mcelog.cron
Jun 19 21:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1278 finished 0anacron
Jun 19 21:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1278 starting mcelog.cron
Jun 19 21:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1262 finished 0anacron
Jun 19 21:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1262 starting mcelog.cron
Jun 19 21:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1278 finished mcelog.cron
Jun 19 21:01:01 weisshorn09 CROND[125942]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 21:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1262 finished mcelog.cron
Jun 19 21:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1259 starting 0anacron
Jun 19 21:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1259 finished 0anacron
Jun 19 21:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1259 starting mcelog.cron
Jun 19 21:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1259 finished mcelog.cron
Jun 19 21:01:01 weisshorn08 CROND[124998]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 21:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1249 starting 0anacron
Jun 19 21:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1250 finished 0anacron
Jun 19 21:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1249 starting mcelog.cron
Jun 19 21:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1250 finished mcelog.cron
Jun 19 21:01:01 weisshorn04 CROND[125684]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 21:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1256 starting 0anacron
Jun 19 21:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1256 finished 0anacron
Jun 19 21:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1256 starting mcelog.cron
Jun 19 21:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1257 finished mcelog.cron
Jun 19 21:01:01 weisshorn02 CROND[48608]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 21:01:02 weisshorn02 run-parts(/etc/cron.hourly)[4860 starting 0anacron
Jun 19 21:01:02 weisshorn02 run-parts(/etc/cron.hourly)[4861 finished 0anacron
Jun 19 21:01:02 weisshorn02 run-parts(/etc/cron.hourly)[4860 starting mcelog.cron
Jun 19 21:01:02 weisshorn02 run-parts(/etc/cron.hourly)[4862 finished mcelog.cron
Jun 19 21:01:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:01:16 weisshorn09 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 3141s: evicting client at 148.187.6.237@o2ib2  ns: filter-scratch-OST0004_UUID lock: ffff880ada86eb40/0x8a4ce0c839fa2ebd lrc: 3/0,0 mode: PW/PW res: 7784095/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->946175) flags: 0x10020 remote: 0x4ca7334191757d1 expref: 1143 pid: 8416 timeout 5612107628
Jun 19 21:01:16 weisshorn05 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 3142s: evicting client at 148.187.6.237@o2ib2  ns: filter-scratch-OST003e_UUID lock: ffff880453755480/0xf8449fdfde128764 lrc: 3/0,0 mode: PW/PW res: 9550819/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->946175) flags: 0x10020 remote: 0x4ca73341917561f expref: 1199 pid: 7781 timeout 5612063276
Jun 19 21:01:17 weisshorn07 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 3142s: evicting client at 148.187.6.237@o2ib2  ns: filter-scratch-OST0006_UUID lock: ffff880cbb34b480/0x6efdd3e8cab8654d lrc: 3/0,0 mode: PW/PW res: 9547378/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->946175) flags: 0x10020 remote: 0x4ca73341917598a expref: 1237 pid: 8747 timeout 5612085003
Jun 19 21:01:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:01:34 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340132438/real 1340132438]  req@ffff880adce85c00 x1403849953987965/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340132494 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:01:34 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
Jun 19 21:01:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:01:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:01:55 weisshorn08 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 120s: evicting client at 148.187.6.236@o2ib2  ns: filter-scratch-OST000c_UUID lock: ffff8809dcfa9240/0xd6d015c7a6fc081e lrc: 3/0,0 mode: PW/PW res: 7785146/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 remote: 0xfc0c4f7b63967eac expref: 514 pid: 8172 timeout 5612123193
Jun 19 21:01:55 weisshorn08 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) Skipped 1 previous similar message
Jun 19 21:02:04 weisshorn12 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 3190s: evicting client at 148.187.6.237@o2ib2  ns: filter-scratch-OST0044_UUID lock: ffff88042c9e0240/0xbc58479c9f79c5b1 lrc: 18/0,0 mode: PW/PW res: 9619268/0 rrc: 17 type: EXT [0->18446744073709551615] (req 0->946175) flags: 0x20 remote: 0x4ca73341917545f expref: 111 pid: 5993 timeout 5612109834
Jun 19 21:02:04 weisshorn12 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) Skipped 7 previous similar messages
Jun 19 21:02:04 weisshorn12 kernel: LustreError: 138-a: scratch-OST0044: A client on nid 148.187.6.237@o2ib2 was evicted due to a lock glimpse callback time out: rc -4
Jun 19 21:02:04 weisshorn12 kernel: LustreError: Skipped 10 previous similar messages
Jun 19 21:02:04 weisshorn12 kernel: Lustre: 5982:0:(service.c:1739:ptlrpc_server_handle_request()) @@@ Request x1403846440482381 took longer than estimated (1166:1117s); client may timeout.  req@ffff88063d73c800 x1403846440482381/t0(0) o101->e57ec748-7ebb-e790-3487-bdc1d97a8337@132@gni:0/0 lens 296/352 e 1 to 0 dl 1340131407 ref 1 fl Complete:/0/0 rc 301/301
Jun 19 21:02:04 weisshorn12 kernel: LNet: Service thread pid 5982 completed after 2282.60s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Jun 19 21:02:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:02:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:02:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:02:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:03:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:03:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:03:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:03:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:04:00 weisshorn12 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 3304s: evicting client at 148.187.6.237@o2ib2  ns: filter-scratch-OST0040_UUID lock: ffff88056973c900/0xbc58479c9f79d00e lrc: 3/0,0 mode: PW/PW res: 9430091/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->946175) flags: 0x10020 remote: 0x4ca73341917733e expref: 4 pid: 8199 timeout 5612225492
Jun 19 21:04:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:04:14 weisshorn08 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 3316s: evicting client at 148.187.6.237@o2ib2  ns: filter-scratch-OST003b_UUID lock: ffff8800b764c6c0/0xd6d015c7a6c7aaca lrc: 3/0,0 mode: PW/PW res: 6728344/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->946175) flags: 0x10020 remote: 0x4ca733419178987 expref: 41 pid: 8112 timeout 5612262483
Jun 19 21:04:14 weisshorn09 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 3316s: evicting client at 148.187.6.237@o2ib2  ns: filter-scratch-OST001a_UUID lock: ffff880f4dd2f6c0/0x8a4ce0c839fa33ef lrc: 3/0,0 mode: PW/PW res: 7945233/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->946175) flags: 0x10020 remote: 0x4ca733419178b47 expref: 1128 pid: 8447 timeout 5612285693
Jun 19 21:04:18 weisshorn07 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 7091s: evicting client at 148.187.6.237@o2ib2  ns: filter-scratch-OST0037_UUID lock: ffff880b5c727480/0x6efdd3e8ca533ae3 lrc: 3/0,0 mode: PW/PW res: 9530456/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 remote: 0x4ca733419157d28 expref: 1106 pid: 8496 timeout 5612266515
Jun 19 21:04:18 weisshorn10 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 7091s: evicting client at 148.187.6.237@o2ib2  ns: filter-scratch-OST003a_UUID lock: ffff8808f370e480/0xd73a6dc0a0a8db55 lrc: 3/0,0 mode: PW/PW res: 9618276/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 remote: 0x4ca733419157d4b expref: 1100 pid: 8628 timeout 5612243494
Jun 19 21:04:18 weisshorn10 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) Skipped 1 previous similar message
Jun 19 21:04:18 weisshorn08 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 7091s: evicting client at 148.187.6.237@o2ib2  ns: filter-scratch-OST0002_UUID lock: ffff8804ccd8c000/0xd6d015c7a67d1754 lrc: 3/0,0 mode: PW/PW res: 6765325/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0x4ca733419157d91 expref: 33 pid: 8428 timeout 5612266463
Jun 19 21:04:18 weisshorn05 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 7091s: evicting client at 148.187.6.237@o2ib2  ns: filter-scratch-OST0046_UUID lock: ffff880e9a031000/0xf8449fdfdd93e656 lrc: 3/0,0 mode: PW/PW res: 9507556/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 remote: 0x4ca733419157d7c expref: 1135 pid: 8039 timeout 5612245343
Jun 19 21:04:18 weisshorn04 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 7091s: evicting client at 148.187.6.237@o2ib2  ns: filter-scratch-OST0045_UUID lock: ffff880ec0a51480/0x6607ebb0f2dbbde6 lrc: 3/0,0 mode: PW/PW res: 9616067/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 remote: 0x4ca733419157d75 expref: 1176 pid: 8467 timeout 5612265333
Jun 19 21:04:18 weisshorn03 kernel: LustreError: 101:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 7091s: evicting client at 148.187.6.237@o2ib2  ns: filter-scratch-OST0043_UUID lock: ffff880e62009900/0x4a8688c95310bc39 lrc: 3/0,0 mode: PW/PW res: 9485552/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 remote: 0x4ca733419157d67 expref: 48 pid: 7781 timeout 5612240326
Jun 19 21:04:18 weisshorn13 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 3319s: evicting client at 148.187.6.237@o2ib2  ns: filter-scratch-OST0030_UUID lock: ffff880c78fb9000/0x24ad26cddd520432 lrc: 3/0,0 mode: PR/PR res: 9515681/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0x4ca733419179cb2 expref: 4 pid: 8660 timeout 5612242304
Jun 19 21:04:18 weisshorn13 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) Skipped 10 previous similar messages
Jun 19 21:04:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:04:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:04:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:05:02 weisshorn02 kernel: LustreError: 47538:0:(mdt_handler.c:2822:mdt_recovery()) operation 41 on unconnected MDS from 12345-148.187.6.7@o2ib2
Jun 19 21:05:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:05:08 weisshorn12 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 3369s: evicting client at 148.187.6.237@o2ib2  ns: filter-scratch-OST0039_UUID lock: ffff8801edaebd80/0xbc58479c9f79e6e3 lrc: 3/0,0 mode: PR/PR res: 9525173/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0x4ca733419179cc0 expref: 4 pid: 8264 timeout 5612293552
Jun 19 21:05:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:05:26 weisshorn07 kernel: Lustre: scratch-OST0006: haven't heard from client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806ef530c00, cur 1340132726 expire 1340132576 last 1340132499
Jun 19 21:05:26 weisshorn07 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 21:05:37 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:05:52 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:06:07 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:06:16 weisshorn12 kernel: Lustre: scratch-OST0044: haven't heard from client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88069912bc00, cur 1340132776 expire 1340132626 last 1340132549
Jun 19 21:06:16 weisshorn12 kernel: Lustre: Skipped 7 previous similar messages
Jun 19 21:06:17 weisshorn13 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 6966s: evicting client at 148.187.6.237@o2ib2  ns: filter-scratch-OST0011_UUID lock: ffff8808c1c0ad80/0x24ad26cddc8c96bf lrc: 3/0,0 mode: PW/PW res: 9422108/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->946175) flags: 0x10020 remote: 0x4ca733419173182 expref: 4 pid: 8371 timeout 5612361664
Jun 19 21:06:17 weisshorn13 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) Skipped 1 previous similar message
Jun 19 21:06:21 weisshorn13 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 6968s: evicting client at 148.187.6.237@o2ib2  ns: filter-scratch-OST000b_UUID lock: ffff880acaeb1480/0x24ad26cddc8ce292 lrc: 3/0,0 mode: PR/PR res: 9542916/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 remote: 0x4ca73341917431e expref: 4 pid: 8436 timeout 5612365667
Jun 19 21:06:22 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:06:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:06:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:06:58 weisshorn04 kernel: Lustre: scratch-OST0025: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 188 seconds. I think it's dead, and I am evicting it. exp ffff880200a40c00, cur 1340132818 expire 1340132668 last 1340132630
Jun 19 21:06:58 weisshorn04 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 21:07:05 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340132769/real 1340132769]  req@ffff880453b47000 x1403849985583072/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340132825 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:07:05 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 21:07:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:07:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:07:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:07:48 weisshorn02 kernel: Lustre: scratch-MDT0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 166 seconds. I think it's dead, and I am evicting it. exp ffff880fbaed4000, cur 1340132868 expire 1340132718 last 1340132702
Jun 19 21:07:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:08:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:08:13 weisshorn10 kernel: Lustre: scratch-OST0016: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88018b82c000, cur 1340132893 expire 1340132743 last 1340132666
Jun 19 21:08:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:08:27 weisshorn10 kernel: Lustre: scratch-OST000a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880acc33e000, cur 1340132907 expire 1340132757 last 1340132680
Jun 19 21:08:31 weisshorn05 kernel: Lustre: scratch-OST001b: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 21:08:31 weisshorn05 kernel: Lustre: Skipped 1 previous similar message
Jun 19 21:08:31 weisshorn05 kernel: Lustre: scratch-OST001b: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) refused reconnection, still busy with 2 active RPCs
Jun 19 21:08:31 weisshorn05 kernel: LustreError: 8374:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc -107  req@ffff88101148a400 x1403837087048340/t0(0) o4->cdd482d3-fa4f-4442-72da-16bfade64b5f@148.187.6.236@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340133042 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 21:08:31 weisshorn05 kernel: Lustre: scratch-OST001b: Bulk IO write error with cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2), client will retry: rc -107
Jun 19 21:08:31 weisshorn05 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 21:08:38 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802e9757800, cur 1340132918 expire 1340132768 last 1340132691
Jun 19 21:08:38 weisshorn08 kernel: Lustre: Skipped 1 previous similar message
Jun 19 21:08:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:08:46 weisshorn09 kernel: Lustre: scratch-OST001a: haven't heard from client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881015d3c400, cur 1340132926 expire 1340132776 last 1340132699
Jun 19 21:08:46 weisshorn09 kernel: Lustre: Skipped 6 previous similar messages
Jun 19 21:08:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:08:54 weisshorn05 kernel: Lustre: scratch-OST001b: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 21:09:01 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340132885/real 1340132885]  req@ffff880407176000 x1403849872204265/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340132941 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:09:01 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:09:03 weisshorn13 kernel: Lustre: scratch-OST001e: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88024a875800, cur 1340132943 expire 1340132793 last 1340132716
Jun 19 21:09:03 weisshorn13 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 21:09:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:09:11 weisshorn03 kernel: Lustre: scratch-OST0038: haven't heard from client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a0a82c400, cur 1340132951 expire 1340132801 last 1340132724
Jun 19 21:09:11 weisshorn03 kernel: Lustre: Skipped 6 previous similar messages
Jun 19 21:09:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:09:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:09:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:09:55 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340132939/real 1340132939]  req@ffff880e633ac800 x1403849943821929/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340132995 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:09:55 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:10:01 weisshorn02 CROND[48665]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:10:01 weisshorn13 CROND[129892]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:10:01 weisshorn14 CROND[54856]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:10:01 weisshorn07 CROND[126444]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:10:01 weisshorn01 CROND[3412]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:10:01 weisshorn01 CROND[3413]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 21:10:01 weisshorn10 CROND[126298]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:10:01 weisshorn14 sshd[54858]: Accepted publickey for root from 148.187.12.231 port 47191 ssh2
Jun 19 21:10:01 weisshorn14 sshd[54858]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 21:10:01 weisshorn06 sshd[127222]: Accepted publickey for root from 148.187.12.231 port 54720 ssh2
Jun 19 21:10:01 weisshorn06 sshd[127222]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 21:10:01 weisshorn03 CROND[126994]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:10:01 weisshorn06 CROND[127231]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:10:01 weisshorn14 sshd[54858]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 21:10:01 weisshorn14 sshd[54858]: pam_unix(sshd:session): session closed for user root
Jun 19 21:10:01 weisshorn10 sshd[126295]: Accepted publickey for root from 148.187.12.231 port 42211 ssh2
Jun 19 21:10:01 weisshorn10 sshd[126295]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 21:10:01 weisshorn12 CROND[127863]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:10:01 weisshorn08 sshd[125071]: Accepted publickey for root from 148.187.12.231 port 39105 ssh2
Jun 19 21:10:01 weisshorn12 sshd[127860]: Accepted publickey for root from 148.187.12.231 port 42335 ssh2
Jun 19 21:10:01 weisshorn05 sshd[126267]: Accepted publickey for root from 148.187.12.231 port 46602 ssh2
Jun 19 21:10:01 weisshorn05 sshd[126267]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 21:10:01 weisshorn08 sshd[125071]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 21:10:01 weisshorn07 sshd[126446]: Accepted publickey for root from 148.187.12.231 port 57373 ssh2
Jun 19 21:10:01 weisshorn07 sshd[126446]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 21:10:01 weisshorn03 sshd[126991]: Accepted publickey for root from 148.187.12.231 port 38217 ssh2
Jun 19 21:10:01 weisshorn03 sshd[126991]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 21:10:01 weisshorn05 CROND[126272]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:10:01 weisshorn12 sshd[127860]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 21:10:01 weisshorn09 sshd[126015]: Accepted publickey for root from 148.187.12.231 port 56754 ssh2
Jun 19 21:10:01 weisshorn09 sshd[126015]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 21:10:01 weisshorn10 sshd[126295]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 21:10:01 weisshorn10 sshd[126295]: pam_unix(sshd:session): session closed for user root
Jun 19 21:10:01 weisshorn06 sshd[127222]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 21:10:01 weisshorn06 sshd[127222]: pam_unix(sshd:session): session closed for user root
Jun 19 21:10:01 weisshorn13 sshd[129893]: Accepted publickey for root from 148.187.12.231 port 40135 ssh2
Jun 19 21:10:01 weisshorn04 sshd[125730]: Accepted publickey for root from 148.187.12.231 port 60171 ssh2
Jun 19 21:10:01 weisshorn04 sshd[125730]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 21:10:01 weisshorn13 sshd[129893]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 21:10:01 weisshorn09 CROND[126026]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:10:01 weisshorn08 sshd[125071]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 21:10:01 weisshorn08 sshd[125071]: pam_unix(sshd:session): session closed for user root
Jun 19 21:10:01 weisshorn07 sshd[126446]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 21:10:01 weisshorn07 sshd[126446]: pam_unix(sshd:session): session closed for user root
Jun 19 21:10:01 weisshorn03 sshd[126991]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 21:10:01 weisshorn03 sshd[126991]: pam_unix(sshd:session): session closed for user root
Jun 19 21:10:01 weisshorn09 sshd[126015]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 21:10:01 weisshorn09 sshd[126015]: pam_unix(sshd:session): session closed for user root
Jun 19 21:10:01 weisshorn12 sshd[127860]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 21:10:01 weisshorn12 sshd[127860]: pam_unix(sshd:session): session closed for user root
Jun 19 21:10:01 weisshorn05 sshd[126267]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 21:10:01 weisshorn05 sshd[126267]: pam_unix(sshd:session): session closed for user root
Jun 19 21:10:01 weisshorn08 CROND[125094]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:10:01 weisshorn04 sshd[125730]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 21:10:01 weisshorn04 sshd[125730]: pam_unix(sshd:session): session closed for user root
Jun 19 21:10:01 weisshorn04 CROND[125753]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:10:01 weisshorn13 sshd[129893]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 21:10:01 weisshorn13 sshd[129893]: pam_unix(sshd:session): session closed for user root
Jun 19 21:10:02 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340132946/real 1340132946]  req@ffff880d1737ac00 x1403849897856014/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340133002 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:10:02 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:10:04 weisshorn01 postfix/pickup[2716]: 59EE792224F: uid=0 from=<root>
Jun 19 21:10:04 weisshorn01 postfix/cleanup[3444]: 59EE792224F: message-id=<20120619191004.59EE792224F@weisshorn01.admin.cscs.ch>
Jun 19 21:10:04 weisshorn01 postfix/qmgr[2717]: 59EE792224F: from=<root@weisshorn01.admin.cscs.ch>, size=1128, nrcpt=1 (queue active)
Jun 19 21:10:04 weisshorn01 postfix/local[3446]: 59EE792224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=3.2, delays=3.1/0.01/0/0.03, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 21:10:04 weisshorn01 postfix/qmgr[2717]: 59EE792224F: removed
Jun 19 21:10:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:10:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:10:27 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340132971/real 1340132971]  req@ffff88081a0c0800 x1403849637147655/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340133027 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:10:27 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:10:32 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340132976/real 1340132976]  req@ffff88081d6a5400 x1403849823218039/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340133032 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:10:32 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 21:10:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:10:42 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340132986/real 1340132986]  req@ffff88102ab96c00 x1403849862201421/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340133042 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:10:42 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:10:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:11:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:11:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:11:33 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340133037/real 1340133037]  req@ffff880aa0a90000 x1403849922314323/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340133093 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:11:33 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:11:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:11:44 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340133048/real 1340133048]  req@ffff881005877000 x1403849873958305/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340133104 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:11:44 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:11:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:12:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:12:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:12:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:12:49 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340133113/real 1340133113]  req@ffff880f699c2800 x1403849954112917/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340133169 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:12:49 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:12:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:13:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:13:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:13:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:13:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:14:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:14:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:14:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:14:42 weisshorn03 kernel: Lustre: scratch-OST0020: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880598a0d800, cur 1340133282 expire 1340133132 last 1340133055
Jun 19 21:14:42 weisshorn06 kernel: Lustre: scratch-OST0041: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881012cf6c00, cur 1340133282 expire 1340133132 last 1340133055
Jun 19 21:14:42 weisshorn06 kernel: Lustre: Skipped 9 previous similar messages
Jun 19 21:14:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:15:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:15:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:15:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:15:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:16:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:16:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:16:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:16:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:17:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:17:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:17:38 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:17:53 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:18:08 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:18:20 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340133444/real 1340133444]  req@ffff8807007d5c00 x1403849985738080/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340133500 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:18:20 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:18:23 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:18:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:18:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:19:01 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340133485/real 1340133485]  req@ffff8810124b6000 x1403849872295786/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340133541 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:19:01 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 21:19:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:19:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:19:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:19:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:20:01 weisshorn04 CROND[125811]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:20:01 weisshorn02 CROND[48761]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:20:01 weisshorn14 CROND[54912]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:20:01 weisshorn01 CROND[3507]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:20:01 weisshorn07 CROND[126495]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:20:01 weisshorn13 CROND[129943]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:20:01 weisshorn10 CROND[126375]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:20:01 weisshorn06 CROND[127269]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:20:01 weisshorn03 CROND[127046]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:20:01 weisshorn05 CROND[126348]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:20:01 weisshorn12 CROND[127924]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:20:01 weisshorn09 CROND[126070]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:20:01 weisshorn08 CROND[125123]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:20:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:20:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:20:32 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340133576/real 1340133576]  req@ffff880807823000 x1403849823290177/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340133632 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:20:32 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 21:20:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:20:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:21:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:21:10 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340133614/real 1340133614]  req@ffff881024ed3c00 x1403849943893533/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340133670 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:21:10 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:21:17 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340133621/real 1340133621]  req@ffff88049534ac00 x1403849897940556/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340133677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:21:17 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:21:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:21:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:21:42 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340133646/real 1340133646]  req@ffff8802c5c7ac00 x1403849637237456/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340133702 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:21:42 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:21:44 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340133648/real 1340133648]  req@ffff8810147f3c00 x1403849874041400/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340133704 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:21:44 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 21:21:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:21:57 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340133661/real 1340133661]  req@ffff881015aa1800 x1403849862265162/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340133717 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:21:57 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:22:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:22:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:22:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:22:48 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340133712/real 1340133712]  req@ffff880a2dbfb400 x1403849922384244/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340133768 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:22:48 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:22:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:23:04 weisshorn05 kernel: Lustre: scratch-OST001b: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 21:23:04 weisshorn05 kernel: Lustre: scratch-OST001b: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) refused reconnection, still busy with 4 active RPCs
Jun 19 21:23:04 weisshorn05 kernel: LustreError: 5230:0:(events.c:389:server_bulk_callback()) event type 4, status -61, desc ffff880791bd70c0
Jun 19 21:23:04 weisshorn05 kernel: LustreError: 6458:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc -107  req@ffff88100e7f8400 x1403837087247460/t0(0) o4->cdd482d3-fa4f-4442-72da-16bfade64b5f@148.187.6.236@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340133799 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 21:23:04 weisshorn05 kernel: LustreError: 5242:0:(events.c:389:server_bulk_callback()) event type 4, status -61, desc ffff880791bd7780
Jun 19 21:23:04 weisshorn05 kernel: LustreError: 5249:0:(events.c:389:server_bulk_callback()) event type 4, status -61, desc ffff880db01e5b40
Jun 19 21:23:04 weisshorn05 kernel: LustreError: 5238:0:(events.c:389:server_bulk_callback()) event type 4, status -61, desc ffff880a4d558780
Jun 19 21:23:04 weisshorn05 kernel: LustreError: 5250:0:(events.c:389:server_bulk_callback()) event type 4, status -61, desc ffff880db01e5a80
Jun 19 21:23:04 weisshorn05 kernel: LustreError: 5250:0:(events.c:389:server_bulk_callback()) event type 2, status -61, desc ffff880db01e5a80
Jun 19 21:23:04 weisshorn05 kernel: Lustre: scratch-OST0027: Bulk IO write error with cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2), client will retry: rc -107
Jun 19 21:23:04 weisshorn05 kernel: LustreError: 5230:0:(events.c:389:server_bulk_callback()) event type 2, status -61, desc ffff880791bd70c0
Jun 19 21:23:04 weisshorn05 kernel: LustreError: 8547:0:(ldlm_lib.c:2706:target_bulk_io()) @@@ Reconnect on bulk GET  req@ffff881013ba8400 x1403837087247500/t0(0) o4->cdd482d3-fa4f-4442-72da-16bfade64b5f@148.187.6.236@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340133799 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 21:23:04 weisshorn05 kernel: LustreError: 8547:0:(ldlm_lib.c:2706:target_bulk_io()) Skipped 12 previous similar messages
Jun 19 21:23:04 weisshorn05 kernel: LustreError: 5242:0:(events.c:389:server_bulk_callback()) event type 2, status -61, desc ffff880791bd7780
Jun 19 21:23:04 weisshorn05 kernel: LustreError: 5249:0:(events.c:389:server_bulk_callback()) event type 2, status -61, desc ffff880db01e5b40
Jun 19 21:23:04 weisshorn05 kernel: LustreError: 5238:0:(events.c:389:server_bulk_callback()) event type 2, status -61, desc ffff880a4d558780
Jun 19 21:23:04 weisshorn05 kernel: LustreError: 6458:0:(ldlm_lib.c:2725:target_bulk_io()) Skipped 1 previous similar message
Jun 19 21:23:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:23:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:23:28 weisshorn05 kernel: Lustre: scratch-OST003e: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 21:23:28 weisshorn05 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 21:23:28 weisshorn06 kernel: LustreError: 137-5: scratch-OST001b: Not available for connect from 148.187.6.236@o2ib2 (no target)
Jun 19 21:23:28 weisshorn06 kernel: LustreError: Skipped 4 previous similar messages
Jun 19 21:23:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:23:53 weisshorn05 kernel: Lustre: scratch-OST001b: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 21:23:53 weisshorn05 kernel: Lustre: Skipped 1 previous similar message
Jun 19 21:23:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:24:04 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340133788/real 1340133788]  req@ffff8809b4ea4400 x1403849954281389/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340133844 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:24:04 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:24:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:24:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:24:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:24:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:25:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:25:15 weisshorn09 nslcd[3954]: [997556] ldap_result() failed: Can't contact LDAP server
Jun 19 21:25:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:25:38 weisshorn08 nslcd[3946]: [6f6d8a] ldap_result() failed: Can't contact LDAP server
Jun 19 21:25:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:25:39 weisshorn06 nslcd[3953]: [0e6897] ldap_result() failed: Can't contact LDAP server
Jun 19 21:25:41 weisshorn04 nslcd[3885]: [997556] ldap_result() failed: Can't contact LDAP server
Jun 19 21:25:42 weisshorn07 nslcd[4297]: [6f6d8a] ldap_result() failed: Can't contact LDAP server
Jun 19 21:25:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:26:02 weisshorn12 nslcd[3954]: [6f6d8a] ldap_result() failed: Can't contact LDAP server
Jun 19 21:26:04 weisshorn13 nslcd[4224]: [6f6d8a] ldap_result() failed: Can't contact LDAP server
Jun 19 21:26:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:26:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:26:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:26:41 weisshorn05 nslcd[4378]: [49361f] ldap_result() failed: Can't contact LDAP server
Jun 19 21:26:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:27:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:27:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:27:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:27:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:28:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:28:20 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340134044/real 1340134044]  req@ffff880573d13800 x1403849985846352/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340134100 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:28:20 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 21:28:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:28:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:28:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:29:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:29:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:29:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:29:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:30:01 weisshorn04 CROND[125843]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:30:01 weisshorn02 CROND[48816]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:30:01 weisshorn14 CROND[54971]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:30:01 weisshorn01 CROND[3543]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:30:01 weisshorn10 CROND[126406]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:30:01 weisshorn07 CROND[126570]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:30:01 weisshorn13 CROND[130009]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:30:01 weisshorn06 CROND[127315]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:30:01 weisshorn05 CROND[126381]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:30:01 weisshorn12 CROND[127985]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:30:01 weisshorn03 CROND[127071]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:30:01 weisshorn09 CROND[126102]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:30:02 weisshorn08 CROND[125156]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:30:09 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:30:16 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340134160/real 1340134160]  req@ffff881011beec00 x1403849872384017/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340134216 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:30:16 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:30:24 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:30:39 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:30:54 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:31:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:31:10 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340134214/real 1340134214]  req@ffff880c0c787000 x1403849943945251/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340134270 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:31:10 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 21:31:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:31:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:31:47 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340134251/real 1340134251]  req@ffff880808071400 x1403849823354352/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340134307 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:31:47 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:31:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:32:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:32:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:32:32 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340134296/real 1340134296]  req@ffff880816b57000 x1403849898006425/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340134352 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:32:32 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:32:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:32:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:32:57 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340134321/real 1340134321]  req@ffff8808d1913400 x1403849637299070/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340134377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:32:57 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:32:59 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340134323/real 1340134323]  req@ffff881013015400 x1403849874101344/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340134379 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:32:59 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:33:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:33:12 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340134336/real 1340134336]  req@ffff881018386400 x1403849862328930/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340134392 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:33:12 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:33:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:33:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:33:45 weisshorn02 kernel: LustreError: 48117:0:(mdt_recovery.c:813:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 55843121142, new: 55843121141 replay: 0. see LU-617.
Jun 19 21:33:45 weisshorn02 kernel: LustreError: 48117:0:(mdt_recovery.c:813:mdt_last_rcvd_update()) Skipped 1 previous similar message
Jun 19 21:33:45 weisshorn02 kernel: LustreError: 48148:0:(mdt_recovery.c:813:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 55843121755, new: 55843121754 replay: 0. see LU-617.
Jun 19 21:33:45 weisshorn02 kernel: LustreError: 48148:0:(mdt_recovery.c:813:mdt_last_rcvd_update()) Skipped 1 previous similar message
Jun 19 21:33:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:34:03 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340134387/real 1340134387]  req@ffff8810121f6000 x1403849922454257/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340134443 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:34:03 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:34:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:34:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:34:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:34:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:35:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:35:19 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340134463/real 1340134463]  req@ffff880be7e43800 x1403849954395258/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340134519 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:35:19 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:35:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:35:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:35:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:35:57 weisshorn03 kernel: Lustre: scratch-OST0020: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881010cc3000, cur 1340134557 expire 1340134407 last 1340134330
Jun 19 21:35:57 weisshorn03 kernel: Lustre: Skipped 1 previous similar message
Jun 19 21:35:57 weisshorn06 kernel: Lustre: scratch-OST0041: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806ff1eec00, cur 1340134557 expire 1340134407 last 1340134330
Jun 19 21:35:57 weisshorn13 kernel: Lustre: scratch-OST000b: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88081cda1400, cur 1340134557 expire 1340134407 last 1340134330
Jun 19 21:35:57 weisshorn13 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 21:36:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:36:22 weisshorn09 kernel: Lustre: scratch-OST0026: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880399f8a800, cur 1340134582 expire 1340134432 last 1340134355
Jun 19 21:36:22 weisshorn09 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 21:36:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:36:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:36:47 weisshorn03 kernel: Lustre: scratch-OST002c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bbefebc00, cur 1340134607 expire 1340134457 last 1340134380
Jun 19 21:36:47 weisshorn04 kernel: Lustre: scratch-OST0019: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810177eec00, cur 1340134607 expire 1340134457 last 1340134380
Jun 19 21:36:47 weisshorn04 kernel: Lustre: Skipped 6 previous similar messages
Jun 19 21:36:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:37:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:37:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:37:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:37:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:38:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:38:20 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340134644/real 1340134644]  req@ffff88012a0d6000 x1403849985972291/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340134700 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:38:20 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 21:38:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:38:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:38:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:39:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:39:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:39:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:39:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:40:01 weisshorn08 CROND[125216]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:40:01 weisshorn04 CROND[125877]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:40:01 weisshorn02 CROND[48913]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:40:01 weisshorn14 CROND[55001]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:40:01 weisshorn01 CROND[3567]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:40:01 weisshorn10 CROND[126439]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:40:01 weisshorn07 CROND[126590]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:40:01 weisshorn13 CROND[130041]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:40:01 weisshorn06 CROND[127371]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:40:01 weisshorn05 CROND[126414]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:40:01 weisshorn03 CROND[127136]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:40:01 weisshorn12 CROND[128010]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:40:01 weisshorn09 CROND[126163]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:40:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:40:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:40:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:40:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:41:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:41:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:41:31 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340134835/real 1340134835]  req@ffff881023173000 x1403849872455436/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340134891 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:41:31 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:41:31 weisshorn14 nslcd[3437]: [06ecb2] ldap_result() failed: Can't contact LDAP server
Jun 19 21:41:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:41:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:42:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:42:25 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340134889/real 1340134889]  req@ffff880c20190000 x1403849943998765/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340134945 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:42:25 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:42:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:42:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:42:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:43:02 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340134926/real 1340134926]  req@ffff880821337000 x1403849823400074/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340134982 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:43:02 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:43:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:43:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:43:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:43:47 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340134971/real 1340134971]  req@ffff880fb9f81000 x1403849898060452/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340135027 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:43:47 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:43:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:44:03 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340134987/real 1340134987]  req@ffff881015e28400 x1403849922501586/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340135043 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:44:03 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 21:44:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:44:12 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340134996/real 1340134996]  req@ffff880effb39c00 x1403849637353212/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340135052 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:44:12 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:44:14 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340134998/real 1340134998]  req@ffff880167e09800 x1403849874171405/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340135054 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:44:14 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:44:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:44:27 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340135011/real 1340135011]  req@ffff881015feec00 x1403849862381253/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340135067 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:44:27 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:44:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:44:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:45:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:45:19 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340135063/real 1340135063]  req@ffff880891446c00 x1403849954477602/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340135119 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:45:19 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 21:45:21 weisshorn03 nslcd[3944]: [333c89] ldap_result() failed: Can't contact LDAP server
Jun 19 21:45:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:45:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:45:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:46:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:46:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:46:32 weisshorn10 nslcd[4294]: [49361f] ldap_result() failed: Can't contact LDAP server
Jun 19 21:46:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:46:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:47:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:47:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:47:40 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:47:55 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:48:10 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:48:25 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:48:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:48:52 weisshorn09 kernel: Lustre: scratch-OST0026: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880965930000, cur 1340135332 expire 1340135182 last 1340135105
Jun 19 21:48:52 weisshorn09 kernel: Lustre: Skipped 1 previous similar message
Jun 19 21:48:52 weisshorn05 kernel: Lustre: scratch-OST0027: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881021745800, cur 1340135332 expire 1340135182 last 1340135105
Jun 19 21:48:52 weisshorn05 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 21:48:52 weisshorn13 kernel: Lustre: scratch-OST0011: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880735699c00, cur 1340135332 expire 1340135182 last 1340135105
Jun 19 21:48:52 weisshorn13 kernel: Lustre: Skipped 1 previous similar message
Jun 19 21:48:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:49:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:49:17 weisshorn13 kernel: Lustre: scratch-OST0005: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101320f000, cur 1340135357 expire 1340135207 last 1340135130
Jun 19 21:49:17 weisshorn03 kernel: Lustre: scratch-OST002c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88058e527000, cur 1340135357 expire 1340135207 last 1340135130
Jun 19 21:49:17 weisshorn09 kernel: Lustre: scratch-OST0004: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f70b6dc00, cur 1340135357 expire 1340135207 last 1340135130
Jun 19 21:49:17 weisshorn04 kernel: Lustre: scratch-OST0031: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881022181000, cur 1340135357 expire 1340135207 last 1340135130
Jun 19 21:49:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:49:35 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340135319/real 1340135319]  req@ffff881011aa0000 x1403849986063585/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340135375 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:49:35 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:49:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:49:42 weisshorn08 kernel: Lustre: scratch-OST000c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bd301a800, cur 1340135382 expire 1340135232 last 1340135155
Jun 19 21:49:42 weisshorn08 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 21:49:42 weisshorn13 kernel: Lustre: scratch-OST000b: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880726d94000, cur 1340135382 expire 1340135232 last 1340135155
Jun 19 21:49:42 weisshorn12 kernel: Lustre: scratch-OST0021: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88067061d400, cur 1340135382 expire 1340135232 last 1340135155
Jun 19 21:49:42 weisshorn12 kernel: Lustre: Skipped 8 previous similar messages
Jun 19 21:49:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:50:01 weisshorn08 CROND[125249]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:50:01 weisshorn04 CROND[125945]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:50:01 weisshorn14 CROND[55042]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:50:01 weisshorn02 CROND[48998]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:50:01 weisshorn01 CROND[3635]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:50:01 weisshorn10 CROND[126507]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:50:01 weisshorn07 CROND[126634]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:50:01 weisshorn13 CROND[130073]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:50:01 weisshorn06 CROND[127410]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:50:01 weisshorn05 CROND[126482]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:50:01 weisshorn03 CROND[127178]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:50:02 weisshorn09 CROND[126204]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:50:02 weisshorn12 CROND[128054]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 21:50:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:50:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:50:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:50:44 weisshorn02 kernel: Lustre: 47776:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200005069:0x14:0x0]] [valid 68719476736]
Jun 19 21:50:46 weisshorn14 nslcd[3437]: [594807] ldap_result() failed: Can't contact LDAP server
Jun 19 21:50:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:51:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:51:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:51:30 weisshorn02 kernel: Lustre: 47752:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200005069:0x19:0x0]] [valid 68719476736]
Jun 19 21:51:31 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340135435/real 1340135435]  req@ffff881013058400 x1403849872510796/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340135491 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:51:31 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 21:51:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:51:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:52:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:52:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:52:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:52:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:53:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:53:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:53:40 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340135564/real 1340135564]  req@ffff880d2eee5400 x1403849944043285/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340135620 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:53:40 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:53:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:53:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:54:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:54:17 weisshorn12 kernel: Lustre: scratch-OST0028: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88047e550400, cur 1340135657 expire 1340135507 last 1340135430
Jun 19 21:54:17 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340135601/real 1340135601]  req@ffff8804bc26a400 x1403849823439812/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340135657 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:54:17 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:54:17 weisshorn07 kernel: Lustre: scratch-OST0013: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881016d82800, cur 1340135657 expire 1340135507 last 1340135430
Jun 19 21:54:17 weisshorn07 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 21:54:17 weisshorn06 kernel: Lustre: scratch-OST002a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804d33da000, cur 1340135657 expire 1340135507 last 1340135430
Jun 19 21:54:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:54:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:54:42 weisshorn10 kernel: Lustre: scratch-OST000a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88081e9dd800, cur 1340135682 expire 1340135532 last 1340135455
Jun 19 21:54:42 weisshorn03 kernel: Lustre: scratch-OST0009: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805c0845c00, cur 1340135682 expire 1340135532 last 1340135455
Jun 19 21:54:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:55:02 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340135646/real 1340135646]  req@ffff880f0dc42400 x1403849898154172/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340135702 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:55:02 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:55:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:55:18 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340135662/real 1340135662]  req@ffff881016c00400 x1403849922554732/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340135718 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:55:18 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:55:19 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340135663/real 1340135663]  req@ffff8808f4b75800 x1403849954590607/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340135719 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:55:19 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 21:55:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:55:27 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340135671/real 1340135671]  req@ffff880ddc8a9c00 x1403849637424705/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340135727 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:55:27 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:55:29 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340135673/real 1340135673]  req@ffff8810169dac00 x1403849874237660/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340135729 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:55:29 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:55:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:55:42 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340135686/real 1340135686]  req@ffff88101aa5b800 x1403849862431728/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340135742 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 21:55:42 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 21:55:54 weisshorn02 kernel: Lustre: 47806:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200005069:0x1e:0x0]] [valid 68719476736]
Jun 19 21:55:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:56:05 weisshorn02 kernel: Lustre: 48051:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200005069:0x23:0x0]] [valid 68719476736]
Jun 19 21:56:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:56:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:56:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:56:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:57:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:57:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:57:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:57:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:58:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:58:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:58:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:58:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:59:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:59:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:59:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 21:59:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:00:01 weisshorn03 CROND[127203]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:00:01 weisshorn12 CROND[128119]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:00:01 weisshorn09 CROND[126228]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:00:01 weisshorn08 CROND[125281]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:00:01 weisshorn04 CROND[125969]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:00:01 weisshorn14 CROND[55096]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:00:01 weisshorn02 CROND[49064]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:00:01 weisshorn01 CROND[3664]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:00:01 weisshorn10 CROND[126533]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:00:01 weisshorn07 CROND[126696]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:00:01 weisshorn06 CROND[127441]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:00:01 weisshorn13 CROND[130136]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:00:02 weisshorn05 CROND[126507]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:00:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:00:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:00:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:00:50 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340135994/real 1340135994]  req@ffff881011a77c00 x1403849986181654/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340136050 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:00:50 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:00:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:01:01 weisshorn03 CROND[127208]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 22:01:01 weisshorn10 nslcd[4294]: [bc6770] ldap_result() failed: Can't contact LDAP server
Jun 19 22:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1272 starting 0anacron
Jun 19 22:01:01 weisshorn05 CROND[126512]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 22:01:01 weisshorn06 nslcd[3953]: [5138de] ldap_result() failed: Can't contact LDAP server
Jun 19 22:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1272 finished 0anacron
Jun 19 22:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1272 starting mcelog.cron
Jun 19 22:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1265 starting 0anacron
Jun 19 22:01:01 weisshorn12 CROND[128124]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 22:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1281 starting 0anacron
Jun 19 22:01:01 weisshorn09 CROND[126233]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 22:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1281 finished 0anacron
Jun 19 22:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1281 starting mcelog.cron
Jun 19 22:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1272 finished mcelog.cron
Jun 19 22:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1262 starting 0anacron
Jun 19 22:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1281 finished mcelog.cron
Jun 19 22:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1265 finished 0anacron
Jun 19 22:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1265 starting mcelog.cron
Jun 19 22:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1262 finished 0anacron
Jun 19 22:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1262 starting mcelog.cron
Jun 19 22:01:01 weisshorn08 CROND[125286]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 22:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1265 finished mcelog.cron
Jun 19 22:01:01 weisshorn09 run-parts(/etc/cron.hourly)[1262 finished mcelog.cron
Jun 19 22:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1252 starting 0anacron
Jun 19 22:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1252 finished 0anacron
Jun 19 22:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1252 starting mcelog.cron
Jun 19 22:01:01 weisshorn05 nslcd[4378]: [bc6770] ldap_result() failed: Can't contact LDAP server
Jun 19 22:01:01 weisshorn13 nslcd[4224]: [49361f] ldap_result() failed: Can't contact LDAP server
Jun 19 22:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1253 finished mcelog.cron
Jun 19 22:01:01 weisshorn04 CROND[125974]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 22:01:01 weisshorn09 nslcd[3954]: [333c89] ldap_result() failed: Can't contact LDAP server
Jun 19 22:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1259 starting 0anacron
Jun 19 22:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1259 finished 0anacron
Jun 19 22:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1259 starting mcelog.cron
Jun 19 22:01:01 weisshorn03 nslcd[3944]: [a9831a] ldap_result() failed: Can't contact LDAP server
Jun 19 22:01:01 weisshorn14 CROND[55102]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 22:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5510 starting 0anacron
Jun 19 22:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1259 finished mcelog.cron
Jun 19 22:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5511 finished 0anacron
Jun 19 22:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5510 starting mcelog.cron
Jun 19 22:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5511 finished mcelog.cron
Jun 19 22:01:01 weisshorn12 nslcd[3954]: [49361f] ldap_result() failed: Can't contact LDAP server
Jun 19 22:01:01 weisshorn07 nslcd[4297]: [49361f] ldap_result() failed: Can't contact LDAP server
Jun 19 22:01:01 weisshorn01 CROND[3669]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 22:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3669 starting 0anacron
Jun 19 22:01:01 weisshorn02 CROND[49068]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 22:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3679 finished 0anacron
Jun 19 22:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3669 starting mcelog.cron
Jun 19 22:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4906 starting 0anacron
Jun 19 22:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3686 finished mcelog.cron
Jun 19 22:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4907 finished 0anacron
Jun 19 22:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4906 starting mcelog.cron
Jun 19 22:01:01 weisshorn08 nslcd[3946]: [49361f] ldap_result() failed: Can't contact LDAP server
Jun 19 22:01:01 weisshorn02 run-parts(/etc/cron.hourly)[4908 finished mcelog.cron
Jun 19 22:01:01 weisshorn10 CROND[126538]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 22:01:01 weisshorn01 nslcd[2168]: [36c40e] ldap_result() failed: Can't contact LDAP server
Jun 19 22:01:01 weisshorn07 CROND[126701]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 22:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1265 starting 0anacron
Jun 19 22:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1265 finished 0anacron
Jun 19 22:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1265 starting mcelog.cron
Jun 19 22:01:01 weisshorn13 CROND[130141]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 22:01:01 weisshorn06 CROND[127446]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 22:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1274 starting 0anacron
Jun 19 22:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1267 starting 0anacron
Jun 19 22:01:01 weisshorn04 nslcd[3885]: [333c89] ldap_result() failed: Can't contact LDAP server
Jun 19 22:01:02 weisshorn06 run-parts(/etc/cron.hourly)[1274 finished 0anacron
Jun 19 22:01:02 weisshorn06 run-parts(/etc/cron.hourly)[1274 starting mcelog.cron
Jun 19 22:01:02 weisshorn10 run-parts(/etc/cron.hourly)[1265 finished mcelog.cron
Jun 19 22:01:02 weisshorn06 run-parts(/etc/cron.hourly)[1274 finished mcelog.cron
Jun 19 22:01:02 weisshorn07 run-parts(/etc/cron.hourly)[1267 finished 0anacron
Jun 19 22:01:02 weisshorn07 run-parts(/etc/cron.hourly)[1267 starting mcelog.cron
Jun 19 22:01:02 weisshorn13 run-parts(/etc/cron.hourly)[1301 starting 0anacron
Jun 19 22:01:02 weisshorn07 run-parts(/etc/cron.hourly)[1267 finished mcelog.cron
Jun 19 22:01:02 weisshorn13 run-parts(/etc/cron.hourly)[1301 finished 0anacron
Jun 19 22:01:02 weisshorn13 run-parts(/etc/cron.hourly)[1301 starting mcelog.cron
Jun 19 22:01:02 weisshorn13 run-parts(/etc/cron.hourly)[1301 finished mcelog.cron
Jun 19 22:01:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:01:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:01:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:01:47 weisshorn04 kernel: Lustre: scratch-OST0019: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ce3ede800, cur 1340136107 expire 1340135957 last 1340135880
Jun 19 22:01:47 weisshorn04 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:01:47 weisshorn03 kernel: Lustre: scratch-OST002c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880274072400, cur 1340136107 expire 1340135957 last 1340135880
Jun 19 22:01:47 weisshorn09 kernel: Lustre: scratch-OST0004: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804c8b32c00, cur 1340136107 expire 1340135957 last 1340135880
Jun 19 22:01:47 weisshorn09 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:01:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:02:10 weisshorn07 rhnsd[126720]: running program /usr/sbin/rhn_check
Jun 19 22:02:10 weisshorn02 kernel: LustreError: 47406:0:(mdt_handler.c:2822:mdt_recovery()) operation 400 on unconnected MDS from 12345-148.187.6.7@o2ib2
Jun 19 22:02:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:02:12 weisshorn13 kernel: Lustre: scratch-OST000b: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88091be53000, cur 1340136132 expire 1340135982 last 1340135905
Jun 19 22:02:12 weisshorn06 kernel: Lustre: scratch-OST0041: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881012bc7000, cur 1340136132 expire 1340135982 last 1340135905
Jun 19 22:02:12 weisshorn03 kernel: Lustre: scratch-OST0020: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800a26b1c00, cur 1340136132 expire 1340135982 last 1340135905
Jun 19 22:02:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:02:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:02:46 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340136110/real 1340136110]  req@ffff88101166a400 x1403849872580895/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340136166 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:02:46 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:02:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:03:02 weisshorn04 kernel: Lustre: scratch-OST003d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f3b0bfc00, cur 1340136182 expire 1340136032 last 1340135955
Jun 19 22:03:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:03:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:03:27 weisshorn09 kernel: Lustre: scratch-OST0010: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881016dec400, cur 1340136207 expire 1340136057 last 1340135980
Jun 19 22:03:27 weisshorn13 kernel: Lustre: scratch-OST0011: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101425c000, cur 1340136207 expire 1340136057 last 1340135980
Jun 19 22:03:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:03:52 weisshorn09 kernel: Lustre: scratch-OST001a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88073cca1c00, cur 1340136232 expire 1340136082 last 1340136005
Jun 19 22:03:52 weisshorn09 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:03:52 weisshorn04 kernel: Lustre: scratch-OST0031: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810196ab000, cur 1340136232 expire 1340136082 last 1340136005
Jun 19 22:03:52 weisshorn13 kernel: Lustre: scratch-OST0005: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100dff5800, cur 1340136232 expire 1340136082 last 1340136005
Jun 19 22:03:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:04:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:04:17 weisshorn08 kernel: Lustre: scratch-OST000c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e536d3c00, cur 1340136257 expire 1340136107 last 1340136030
Jun 19 22:04:17 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340136201/real 1340136201]  req@ffff8808ab3e2c00 x1403849823477733/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340136257 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:04:17 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 22:04:17 weisshorn07 kernel: Lustre: scratch-OST0042: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c91769800, cur 1340136257 expire 1340136107 last 1340136030
Jun 19 22:04:17 weisshorn07 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:04:17 weisshorn12 kernel: Lustre: scratch-OST0021: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809a52edc00, cur 1340136257 expire 1340136107 last 1340136030
Jun 19 22:04:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:04:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:04:42 weisshorn05 kernel: Lustre: scratch-OST0027: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806b2ae5c00, cur 1340136282 expire 1340136132 last 1340136055
Jun 19 22:04:42 weisshorn06 kernel: Lustre: scratch-OST0012: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881012b96400, cur 1340136282 expire 1340136132 last 1340136055
Jun 19 22:04:55 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340136239/real 1340136239]  req@ffff880f1dfa9c00 x1403849944085275/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340136295 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:04:55 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:04:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:05:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:05:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:05:27 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340136271/real 1340136271]  req@ffff88100fb36800 x1403849637477438/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340136327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:05:27 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 22:05:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:05:42 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340136286/real 1340136286]  req@ffff88101a65c800 x1403849862469747/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340136342 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:05:42 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 22:05:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:06:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:06:17 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340136321/real 1340136321]  req@ffff88102b3a3800 x1403849898225657/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340136377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:06:17 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:06:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:06:33 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340136337/real 1340136337]  req@ffff88101595d000 x1403849922622400/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340136393 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:06:33 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:06:34 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340136338/real 1340136338]  req@ffff88035b849400 x1403849954684628/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340136394 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:06:34 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:06:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:06:44 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340136348/real 1340136348]  req@ffff880d6c4d4c00 x1403849874303361/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340136404 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:06:44 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:06:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:07:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:07:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:07:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:07:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:08:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:08:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:08:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:08:49 weisshorn02 kernel: Lustre: scratch-MDT0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807f12b2800, cur 1340136529 expire 1340136379 last 1340136302
Jun 19 22:08:49 weisshorn03 kernel: Lustre: scratch-OST002c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810288fd800, cur 1340136529 expire 1340136379 last 1340136302
Jun 19 22:08:49 weisshorn09 kernel: Lustre: scratch-OST0026: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881016833400, cur 1340136529 expire 1340136379 last 1340136302
Jun 19 22:08:49 weisshorn04 kernel: Lustre: scratch-OST0019: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881015f28400, cur 1340136529 expire 1340136379 last 1340136302
Jun 19 22:08:49 weisshorn07 kernel: Lustre: scratch-OST002b: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102a6e2400, cur 1340136529 expire 1340136379 last 1340136302
Jun 19 22:08:49 weisshorn13 kernel: Lustre: scratch-OST0018: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88021ddca400, cur 1340136529 expire 1340136379 last 1340136302
Jun 19 22:08:49 weisshorn06 kernel: Lustre: scratch-OST002a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080375c000, cur 1340136529 expire 1340136379 last 1340136302
Jun 19 22:08:49 weisshorn10 kernel: Lustre: scratch-OST000a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d9351ec00, cur 1340136529 expire 1340136379 last 1340136302
Jun 19 22:08:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:09:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:09:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:09:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:09:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:10:01 weisshorn10 CROND[126585]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:10:01 weisshorn06 CROND[127517]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:10:01 weisshorn07 CROND[126738]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:10:01 weisshorn13 CROND[130187]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:10:01 weisshorn03 CROND[127282]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:10:01 weisshorn12 CROND[128159]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:10:01 weisshorn09 CROND[126309]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:10:01 weisshorn05 CROND[126558]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:10:01 weisshorn08 CROND[125360]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:10:01 weisshorn04 CROND[126023]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:10:01 weisshorn14 CROND[55146]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:10:01 weisshorn01 CROND[3708]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:10:01 weisshorn01 CROND[3710]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 22:10:01 weisshorn02 CROND[49117]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:10:01 weisshorn14 sshd[55148]: Accepted publickey for root from 148.187.12.231 port 47200 ssh2
Jun 19 22:10:01 weisshorn14 sshd[55148]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 22:10:01 weisshorn08 sshd[125362]: Accepted publickey for root from 148.187.12.231 port 39119 ssh2
Jun 19 22:10:01 weisshorn08 sshd[125362]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 22:10:01 weisshorn06 sshd[127519]: Accepted publickey for root from 148.187.12.231 port 54729 ssh2
Jun 19 22:10:01 weisshorn12 sshd[128161]: Accepted publickey for root from 148.187.12.231 port 42348 ssh2
Jun 19 22:10:01 weisshorn12 sshd[128161]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 22:10:01 weisshorn06 sshd[127519]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 22:10:01 weisshorn03 sshd[127284]: Accepted publickey for root from 148.187.12.231 port 38229 ssh2
Jun 19 22:10:02 weisshorn14 sshd[55148]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 22:10:02 weisshorn14 sshd[55148]: pam_unix(sshd:session): session closed for user root
Jun 19 22:10:02 weisshorn03 sshd[127284]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 22:10:02 weisshorn10 sshd[126587]: Accepted publickey for root from 148.187.12.231 port 42224 ssh2
Jun 19 22:10:02 weisshorn05 sshd[126560]: Accepted publickey for root from 148.187.12.231 port 46613 ssh2
Jun 19 22:10:02 weisshorn05 sshd[126560]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 22:10:02 weisshorn10 sshd[126587]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 22:10:02 weisshorn07 sshd[126740]: Accepted publickey for root from 148.187.12.231 port 57386 ssh2
Jun 19 22:10:02 weisshorn09 sshd[126311]: Accepted publickey for root from 148.187.12.231 port 56772 ssh2
Jun 19 22:10:02 weisshorn09 sshd[126311]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 22:10:02 weisshorn07 sshd[126740]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 22:10:02 weisshorn04 sshd[126025]: Accepted publickey for root from 148.187.12.231 port 60185 ssh2
Jun 19 22:10:02 weisshorn08 sshd[125362]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 22:10:02 weisshorn08 sshd[125362]: pam_unix(sshd:session): session closed for user root
Jun 19 22:10:02 weisshorn12 sshd[128161]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 22:10:02 weisshorn12 sshd[128161]: pam_unix(sshd:session): session closed for user root
Jun 19 22:10:02 weisshorn04 sshd[126025]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 22:10:02 weisshorn06 sshd[127519]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 22:10:02 weisshorn06 sshd[127519]: pam_unix(sshd:session): session closed for user root
Jun 19 22:10:02 weisshorn13 sshd[130189]: Accepted publickey for root from 148.187.12.231 port 40152 ssh2
Jun 19 22:10:02 weisshorn13 sshd[130189]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 22:10:02 weisshorn09 sshd[126311]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 22:10:02 weisshorn09 sshd[126311]: pam_unix(sshd:session): session closed for user root
Jun 19 22:10:02 weisshorn10 sshd[126587]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 22:10:02 weisshorn10 sshd[126587]: pam_unix(sshd:session): session closed for user root
Jun 19 22:10:02 weisshorn03 sshd[127284]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 22:10:02 weisshorn03 sshd[127284]: pam_unix(sshd:session): session closed for user root
Jun 19 22:10:02 weisshorn05 sshd[126560]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 22:10:02 weisshorn05 sshd[126560]: pam_unix(sshd:session): session closed for user root
Jun 19 22:10:02 weisshorn04 sshd[126025]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 22:10:02 weisshorn04 sshd[126025]: pam_unix(sshd:session): session closed for user root
Jun 19 22:10:02 weisshorn07 sshd[126740]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 22:10:02 weisshorn07 sshd[126740]: pam_unix(sshd:session): session closed for user root
Jun 19 22:10:02 weisshorn13 sshd[130189]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 22:10:02 weisshorn13 sshd[130189]: pam_unix(sshd:session): session closed for user root
Jun 19 22:10:04 weisshorn01 nslcd[2168]: [901d82] ldap_result() failed: Can't contact LDAP server
Jun 19 22:10:04 weisshorn01 postfix/pickup[3521]: DCD8B92224F: uid=0 from=<root>
Jun 19 22:10:04 weisshorn01 postfix/cleanup[3740]: DCD8B92224F: message-id=<20120619201004.DCD8B92224F@weisshorn01.admin.cscs.ch>
Jun 19 22:10:04 weisshorn01 postfix/qmgr[2717]: DCD8B92224F: from=<root@weisshorn01.admin.cscs.ch>, size=1128, nrcpt=1 (queue active)
Jun 19 22:10:05 weisshorn01 postfix/local[3742]: DCD8B92224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=3.2, delays=3.1/0.01/0/0.04, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 22:10:05 weisshorn01 postfix/qmgr[2717]: DCD8B92224F: removed
Jun 19 22:10:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:10:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:10:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:10:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:10:57 weisshorn12 kernel: Lustre: scratch-OST0021: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880068872800, cur 1340136657 expire 1340136507 last 1340136430
Jun 19 22:10:57 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809cbfd5400, cur 1340136657 expire 1340136507 last 1340136430
Jun 19 22:11:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:11:22 weisshorn05 kernel: Lustre: scratch-OST0027: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880917ce0400, cur 1340136682 expire 1340136532 last 1340136455
Jun 19 22:11:22 weisshorn06 kernel: Lustre: scratch-OST0012: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881013cb6400, cur 1340136682 expire 1340136532 last 1340136455
Jun 19 22:11:22 weisshorn06 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 22:11:22 weisshorn02 kernel: Lustre: MGS: haven't heard from client 02e8b9ff-fc43-5841-d3bb-815052085829 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fb5fe6000, cur 1340136682 expire 1340136532 last 1340136455
Jun 19 22:11:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:11:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:11:47 weisshorn09 kernel: Lustre: scratch-OST0034: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806d11acc00, cur 1340136707 expire 1340136557 last 1340136480
Jun 19 22:11:47 weisshorn09 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 22:11:47 weisshorn13 kernel: Lustre: scratch-OST0005: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88060e82b400, cur 1340136707 expire 1340136557 last 1340136480
Jun 19 22:11:47 weisshorn13 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 22:11:47 weisshorn04 kernel: Lustre: scratch-OST0031: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102587a800, cur 1340136707 expire 1340136557 last 1340136480
Jun 19 22:11:47 weisshorn04 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:11:47 weisshorn06 kernel: Lustre: scratch-OST001d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f0571d000, cur 1340136707 expire 1340136557 last 1340136480
Jun 19 22:11:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:12:05 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340136669/real 1340136669]  req@ffff88020157e800 x1403849986299060/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340136725 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:12:05 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:12:10 weisshorn02 kernel: LustreError: 47510:0:(mdt_handler.c:2822:mdt_recovery()) operation 400 on unconnected MDS from 12345-148.187.6.7@o2ib2
Jun 19 22:12:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:12:12 weisshorn05 kernel: Lustre: scratch-OST000e: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100e8e9800, cur 1340136732 expire 1340136582 last 1340136505
Jun 19 22:12:12 weisshorn04 kernel: Lustre: scratch-OST000d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101aa02000, cur 1340136732 expire 1340136582 last 1340136505
Jun 19 22:12:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:12:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:12:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:13:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:13:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:13:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:13:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:14:01 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340136785/real 1340136785]  req@ffff880e68af1000 x1403849872657938/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340136841 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:14:01 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:14:06 weisshorn05 kernel: Lustre: scratch-OST0027: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 22:14:06 weisshorn05 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:14:06 weisshorn05 kernel: Lustre: scratch-OST0027: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) refused reconnection, still busy with 2 active RPCs
Jun 19 22:14:06 weisshorn05 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 22:14:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:14:17 weisshorn07 kernel: Lustre: scratch-OST001f: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806370fbc00, cur 1340136857 expire 1340136707 last 1340136630
Jun 19 22:14:17 weisshorn07 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 22:14:17 weisshorn04 kernel: Lustre: scratch-OST003d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810190b0800, cur 1340136857 expire 1340136707 last 1340136630
Jun 19 22:14:17 weisshorn10 kernel: Lustre: scratch-OST000a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88093d4d7400, cur 1340136857 expire 1340136707 last 1340136630
Jun 19 22:14:23 weisshorn14 rhnsd[55192]: running program /usr/sbin/rhn_check
Jun 19 22:14:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:14:30 weisshorn05 kernel: Lustre: scratch-OST0027: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 22:14:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:14:42 weisshorn09 kernel: Lustre: scratch-OST0010: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88081919d400, cur 1340136882 expire 1340136732 last 1340136655
Jun 19 22:14:42 weisshorn09 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:14:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:15:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:15:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:15:32 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100ba4a800, cur 1340136932 expire 1340136782 last 1340136705
Jun 19 22:15:32 weisshorn08 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:15:32 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340136876/real 1340136876]  req@ffff880987343400 x1403849823514593/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340136932 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:15:32 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:15:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:15:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:15:57 weisshorn02 kernel: Lustre: scratch-MDT0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880752cd5400, cur 1340136957 expire 1340136807 last 1340136730
Jun 19 22:15:57 weisshorn03 kernel: Lustre: scratch-OST0014: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881012281c00, cur 1340136957 expire 1340136807 last 1340136730
Jun 19 22:15:57 weisshorn03 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 22:15:57 weisshorn13 kernel: Lustre: scratch-OST0018: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880973d03400, cur 1340136957 expire 1340136807 last 1340136730
Jun 19 22:15:57 weisshorn13 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:16:06 weisshorn02 kernel: Lustre: MGS: haven't heard from client 02e8b9ff-fc43-5841-d3bb-815052085829 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804042c1800, cur 1340136966 expire 1340136816 last 1340136739
Jun 19 22:16:10 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340136914/real 1340136914]  req@ffff880eda0a3000 x1403849944121032/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340136970 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:16:10 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:16:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:16:17 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340136921/real 1340136921]  req@ffff88102a28f000 x1403849898278516/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340136977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:16:17 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 22:16:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:16:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:16:42 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340136946/real 1340136946]  req@ffff88100e482000 x1403849637533410/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340137002 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:16:42 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:16:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:16:57 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340136961/real 1340136961]  req@ffff8810222e5c00 x1403849862525234/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340137017 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:16:57 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:17:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:17:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:17:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:17:48 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340137012/real 1340137012]  req@ffff88101733d000 x1403849922681107/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340137068 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:17:48 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:17:49 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340137013/real 1340137013]  req@ffff880a27551800 x1403849954803809/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340137069 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:17:49 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:17:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:17:59 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340137023/real 1340137023]  req@ffff880b5383b800 x1403849874350744/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340137079 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:17:59 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:18:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:18:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:18:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:18:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:19:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:19:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:19:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:19:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:20:01 weisshorn06 CROND[127577]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:20:01 weisshorn10 CROND[126673]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:20:01 weisshorn07 CROND[126838]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:20:01 weisshorn12 CROND[128227]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:20:01 weisshorn09 CROND[126376]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:20:01 weisshorn13 CROND[130240]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:20:01 weisshorn03 CROND[127343]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:20:01 weisshorn05 CROND[126647]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:20:01 weisshorn08 CROND[125417]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:20:01 weisshorn14 CROND[55209]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:20:01 weisshorn04 CROND[126112]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:20:01 weisshorn01 CROND[3809]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:20:01 weisshorn02 CROND[382]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:20:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:20:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:20:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:20:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:21:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:21:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:21:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:21:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:22:07 weisshorn05 kernel: Lustre: scratch-OST0027: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 22:22:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:22:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:22:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:22:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:23:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:23:20 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340137344/real 1340137344]  req@ffff88010db76800 x1403849986384231/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340137400 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:23:20 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:23:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:23:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:23:47 weisshorn05 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 148.187.6.236@o2ib2  ns: filter-scratch-OST0027_UUID lock: ffff880374e1fd80/0xf8449fdfde9f1ac6 lrc: 3/0,0 mode: PW/PW res: 9552916/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) flags: 0x20 remote: 0xfc0c4f7b63b39276 expref: 2522 pid: 6275 timeout 5617014647
Jun 19 22:23:47 weisshorn05 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) Skipped 1 previous similar message
Jun 19 22:23:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:24:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:24:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:24:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:24:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:25:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:25:16 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340137460/real 1340137460]  req@ffff8810111f4000 x1403849872714545/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340137516 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:25:16 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:25:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:25:30 weisshorn09 nslcd[3954]: [a9831a] ldap_result() failed: Can't contact LDAP server
Jun 19 22:25:32 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340137476/real 1340137476]  req@ffff880f71d72800 x1403849823559760/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340137532 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:25:32 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 22:25:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:25:53 weisshorn08 nslcd[3946]: [bc6770] ldap_result() failed: Can't contact LDAP server
Jun 19 22:25:54 weisshorn06 nslcd[3953]: [22ad0c] ldap_result() failed: Can't contact LDAP server
Jun 19 22:25:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:25:56 weisshorn04 nslcd[3885]: [a9831a] ldap_result() failed: Can't contact LDAP server
Jun 19 22:25:57 weisshorn07 nslcd[4297]: [bc6770] ldap_result() failed: Can't contact LDAP server
Jun 19 22:26:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:26:17 weisshorn12 nslcd[3954]: [bc6770] ldap_result() failed: Can't contact LDAP server
Jun 19 22:26:19 weisshorn13 nslcd[4224]: [bc6770] ldap_result() failed: Can't contact LDAP server
Jun 19 22:26:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:26:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:26:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:26:56 weisshorn05 nslcd[4378]: [0e6897] ldap_result() failed: Can't contact LDAP server
Jun 19 22:27:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:27:25 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340137589/real 1340137589]  req@ffff88100dae9000 x1403849944152694/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340137645 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:27:25 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:27:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:27:32 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340137596/real 1340137596]  req@ffff881013bcc000 x1403849898313015/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340137652 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:27:32 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:27:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:27:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:27:57 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340137621/real 1340137621]  req@ffff8809b1e2a800 x1403849637561133/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340137677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:27:57 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:28:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:28:12 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340137636/real 1340137636]  req@ffff88101a8a1400 x1403849862564902/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340137692 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:28:12 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:28:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:28:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:28:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:29:03 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340137687/real 1340137687]  req@ffff881024f77800 x1403849922718342/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340137743 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:29:03 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:29:04 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340137688/real 1340137688]  req@ffff88091064fc00 x1403849954880315/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340137744 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:29:04 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:29:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:29:14 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340137698/real 1340137698]  req@ffff88101392c400 x1403849874387088/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340137754 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:29:14 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:29:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:29:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:29:41 weisshorn01 rhnsd[3837]: running program /usr/sbin/rhn_check
Jun 19 22:29:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:30:01 weisshorn02 CROND[419]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:30:01 weisshorn06 CROND[127609]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:30:01 weisshorn10 CROND[126698]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:30:01 weisshorn07 CROND[126865]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:30:01 weisshorn12 CROND[128288]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:30:01 weisshorn09 CROND[126397]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:30:01 weisshorn05 CROND[126675]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:30:01 weisshorn08 CROND[125445]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:30:01 weisshorn03 CROND[127368]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:30:01 weisshorn13 CROND[130306]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:30:01 weisshorn14 CROND[55263]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:30:01 weisshorn04 CROND[126138]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:30:01 weisshorn01 CROND[3840]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:30:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:30:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:30:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:30:47 weisshorn05 kernel: Lustre: scratch-OST003e: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 22:30:47 weisshorn05 kernel: Lustre: scratch-OST003e: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) refused reconnection, still busy with 3 active RPCs
Jun 19 22:30:47 weisshorn05 kernel: LustreError: 8339:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc -107  req@ffff881015937050 x1403837088145753/t0(0) o4->cdd482d3-fa4f-4442-72da-16bfade64b5f@148.187.6.236@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340137862 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 22:30:47 weisshorn05 kernel: Lustre: scratch-OST003e: Bulk IO write error with cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2), client will retry: rc -107
Jun 19 22:30:47 weisshorn05 kernel: Lustre: Skipped 10 previous similar messages
Jun 19 22:30:47 weisshorn05 kernel: LustreError: 8339:0:(ldlm_lib.c:2725:target_bulk_io()) Skipped 1 previous similar message
Jun 19 22:30:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:31:09 weisshorn05 kernel: Lustre: scratch-OST001b: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 22:31:09 weisshorn05 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:31:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:31:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:31:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:31:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:32:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:32:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:32:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:32:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:33:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:33:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:33:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:33:51 weisshorn13 rhnsd[130314]: running program /usr/sbin/rhn_check
Jun 19 22:33:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:34:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:34:15 weisshorn02 kernel: LustreError: 47791:0:(mdt_handler.c:2822:mdt_recovery()) operation 400 on unconnected MDS from 12345-148.187.6.7@o2ib2
Jun 19 22:34:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:34:35 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340138019/real 1340138019]  req@ffff880c74acbc00 x1403849986431571/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340138075 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:34:35 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:34:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:34:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:35:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:35:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:35:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:35:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:36:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:36:22 weisshorn06 kernel: Lustre: scratch-OST0041: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102a0cfc00, cur 1340138182 expire 1340138032 last 1340137955
Jun 19 22:36:22 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:36:22 weisshorn03 kernel: Lustre: scratch-OST0020: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880809d13c00, cur 1340138182 expire 1340138032 last 1340137955
Jun 19 22:36:22 weisshorn07 kernel: Lustre: scratch-OST0042: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88099a156400, cur 1340138182 expire 1340138032 last 1340137955
Jun 19 22:36:22 weisshorn07 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:36:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:36:31 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340138135/real 1340138135]  req@ffff8803c056f400 x1403849872749713/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340138191 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:36:31 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:36:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:36:47 weisshorn05 kernel: Lustre: scratch-OST0027: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100e490000, cur 1340138207 expire 1340138057 last 1340137980
Jun 19 22:36:47 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340138151/real 1340138151]  req@ffff8809e7aea800 x1403849823578870/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340138207 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:36:47 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:36:47 weisshorn06 kernel: Lustre: scratch-OST0012: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881011b9cc00, cur 1340138207 expire 1340138057 last 1340137980
Jun 19 22:36:47 weisshorn13 kernel: Lustre: scratch-OST0011: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe4545000, cur 1340138207 expire 1340138057 last 1340137980
Jun 19 22:36:47 weisshorn09 kernel: Lustre: scratch-OST0026: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880823aa1000, cur 1340138207 expire 1340138057 last 1340137980
Jun 19 22:36:47 weisshorn09 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:36:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:37:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:37:12 weisshorn13 kernel: Lustre: scratch-OST0005: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880207f1cc00, cur 1340138232 expire 1340138082 last 1340138005
Jun 19 22:37:12 weisshorn09 kernel: Lustre: scratch-OST0004: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b4aa1fc00, cur 1340138232 expire 1340138082 last 1340138005
Jun 19 22:37:25 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340138189/real 1340138189]  req@ffff880ec3ba1800 x1403849944165217/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340138245 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:37:25 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 22:37:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:37:37 weisshorn12 kernel: Lustre: scratch-OST0021: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808804ea800, cur 1340138257 expire 1340138107 last 1340138030
Jun 19 22:37:37 weisshorn12 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:37:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:37:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:37:57 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340138221/real 1340138221]  req@ffff880f79783000 x1403849637574622/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340138277 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:37:57 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 19 previous similar messages
Jun 19 22:38:10 weisshorn02 kernel: Lustre: MGS: haven't heard from client 02e8b9ff-fc43-5841-d3bb-815052085829 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bb2339400, cur 1340138290 expire 1340138140 last 1340138063
Jun 19 22:38:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:38:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:38:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:38:47 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340138271/real 1340138271]  req@ffff8810110bbc00 x1403849898339658/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340138327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:38:47 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:38:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:39:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:39:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:39:27 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340138311/real 1340138311]  req@ffff881019676c00 x1403849862597285/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340138367 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:39:27 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:39:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:39:42 weisshorn04 kernel: Lustre: scratch-OST0019: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805fd094c00, cur 1340138382 expire 1340138232 last 1340138155
Jun 19 22:39:42 weisshorn04 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:39:42 weisshorn03 kernel: Lustre: scratch-OST002c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101350ac00, cur 1340138382 expire 1340138232 last 1340138155
Jun 19 22:39:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:40:01 weisshorn01 CROND[3871]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:40:01 weisshorn04 CROND[126206]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:40:01 weisshorn02 CROND[489]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:40:01 weisshorn06 CROND[127670]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:40:01 weisshorn10 CROND[126766]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:40:01 weisshorn07 CROND[126884]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:40:01 weisshorn12 CROND[128318]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:40:01 weisshorn09 CROND[126461]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:40:01 weisshorn08 CROND[125505]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:40:01 weisshorn05 CROND[126744]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:40:01 weisshorn03 CROND[127434]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:40:01 weisshorn13 CROND[130345]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:40:01 weisshorn14 CROND[55299]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:40:07 weisshorn13 kernel: Lustre: scratch-OST0024: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100e0df000, cur 1340138407 expire 1340138257 last 1340138180
Jun 19 22:40:07 weisshorn13 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:40:07 weisshorn04 kernel: Lustre: scratch-OST000d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c39859000, cur 1340138407 expire 1340138257 last 1340138180
Jun 19 22:40:07 weisshorn04 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:40:07 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802b4020400, cur 1340138407 expire 1340138257 last 1340138180
Jun 19 22:40:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:40:18 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340138362/real 1340138362]  req@ffff8810161f8400 x1403849922740824/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340138418 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:40:18 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:40:19 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340138363/real 1340138363]  req@ffff880fc7539000 x1403849954916790/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340138419 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:40:19 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:40:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:40:29 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340138373/real 1340138373]  req@ffff881016fc9400 x1403849874404663/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340138429 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:40:29 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:40:32 weisshorn02 kernel: Lustre: scratch-MDT0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803ff425000, cur 1340138432 expire 1340138282 last 1340138205
Jun 19 22:40:32 weisshorn06 kernel: Lustre: scratch-OST002a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d1278f800, cur 1340138432 expire 1340138282 last 1340138205
Jun 19 22:40:32 weisshorn07 kernel: Lustre: scratch-OST002b: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88081bcf4800, cur 1340138432 expire 1340138282 last 1340138205
Jun 19 22:40:32 weisshorn09 kernel: Lustre: scratch-OST0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101626dc00, cur 1340138432 expire 1340138282 last 1340138205
Jun 19 22:40:32 weisshorn09 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:40:32 weisshorn03 kernel: Lustre: scratch-OST0014: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100dd41000, cur 1340138432 expire 1340138282 last 1340138205
Jun 19 22:40:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:40:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:40:57 weisshorn10 kernel: Lustre: scratch-OST000a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880805b3f400, cur 1340138457 expire 1340138307 last 1340138230
Jun 19 22:41:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:41:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:41:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:41:46 weisshorn14 nslcd[3437]: [aa2304] ldap_result() failed: Can't contact LDAP server
Jun 19 22:41:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:42:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:42:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:42:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:42:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:43:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:43:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:43:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:43:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:44:11 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:44:26 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:44:41 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:44:56 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:45:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:45:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:45:36 weisshorn03 nslcd[3944]: [d71a2b] ldap_result() failed: Can't contact LDAP server
Jun 19 22:45:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:45:50 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340138694/real 1340138694]  req@ffff880b35d89800 x1403849986473380/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340138750 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:45:50 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:45:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:46:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:46:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:46:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:46:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:47:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:47:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:47:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:47:46 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340138810/real 1340138810]  req@ffff880027080400 x1403849872807779/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340138866 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:47:46 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:47:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:48:02 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340138826/real 1340138826]  req@ffff880938b1cc00 x1403849823621051/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340138882 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:48:02 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:48:11 weisshorn05 kernel: Lustre: 7953:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340138884/real 1340138884]  req@ffff881014bdd800 x1403849637594401/t0(0) o106->scratch-OST000e@148.187.6.236@o2ib2:15/16 lens 296/232 e 0 to 1 dl 1340138891 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jun 19 22:48:11 weisshorn05 kernel: Lustre: 7864:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340138884/real 1340138884]  req@ffff88100e565400 x1403849637594402/t0(0) o106->scratch-OST001b@148.187.6.236@o2ib2:15/16 lens 296/232 e 0 to 1 dl 1340138891 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jun 19 22:48:11 weisshorn05 kernel: Lustre: 7953:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:48:11 weisshorn05 kernel: Lustre: 7864:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:48:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:48:19 weisshorn05 kernel: Lustre: scratch-OST001b: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 22:48:19 weisshorn05 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:48:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:48:38 weisshorn05 kernel: Lustre: scratch-OST003e: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bab52d800, cur 1340138918 expire 1340138768 last 1340138691
Jun 19 22:48:38 weisshorn13 kernel: Lustre: scratch-OST0024: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88046e07f400, cur 1340138918 expire 1340138768 last 1340138691
Jun 19 22:48:38 weisshorn13 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:48:38 weisshorn03 kernel: Lustre: scratch-OST0009: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88043d9b7800, cur 1340138918 expire 1340138768 last 1340138691
Jun 19 22:48:38 weisshorn03 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:48:40 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340138864/real 1340138864]  req@ffff880bcb7c5c00 x1403849944184739/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340138920 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:48:40 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:48:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:48:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:49:03 weisshorn07 kernel: Lustre: scratch-OST001f: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810158db400, cur 1340138943 expire 1340138793 last 1340138716
Jun 19 22:49:03 weisshorn07 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:49:03 weisshorn06 kernel: Lustre: scratch-OST0036: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880dc6950400, cur 1340138943 expire 1340138793 last 1340138716
Jun 19 22:49:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:49:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:49:28 weisshorn02 kernel: Lustre: MGS: haven't heard from client aca8aee1-5c22-ca9e-453b-0ad0e5295f34 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803f29fb000, cur 1340138968 expire 1340138818 last 1340138741
Jun 19 22:49:28 weisshorn09 kernel: Lustre: scratch-OST001a: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881018982400, cur 1340138968 expire 1340138818 last 1340138741
Jun 19 22:49:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:49:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:50:01 weisshorn13 CROND[130376]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:50:01 weisshorn01 CROND[3936]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:50:01 weisshorn04 CROND[126235]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:50:01 weisshorn02 CROND[581]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:50:01 weisshorn06 CROND[127709]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:50:01 weisshorn10 CROND[126797]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:50:01 weisshorn07 CROND[126968]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:50:01 weisshorn12 CROND[128353]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:50:01 weisshorn08 CROND[125549]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:50:01 weisshorn09 CROND[126500]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:50:01 weisshorn05 CROND[126775]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:50:01 weisshorn03 CROND[127468]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:50:01 weisshorn14 CROND[55334]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 22:50:02 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340138946/real 1340138946]  req@ffff881016987000 x1403849898369713/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340139002 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:50:02 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:50:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:50:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:50:42 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340138986/real 1340138986]  req@ffff88101758d800 x1403849862616623/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340139042 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:50:42 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:50:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:50:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:51:01 weisshorn14 nslcd[3437]: [7c2ff4] ldap_result() failed: Can't contact LDAP server
Jun 19 22:51:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:51:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:51:33 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340139037/real 1340139037]  req@ffff8809efa96800 x1403849922771451/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340139093 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:51:33 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:51:34 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340139038/real 1340139038]  req@ffff880e04955000 x1403849954956530/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340139094 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:51:34 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:51:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:51:44 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340139048/real 1340139048]  req@ffff880f907bd000 x1403849874422078/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340139104 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:51:44 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:51:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:51:59 weisshorn05 kernel: Lustre: scratch-OST0033: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 22:51:59 weisshorn05 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:51:59 weisshorn05 kernel: Lustre: scratch-OST0033: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) refused reconnection, still busy with 2 active RPCs
Jun 19 22:51:59 weisshorn05 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:51:59 weisshorn05 kernel: LustreError: 8437:0:(ldlm_lib.c:2706:target_bulk_io()) @@@ Reconnect on bulk GET  req@ffff88100d042400 x1403837088348356/t0(0) o4->cdd482d3-fa4f-4442-72da-16bfade64b5f@148.187.6.236@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340139167 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 22:51:59 weisshorn05 kernel: LustreError: 8437:0:(ldlm_lib.c:2706:target_bulk_io()) Skipped 8 previous similar messages
Jun 19 22:51:59 weisshorn05 kernel: Lustre: scratch-OST0033: Bulk IO write error with cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2), client will retry: rc -110
Jun 19 22:51:59 weisshorn05 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:52:10 weisshorn02 kernel: LustreError: 47672:0:(mdt_handler.c:2822:mdt_recovery()) operation 400 on unconnected MDS from 12345-148.187.6.7@o2ib2
Jun 19 22:52:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:52:23 weisshorn05 kernel: Lustre: scratch-OST0033: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 22:52:27 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:52:42 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:52:57 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:53:12 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:53:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:53:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:53:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:54:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:54:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:54:42 weisshorn09 kernel: Lustre: scratch-OST0026: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810158fd000, cur 1340139282 expire 1340139132 last 1340139055
Jun 19 22:54:42 weisshorn05 kernel: Lustre: scratch-OST0027: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88011a8a5c00, cur 1340139282 expire 1340139132 last 1340139055
Jun 19 22:54:42 weisshorn13 kernel: Lustre: scratch-OST0011: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881013002000, cur 1340139282 expire 1340139132 last 1340139055
Jun 19 22:54:42 weisshorn13 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:54:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:54:46 weisshorn05 kernel: Lustre: scratch-OST003e: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 22:54:46 weisshorn05 kernel: Lustre: scratch-OST003e: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) refused reconnection, still busy with 2 active RPCs
Jun 19 22:54:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:55:07 weisshorn06 kernel: Lustre: scratch-OST0007: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ebe927800, cur 1340139307 expire 1340139157 last 1340139080
Jun 19 22:55:07 weisshorn06 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:55:11 weisshorn05 kernel: Lustre: scratch-OST003e: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 22:55:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:55:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:55:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:55:58 weisshorn03 kernel: Lustre: scratch-OST0014: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88006ce2c800, cur 1340139358 expire 1340139208 last 1340139131
Jun 19 22:55:58 weisshorn03 kernel: Lustre: Skipped 1 previous similar message
Jun 19 22:55:58 weisshorn07 kernel: Lustre: scratch-OST002b: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880676866000, cur 1340139358 expire 1340139208 last 1340139131
Jun 19 22:55:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:56:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:56:22 weisshorn06 kernel: Lustre: scratch-OST0036: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881012a1b400, cur 1340139382 expire 1340139232 last 1340139155
Jun 19 22:56:22 weisshorn06 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 22:56:22 weisshorn04 kernel: Lustre: scratch-OST003d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881015419c00, cur 1340139382 expire 1340139232 last 1340139155
Jun 19 22:56:22 weisshorn10 kernel: Lustre: scratch-OST000a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a48629800, cur 1340139382 expire 1340139232 last 1340139155
Jun 19 22:56:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:56:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:56:47 weisshorn04 kernel: Lustre: scratch-OST0025: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101a89f800, cur 1340139407 expire 1340139257 last 1340139180
Jun 19 22:56:47 weisshorn05 kernel: Lustre: scratch-OST000e: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bd21a6000, cur 1340139407 expire 1340139257 last 1340139180
Jun 19 22:56:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:57:05 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340139369/real 1340139369]  req@ffff88014e3ca800 x1403849986500571/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340139425 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:57:05 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:57:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:57:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:57:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:57:46 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340139410/real 1340139410]  req@ffff88070ea1b000 x1403849872833967/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340139466 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:57:46 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 22:57:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:58:02 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340139426/real 1340139426]  req@ffff881012346800 x1403849823632120/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340139482 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:58:02 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 22:58:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:58:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:58:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:58:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:59:12 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340139496/real 1340139496]  req@ffff88100f2bb400 x1403849637607757/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340139552 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:59:12 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 30 previous similar messages
Jun 19 22:59:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:59:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:59:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 22:59:55 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340139539/real 1340139539]  req@ffff880d7306f800 x1403849944211964/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340139595 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 22:59:55 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 22:59:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:00:01 weisshorn14 CROND[55388]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:00:01 weisshorn13 CROND[130437]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:00:01 weisshorn01 CROND[3960]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:00:01 weisshorn04 CROND[126260]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:00:01 weisshorn02 CROND[683]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:00:01 weisshorn06 CROND[127734]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:00:01 weisshorn10 CROND[126823]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:00:01 weisshorn07 CROND[126988]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:00:01 weisshorn08 CROND[125569]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:00:01 weisshorn12 CROND[128413]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:00:01 weisshorn05 CROND[126803]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:00:01 weisshorn03 CROND[127494]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:00:01 weisshorn09 CROND[126520]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:00:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:00:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:00:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:00:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:01:01 weisshorn14 CROND[55393]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 23:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5539 starting 0anacron
Jun 19 23:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5540 finished 0anacron
Jun 19 23:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5539 starting mcelog.cron
Jun 19 23:01:01 weisshorn14 run-parts(/etc/cron.hourly)[5541 finished mcelog.cron
Jun 19 23:01:01 weisshorn13 CROND[130442]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 23:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1304 starting 0anacron
Jun 19 23:01:01 weisshorn01 CROND[3965]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 23:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3965 starting 0anacron
Jun 19 23:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1304 finished 0anacron
Jun 19 23:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1304 starting mcelog.cron
Jun 19 23:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3975 finished 0anacron
Jun 19 23:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3965 starting mcelog.cron
Jun 19 23:01:01 weisshorn01 run-parts(/etc/cron.hourly)[3982 finished mcelog.cron
Jun 19 23:01:01 weisshorn04 CROND[126273]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 23:01:01 weisshorn13 run-parts(/etc/cron.hourly)[1304 finished mcelog.cron
Jun 19 23:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1262 starting 0anacron
Jun 19 23:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1262 finished 0anacron
Jun 19 23:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1262 starting mcelog.cron
Jun 19 23:01:01 weisshorn02 CROND[694]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 23:01:01 weisshorn04 run-parts(/etc/cron.hourly)[1262 finished mcelog.cron
Jun 19 23:01:01 weisshorn02 run-parts(/etc/cron.hourly)[694] starting 0anacron
Jun 19 23:01:01 weisshorn02 run-parts(/etc/cron.hourly)[703] finished 0anacron
Jun 19 23:01:01 weisshorn02 run-parts(/etc/cron.hourly)[694] starting mcelog.cron
Jun 19 23:01:01 weisshorn06 CROND[127739]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 23:01:01 weisshorn02 run-parts(/etc/cron.hourly)[710] finished mcelog.cron
Jun 19 23:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1277 starting 0anacron
Jun 19 23:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1277 finished 0anacron
Jun 19 23:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1277 starting mcelog.cron
Jun 19 23:01:01 weisshorn06 run-parts(/etc/cron.hourly)[1277 finished mcelog.cron
Jun 19 23:01:01 weisshorn10 CROND[126835]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 23:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1268 starting 0anacron
Jun 19 23:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1268 finished 0anacron
Jun 19 23:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1268 starting mcelog.cron
Jun 19 23:01:01 weisshorn07 CROND[126993]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 23:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1269 starting 0anacron
Jun 19 23:01:01 weisshorn10 run-parts(/etc/cron.hourly)[1268 finished mcelog.cron
Jun 19 23:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1270 finished 0anacron
Jun 19 23:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1269 starting mcelog.cron
Jun 19 23:01:01 weisshorn07 run-parts(/etc/cron.hourly)[1270 finished mcelog.cron
Jun 19 23:01:01 weisshorn08 CROND[125609]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 23:01:01 weisshorn12 CROND[128418]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 23:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1256 starting 0anacron
Jun 19 23:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1256 finished 0anacron
Jun 19 23:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1256 starting mcelog.cron
Jun 19 23:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1284 starting 0anacron
Jun 19 23:01:01 weisshorn05 CROND[126815]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 23:01:01 weisshorn08 run-parts(/etc/cron.hourly)[1256 finished mcelog.cron
Jun 19 23:01:01 weisshorn03 CROND[127534]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 23:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1284 finished 0anacron
Jun 19 23:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1284 starting mcelog.cron
Jun 19 23:01:01 weisshorn03 run-parts(/etc/cron.hourly)[1275 starting 0anacron
Jun 19 23:01:01 weisshorn12 run-parts(/etc/cron.hourly)[1284 finished mcelog.cron
Jun 19 23:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1268 starting 0anacron
Jun 19 23:01:01 weisshorn09 CROND[126560]: (root) CMD (run-parts /etc/cron.hourly)
Jun 19 23:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1268 finished 0anacron
Jun 19 23:01:01 weisshorn05 run-parts(/etc/cron.hourly)[1268 starting mcelog.cron
Jun 19 23:01:02 weisshorn04 nslcd[3885]: [d71a2b] ldap_result() failed: Can't contact LDAP server
Jun 19 23:01:02 weisshorn03 run-parts(/etc/cron.hourly)[1275 finished 0anacron
Jun 19 23:01:02 weisshorn03 run-parts(/etc/cron.hourly)[1275 starting mcelog.cron
Jun 19 23:01:02 weisshorn05 run-parts(/etc/cron.hourly)[1268 finished mcelog.cron
Jun 19 23:01:02 weisshorn09 run-parts(/etc/cron.hourly)[1265 starting 0anacron
Jun 19 23:01:02 weisshorn03 run-parts(/etc/cron.hourly)[1275 finished mcelog.cron
Jun 19 23:01:02 weisshorn09 run-parts(/etc/cron.hourly)[1265 finished 0anacron
Jun 19 23:01:02 weisshorn09 run-parts(/etc/cron.hourly)[1265 starting mcelog.cron
Jun 19 23:01:02 weisshorn09 run-parts(/etc/cron.hourly)[1265 finished mcelog.cron
Jun 19 23:01:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:01:16 weisshorn10 nslcd[4294]: [0e6897] ldap_result() failed: Can't contact LDAP server
Jun 19 23:01:16 weisshorn06 nslcd[3953]: [997556] ldap_result() failed: Can't contact LDAP server
Jun 19 23:01:16 weisshorn05 nslcd[4378]: [5138de] ldap_result() failed: Can't contact LDAP server
Jun 19 23:01:16 weisshorn13 nslcd[4224]: [0e6897] ldap_result() failed: Can't contact LDAP server
Jun 19 23:01:16 weisshorn09 nslcd[3954]: [d71a2b] ldap_result() failed: Can't contact LDAP server
Jun 19 23:01:16 weisshorn03 nslcd[3944]: [fc7321] ldap_result() failed: Can't contact LDAP server
Jun 19 23:01:16 weisshorn12 nslcd[3954]: [0e6897] ldap_result() failed: Can't contact LDAP server
Jun 19 23:01:16 weisshorn07 nslcd[4297]: [0e6897] ldap_result() failed: Can't contact LDAP server
Jun 19 23:01:16 weisshorn08 nslcd[3946]: [0e6897] ldap_result() failed: Can't contact LDAP server
Jun 19 23:01:16 weisshorn01 nslcd[2168]: [95f874] ldap_result() failed: Can't contact LDAP server
Jun 19 23:01:17 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340139621/real 1340139621]  req@ffff881017577800 x1403849898404792/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340139677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:01:17 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:01:22 weisshorn03 kernel: Lustre: scratch-OST002c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88100fa1b400, cur 1340139682 expire 1340139532 last 1340139455
Jun 19 23:01:22 weisshorn03 kernel: Lustre: Skipped 1 previous similar message
Jun 19 23:01:22 weisshorn04 kernel: Lustre: scratch-OST0031: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101b635000, cur 1340139682 expire 1340139532 last 1340139455
Jun 19 23:01:25 weisshorn10 rhnsd[126854]: running program /usr/sbin/rhn_check
Jun 19 23:01:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:01:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:01:47 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804abf77c00, cur 1340139707 expire 1340139557 last 1340139480
Jun 19 23:01:47 weisshorn08 kernel: Lustre: Skipped 1 previous similar message
Jun 19 23:01:47 weisshorn04 kernel: Lustre: scratch-OST000d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880133709c00, cur 1340139707 expire 1340139557 last 1340139480
Jun 19 23:01:47 weisshorn04 kernel: Lustre: Skipped 1 previous similar message
Jun 19 23:01:57 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340139661/real 1340139661]  req@ffff8809a6ffe000 x1403849862650904/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340139717 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:01:57 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:01:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:02:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:02:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:02:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:02:48 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340139712/real 1340139712]  req@ffff8810158bfc00 x1403849922799679/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340139768 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:02:48 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:02:49 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340139713/real 1340139713]  req@ffff880a48e7f400 x1403849955013867/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340139769 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:02:49 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:02:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:02:59 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340139723/real 1340139723]  req@ffff880e7b247c00 x1403849874444858/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340139779 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:02:59 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:03:09 weisshorn08 kernel: LustreError: 5765:0:(events.c:389:server_bulk_callback()) event type 4, status -103, desc ffff88057f314000
Jun 19 23:03:09 weisshorn08 kernel: LustreError: 5765:0:(events.c:389:server_bulk_callback()) event type 2, status -103, desc ffff88057f314000
Jun 19 23:03:09 weisshorn08 kernel: LustreError: 7971:0:(ldlm_lib.c:2717:target_bulk_io()) @@@ network error on bulk GET 0(1048576)  req@ffff880a3a8e4400 x1403837088426080/t0(0) o4->cdd482d3-fa4f-4442-72da-16bfade64b5f@148.187.6.236@o2ib2:0/0 lens 456/416 e 1 to 0 dl 1340139815 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 23:03:09 weisshorn08 kernel: Lustre: scratch-OST000c: Bulk IO write error with cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2), client will retry: rc -110
Jun 19 23:03:09 weisshorn08 kernel: Lustre: Skipped 1 previous similar message
Jun 19 23:03:09 weisshorn08 kernel: Lustre: scratch-OST0017: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 23:03:09 weisshorn08 kernel: Lustre: scratch-OST0017: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) refused reconnection, still busy with 1 active RPCs
Jun 19 23:03:09 weisshorn08 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 23:03:09 weisshorn08 kernel: LustreError: 7773:0:(ldlm_lib.c:2706:target_bulk_io()) @@@ Reconnect on bulk GET  req@ffff88080ca32850 x1403837088426243/t0(0) o4->cdd482d3-fa4f-4442-72da-16bfade64b5f@148.187.6.236@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340139802 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 23:03:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:03:27 weisshorn06 kernel: Lustre: scratch-OST002a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b35a80000, cur 1340139807 expire 1340139657 last 1340139580
Jun 19 23:03:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:03:34 weisshorn08 kernel: Lustre: scratch-OST0002: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 23:03:34 weisshorn08 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 23:03:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:03:52 weisshorn10 kernel: Lustre: scratch-OST000a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fc1391000, cur 1340139832 expire 1340139682 last 1340139605
Jun 19 23:03:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:04:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:04:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:04:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:04:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:05:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:05:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:05:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:05:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:06:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:06:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:06:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:06:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:07:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:07:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:07:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:07:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:08:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:08:20 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340140044/real 1340140044]  req@ffff880804efe000 x1403849986554950/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340140100 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:08:20 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:08:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:08:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:08:49 weisshorn04 kernel: Lustre: scratch-OST000d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881016702c00, cur 1340140129 expire 1340139979 last 1340139902
Jun 19 23:08:49 weisshorn13 kernel: Lustre: scratch-OST0005: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807d5219800, cur 1340140129 expire 1340139979 last 1340139902
Jun 19 23:08:49 weisshorn13 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 23:08:49 weisshorn05 kernel: Lustre: scratch-OST000e: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe0bf3400, cur 1340140129 expire 1340139979 last 1340139902
Jun 19 23:08:49 weisshorn09 kernel: Lustre: scratch-OST0034: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88076e9cb800, cur 1340140129 expire 1340139979 last 1340139902
Jun 19 23:08:49 weisshorn09 kernel: Lustre: Skipped 1 previous similar message
Jun 19 23:08:49 weisshorn06 kernel: Lustre: scratch-OST0036: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c0473d000, cur 1340140129 expire 1340139979 last 1340139902
Jun 19 23:08:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:09:01 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340140085/real 1340140085]  req@ffff88100f7cf800 x1403849872876798/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340140141 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:09:01 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:09:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:09:17 weisshorn09 kernel: Lustre: scratch-OST0004: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88042ee01400, cur 1340140157 expire 1340140007 last 1340139930
Jun 19 23:09:17 weisshorn09 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 23:09:17 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340140101/real 1340140101]  req@ffff88095a383400 x1403849823660562/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340140157 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:09:17 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 26 previous similar messages
Jun 19 23:09:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:09:42 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c32c8b800, cur 1340140182 expire 1340140032 last 1340139955
Jun 19 23:09:42 weisshorn08 kernel: Lustre: Skipped 1 previous similar message
Jun 19 23:09:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:09:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:10:01 weisshorn05 CROND[126890]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:10:01 weisshorn03 CROND[127587]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:10:01 weisshorn09 CROND[126614]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:10:01 weisshorn14 CROND[55453]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:10:01 weisshorn01 CROND[4048]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:10:01 weisshorn01 CROND[4050]: (root) CMD (/root/bin/lustre_tuning.sh)
Jun 19 23:10:01 weisshorn14 sshd[55455]: Accepted publickey for root from 148.187.12.231 port 47222 ssh2
Jun 19 23:10:01 weisshorn14 sshd[55455]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 23:10:01 weisshorn14 sshd[55455]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 23:10:01 weisshorn14 sshd[55455]: pam_unix(sshd:session): session closed for user root
Jun 19 23:10:01 weisshorn09 sshd[126615]: Accepted publickey for root from 148.187.12.231 port 56797 ssh2
Jun 19 23:10:01 weisshorn06 sshd[127814]: Accepted publickey for root from 148.187.12.231 port 54751 ssh2
Jun 19 23:10:01 weisshorn02 CROND[782]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:10:01 weisshorn07 sshd[127032]: Accepted publickey for root from 148.187.12.231 port 57413 ssh2
Jun 19 23:10:01 weisshorn03 sshd[127588]: Accepted publickey for root from 148.187.12.231 port 38263 ssh2
Jun 19 23:10:01 weisshorn05 sshd[126892]: Accepted publickey for root from 148.187.12.231 port 46638 ssh2
Jun 19 23:10:01 weisshorn09 sshd[126615]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 23:10:01 weisshorn05 sshd[126892]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 23:10:01 weisshorn07 sshd[127032]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 23:10:01 weisshorn03 sshd[127588]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 23:10:01 weisshorn04 CROND[126355]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:10:01 weisshorn06 sshd[127814]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 23:10:01 weisshorn06 CROND[127819]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:10:01 weisshorn08 sshd[125660]: Accepted publickey for root from 148.187.12.231 port 39140 ssh2
Jun 19 23:10:01 weisshorn13 CROND[130495]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:10:01 weisshorn08 sshd[125660]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 23:10:01 weisshorn12 sshd[128474]: Accepted publickey for root from 148.187.12.231 port 42371 ssh2
Jun 19 23:10:01 weisshorn12 sshd[128474]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 23:10:01 weisshorn10 sshd[126910]: Accepted publickey for root from 148.187.12.231 port 42246 ssh2
Jun 19 23:10:01 weisshorn04 sshd[126352]: Accepted publickey for root from 148.187.12.231 port 60206 ssh2
Jun 19 23:10:01 weisshorn10 sshd[126910]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 23:10:01 weisshorn04 sshd[126352]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 23:10:01 weisshorn10 CROND[126924]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:10:01 weisshorn09 sshd[126615]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 23:10:01 weisshorn09 sshd[126615]: pam_unix(sshd:session): session closed for user root
Jun 19 23:10:01 weisshorn03 sshd[127588]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 23:10:01 weisshorn03 sshd[127588]: pam_unix(sshd:session): session closed for user root
Jun 19 23:10:01 weisshorn06 sshd[127814]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 23:10:01 weisshorn06 sshd[127814]: pam_unix(sshd:session): session closed for user root
Jun 19 23:10:01 weisshorn05 sshd[126892]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 23:10:01 weisshorn05 sshd[126892]: pam_unix(sshd:session): session closed for user root
Jun 19 23:10:01 weisshorn07 CROND[127055]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:10:01 weisshorn07 sshd[127032]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 23:10:01 weisshorn07 sshd[127032]: pam_unix(sshd:session): session closed for user root
Jun 19 23:10:01 weisshorn08 sshd[125660]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 23:10:01 weisshorn08 sshd[125660]: pam_unix(sshd:session): session closed for user root
Jun 19 23:10:01 weisshorn13 sshd[130494]: Accepted publickey for root from 148.187.12.231 port 40175 ssh2
Jun 19 23:10:01 weisshorn12 sshd[128474]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 23:10:01 weisshorn12 sshd[128474]: pam_unix(sshd:session): session closed for user root
Jun 19 23:10:01 weisshorn13 sshd[130494]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 19 23:10:01 weisshorn08 CROND[125682]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:10:01 weisshorn10 sshd[126910]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 23:10:01 weisshorn10 sshd[126910]: pam_unix(sshd:session): session closed for user root
Jun 19 23:10:01 weisshorn12 CROND[128497]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:10:02 weisshorn04 sshd[126352]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 23:10:02 weisshorn04 sshd[126352]: pam_unix(sshd:session): session closed for user root
Jun 19 23:10:02 weisshorn13 sshd[130494]: Received disconnect from 148.187.12.231: 11: disconnected by user
Jun 19 23:10:02 weisshorn13 sshd[130494]: pam_unix(sshd:session): session closed for user root
Jun 19 23:10:04 weisshorn01 postfix/pickup[3990]: 6B5C492224F: uid=0 from=<root>
Jun 19 23:10:04 weisshorn01 postfix/cleanup[4080]: 6B5C492224F: message-id=<20120619211004.6B5C492224F@weisshorn01.admin.cscs.ch>
Jun 19 23:10:04 weisshorn01 postfix/qmgr[2717]: 6B5C492224F: from=<root@weisshorn01.admin.cscs.ch>, size=1128, nrcpt=1 (queue active)
Jun 19 23:10:04 weisshorn01 postfix/local[4082]: 6B5C492224F: to=<root@weisshorn01.admin.cscs.ch>, orig_to=<root>, relay=local, delay=3.1, delays=3.1/0.01/0/0.04, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 19 23:10:04 weisshorn01 postfix/qmgr[2717]: 6B5C492224F: removed
Jun 19 23:10:07 weisshorn02 kernel: Lustre: scratch-MDT0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bb9c11400, cur 1340140207 expire 1340140057 last 1340139980
Jun 19 23:10:07 weisshorn07 kernel: Lustre: scratch-OST002b: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101462b400, cur 1340140207 expire 1340140057 last 1340139980
Jun 19 23:10:07 weisshorn07 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 23:10:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:10:14 weisshorn02 kernel: Lustre: MGS: haven't heard from client 02e8b9ff-fc43-5841-d3bb-815052085829 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88072df28800, cur 1340140214 expire 1340140064 last 1340139987
Jun 19 23:10:16 weisshorn01 nslcd[2168]: [138641] ldap_result() failed: Can't contact LDAP server
Jun 19 23:10:27 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340140171/real 1340140171]  req@ffff88100fd6b000 x1403849637628278/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340140227 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:10:27 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jun 19 23:10:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:10:32 weisshorn12 kernel: Lustre: scratch-OST0021: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fa7279400, cur 1340140232 expire 1340140082 last 1340140005
Jun 19 23:10:32 weisshorn08 kernel: Lustre: scratch-OST000c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fda392400, cur 1340140232 expire 1340140082 last 1340140005
Jun 19 23:10:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:10:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:11:03 weisshorn06 rhnsd[127840]: running program /usr/sbin/rhn_check
Jun 19 23:11:10 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340140214/real 1340140214]  req@ffff880c76033800 x1403849944230797/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340140270 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:11:10 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:11:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:11:17 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340140221/real 1340140221]  req@ffff8810124b2c00 x1403849898437006/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340140277 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:11:17 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 23:11:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:11:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:11:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:12:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:12:28 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:12:43 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:12:58 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:13:12 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340140336/real 1340140336]  req@ffff880d4ba12400 x1403849862666288/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340140392 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:13:12 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:13:13 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:13:25 weisshorn02 kernel: LustreError: 47784:0:(mdt_handler.c:2822:mdt_recovery()) operation 400 on unconnected MDS from 12345-148.187.6.7@o2ib2
Jun 19 23:13:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:13:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:13:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:14:03 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340140387/real 1340140387]  req@ffff88042d762400 x1403849922827197/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340140443 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:14:03 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:14:04 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340140388/real 1340140388]  req@ffff880d035c5c00 x1403849955054156/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340140444 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:14:04 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:14:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:14:14 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340140398/real 1340140398]  req@ffff880e2291e800 x1403849874462512/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340140454 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:14:14 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:14:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:14:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:14:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:15:06 weisshorn02 kernel: Lustre: 47772:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004f65:0x414:0x0]] [valid 68719476736]
Jun 19 23:15:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:15:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:15:32 weisshorn03 kernel: Lustre: scratch-OST0020: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804f7a65000, cur 1340140532 expire 1340140382 last 1340140305
Jun 19 23:15:32 weisshorn03 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 23:15:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:15:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:16:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:16:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:16:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:16:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:17:12 weisshorn12 kernel: Lustre: scratch-OST0028: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e486a5000, cur 1340140632 expire 1340140482 last 1340140405
Jun 19 23:17:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:17:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:17:44 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:17:59 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:18:02 weisshorn05 kernel: Lustre: scratch-OST000e: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804fc77a400, cur 1340140682 expire 1340140532 last 1340140455
Jun 19 23:18:02 weisshorn05 kernel: Lustre: Skipped 1 previous similar message
Jun 19 23:18:02 weisshorn04 kernel: Lustre: scratch-OST0025: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880802869800, cur 1340140682 expire 1340140532 last 1340140455
Jun 19 23:18:02 weisshorn04 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 23:18:14 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:18:27 weisshorn02 kernel: Lustre: scratch-MDT0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fb5d9d000, cur 1340140707 expire 1340140557 last 1340140480
Jun 19 23:18:27 weisshorn09 kernel: Lustre: scratch-OST0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c95eb7800, cur 1340140707 expire 1340140557 last 1340140480
Jun 19 23:18:27 weisshorn09 kernel: Lustre: Skipped 1 previous similar message
Jun 19 23:18:29 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:18:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:19:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:19:01 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340140685/real 1340140685]  req@ffff8809de7af800 x1403849872954768/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340140741 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:19:01 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 23:19:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:19:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:19:35 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340140719/real 1340140719]  req@ffff8807c1516800 x1403849986649522/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340140775 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:19:35 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:19:39 weisshorn08 kernel: Lustre: 8441:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340140772/real 1340140772]  req@ffff880807cbbc00 x1403849823733503/t0(0) o106->scratch-OST0017@148.187.6.236@o2ib2:15/16 lens 296/232 e 0 to 1 dl 1340140779 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Jun 19 23:19:39 weisshorn08 kernel: Lustre: 8441:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jun 19 23:19:40 weisshorn02 kernel: LustreError: 47510:0:(mdt_handler.c:2822:mdt_recovery()) operation 400 on unconnected MDS from 12345-148.187.6.7@o2ib2
Jun 19 23:19:42 weisshorn06 kernel: Lustre: scratch-OST001d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881014072000, cur 1340140782 expire 1340140632 last 1340140555
Jun 19 23:19:42 weisshorn06 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 23:19:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:20:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:20:01 weisshorn08 CROND[125714]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:20:01 weisshorn12 CROND[128522]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:20:01 weisshorn05 CROND[126942]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:20:01 weisshorn14 CROND[55498]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:20:01 weisshorn09 CROND[126666]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:20:01 weisshorn03 CROND[127634]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:20:01 weisshorn01 CROND[4111]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:20:01 weisshorn02 CROND[861]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:20:01 weisshorn04 CROND[126401]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:20:01 weisshorn13 CROND[130545]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:20:01 weisshorn06 CROND[127877]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:20:01 weisshorn10 CROND[126968]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:20:02 weisshorn07 CROND[127133]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:20:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:20:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:20:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:21:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:21:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:21:22 weisshorn07 kernel: Lustre: scratch-OST002b: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807fd2fe800, cur 1340140882 expire 1340140732 last 1340140655
Jun 19 23:21:22 weisshorn07 kernel: Lustre: Skipped 1 previous similar message
Jun 19 23:21:22 weisshorn02 kernel: Lustre: MGS: haven't heard from client 02e8b9ff-fc43-5841-d3bb-815052085829 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bb4ef6c00, cur 1340140882 expire 1340140732 last 1340140655
Jun 19 23:21:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:21:42 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340140846/real 1340140846]  req@ffff88101061cc00 x1403849637697146/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340140902 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:21:42 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
Jun 19 23:21:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:21:47 weisshorn04 kernel: Lustre: scratch-OST003d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881024aa5c00, cur 1340140907 expire 1340140757 last 1340140680
Jun 19 23:22:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:22:11 weisshorn05 kernel: Lustre: scratch-OST0046: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 23:22:12 weisshorn09 kernel: Lustre: scratch-OST0010: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880344757400, cur 1340140932 expire 1340140782 last 1340140705
Jun 19 23:22:12 weisshorn05 kernel: Lustre: scratch-OST0027: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804a0a4a400, cur 1340140932 expire 1340140782 last 1340140705
Jun 19 23:22:12 weisshorn09 kernel: Lustre: Skipped 1 previous similar message
Jun 19 23:22:12 weisshorn13 kernel: Lustre: scratch-OST0011: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88081932e000, cur 1340140932 expire 1340140782 last 1340140705
Jun 19 23:22:12 weisshorn13 kernel: Lustre: Skipped 7 previous similar messages
Jun 19 23:22:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:22:25 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340140889/real 1340140889]  req@ffff881016aa4400 x1403849944307413/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340140945 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:22:25 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:22:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:22:32 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340140896/real 1340140896]  req@ffff880f63cde800 x1403849898519218/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340140952 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:22:32 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:22:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:23:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:23:12 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340140936/real 1340140936]  req@ffff880f69f69400 x1403849862765261/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340140992 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:23:12 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 23:23:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:23:27 weisshorn02 kernel: Lustre: 47790:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004f65:0x41c:0x0]] [valid 68719476736]
Jun 19 23:23:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:23:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:23:52 weisshorn08 kernel: Lustre: scratch-OST000c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88037b0aa000, cur 1340141032 expire 1340140882 last 1340140805
Jun 19 23:23:52 weisshorn12 kernel: Lustre: scratch-OST0021: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88050430d000, cur 1340141032 expire 1340140882 last 1340140805
Jun 19 23:23:52 weisshorn05 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 148.187.6.236@o2ib2  ns: filter-scratch-OST0046_UUID lock: ffff880c65343000/0xf8449fdfdec3b14b lrc: 3/0,0 mode: PW/PW res: 9614518/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) flags: 0x20 remote: 0xfc0c4f7b63c0363d expref: 9244 pid: 7785 timeout 5620619111
Jun 19 23:24:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:24:14 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340140998/real 1340140998]  req@ffff881015c31000 x1403849874551372/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340141054 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:24:14 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 23:24:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:24:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:24:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:25:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:25:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:25:16 weisshorn10 nslcd[4294]: [5138de] ldap_result() failed: Can't contact LDAP server
Jun 19 23:25:18 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340141062/real 1340141062]  req@ffff881015988c00 x1403849922954883/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340141118 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:25:18 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:25:19 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340141063/real 1340141063]  req@ffff880d3ff81000 x1403849955224220/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340141119 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:25:19 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:25:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:25:32 weisshorn09 kernel: Lustre: scratch-OST0034: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881017297800, cur 1340141132 expire 1340140982 last 1340140905
Jun 19 23:25:32 weisshorn09 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 23:25:45 weisshorn09 nslcd[3954]: [fc7321] ldap_result() failed: Can't contact LDAP server
Jun 19 23:25:45 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:26:00 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:26:08 weisshorn08 nslcd[3946]: [5138de] ldap_result() failed: Can't contact LDAP server
Jun 19 23:26:09 weisshorn06 nslcd[3953]: [333c89] ldap_result() failed: Can't contact LDAP server
Jun 19 23:26:11 weisshorn04 nslcd[3885]: [fc7321] ldap_result() failed: Can't contact LDAP server
Jun 19 23:26:12 weisshorn07 nslcd[4297]: [5138de] ldap_result() failed: Can't contact LDAP server
Jun 19 23:26:15 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:26:22 weisshorn12 kernel: Lustre: scratch-OST0028: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88033407a800, cur 1340141182 expire 1340141032 last 1340140955
Jun 19 23:26:22 weisshorn03 kernel: Lustre: scratch-OST0014: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88020dc91c00, cur 1340141182 expire 1340141032 last 1340140955
Jun 19 23:26:22 weisshorn03 kernel: Lustre: Skipped 3 previous similar messages
Jun 19 23:26:29 weisshorn02 kernel: Lustre: MGS: haven't heard from client 02e8b9ff-fc43-5841-d3bb-815052085829 (at 148.187.6.7@o2ib2) in 228 seconds. I think it's dead, and I am evicting it. exp ffff880fb5fe8000, cur 1340141189 expire 1340141039 last 1340140961
Jun 19 23:26:30 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:26:32 weisshorn12 nslcd[3954]: [5138de] ldap_result() failed: Can't contact LDAP server
Jun 19 23:26:33 weisshorn02 kernel: Lustre: 47761:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004f65:0x426:0x0]] [valid 68719476736]
Jun 19 23:26:34 weisshorn13 nslcd[4224]: [5138de] ldap_result() failed: Can't contact LDAP server
Jun 19 23:26:46 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:26:47 weisshorn10 kernel: Lustre: scratch-OST000a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c9d1d4800, cur 1340141207 expire 1340141057 last 1340140980
Jun 19 23:26:47 weisshorn04 kernel: Lustre: scratch-OST003d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c0919c400, cur 1340141207 expire 1340141057 last 1340140980
Jun 19 23:26:47 weisshorn04 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 23:27:01 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:27:11 weisshorn05 nslcd[4378]: [22ad0c] ldap_result() failed: Can't contact LDAP server
Jun 19 23:27:12 weisshorn05 kernel: Lustre: scratch-OST000e: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880652de7800, cur 1340141232 expire 1340141082 last 1340141005
Jun 19 23:27:16 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:27:29 weisshorn02 kernel: Lustre: 47503:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004f65:0x42b:0x0]] [valid 68719476736]
Jun 19 23:27:31 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:27:46 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:27:48 weisshorn12 kernel: Lustre: scratch-OST0021: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807b6032000, cur 1340141268 expire 1340141118 last 1340141041
Jun 19 23:27:48 weisshorn12 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 23:28:01 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:28:08 weisshorn02 kernel: Lustre: 48049:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004f65:0x430:0x0]] [valid 68719476736]
Jun 19 23:28:16 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:28:31 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:28:40 weisshorn02 kernel: Lustre: 47807:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004f65:0x435:0x0]] [valid 68719476736]
Jun 19 23:28:46 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:29:01 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:29:10 weisshorn02 kernel: Lustre: 48047:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004f65:0x43a:0x0]] [valid 68719476736]
Jun 19 23:29:16 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:29:31 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:29:45 weisshorn02 kernel: Lustre: 48052:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004f65:0x442:0x0]] [valid 68719476736]
Jun 19 23:29:46 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:30:01 weisshorn10 CROND[126999]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:30:01 weisshorn08 CROND[125770]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:30:01 weisshorn07 CROND[127156]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:30:01 weisshorn14 CROND[55552]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:30:01 weisshorn05 CROND[126978]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:30:01 weisshorn12 CROND[128586]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:30:01 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:30:01 weisshorn09 CROND[126723]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:30:01 weisshorn03 CROND[127694]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:30:01 weisshorn01 CROND[4135]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:30:01 weisshorn02 CROND[910]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:30:01 weisshorn04 CROND[126437]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:30:01 weisshorn13 CROND[130606]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:30:02 weisshorn06 CROND[127903]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:30:16 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340141360/real 1340141360]  req@ffff88100fdaac00 x1403849873069118/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340141416 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:30:16 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:30:16 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:30:25 weisshorn02 kernel: Lustre: 47765:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004f65:0x44a:0x0]] [valid 68719476736]
Jun 19 23:30:31 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:30:32 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340141376/real 1340141376]  req@ffff8809bfaba400 x1403849823802799/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340141432 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:30:32 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
Jun 19 23:30:46 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:30:50 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340141394/real 1340141394]  req@ffff8810127ed000 x1403849986839254/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340141450 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:30:50 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:30:57 weisshorn09 kernel: Lustre: scratch-OST0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88081e9ecc00, cur 1340141457 expire 1340141307 last 1340141230
Jun 19 23:30:57 weisshorn09 kernel: Lustre: Skipped 2 previous similar messages
Jun 19 23:31:01 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:31:08 weisshorn05 kernel: Lustre: scratch-OST0027: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e5fc09400, cur 1340141468 expire 1340141318 last 1340141241
Jun 19 23:31:08 weisshorn06 kernel: Lustre: scratch-OST0041: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808086d2400, cur 1340141468 expire 1340141318 last 1340141241
Jun 19 23:31:08 weisshorn06 kernel: Lustre: Skipped 7 previous similar messages
Jun 19 23:31:16 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:31:22 weisshorn12 kernel: Lustre: scratch-OST0021: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806c5d2dc00, cur 1340141482 expire 1340141332 last 1340141255
Jun 19 23:31:22 weisshorn12 kernel: Lustre: Skipped 1 previous similar message
Jun 19 23:31:22 weisshorn08 kernel: Lustre: scratch-OST000c: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88033de70c00, cur 1340141482 expire 1340141332 last 1340141255
Jun 19 23:31:31 weisshorn02 kernel: Lustre: 47715:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004f65:0x459:0x0]] [valid 68719476736]
Jun 19 23:31:31 weisshorn02 kernel: Lustre: 47715:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 2 previous similar messages
Jun 19 23:31:31 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:31:46 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:32:01 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:32:16 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:32:25 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340141489/real 1340141489]  req@ffff880994cea400 x1403849944402638/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340141545 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:32:25 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 23:32:31 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:32:46 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:32:57 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340141521/real 1340141521]  req@ffff8806e2648000 x1403849637797419/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340141577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:32:57 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Jun 19 23:33:01 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:33:02 weisshorn07 kernel: Lustre: scratch-OST0042: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802de65a400, cur 1340141582 expire 1340141432 last 1340141355
Jun 19 23:33:02 weisshorn07 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 23:33:16 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:33:31 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:33:46 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:33:47 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340141571/real 1340141571]  req@ffff8810149c2c00 x1403849898640425/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340141627 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:33:47 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:34:01 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:34:16 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:34:17 weisshorn04 kernel: Lustre: scratch-OST000d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e64858400, cur 1340141657 expire 1340141507 last 1340141430
Jun 19 23:34:17 weisshorn04 kernel: Lustre: Skipped 5 previous similar messages
Jun 19 23:34:17 weisshorn08 kernel: Lustre: scratch-OST0023: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eee782000, cur 1340141657 expire 1340141507 last 1340141430
Jun 19 23:34:27 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340141611/real 1340141611]  req@ffff88101aef3800 x1403849862862420/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340141667 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:34:27 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:34:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:34:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:34:51 weisshorn02 kernel: Lustre: MGS: haven't heard from client 02e8b9ff-fc43-5841-d3bb-815052085829 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fb5f23c00, cur 1340141691 expire 1340141541 last 1340141464
Jun 19 23:35:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:35:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:35:19 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340141663/real 1340141663]  req@ffff880b34a9b000 x1403849955398506/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340141719 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:35:19 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 23:35:25 weisshorn02 kernel: Lustre: 47789:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004f65:0x47f:0x0]] [valid 68719476736]
Jun 19 23:35:25 weisshorn02 kernel: Lustre: 47789:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 3 previous similar messages
Jun 19 23:35:29 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340141673/real 1340141673]  req@ffff881013f47c00 x1403849874681041/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340141729 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:35:29 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:35:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:35:40 weisshorn05 kernel: Lustre: scratch-OST0046: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 23:35:40 weisshorn05 kernel: Lustre: Skipped 1 previous similar message
Jun 19 23:35:40 weisshorn05 kernel: Lustre: scratch-OST0046: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) refused reconnection, still busy with 2 active RPCs
Jun 19 23:35:40 weisshorn05 kernel: LustreError: 8533:0:(ldlm_lib.c:2706:target_bulk_io()) @@@ Reconnect on bulk GET  req@ffff88100d501c00 x1403837088647868/t0(0) o4->cdd482d3-fa4f-4442-72da-16bfade64b5f@148.187.6.236@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340141786 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 23:35:40 weisshorn05 kernel: LustreError: 8219:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc -107  req@ffff881016a43050 x1403837088647865/t0(0) o4->cdd482d3-fa4f-4442-72da-16bfade64b5f@148.187.6.236@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340141786 ref 1 fl Interpret:/0/0 rc 0/0
Jun 19 23:35:40 weisshorn05 kernel: Lustre: scratch-OST0046: Bulk IO write error with cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2), client will retry: rc -107
Jun 19 23:35:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:36:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:36:05 weisshorn05 kernel: Lustre: scratch-OST0046: Client cdd482d3-fa4f-4442-72da-16bfade64b5f (at 148.187.6.236@o2ib2) reconnecting
Jun 19 23:36:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:36:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:36:33 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340141737/real 1340141737]  req@ffff881016269400 x1403849923090364/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340141793 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:36:33 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:36:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:36:50 weisshorn03 rhnsd[127717]: running program /usr/sbin/rhn_check
Jun 19 23:37:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:37:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:37:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:37:41 weisshorn04 rhnsd[126493]: running program /usr/sbin/rhn_check
Jun 19 23:37:45 weisshorn05 kernel: LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 125s: evicting client at 148.187.6.236@o2ib2  ns: filter-scratch-OST0046_UUID lock: ffff880c61ab3480/0xf8449fdfded43528 lrc: 3/0,0 mode: PW/PW res: 9618100/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x20 remote: 0xfc0c4f7b63c265b1 expref: 195 pid: 7919 timeout 5621452856
Jun 19 23:37:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:38:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:38:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:38:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:38:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:39:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:39:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:39:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:39:42 weisshorn02 kernel: Lustre: scratch-MDT0000: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803aa271000, cur 1340141982 expire 1340141832 last 1340141755
Jun 19 23:39:42 weisshorn03 kernel: Lustre: scratch-OST0014: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88030a572000, cur 1340141982 expire 1340141832 last 1340141755
Jun 19 23:39:42 weisshorn03 kernel: Lustre: Skipped 4 previous similar messages
Jun 19 23:39:42 weisshorn13 kernel: Lustre: scratch-OST0018: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807ce093c00, cur 1340141982 expire 1340141832 last 1340141755
Jun 19 23:39:42 weisshorn13 kernel: Lustre: Skipped 7 previous similar messages
Jun 19 23:39:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:40:01 weisshorn13 CROND[130648]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:40:01 weisshorn06 CROND[127964]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:40:01 weisshorn08 CROND[125806]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:40:01 weisshorn07 CROND[127187]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:40:01 weisshorn10 CROND[127059]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:40:01 weisshorn14 CROND[55598]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:40:01 weisshorn05 CROND[127046]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:40:01 weisshorn12 CROND[128629]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:40:01 weisshorn09 CROND[126761]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:40:01 weisshorn01 CROND[4202]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:40:01 weisshorn03 CROND[127734]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:40:01 weisshorn02 CROND[1006]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:40:02 weisshorn04 CROND[126504]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:40:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:40:07 weisshorn10 kernel: Lustre: scratch-OST000a: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880807c75c00, cur 1340142007 expire 1340141857 last 1340141780
Jun 19 23:40:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:40:29 weisshorn02 kernel: Lustre: 47750:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004f65:0x4b4:0x0]] [valid 68719476736]
Jun 19 23:40:29 weisshorn02 kernel: Lustre: 47750:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 9 previous similar messages
Jun 19 23:40:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:40:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:41:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:41:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:41:31 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142035/real 1340142035]  req@ffff88081d0f0000 x1403849873201078/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340142091 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:41:31 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:41:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:41:47 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142051/real 1340142051]  req@ffff880a00dc8000 x1403849823882731/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340142107 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:41:47 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
Jun 19 23:41:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:42:01 weisshorn14 nslcd[3437]: [413bec] ldap_result() failed: Can't contact LDAP server
Jun 19 23:42:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:42:05 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142069/real 1340142069]  req@ffff8802b72c2400 x1403849987030765/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340142125 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:42:05 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:42:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:42:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:42:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:42:57 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142121/real 1340142121]  req@ffff88059e986400 x1403849637906703/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340142177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:42:57 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 23:43:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:43:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:43:25 weisshorn07 ntpd[4804]: synchronized to 148.187.18.120, stratum 2
Jun 19 23:43:32 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:43:40 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142164/real 1340142164]  req@ffff880a77abb400 x1403849944524041/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340142220 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:43:40 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:43:47 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:44:02 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:44:17 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:44:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:44:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:45:02 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142246/real 1340142246]  req@ffff880fd53ea400 x1403849898763687/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340142302 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:45:02 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:45:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:45:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:45:29 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142273/real 1340142273]  req@ffff881013f47c00 x1403849874779418/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340142329 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:45:29 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 23:45:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:45:42 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142286/real 1340142286]  req@ffff880b0dd18400 x1403849862959315/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340142342 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:45:42 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:45:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:45:51 weisshorn03 nslcd[3944]: [21971b] ldap_result() failed: Can't contact LDAP server
Jun 19 23:46:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:46:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:46:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:46:34 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142338/real 1340142338]  req@ffff8808c8816c00 x1403849955626741/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340142394 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:46:34 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:46:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:47:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:47:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:47:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:47:48 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142412/real 1340142412]  req@ffff88101659a000 x1403849923238419/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340142468 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:47:48 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:47:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:48:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:48:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:48:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:48:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:49:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:49:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:49:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:49:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:50:01 weisshorn02 CROND[1062]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:50:01 weisshorn04 CROND[126530]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:50:01 weisshorn13 CROND[130672]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:50:01 weisshorn06 CROND[128003]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:50:01 weisshorn08 CROND[125839]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:50:01 weisshorn07 CROND[127262]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:50:01 weisshorn10 CROND[127095]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:50:01 weisshorn14 CROND[55625]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:50:01 weisshorn05 CROND[127072]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:50:01 weisshorn09 CROND[126792]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:50:01 weisshorn12 CROND[128654]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:50:01 weisshorn01 CROND[4233]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:50:01 weisshorn03 CROND[127760]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1)
Jun 19 23:50:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:50:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:50:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:50:42 weisshorn02 kernel: Lustre: 47402:0:(mdt_xattr.c:367:mdt_reint_setxattr()) scratch-MDT0000: client miss to set OBD_MD_FLCTIME when setxattr system.posix_acl_access: [object [0x200004f65:0x4c9:0x0]] [valid 68719476736]
Jun 19 23:50:42 weisshorn02 kernel: Lustre: 47402:0:(mdt_xattr.c:367:mdt_reint_setxattr()) Skipped 2 previous similar messages
Jun 19 23:50:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:51:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:51:16 weisshorn14 nslcd[3437]: [180b0b] ldap_result() failed: Can't contact LDAP server
Jun 19 23:51:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:51:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:51:36 weisshorn12 rhnsd[128693]: running program /usr/sbin/rhn_check
Jun 19 23:51:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:52:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:52:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:52:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:52:46 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142710/real 1340142710]  req@ffff88081d0f0000 x1403849873316845/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340142766 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:52:46 weisshorn03 kernel: Lustre: 5777:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:52:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:53:01 weisshorn02 CROND[1076]: (root) CMD (/usr/lib64/sa/sa2 -A)
Jun 19 23:53:01 weisshorn06 CROND[128015]: (root) CMD (/usr/lib64/sa/sa2 -A)
Jun 19 23:53:01 weisshorn13 CROND[130720]: (root) CMD (/usr/lib64/sa/sa2 -A)
Jun 19 23:53:01 weisshorn04 CROND[126544]: (root) CMD (/usr/lib64/sa/sa2 -A)
Jun 19 23:53:01 weisshorn08 CROND[125848]: (root) CMD (/usr/lib64/sa/sa2 -A)
Jun 19 23:53:01 weisshorn07 CROND[127270]: (root) CMD (/usr/lib64/sa/sa2 -A)
Jun 19 23:53:01 weisshorn10 CROND[127103]: (root) CMD (/usr/lib64/sa/sa2 -A)
Jun 19 23:53:01 weisshorn14 CROND[55669]: (root) CMD (/usr/lib64/sa/sa2 -A)
Jun 19 23:53:01 weisshorn05 CROND[127085]: (root) CMD (/usr/lib64/sa/sa2 -A)
Jun 19 23:53:01 weisshorn09 CROND[126801]: (root) CMD (/usr/lib64/sa/sa2 -A)
Jun 19 23:53:01 weisshorn12 CROND[128698]: (root) CMD (/usr/lib64/sa/sa2 -A)
Jun 19 23:53:01 weisshorn01 CROND[4245]: (root) CMD (/usr/lib64/sa/sa2 -A)
Jun 19 23:53:01 weisshorn03 CROND[127769]: (root) CMD (/usr/lib64/sa/sa2 -A)
Jun 19 23:53:02 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142726/real 1340142726]  req@ffff88100f217c00 x1403849823938140/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340142782 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:53:02 weisshorn08 kernel: Lustre: 5767:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:53:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:53:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:53:20 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142744/real 1340142744]  req@ffff88100dc7b800 x1403849987254453/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340142800 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:53:20 weisshorn13 kernel: Lustre: 6042:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:53:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:53:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:54:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:54:12 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142796/real 1340142796]  req@ffff880807b3cc00 x1403849638022361/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340142852 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:54:12 weisshorn05 kernel: Lustre: 5258:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:54:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:54:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:54:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:54:55 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142839/real 1340142839]  req@ffff881013c0f400 x1403849944634257/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340142895 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:54:55 weisshorn10 kernel: Lustre: 6085:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:55:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:55:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:55:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:55:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:56:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:56:17 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142921/real 1340142921]  req@ffff881012316800 x1403849898868381/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340142977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:56:17 weisshorn06 kernel: Lustre: 5990:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:56:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:56:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:56:34 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142938/real 1340142938]  req@ffff880bfb981c00 x1403849955803943/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340142994 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:56:34 weisshorn12 kernel: Lustre: 5747:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Jun 19 23:56:44 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142948/real 1340142948]  req@ffff8808e60ab800 x1403849874899838/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340143004 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:56:44 weisshorn07 kernel: Lustre: 6136:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:56:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:56:57 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340142961/real 1340142961]  req@ffff88101b3b4c00 x1403849863033411/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340143017 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:56:57 weisshorn04 kernel: Lustre: 5726:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:57:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:57:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:57:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:57:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:58:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:58:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:58:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:58:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:59:03 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1340143087/real 1340143087]  req@ffff8810251a8000 x1403849923337983/t0(0) o250->MGC148.187.7.101@o2ib2@148.187.7.101@o2ib2:26/25 lens 368/512 e 0 to 1 dl 1340143143 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Jun 19 23:59:03 weisshorn09 kernel: Lustre: 5842:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Jun 19 23:59:03 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:59:18 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:59:33 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database
Jun 19 23:59:48 weisshorn02 /usr/sbin/cerebrod[45710]: lmt_mysql: failed to connect to database