Jun 19 00:00:01 weisshorn11 CROND[120224]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 00:01:01 weisshorn11 CROND[120229]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 00:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1202 starting 0anacron Jun 19 00:01:01 weisshorn11 anacron[120241]: Anacron started on 2012-06-19 Jun 19 00:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1202 finished 0anacron Jun 19 00:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1202 starting mcelog.cron Jun 19 00:01:01 weisshorn11 anacron[120241]: Jobs will be executed sequentially Jun 19 00:01:01 weisshorn11 anacron[120241]: Normal exit (0 jobs run) Jun 19 00:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1202 finished mcelog.cron Jun 19 00:10:01 weisshorn11 sshd[120320]: Accepted publickey for root from 148.187.12.231 port 45102 ssh2 Jun 19 00:10:01 weisshorn11 sshd[120320]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 00:10:01 weisshorn11 sshd[120320]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 00:10:01 weisshorn11 sshd[120320]: pam_unix(sshd:session): session closed for user root Jun 19 00:10:01 weisshorn11 CROND[120343]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 00:20:01 weisshorn11 CROND[120366]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 00:25:47 weisshorn11 nslcd[3954]: [9efb0d] ldap_result() failed: Can't contact LDAP server Jun 19 00:30:01 weisshorn11 CROND[120399]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 00:30:17 weisshorn11 nslcd[3954]: [0802be] ldap_result() failed: Can't contact LDAP server Jun 19 00:40:02 weisshorn11 CROND[120464]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 00:50:01 weisshorn11 CROND[120488]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 01:00:01 weisshorn11 CROND[120525]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 01:01:01 weisshorn11 CROND[120530]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 01:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1205 starting 0anacron Jun 19 01:01:01 weisshorn11 anacron[120542]: Anacron started on 2012-06-19 Jun 19 01:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1205 finished 0anacron Jun 19 01:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1205 starting mcelog.cron Jun 19 01:01:01 weisshorn11 anacron[120542]: Jobs will be executed sequentially Jun 19 01:01:01 weisshorn11 anacron[120542]: Normal exit (0 jobs run) Jun 19 01:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1205 finished mcelog.cron Jun 19 01:10:01 weisshorn11 CROND[120608]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 01:10:01 weisshorn11 sshd[120610]: Accepted publickey for root from 148.187.12.231 port 45124 ssh2 Jun 19 01:10:01 weisshorn11 sshd[120610]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 01:10:02 weisshorn11 sshd[120610]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 01:10:02 weisshorn11 sshd[120610]: pam_unix(sshd:session): session closed for user root Jun 19 01:20:01 weisshorn11 CROND[120653]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 01:26:02 weisshorn11 nslcd[3954]: [66459b] ldap_result() failed: Can't contact LDAP server Jun 19 01:30:01 weisshorn11 CROND[120691]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 01:30:32 weisshorn11 nslcd[3954]: [8b0cbf] ldap_result() failed: Can't contact LDAP server Jun 19 01:40:01 weisshorn11 CROND[120751]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 01:50:01 weisshorn11 CROND[120781]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 02:00:01 weisshorn11 CROND[120854]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 02:01:01 weisshorn11 CROND[120859]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 02:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1208 starting 0anacron Jun 19 02:01:01 weisshorn11 anacron[120871]: Anacron started on 2012-06-19 Jun 19 02:01:01 weisshorn11 anacron[120871]: Jobs will be executed sequentially Jun 19 02:01:01 weisshorn11 anacron[120871]: Normal exit (0 jobs run) Jun 19 02:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1208 finished 0anacron Jun 19 02:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1208 starting mcelog.cron Jun 19 02:01:02 weisshorn11 run-parts(/etc/cron.hourly)[1208 finished mcelog.cron Jun 19 02:10:01 weisshorn11 CROND[120896]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 02:10:01 weisshorn11 sshd[120898]: Accepted publickey for root from 148.187.12.231 port 45139 ssh2 Jun 19 02:10:01 weisshorn11 sshd[120898]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 02:10:02 weisshorn11 sshd[120898]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 02:10:02 weisshorn11 sshd[120898]: pam_unix(sshd:session): session closed for user root Jun 19 02:20:01 weisshorn11 CROND[120946]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 02:26:17 weisshorn11 nslcd[3954]: [e203ab] ldap_result() failed: Can't contact LDAP server Jun 19 02:30:01 weisshorn11 CROND[121020]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 02:30:47 weisshorn11 nslcd[3954]: [86d60e] ldap_result() failed: Can't contact LDAP server Jun 19 02:40:01 weisshorn11 CROND[121044]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 02:50:01 weisshorn11 CROND[121068]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 03:00:01 weisshorn11 CROND[121141]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 03:01:01 weisshorn11 CROND[121146]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 03:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1211 starting 0anacron Jun 19 03:01:01 weisshorn11 anacron[121158]: Anacron started on 2012-06-19 Jun 19 03:01:01 weisshorn11 anacron[121158]: Will run job `cron.daily' in 46 min. Jun 19 03:01:01 weisshorn11 anacron[121158]: Jobs will be executed sequentially Jun 19 03:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1211 finished 0anacron Jun 19 03:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1211 starting mcelog.cron Jun 19 03:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1211 finished mcelog.cron Jun 19 03:10:01 weisshorn11 sshd[121188]: Accepted publickey for root from 148.187.12.231 port 45146 ssh2 Jun 19 03:10:01 weisshorn11 sshd[121188]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 03:10:01 weisshorn11 sshd[121188]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 03:10:01 weisshorn11 sshd[121188]: pam_unix(sshd:session): session closed for user root Jun 19 03:10:01 weisshorn11 CROND[121211]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 03:20:01 weisshorn11 CROND[121247]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 03:26:32 weisshorn11 nslcd[3954]: [3da4d5] ldap_result() failed: Can't contact LDAP server Jun 19 03:26:32 weisshorn11 nslcd[3954]: [3da4d5] ldap_abandon() failed to abandon search: Other (e.g., implementation specific) error Jun 19 03:30:01 weisshorn11 CROND[121309]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 03:31:02 weisshorn11 nslcd[3954]: [c93698] ldap_result() failed: Can't contact LDAP server Jun 19 03:34:50 weisshorn11 ntpd[4454]: synchronized to 148.187.18.120, stratum 2 Jun 19 03:36:45 weisshorn11 rhnsd[121328]: running program /usr/sbin/rhn_check Jun 19 03:40:02 weisshorn11 CROND[121337]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 03:47:01 weisshorn11 anacron[121158]: Job `cron.daily' started Jun 19 03:47:01 weisshorn11 run-parts(/etc/cron.daily)[12135 starting cups Jun 19 03:47:01 weisshorn11 run-parts(/etc/cron.daily)[12136 finished cups Jun 19 03:47:01 weisshorn11 run-parts(/etc/cron.daily)[12135 starting logrotate Jun 19 03:47:01 weisshorn11 run-parts(/etc/cron.daily)[12137 finished logrotate Jun 19 03:47:01 weisshorn11 run-parts(/etc/cron.daily)[12135 starting makewhatis.cron Jun 19 03:47:02 weisshorn11 run-parts(/etc/cron.daily)[12149 finished makewhatis.cron Jun 19 03:47:02 weisshorn11 run-parts(/etc/cron.daily)[12135 starting mlocate.cron Jun 19 03:47:04 weisshorn11 run-parts(/etc/cron.daily)[12151 finished mlocate.cron Jun 19 03:47:04 weisshorn11 run-parts(/etc/cron.daily)[12135 starting prelink Jun 19 03:47:30 weisshorn11 run-parts(/etc/cron.daily)[12397 finished prelink Jun 19 03:47:30 weisshorn11 run-parts(/etc/cron.daily)[12135 starting readahead.cron Jun 19 03:47:30 weisshorn11 run-parts(/etc/cron.daily)[12398 finished readahead.cron Jun 19 03:47:30 weisshorn11 run-parts(/etc/cron.daily)[12135 starting rhsmd Jun 19 03:47:31 weisshorn11 rhsmd: This system is registered to RHN Classic Jun 19 03:47:31 weisshorn11 run-parts(/etc/cron.daily)[12399 finished rhsmd Jun 19 03:47:31 weisshorn11 run-parts(/etc/cron.daily)[12135 starting tmpwatch Jun 19 03:47:32 weisshorn11 run-parts(/etc/cron.daily)[12402 finished tmpwatch Jun 19 03:47:32 weisshorn11 anacron[121158]: Job `cron.daily' terminated Jun 19 03:47:32 weisshorn11 anacron[121158]: Normal exit (1 job run) Jun 19 03:50:01 weisshorn11 CROND[124048]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 04:00:01 weisshorn11 CROND[124108]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 04:01:01 weisshorn11 CROND[124113]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 04:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1241 starting 0anacron Jun 19 04:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1241 finished 0anacron Jun 19 04:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1241 starting mcelog.cron Jun 19 04:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1241 finished mcelog.cron Jun 19 04:10:01 weisshorn11 CROND[124157]: (root) CMD (/usr/share/spamassassin/sa-update.cron 2>&1 | tee -a /var/log/sa-update.log) Jun 19 04:10:01 weisshorn11 CROND[124158]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 04:10:01 weisshorn11 sshd[124162]: Accepted publickey for root from 148.187.12.231 port 45165 ssh2 Jun 19 04:10:01 weisshorn11 sshd[124162]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 04:10:02 weisshorn11 sshd[124162]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 04:10:02 weisshorn11 sshd[124162]: pam_unix(sshd:session): session closed for user root Jun 19 04:20:01 weisshorn11 CROND[124256]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 04:30:01 weisshorn11 CROND[124275]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 04:31:17 weisshorn11 nslcd[3954]: [781401] ldap_result() failed: Can't contact LDAP server Jun 19 04:40:01 weisshorn11 CROND[124305]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 04:50:01 weisshorn11 CROND[124378]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 05:00:01 weisshorn11 CROND[124402]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 05:01:01 weisshorn11 CROND[124407]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 05:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1244 starting 0anacron Jun 19 05:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1244 finished 0anacron Jun 19 05:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1244 starting mcelog.cron Jun 19 05:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1244 finished mcelog.cron Jun 19 05:05:17 weisshorn11 nslcd[3954]: [6928d6] ldap_result() failed: Can't contact LDAP server Jun 19 05:10:01 weisshorn11 CROND[124447]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 05:10:02 weisshorn11 sshd[124449]: Accepted publickey for root from 148.187.12.231 port 45185 ssh2 Jun 19 05:10:02 weisshorn11 sshd[124449]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 05:10:02 weisshorn11 sshd[124449]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 05:10:02 weisshorn11 sshd[124449]: pam_unix(sshd:session): session closed for user root Jun 19 05:20:01 weisshorn11 CROND[124541]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 05:30:01 weisshorn11 CROND[124565]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 05:40:01 weisshorn11 nslcd[3954]: [7195e5] ldap_result() failed: Can't contact LDAP server Jun 19 05:40:01 weisshorn11 CROND[124598]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 05:50:01 weisshorn11 CROND[124662]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 06:00:02 weisshorn11 CROND[124686]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 06:01:01 weisshorn11 CROND[124691]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 06:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1246 starting 0anacron Jun 19 06:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1247 finished 0anacron Jun 19 06:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1246 starting mcelog.cron Jun 19 06:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1247 finished mcelog.cron Jun 19 06:05:32 weisshorn11 nslcd[3954]: [46a5f2] ldap_result() failed: Can't contact LDAP server Jun 19 06:10:01 weisshorn11 CROND[124742]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 06:10:02 weisshorn11 sshd[124744]: Accepted publickey for root from 148.187.12.231 port 45203 ssh2 Jun 19 06:10:02 weisshorn11 sshd[124744]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 06:10:02 weisshorn11 sshd[124744]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 06:10:02 weisshorn11 sshd[124744]: pam_unix(sshd:session): session closed for user root Jun 19 06:20:01 weisshorn11 CROND[124823]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 06:30:01 weisshorn11 CROND[124852]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 06:40:01 weisshorn11 CROND[124884]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 06:40:16 weisshorn11 nslcd[3954]: [2dfed6] ldap_result() failed: Can't contact LDAP server Jun 19 06:50:01 weisshorn11 CROND[124945]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 07:00:01 weisshorn11 CROND[124974]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 07:01:01 weisshorn11 CROND[124979]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 07:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1249 starting 0anacron Jun 19 07:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1249 finished 0anacron Jun 19 07:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1249 starting mcelog.cron Jun 19 07:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1249 finished mcelog.cron Jun 19 07:05:47 weisshorn11 nslcd[3954]: [6ed7f6] ldap_result() failed: Can't contact LDAP server Jun 19 07:10:01 weisshorn11 CROND[125068]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 07:10:02 weisshorn11 sshd[125070]: Accepted publickey for root from 148.187.12.231 port 45215 ssh2 Jun 19 07:10:02 weisshorn11 sshd[125070]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 07:10:02 weisshorn11 sshd[125070]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 07:10:02 weisshorn11 sshd[125070]: pam_unix(sshd:session): session closed for user root Jun 19 07:20:02 weisshorn11 CROND[125113]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 07:30:01 weisshorn11 CROND[125145]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 07:36:45 weisshorn11 rhnsd[125162]: running program /usr/sbin/rhn_check Jun 19 07:40:01 weisshorn11 CROND[125212]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 07:40:31 weisshorn11 nslcd[3954]: [9df57d] ldap_result() failed: Can't contact LDAP server Jun 19 07:50:01 weisshorn11 CROND[125236]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 08:00:01 weisshorn11 CROND[125268]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 08:01:01 weisshorn11 CROND[125273]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 08:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1252 starting 0anacron Jun 19 08:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1252 finished 0anacron Jun 19 08:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1252 starting mcelog.cron Jun 19 08:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1252 finished mcelog.cron Jun 19 08:06:02 weisshorn11 nslcd[3954]: [bd7a3e] ldap_result() failed: Can't contact LDAP server Jun 19 08:10:01 weisshorn11 CROND[125349]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 08:10:01 weisshorn11 sshd[125346]: Accepted publickey for root from 148.187.12.231 port 45222 ssh2 Jun 19 08:10:01 weisshorn11 sshd[125346]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 08:10:02 weisshorn11 sshd[125346]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 08:10:02 weisshorn11 sshd[125346]: pam_unix(sshd:session): session closed for user root Jun 19 08:20:01 weisshorn11 CROND[125400]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 08:30:01 weisshorn11 CROND[125465]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 08:40:01 weisshorn11 CROND[125495]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 08:40:46 weisshorn11 nslcd[3954]: [90b7c5] ldap_result() failed: Can't contact LDAP server Jun 19 08:50:01 weisshorn11 CROND[125522]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 08:55:32 weisshorn11 kernel: Lustre: scratch-OST0021: haven't heard from client 8dc459e1-7a2a-f1d1-5f90-1a43bf55749c (at 1197@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080374a400, cur 1340088932 expire 1340088782 last 1340088705 Jun 19 08:55:32 weisshorn11 kernel: Lustre: Skipped 5 previous similar messages Jun 19 08:55:33 weisshorn11 kernel: Lustre: scratch-OST0015: haven't heard from client 8dc459e1-7a2a-f1d1-5f90-1a43bf55749c (at 1197@gni) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801fe0f4400, cur 1340088933 expire 1340088783 last 1340088706 Jun 19 08:55:33 weisshorn11 kernel: Lustre: Skipped 19 previous similar messages Jun 19 09:00:01 weisshorn11 CROND[125552]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 09:01:01 weisshorn11 CROND[125557]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 09:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1255 starting 0anacron Jun 19 09:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1255 finished 0anacron Jun 19 09:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1255 starting mcelog.cron Jun 19 09:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1255 finished mcelog.cron Jun 19 09:03:03 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting Jun 19 09:03:03 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 7 active RPCs Jun 19 09:03:03 weisshorn11 kernel: Lustre: Skipped 1 previous similar message Jun 19 09:03:03 weisshorn11 kernel: LustreError: 7658:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc -107 req@ffff8810190bdc00 x1403838203911626/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340089426 ref 1 fl Interpret:/0/0 rc 0/0 Jun 19 09:03:03 weisshorn11 kernel: Lustre: scratch-OST0008: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107 Jun 19 09:03:03 weisshorn11 kernel: LustreError: 7658:0:(ldlm_lib.c:2725:target_bulk_io()) Skipped 2 previous similar messages Jun 19 09:03:03 weisshorn11 kernel: Lustre: scratch-OST0008: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107 Jun 19 09:03:03 weisshorn11 kernel: Lustre: Skipped 2 previous similar messages Jun 19 09:03:03 weisshorn11 kernel: LustreError: 6137:0:(ldlm_lib.c:2706:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff881018c7d000 x1403838203911480/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340089426 ref 1 fl Interpret:/0/0 rc 0/0 Jun 19 09:03:03 weisshorn11 kernel: LustreError: 6137:0:(ldlm_lib.c:2706:target_bulk_io()) Skipped 3 previous similar messages Jun 19 09:03:27 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting Jun 19 09:04:56 weisshorn11 kernel: Lustre: scratch-OST002d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting Jun 19 09:04:56 weisshorn11 kernel: Lustre: scratch-OST002d: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 8 active RPCs Jun 19 09:04:56 weisshorn11 kernel: LustreError: 7658:0:(ldlm_lib.c:2706:target_bulk_io()) @@@ Reconnect on bulk GET req@ffff88080b296850 x1403838203920820/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340089539 ref 1 fl Interpret:/0/0 rc 0/0 Jun 19 09:04:56 weisshorn11 kernel: Lustre: scratch-OST002d: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -110 Jun 19 09:04:56 weisshorn11 kernel: Lustre: Skipped 2 previous similar messages Jun 19 09:05:39 weisshorn11 kernel: Lustre: scratch-OST0044: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting Jun 19 09:05:39 weisshorn11 kernel: Lustre: Skipped 5 previous similar messages Jun 19 09:05:39 weisshorn11 kernel: Lustre: scratch-OST0044: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 8 active RPCs Jun 19 09:05:39 weisshorn11 kernel: Lustre: Skipped 3 previous similar messages Jun 19 09:05:39 weisshorn11 kernel: LustreError: 7916:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc -107 req@ffff88080afda850 x1403838203922150/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340089582 ref 1 fl Interpret:/0/0 rc 0/0 Jun 19 09:05:39 weisshorn11 kernel: Lustre: scratch-OST0044: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107 Jun 19 09:05:39 weisshorn11 kernel: Lustre: Skipped 17 previous similar messages Jun 19 09:05:39 weisshorn11 kernel: LustreError: 7916:0:(ldlm_lib.c:2725:target_bulk_io()) Skipped 4 previous similar messages Jun 19 09:06:17 weisshorn11 nslcd[3954]: [03a216] ldap_result() failed: Can't contact LDAP server Jun 19 09:10:01 weisshorn11 CROND[125638]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 09:10:02 weisshorn11 sshd[125640]: Accepted publickey for root from 148.187.12.231 port 45248 ssh2 Jun 19 09:10:02 weisshorn11 sshd[125640]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 09:10:02 weisshorn11 sshd[125640]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 09:10:02 weisshorn11 sshd[125640]: pam_unix(sshd:session): session closed for user root Jun 19 09:12:53 weisshorn11 kernel: Lustre: 6010:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1340089966/real 1340089966] req@ffff880803848800 x1403849891133751/t0(0) o106->scratch-OST0015@148.187.6.8@o2ib2:15/16 lens 296/232 e 0 to 1 dl 1340089973 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 19 09:13:00 weisshorn11 kernel: Lustre: 6010:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1340089973/real 1340089973] req@ffff880803848800 x1403849891133751/t0(0) o106->scratch-OST0015@148.187.6.8@o2ib2:15/16 lens 296/232 e 0 to 1 dl 1340089980 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Jun 19 09:20:01 weisshorn11 CROND[125687]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 09:26:52 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting Jun 19 09:26:52 weisshorn11 kernel: Lustre: Skipped 3 previous similar messages Jun 19 09:26:52 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 9 active RPCs Jun 19 09:27:16 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting Jun 19 09:30:01 weisshorn11 CROND[125752]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 09:40:01 weisshorn11 CROND[125782]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 09:41:01 weisshorn11 nslcd[3954]: [b8a08e] ldap_result() failed: Can't contact LDAP server Jun 19 09:50:01 weisshorn11 CROND[125814]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 10:00:01 weisshorn11 CROND[125873]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 10:01:01 weisshorn11 CROND[125878]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 10:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1258 starting 0anacron Jun 19 10:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1258 finished 0anacron Jun 19 10:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1258 starting mcelog.cron Jun 19 10:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1258 finished mcelog.cron Jun 19 10:06:13 weisshorn11 kernel: Lustre: scratch-OST0044: Client 77ed2fd6-7ad2-3010-3442-924b80b07058 (at 148.187.6.8@o2ib2) reconnecting Jun 19 10:06:13 weisshorn11 kernel: Lustre: scratch-OST0044: Client 77ed2fd6-7ad2-3010-3442-924b80b07058 (at 148.187.6.8@o2ib2) refused reconnection, still busy with 1 active RPCs Jun 19 10:06:32 weisshorn11 nslcd[3954]: [b59621] ldap_result() failed: Can't contact LDAP server Jun 19 10:06:37 weisshorn11 kernel: Lustre: scratch-OST0044: Client 77ed2fd6-7ad2-3010-3442-924b80b07058 (at 148.187.6.8@o2ib2) reconnecting Jun 19 10:06:37 weisshorn11 kernel: Lustre: Skipped 5 previous similar messages Jun 19 10:10:01 weisshorn11 sshd[125923]: Accepted publickey for root from 148.187.12.231 port 45262 ssh2 Jun 19 10:10:01 weisshorn11 sshd[125923]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 10:10:01 weisshorn11 CROND[125928]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 10:10:01 weisshorn11 sshd[125923]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 10:10:01 weisshorn11 sshd[125923]: pam_unix(sshd:session): session closed for user root Jun 19 10:18:47 weisshorn11 ntpd[4454]: synchronized to 148.187.18.145, stratum 2 Jun 19 10:20:01 weisshorn11 CROND[125983]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 10:26:32 weisshorn11 kernel: Lustre: 5780:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1340094378/real 1340094378] req@ffff88050664a400 x1403849892120495/t0(0) o104->scratch-OST0008@148.187.6.8@o2ib2:15/16 lens 296/192 e 0 to 1 dl 1340094392 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 19 10:26:32 weisshorn11 kernel: LustreError: 138-a: scratch-OST0008: A client on nid 148.187.6.8@o2ib2 was evicted due to a lock blocking callback time out: rc -107 Jun 19 10:26:32 weisshorn11 kernel: LustreError: Skipped 1 previous similar message Jun 19 10:26:36 weisshorn11 kernel: Lustre: 5807:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1340094380/real 1340094380] req@ffff880e1abe9000 x1403849892120952/t0(0) o104->scratch-OST0021@148.187.6.8@o2ib2:15/16 lens 296/192 e 0 to 1 dl 1340094394 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 19 10:26:36 weisshorn11 kernel: LustreError: 138-a: scratch-OST0021: A client on nid 148.187.6.8@o2ib2 was evicted due to a lock blocking callback time out: rc -107 Jun 19 10:26:36 weisshorn11 kernel: LustreError: Skipped 7 previous similar messages Jun 19 10:26:37 weisshorn11 kernel: Lustre: 5778:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1340094380/real 0] req@ffff881013625800 x1403849892120956/t0(0) o104->scratch-OST002d@148.187.6.8@o2ib2:15/16 lens 296/192 e 0 to 1 dl 1340094394 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jun 19 10:26:37 weisshorn11 kernel: LustreError: 138-a: scratch-OST002d: A client on nid 148.187.6.8@o2ib2 was evicted due to a lock blocking callback time out: rc -107 Jun 19 10:26:37 weisshorn11 kernel: LustreError: Skipped 3 previous similar messages Jun 19 10:26:38 weisshorn11 kernel: LustreError: 8487:0:(ldlm_lockd.c:2077:ldlm_cancel_handler()) ldlm_cancel from 148.187.6.8@o2ib2 arrived at 1340094398 with bad export cookie 13982053158071586955 Jun 19 10:26:38 weisshorn11 kernel: LustreError: 8487:0:(ldlm_lockd.c:2077:ldlm_cancel_handler()) Skipped 1 previous similar message Jun 19 10:30:01 weisshorn11 CROND[126043]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 10:40:01 weisshorn11 CROND[126076]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 10:41:16 weisshorn11 nslcd[3954]: [92a02e] ldap_result() failed: Can't contact LDAP server Jun 19 10:50:01 weisshorn11 CROND[126106]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 11:00:01 weisshorn11 CROND[126171]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 11:01:01 weisshorn11 CROND[126176]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 11:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1261 starting 0anacron Jun 19 11:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1261 finished 0anacron Jun 19 11:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1261 starting mcelog.cron Jun 19 11:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1261 finished mcelog.cron Jun 19 11:10:01 weisshorn11 sshd[126217]: Accepted publickey for root from 148.187.12.231 port 45269 ssh2 Jun 19 11:10:01 weisshorn11 sshd[126217]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 11:10:01 weisshorn11 CROND[126231]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 11:10:02 weisshorn11 sshd[126217]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 11:10:02 weisshorn11 sshd[126217]: pam_unix(sshd:session): session closed for user root Jun 19 11:20:01 weisshorn11 CROND[126268]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 11:27:59 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting Jun 19 11:27:59 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 8 active RPCs Jun 19 11:27:59 weisshorn11 kernel: LustreError: 7941:0:(ldlm_lib.c:2725:target_bulk_io()) @@@ bulk GET failed: rc -107 req@ffff881016502400 x1403838204880428/t0(0) o4->79265048-d9a9-2078-6cdf-acef4f74b56d@148.187.6.8@o2ib2:0/0 lens 456/416 e 0 to 0 dl 1340098120 ref 1 fl Interpret:/0/0 rc 0/0 Jun 19 11:27:59 weisshorn11 kernel: LustreError: 7941:0:(ldlm_lib.c:2725:target_bulk_io()) Skipped 9 previous similar messages Jun 19 11:27:59 weisshorn11 kernel: Lustre: scratch-OST0008: Bulk IO write error with 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2), client will retry: rc -107 Jun 19 11:27:59 weisshorn11 kernel: Lustre: Skipped 13 previous similar messages Jun 19 11:28:22 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting Jun 19 11:30:01 weisshorn11 CROND[126334]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 11:36:45 weisshorn11 rhnsd[126354]: running program /usr/sbin/rhn_check Jun 19 11:39:47 weisshorn11 kernel: Lustre: scratch-OST0015: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting Jun 19 11:39:47 weisshorn11 kernel: Lustre: scratch-OST0044: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) refused reconnection, still busy with 1 active RPCs Jun 19 11:39:47 weisshorn11 kernel: Lustre: Skipped 2 previous similar messages Jun 19 11:40:01 weisshorn11 CROND[126364]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 11:40:11 weisshorn11 kernel: Lustre: scratch-OST0008: Client 79265048-d9a9-2078-6cdf-acef4f74b56d (at 148.187.6.8@o2ib2) reconnecting Jun 19 11:40:11 weisshorn11 kernel: Lustre: Skipped 5 previous similar messages Jun 19 11:45:17 weisshorn11 nslcd[3954]: [ef532d] ldap_result() failed: Can't contact LDAP server Jun 19 11:50:01 weisshorn11 nslcd[3954]: [d601af] ldap_result() failed: Can't contact LDAP server Jun 19 11:50:01 weisshorn11 CROND[126430]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 12:00:01 weisshorn11 CROND[126459]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 12:01:01 weisshorn11 CROND[126472]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 12:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1264 starting 0anacron Jun 19 12:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1264 finished 0anacron Jun 19 12:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1264 starting mcelog.cron Jun 19 12:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1264 finished mcelog.cron Jun 19 12:10:01 weisshorn11 CROND[126515]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 12:10:02 weisshorn11 sshd[126517]: Accepted publickey for root from 148.187.12.231 port 45284 ssh2 Jun 19 12:10:02 weisshorn11 sshd[126517]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 12:10:02 weisshorn11 sshd[126517]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 12:10:02 weisshorn11 sshd[126517]: pam_unix(sshd:session): session closed for user root Jun 19 12:20:01 weisshorn11 CROND[126590]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 12:20:35 weisshorn11 kernel: Lustre: 6030:0:(client.c:1762:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1340101142/real 1340101142] req@ffff880803e5a400 x1403849893611355/t0(0) o106->scratch-OST0021@6@gni:15/16 lens 296/232 e 0 to 1 dl 1340101235 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Jun 19 12:20:35 weisshorn11 kernel: Lustre: 6030:0:(client.c:1762:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jun 19 12:30:01 weisshorn11 CROND[126628]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 12:40:01 weisshorn11 CROND[126657]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 12:45:32 weisshorn11 nslcd[3954]: [c77402] ldap_result() failed: Can't contact LDAP server Jun 19 12:50:01 weisshorn11 CROND[126717]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 12:50:16 weisshorn11 nslcd[3954]: [b5b1c6] ldap_result() failed: Can't contact LDAP server Jun 19 13:00:01 weisshorn11 CROND[126750]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 13:01:01 weisshorn11 CROND[126755]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 13:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1267 starting 0anacron Jun 19 13:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1267 finished 0anacron Jun 19 13:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1267 starting mcelog.cron Jun 19 13:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1267 finished mcelog.cron Jun 19 13:07:26 weisshorn11 ntpd[4454]: synchronized to 148.187.18.120, stratum 2 Jun 19 13:10:01 weisshorn11 CROND[126799]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 13:10:02 weisshorn11 sshd[126801]: Accepted publickey for root from 148.187.12.231 port 45298 ssh2 Jun 19 13:10:02 weisshorn11 sshd[126801]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 13:10:02 weisshorn11 sshd[126801]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 13:10:02 weisshorn11 sshd[126801]: pam_unix(sshd:session): session closed for user root Jun 19 13:20:01 weisshorn11 CROND[126885]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 13:30:01 weisshorn11 CROND[126913]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 13:40:01 weisshorn11 CROND[126942]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 13:45:47 weisshorn11 nslcd[3954]: [6be6b9] ldap_result() failed: Can't contact LDAP server Jun 19 13:50:01 weisshorn11 CROND[127008]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 13:50:31 weisshorn11 nslcd[3954]: [234363] ldap_result() failed: Can't contact LDAP server Jun 19 14:00:01 weisshorn11 CROND[127035]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 14:01:01 weisshorn11 CROND[127040]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 14:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1270 starting 0anacron Jun 19 14:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1270 finished 0anacron Jun 19 14:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1270 starting mcelog.cron Jun 19 14:01:02 weisshorn11 run-parts(/etc/cron.hourly)[1270 finished mcelog.cron Jun 19 14:10:01 weisshorn11 CROND[127118]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 14:10:01 weisshorn11 sshd[127120]: Accepted publickey for root from 148.187.12.231 port 45310 ssh2 Jun 19 14:10:01 weisshorn11 sshd[127120]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 14:10:01 weisshorn11 sshd[127120]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 14:10:01 weisshorn11 sshd[127120]: pam_unix(sshd:session): session closed for user root Jun 19 14:20:01 weisshorn11 CROND[127169]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 14:30:01 weisshorn11 CROND[127206]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 14:40:01 weisshorn11 CROND[127260]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 14:46:02 weisshorn11 nslcd[3954]: [d385c9] ldap_result() failed: Can't contact LDAP server Jun 19 14:50:01 weisshorn11 CROND[127299]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 14:50:46 weisshorn11 nslcd[3954]: [6f6d8a] ldap_result() failed: Can't contact LDAP server Jun 19 15:00:01 weisshorn11 CROND[127328]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 15:01:01 weisshorn11 CROND[127333]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 15:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1273 starting 0anacron Jun 19 15:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1273 finished 0anacron Jun 19 15:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1273 starting mcelog.cron Jun 19 15:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1273 finished mcelog.cron Jun 19 15:10:02 weisshorn11 CROND[127417]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 15:10:02 weisshorn11 sshd[127414]: Accepted publickey for root from 148.187.12.231 port 45323 ssh2 Jun 19 15:10:02 weisshorn11 sshd[127414]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 15:10:02 weisshorn11 sshd[127414]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 15:10:02 weisshorn11 sshd[127414]: pam_unix(sshd:session): session closed for user root Jun 19 15:20:01 weisshorn11 CROND[127463]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 15:30:01 weisshorn11 CROND[127492]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 15:36:45 weisshorn11 rhnsd[127550]: running program /usr/sbin/rhn_check Jun 19 15:40:01 weisshorn11 CROND[127559]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 15:46:17 weisshorn11 nslcd[3954]: [49361f] ldap_result() failed: Can't contact LDAP server Jun 19 15:50:01 weisshorn11 CROND[127587]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 15:51:01 weisshorn11 nslcd[3954]: [bc6770] ldap_result() failed: Can't contact LDAP server Jun 19 16:00:01 weisshorn11 CROND[127651]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 16:01:01 weisshorn11 CROND[127656]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 16:01:02 weisshorn11 run-parts(/etc/cron.hourly)[1276 starting 0anacron Jun 19 16:01:02 weisshorn11 run-parts(/etc/cron.hourly)[1276 finished 0anacron Jun 19 16:01:02 weisshorn11 run-parts(/etc/cron.hourly)[1276 starting mcelog.cron Jun 19 16:01:02 weisshorn11 run-parts(/etc/cron.hourly)[1276 finished mcelog.cron Jun 19 16:10:01 weisshorn11 CROND[127700]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 16:10:02 weisshorn11 sshd[127702]: Accepted publickey for root from 148.187.12.231 port 45334 ssh2 Jun 19 16:10:02 weisshorn11 sshd[127702]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 16:10:02 weisshorn11 sshd[127702]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 16:10:02 weisshorn11 sshd[127702]: pam_unix(sshd:session): session closed for user root Jun 19 16:20:01 weisshorn11 CROND[127748]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 16:30:01 weisshorn11 CROND[127812]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 16:34:19 weisshorn11 ntpd[4454]: synchronized to 148.187.18.145, stratum 2 Jun 19 16:40:01 weisshorn11 CROND[127843]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 16:46:32 weisshorn11 nslcd[3954]: [0e6897] ldap_result() failed: Can't contact LDAP server Jun 19 16:50:01 weisshorn11 CROND[127882]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 16:51:16 weisshorn11 nslcd[3954]: [5138de] ldap_result() failed: Can't contact LDAP server Jun 19 17:00:01 weisshorn11 CROND[127937]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 17:01:01 weisshorn11 CROND[127947]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 17:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1279 starting 0anacron Jun 19 17:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1279 finished 0anacron Jun 19 17:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1279 starting mcelog.cron Jun 19 17:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1279 finished mcelog.cron Jun 19 17:10:01 weisshorn11 CROND[127994]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 17:10:02 weisshorn11 sshd[127996]: Accepted publickey for root from 148.187.12.231 port 45352 ssh2 Jun 19 17:10:02 weisshorn11 sshd[127996]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 17:10:02 weisshorn11 sshd[127996]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 17:10:02 weisshorn11 sshd[127996]: pam_unix(sshd:session): session closed for user root Jun 19 17:20:01 weisshorn11 CROND[128044]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 17:30:01 weisshorn11 CROND[128103]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 17:40:01 weisshorn11 CROND[128136]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 17:40:55 weisshorn11 kernel: Lustre: scratch-OST0044: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880800ab0800, cur 1340120455 expire 1340120305 last 1340120228 Jun 19 17:40:55 weisshorn11 kernel: Lustre: Skipped 3 previous similar messages Jun 19 17:41:04 weisshorn11 kernel: Lustre: scratch-OST0015: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880696b33400, cur 1340120464 expire 1340120314 last 1340120237 Jun 19 17:41:04 weisshorn11 kernel: Lustre: Skipped 2 previous similar messages Jun 19 17:42:10 weisshorn11 kernel: Lustre: scratch-OST0039: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88064f251000, cur 1340120530 expire 1340120380 last 1340120303 Jun 19 17:45:05 weisshorn11 kernel: Lustre: scratch-OST0015: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881025cff800, cur 1340120705 expire 1340120555 last 1340120478 Jun 19 17:45:09 weisshorn11 kernel: Lustre: scratch-OST0021: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880623787000, cur 1340120709 expire 1340120559 last 1340120482 Jun 19 17:45:09 weisshorn11 kernel: Lustre: Skipped 1 previous similar message Jun 19 17:45:18 weisshorn11 kernel: Lustre: scratch-OST0044: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880950f64800, cur 1340120718 expire 1340120568 last 1340120491 Jun 19 17:45:18 weisshorn11 kernel: Lustre: Skipped 1 previous similar message Jun 19 17:45:30 weisshorn11 kernel: Lustre: scratch-OST0008: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880818106800, cur 1340120730 expire 1340120580 last 1340120503 Jun 19 17:46:08 weisshorn11 kernel: Lustre: scratch-OST0039: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880804b77c00, cur 1340120768 expire 1340120618 last 1340120541 Jun 19 17:46:08 weisshorn11 kernel: Lustre: Skipped 1 previous similar message Jun 19 17:48:13 weisshorn11 kernel: Lustre: scratch-OST002d: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d3b057c00, cur 1340120893 expire 1340120743 last 1340120666 Jun 19 17:49:53 weisshorn11 kernel: Lustre: scratch-OST0008: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880802b5dc00, cur 1340120993 expire 1340120843 last 1340120766 Jun 19 17:50:01 weisshorn11 CROND[128169]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 17:52:10 weisshorn11 kernel: Lustre: scratch-OST0021: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101622c800, cur 1340121130 expire 1340120980 last 1340120903 Jun 19 17:52:10 weisshorn11 kernel: Lustre: Skipped 2 previous similar messages Jun 19 17:59:15 weisshorn11 kernel: Lustre: scratch-OST0021: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881016182800, cur 1340121555 expire 1340121405 last 1340121328 Jun 19 17:59:15 weisshorn11 kernel: Lustre: Skipped 4 previous similar messages Jun 19 18:00:01 weisshorn11 nslcd[3954]: [22ad0c] ldap_result() failed: Can't contact LDAP server Jun 19 18:00:01 weisshorn11 CROND[128235]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 18:01:01 weisshorn11 CROND[128239]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 18:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1282 starting 0anacron Jun 19 18:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1282 finished 0anacron Jun 19 18:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1282 starting mcelog.cron Jun 19 18:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1282 finished mcelog.cron Jun 19 18:09:03 weisshorn11 kernel: Lustre: scratch-OST0021: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b4946dc00, cur 1340122143 expire 1340121993 last 1340121916 Jun 19 18:09:03 weisshorn11 kernel: Lustre: Skipped 3 previous similar messages Jun 19 18:10:01 weisshorn11 sshd[128280]: Accepted publickey for root from 148.187.12.231 port 45373 ssh2 Jun 19 18:10:01 weisshorn11 sshd[128280]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 18:10:01 weisshorn11 CROND[128293]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 18:10:01 weisshorn11 sshd[128280]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 18:10:01 weisshorn11 sshd[128280]: pam_unix(sshd:session): session closed for user root Jun 19 18:20:01 weisshorn11 CROND[128365]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 18:25:17 weisshorn11 nslcd[3954]: [997556] ldap_result() failed: Can't contact LDAP server Jun 19 18:25:55 weisshorn11 kernel: Lustre: scratch-OST0008: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808015ee800, cur 1340123155 expire 1340123005 last 1340122928 Jun 19 18:25:55 weisshorn11 kernel: Lustre: Skipped 1 previous similar message Jun 19 18:30:01 weisshorn11 CROND[128397]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 18:40:01 weisshorn11 CROND[128424]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 18:45:18 weisshorn11 kernel: Lustre: scratch-OST0021: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c05248c00, cur 1340124318 expire 1340124168 last 1340124091 Jun 19 18:45:18 weisshorn11 kernel: Lustre: Skipped 1 previous similar message Jun 19 18:50:01 weisshorn11 CROND[128489]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 19:00:01 weisshorn11 CROND[128519]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 19:00:16 weisshorn11 nslcd[3954]: [333c89] ldap_result() failed: Can't contact LDAP server Jun 19 19:01:01 weisshorn11 CROND[128524]: (root) CMD (run-parts /etc/cron.hourly) Jun 19 19:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1285 starting 0anacron Jun 19 19:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1285 finished 0anacron Jun 19 19:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1285 starting mcelog.cron Jun 19 19:01:01 weisshorn11 run-parts(/etc/cron.hourly)[1285 finished mcelog.cron Jun 19 19:06:19 weisshorn11 kernel: Lustre: scratch-OST0015: Client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) reconnecting Jun 19 19:06:19 weisshorn11 kernel: Lustre: Skipped 4 previous similar messages Jun 19 19:06:19 weisshorn11 kernel: Lustre: scratch-OST0015: Client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) refused reconnection, still busy with 1 active RPCs Jun 19 19:06:19 weisshorn11 kernel: Lustre: Skipped 1 previous similar message Jun 19 19:06:43 weisshorn11 kernel: Lustre: scratch-OST0015: Client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) reconnecting Jun 19 19:08:44 weisshorn11 kernel: Lustre: scratch-OST0015: Client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) reconnecting Jun 19 19:08:44 weisshorn11 kernel: Lustre: scratch-OST0015: Client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) refused reconnection, still busy with 1 active RPCs Jun 19 19:08:49 weisshorn11 kernel: Lustre: scratch-OST0021: haven't heard from client 8b2888d9-d18e-04dc-f5db-2c5d5cc76db7 (at 148.187.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880334a57000, cur 1340125729 expire 1340125579 last 1340125502 Jun 19 19:09:20 weisshorn11 kernel: Lustre: scratch-OST0015: Client 1d3575c8-d2c3-3917-0d4b-ae0b2c9af8d8 (at 148.187.6.237@o2ib2) reconnecting Jun 19 19:10:01 weisshorn11 CROND[128577]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 19:10:02 weisshorn11 sshd[128578]: Accepted publickey for root from 148.187.12.231 port 45380 ssh2 Jun 19 19:10:02 weisshorn11 sshd[128578]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 19 19:10:02 weisshorn11 sshd[128578]: Received disconnect from 148.187.12.231: 11: disconnected by user Jun 19 19:10:02 weisshorn11 sshd[128578]: pam_unix(sshd:session): session closed for user root Jun 19 19:20:01 weisshorn11 CROND[128656]: (root) CMD (/usr/lib64/sa/sa1 -S DISK 1 1) Jun 19 19:22:35 weisshorn11 kernel: Lustre: scratch-OST0039: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805b03ec000, cur 1340126555 expire 1340126405 last 1340126328 Jun 19 19:24:15 weisshorn11 kernel: Lustre: scratch-OST0008: haven't heard from client 2a0b41f6-aa96-1ba0-1f63-b9a7076f8b03 (at 148.187.6.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808242cc000, cur 1340126655 expire 1340126505 last 1340126428 Jun 19 19:25:32 weisshorn11 nslcd[3954]: [a9831a] ldap_result() failed: Can't contact LDAP server Jun 19 19:27:46 weisshorn11 kernel: ------------[ cut here ]------------ Jun 19 19:27:46 weisshorn11 kernel: kernel BUG at /var/lib/jenkins/workspace/lustre-master/arch/x86_64/build_type/server/distro/el6/ib_stack/inkernel/BUILD/BUILD/lustre-2.2.51/lustre/lvfs/fsfilt-ldiskfs.c:1027! Jun 19 19:27:46 weisshorn11 kernel: invalid opcode: 0000 [#1] SMP Jun 19 19:27:46 weisshorn11 kernel: last sysfs file: /sys/devices/system/cpu/cpu31/cache/index2/shared_cpu_map Jun 19 19:27:46 weisshorn11 kernel: CPU 8 Jun 19 19:27:46 weisshorn11 kernel: Modules linked in: lnet_selftest(U) ib_uverbs obdfilter(U) fsfilt_ldiskfs(U) exportfs ost(U) mgc(U) lustre(U) lquota(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) ldiskfs(U) ko2iblnd(U) rdma_cm iw_cm ib_addr obdclass(U) lnet(U) lvfs(U) libcfs(U) autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf dm_round_robin ib_ipoib ib_cm ib_sa ipv6 microcode sg sb_edac edac_core iTCO_wdt iTCO_vendor_support i2c_i801 i2c_core mlx4_ib ib_mad ib_core mlx4_en mlx4_core ioatdma igb dca ext4 mbcache jbd2 sd_mod crc_t10dif qla2xxx scsi_transport_fc scsi_tgt ahci isci libsas scsi_transport_sas wmi dm_multipath dm_mirror dm_region_hash dm_log dm_mod scsi_dh_rdac [last unloaded: scsi_wait_scan] Jun 19 19:27:46 weisshorn11 kernel: Jun 19 19:27:46 weisshorn11 kernel: Pid: 7800, comm: ll_ost_io_296 Not tainted 2.6.32-220.7.1.el6_lustre.g9c8f747.x86_64 #1 Intel Corporation S2600GZ/S2600GZ Jun 19 19:27:46 weisshorn11 kernel: RIP: 0010:[] [] ldiskfs_ext_new_extent_cb+0x587/0x5f0 [fsfilt_ldiskfs] Jun 19 19:27:46 weisshorn11 kernel: RSP: 0018:ffff8807bfd7b620 EFLAGS: 00010246 Jun 19 19:27:46 weisshorn11 kernel: RAX: 0000000000000038 RBX: ffff8807bfd7b780 RCX: 0000000000000008 Jun 19 19:27:46 weisshorn11 kernel: RDX: ffff880e9b5bc200 RSI: ffff880d25d06340 RDI: ffff880e9b5bc2f8 Jun 19 19:27:46 weisshorn11 kernel: RBP: ffff8807bfd7b6c0 R08: ffff8807bfd7b780 R09: ffff880d25d06340 Jun 19 19:27:46 weisshorn11 kernel: R10: ffff880f8aab8000 R11: 0000000000000001 R12: ffff8807bfd7b700 Jun 19 19:27:46 weisshorn11 kernel: R13: ffff880e9b5bc2f8 R14: ffff880d25d06340 R15: ffff880e9b5bc2f8 Jun 19 19:27:46 weisshorn11 kernel: FS: 00007f790ee17700(0000) GS:ffff88085c400000(0000) knlGS:0000000000000000 Jun 19 19:27:46 weisshorn11 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Jun 19 19:27:46 weisshorn11 kernel: CR2: 00007f98e308d030 CR3: 0000000821784000 CR4: 00000000000406e0 Jun 19 19:27:46 weisshorn11 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jun 19 19:27:46 weisshorn11 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jun 19 19:27:46 weisshorn11 kernel: Process ll_ost_io_296 (pid: 7800, threadinfo ffff8807bfd7a000, task ffff8807c0d94a80) Jun 19 19:27:46 weisshorn11 kernel: Stack: Jun 19 19:27:46 weisshorn11 kernel: ffff880e9b5bc224 ffff880e9b5bc218 0000000000000000 ffffffff8100bdae Jun 19 19:27:46 weisshorn11 kernel: <0> ffff8807bfd7b6c0 0000000000000001 ffff880f8aab8000 ffff880d25d06340 Jun 19 19:27:46 weisshorn11 kernel: <0> 0000000000000000 ffff880e9b5bc2d8 0000000000000009 0000000000000001 Jun 19 19:27:46 weisshorn11 kernel: Call Trace: Jun 19 19:27:46 weisshorn11 kernel: [] ? call_function_single_interrupt+0xe/0x20 Jun 19 19:27:46 weisshorn11 kernel: [] ldiskfs_ext_walk_space+0x14f/0x340 [ldiskfs] Jun 19 19:27:46 weisshorn11 kernel: [] ? ldiskfs_ext_new_extent_cb+0x0/0x5f0 [fsfilt_ldiskfs] Jun 19 19:27:46 weisshorn11 kernel: [] fsfilt_map_nblocks+0xa3/0xd0 [fsfilt_ldiskfs] Jun 19 19:27:46 weisshorn11 kernel: [] fsfilt_ldiskfs_map_ext_inode_pages+0x123/0x1e0 [fsfilt_ldiskfs] Jun 19 19:27:46 weisshorn11 kernel: [] fsfilt_ldiskfs_map_inode_pages+0x85/0x90 [fsfilt_ldiskfs] Jun 19 19:27:46 weisshorn11 kernel: [] ? add_to_page_cache_lru+0x68/0x80 Jun 19 19:27:46 weisshorn11 kernel: [] filter_do_bio+0xf36/0x1570 [obdfilter] Jun 19 19:27:46 weisshorn11 kernel: [] ? find_or_create_page+0x68/0xb0 Jun 19 19:27:46 weisshorn11 kernel: [] ? read_tsc+0x9/0x20 Jun 19 19:27:46 weisshorn11 kernel: [] filter_preprw_read+0x673/0xb20 [obdfilter] Jun 19 19:27:46 weisshorn11 kernel: [] ? null_alloc_rs+0x172/0x2e0 [ptlrpc] Jun 19 19:27:46 weisshorn11 kernel: [] ? sptlrpc_svc_alloc_rs+0x7e/0x2a0 [ptlrpc] Jun 19 19:27:46 weisshorn11 kernel: [] filter_preprw+0x40/0x80 [obdfilter] Jun 19 19:27:46 weisshorn11 kernel: [] obd_preprw+0x128/0x2f0 [ost] Jun 19 19:27:46 weisshorn11 kernel: [] ost_brw_read+0xbf4/0x1120 [ost] Jun 19 19:27:46 weisshorn11 kernel: [] ? lustre_msg_get_version+0x54/0x90 [ptlrpc] Jun 19 19:27:46 weisshorn11 kernel: [] ? lustre_msg_check_version+0x78/0x90 [ptlrpc] Jun 19 19:27:46 weisshorn11 kernel: [] ost_handle+0x1da0/0x39e0 [ost] Jun 19 19:27:46 weisshorn11 kernel: [] ? lustre_msg_get_transno+0x54/0x90 [ptlrpc] Jun 19 19:27:46 weisshorn11 kernel: [] ? ptlrpc_update_export_timer+0x55/0x360 [ptlrpc] Jun 19 19:27:46 weisshorn11 kernel: [] ptlrpc_server_handle_request+0x3c1/0xcb0 [ptlrpc] Jun 19 19:27:46 weisshorn11 kernel: [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jun 19 19:27:46 weisshorn11 kernel: [] ? lc_watchdog_touch+0x79/0x110 [libcfs] Jun 19 19:27:46 weisshorn11 kernel: [] ? ptlrpc_wait_event+0xb2/0x2c0 [ptlrpc] Jun 19 19:27:46 weisshorn11 kernel: [] ? __wake_up+0x53/0x70 Jun 19 19:27:46 weisshorn11 kernel: [] ptlrpc_main+0x71f/0x1210 [ptlrpc] Jun 19 19:27:46 weisshorn11 kernel: [] ? ptlrpc_main+0x0/0x1210 [ptlrpc] Jun 19 19:27:46 weisshorn11 kernel: [] child_rip+0xa/0x20 Jun 19 19:27:46 weisshorn11 kernel: [] ? ptlrpc_main+0x0/0x1210 [ptlrpc] Jun 19 19:27:46 weisshorn11 kernel: [] ? ptlrpc_main+0x0/0x1210 [ptlrpc] Jun 19 19:27:46 weisshorn11 kernel: [] ? child_rip+0x0/0x20 Jun 19 19:29:15 weisshorn12 heartbeat: [5359]: WARN: node weisshorn11.admin.cscs.ch: is dead Jun 19 19:29:15 weisshorn12 heartbeat: [5359]: info: Resources being acquired from weisshorn11.admin.cscs.ch. Jun 19 19:29:15 weisshorn12 heartbeat: [5359]: info: Link weisshorn11.admin.cscs.ch:eth4 dead. Jun 19 19:29:15 weisshorn12 heartbeat: [5359]: info: Link weisshorn11.admin.cscs.ch:ib0 dead. Jun 19 19:29:15 weisshorn12 ResourceManager[125658]: info: Acquiring resource group: weisshorn11.admin.cscs.ch Filesystem::/dev/mapper/Lun00::/mnt/Lun00::lustre Filesystem::/dev/mapper/Lun02::/mnt/Lun02::lustre Filesystem::/dev/mapper/Lun04::/mnt/Lun04::lustre Filesystem::/dev/mapper/Lun06::/mnt/Lun06::lustre Filesystem::/dev/mapper/Lun08::/mnt/Lun08::lustre Filesystem::/dev/mapper/Lun10::/mnt/Lun10::lustre Jun 19 19:38:12 weisshorn12 mach_down[125619]: info: mach_down takeover complete for node weisshorn11.admin.cscs.ch.