May 9 03:48:01 uc1n129 anacron[91116]: Job `cron.daily' started May 9 03:48:02 uc1n129 run-parts(/etc/cron.daily)[91269]: starting cups May 9 03:48:02 uc1n129 run-parts(/etc/cron.daily)[91277]: finished cups May 9 03:48:02 uc1n129 run-parts(/etc/cron.daily)[91269]: starting ibchecknet_cron.sh May 9 03:48:02 uc1n129 run-parts(/etc/cron.daily)[91296]: finished ibchecknet_cron.sh May 9 03:48:02 uc1n129 run-parts(/etc/cron.daily)[91269]: starting logrotate May 9 03:48:02 uc1n129 run-parts(/etc/cron.daily)[91303]: finished logrotate May 9 03:48:02 uc1n129 run-parts(/etc/cron.daily)[91269]: starting lustre-mk-diskusage May 9 03:48:02 uc1n129 run-parts(/etc/cron.daily)[91309]: finished lustre-mk-diskusage May 9 03:48:02 uc1n129 run-parts(/etc/cron.daily)[91269]: starting makewhatis.cron May 9 03:48:02 uc1n129 run-parts(/etc/cron.daily)[91469]: finished makewhatis.cron May 9 03:48:02 uc1n129 run-parts(/etc/cron.daily)[91269]: starting prelink May 9 03:48:03 uc1n129 run-parts(/etc/cron.daily)[91542]: finished prelink May 9 03:48:03 uc1n129 run-parts(/etc/cron.daily)[91269]: starting readahead.cron May 9 03:48:03 uc1n129 run-parts(/etc/cron.daily)[91553]: finished readahead.cron May 9 03:48:03 uc1n129 run-parts(/etc/cron.daily)[91269]: starting tmpwatch May 9 03:48:03 uc1n129 run-parts(/etc/cron.daily)[91591]: finished tmpwatch May 9 03:48:03 uc1n129 anacron[91116]: Job `cron.daily' terminated May 9 03:50:01 uc1n129 CROND[91595]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 04:00:01 uc1n129 CROND[91609]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 04:01:01 uc1n129 CROND[91612]: (root) CMD (run-parts /etc/cron.hourly) May 9 04:01:01 uc1n129 run-parts(/etc/cron.hourly)[91612]: starting 0anacron May 9 04:01:01 uc1n129 run-parts(/etc/cron.hourly)[91621]: finished 0anacron May 9 04:01:01 uc1n129 run-parts(/etc/cron.hourly)[91612]: starting mcelog.cron May 9 04:01:01 uc1n129 run-parts(/etc/cron.hourly)[91635]: finished mcelog.cron May 9 04:01:01 uc1n129 run-parts(/etc/cron.hourly)[91612]: starting nscd_restart_cron.sh May 9 04:01:01 uc1n129 run-parts(/etc/cron.hourly)[91659]: finished nscd_restart_cron.sh May 9 04:08:01 uc1n129 anacron[91116]: Job `cron.weekly' started May 9 04:08:02 uc1n129 anacron[91116]: Job `cron.weekly' terminated May 9 04:08:02 uc1n129 anacron[91116]: Normal exit (2 jobs run) May 9 04:10:01 uc1n129 CROND[91674]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 04:20:01 uc1n129 CROND[91695]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 04:23:46 uc1n129 smartd[3295]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 72 to 73 May 9 04:23:46 uc1n129 smartd[3295]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 28 to 27 May 9 04:30:01 uc1n129 CROND[91709]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 04:40:01 uc1n129 CROND[91723]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 04:50:01 uc1n129 CROND[91759]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 05:00:01 uc1n129 CROND[91774]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 05:01:01 uc1n129 CROND[91777]: (root) CMD (run-parts /etc/cron.hourly) May 9 05:01:01 uc1n129 run-parts(/etc/cron.hourly)[91777]: starting 0anacron May 9 05:01:01 uc1n129 run-parts(/etc/cron.hourly)[91786]: finished 0anacron May 9 05:01:01 uc1n129 run-parts(/etc/cron.hourly)[91777]: starting mcelog.cron May 9 05:01:01 uc1n129 run-parts(/etc/cron.hourly)[91800]: finished mcelog.cron May 9 05:01:01 uc1n129 run-parts(/etc/cron.hourly)[91777]: starting nscd_restart_cron.sh May 9 05:01:02 uc1n129 run-parts(/etc/cron.hourly)[91824]: finished nscd_restart_cron.sh May 9 05:10:01 uc1n129 CROND[91854]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 05:20:01 uc1n129 CROND[91868]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 05:30:01 uc1n129 CROND[91882]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 05:40:01 uc1n129 CROND[91903]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 05:50:01 uc1n129 CROND[91917]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 06:00:01 uc1n129 CROND[91931]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 06:01:01 uc1n129 CROND[91934]: (root) CMD (run-parts /etc/cron.hourly) May 9 06:01:01 uc1n129 run-parts(/etc/cron.hourly)[91934]: starting 0anacron May 9 06:01:01 uc1n129 run-parts(/etc/cron.hourly)[91943]: finished 0anacron May 9 06:01:01 uc1n129 run-parts(/etc/cron.hourly)[91934]: starting mcelog.cron May 9 06:01:01 uc1n129 run-parts(/etc/cron.hourly)[91957]: finished mcelog.cron May 9 06:01:01 uc1n129 run-parts(/etc/cron.hourly)[91934]: starting nscd_restart_cron.sh May 9 06:01:01 uc1n129 run-parts(/etc/cron.hourly)[91981]: finished nscd_restart_cron.sh May 9 06:10:01 uc1n129 CROND[91994]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 06:20:01 uc1n129 CROND[92648]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 06:30:01 uc1n129 CROND[92661]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 06:40:01 uc1n129 CROND[92675]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 06:50:01 uc1n129 CROND[92751]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 07:00:01 uc1n129 CROND[92825]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 07:01:01 uc1n129 CROND[92844]: (root) CMD (run-parts /etc/cron.hourly) May 9 07:01:01 uc1n129 run-parts(/etc/cron.hourly)[92844]: starting 0anacron May 9 07:01:01 uc1n129 run-parts(/etc/cron.hourly)[92853]: finished 0anacron May 9 07:01:01 uc1n129 run-parts(/etc/cron.hourly)[92844]: starting mcelog.cron May 9 07:01:01 uc1n129 run-parts(/etc/cron.hourly)[92867]: finished mcelog.cron May 9 07:01:01 uc1n129 run-parts(/etc/cron.hourly)[92844]: starting nscd_restart_cron.sh May 9 07:01:01 uc1n129 run-parts(/etc/cron.hourly)[92891]: finished nscd_restart_cron.sh May 9 07:10:01 uc1n129 CROND[92934]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 07:20:01 uc1n129 CROND[92994]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 07:23:46 uc1n129 smartd[3295]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 73 to 72 May 9 07:23:46 uc1n129 smartd[3295]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 27 to 28 May 9 07:30:01 uc1n129 CROND[93064]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 07:40:01 uc1n129 CROND[93109]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 07:50:01 uc1n129 CROND[109811]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 08:00:01 uc1n129 CROND[9041]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 08:01:01 uc1n129 CROND[10143]: (root) CMD (run-parts /etc/cron.hourly) May 9 08:01:01 uc1n129 run-parts(/etc/cron.hourly)[10143]: starting 0anacron May 9 08:01:01 uc1n129 run-parts(/etc/cron.hourly)[10152]: finished 0anacron May 9 08:01:01 uc1n129 run-parts(/etc/cron.hourly)[10143]: starting mcelog.cron May 9 08:01:01 uc1n129 run-parts(/etc/cron.hourly)[10166]: finished mcelog.cron May 9 08:01:01 uc1n129 run-parts(/etc/cron.hourly)[10143]: starting nscd_restart_cron.sh May 9 08:01:01 uc1n129 run-parts(/etc/cron.hourly)[10190]: finished nscd_restart_cron.sh May 9 08:10:01 uc1n129 CROND[35854]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 08:20:01 uc1n129 CROND[70259]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 08:30:01 uc1n129 CROND[100973]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 08:40:01 uc1n129 CROND[129323]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 08:50:01 uc1n129 CROND[23598]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 09:00:01 uc1n129 CROND[41068]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 09:01:01 uc1n129 CROND[42710]: (root) CMD (run-parts /etc/cron.hourly) May 9 09:01:01 uc1n129 run-parts(/etc/cron.hourly)[42710]: starting 0anacron May 9 09:01:01 uc1n129 run-parts(/etc/cron.hourly)[42719]: finished 0anacron May 9 09:01:01 uc1n129 run-parts(/etc/cron.hourly)[42710]: starting mcelog.cron May 9 09:01:01 uc1n129 run-parts(/etc/cron.hourly)[42733]: finished mcelog.cron May 9 09:01:01 uc1n129 run-parts(/etc/cron.hourly)[42710]: starting nscd_restart_cron.sh May 9 09:01:01 uc1n129 run-parts(/etc/cron.hourly)[42757]: finished nscd_restart_cron.sh May 9 09:06:28 uc1n129 sshd[52050]: Accepted publickey for root from 10.20.7.229 port 48704 ssh2 May 9 09:06:28 uc1n129 sshd[52050]: pam_unix(sshd:session): session opened for user root by (uid=0) May 9 09:06:28 uc1n129 sshd[52050]: Received disconnect from 10.20.7.229: 11: disconnected by user May 9 09:06:28 uc1n129 sshd[52050]: pam_unix(sshd:session): session closed for user root May 9 09:10:01 uc1n129 CROND[61336]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 09:14:39 uc1n129 sshd[74526]: Accepted publickey for root from 10.20.7.231 port 46513 ssh2 May 9 09:14:39 uc1n129 sshd[74526]: pam_unix(sshd:session): session opened for user root by (uid=0) May 9 09:14:46 uc1n129 sshd[74526]: Received disconnect from 10.20.7.231: 11: disconnected by user May 9 09:14:46 uc1n129 sshd[74526]: pam_unix(sshd:session): session closed for user root May 9 09:20:01 uc1n129 CROND[85301]: (root) CMD (/usr/lib64/sa/sa1 1 1) May 9 09:21:08 uc1n129 kernel: LustreError: 93767:0:(statahead.c:1698:do_statahead_enter()) can't start ll_sa thread, rc: -2752 May 9 09:21:08 uc1n129 kernel: LustreError: 93767:0:(statahead.c:1704:do_statahead_enter()) ASSERTION( lli->u.d.d_sai == ((void *)0) ) failed: May 9 09:21:08 uc1n129 kernel: LustreError: 93767:0:(statahead.c:1704:do_statahead_enter()) LBUG May 9 09:21:08 uc1n129 kernel: Pid: 93767, comm: lesTableFoam May 9 09:21:08 uc1n129 kernel: May 9 09:21:08 uc1n129 kernel: Call Trace: May 9 09:21:08 uc1n129 kernel: [] libcfs_debug_dumpstack+0x55/0x80 [libcfs] May 9 09:21:08 uc1n129 kernel: [] lbug_with_loc+0x47/0xb0 [libcfs] May 9 09:21:08 uc1n129 kernel: [] do_statahead_enter+0x1717/0x1890 [lustre] May 9 09:21:08 uc1n129 kernel: [] ? ldlm_lock_add_to_lru_nolock+0x4a/0x110 [ptlrpc] May 9 09:21:08 uc1n129 kernel: [] ? ldlm_lock_add_to_lru+0x43/0x120 [ptlrpc] May 9 09:21:08 uc1n129 kernel: [] ? class_handle2object+0x95/0x190 [obdclass] May 9 09:21:08 uc1n129 kernel: [] ? ldlm_lock_decref+0x41/0x90 [ptlrpc] May 9 09:21:08 uc1n129 kernel: [] ll_revalidate_it+0xf97/0x1b20 [lustre] May 9 09:21:08 uc1n129 kernel: [] ll_revalidate_nd+0x133/0x3e0 [lustre] May 9 09:21:08 uc1n129 kernel: [] do_lookup+0x66/0x230 May 9 09:21:08 uc1n129 kernel: [] __link_path_walk+0x794/0xff0 May 9 09:21:08 uc1n129 kernel: [] ? generic_permission+0x23/0xb0 May 9 09:21:08 uc1n129 kernel: [] path_walk+0x6a/0xe0 May 9 09:21:08 uc1n129 kernel: [] filename_lookup+0x6b/0xc0 May 9 09:21:08 uc1n129 kernel: [] user_path_at+0x57/0xa0 May 9 09:21:08 uc1n129 kernel: [] ? touch_atime+0x71/0x1a0 May 9 09:21:08 uc1n129 kernel: [] ? current_fs_time+0x27/0x30 May 9 09:21:08 uc1n129 kernel: [] ? ll_stats_ops_tally+0x7b/0xa0 [lustre] May 9 09:21:08 uc1n129 kernel: [] vfs_fstatat+0x50/0xa0 May 9 09:21:08 uc1n129 kernel: [] ? touch_atime+0x71/0x1a0