Nov 4 00:00:01 lola-4 CROND[26808]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 00:00:01 lola-4 CROND[26809]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 00:00:01 lola-4 TIME: Time stamp for console Nov 4 00:01:01 lola-4 CROND[26824]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 00:01:01 lola-4 run-parts(/etc/cron.hourly)[2682 starting 0anacron Nov 4 00:01:01 lola-4 anacron[26835]: Anacron started on 2016-11-04 Nov 4 00:01:01 lola-4 run-parts(/etc/cron.hourly)[2683 finished 0anacron Nov 4 00:01:01 lola-4 anacron[26835]: Jobs will be executed sequentially Nov 4 00:01:01 lola-4 anacron[26835]: Normal exit (0 jobs run) Nov 4 00:05:01 lola-4 CROND[26864]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 00:05:01 lola-4 TIME: Time stamp for console Nov 4 00:10:01 lola-4 CROND[26898]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 00:10:01 lola-4 CROND[26899]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 00:10:01 lola-4 TIME: Time stamp for console Nov 4 00:15:01 lola-4 CROND[26935]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 00:15:01 lola-4 TIME: Time stamp for console Nov 4 00:20:01 lola-4 CROND[26969]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 00:20:01 lola-4 CROND[26970]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 00:20:01 lola-4 TIME: Time stamp for console Nov 4 00:25:01 lola-4 CROND[27006]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 00:25:01 lola-4 TIME: Time stamp for console Nov 4 00:30:01 lola-4 CROND[27040]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 00:30:01 lola-4 TIME: Time stamp for console Nov 4 00:30:01 lola-4 CROND[27041]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 00:35:01 lola-4 CROND[27078]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 00:35:01 lola-4 TIME: Time stamp for console Nov 4 00:40:01 lola-4 CROND[27112]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 00:40:01 lola-4 CROND[27113]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 00:40:01 lola-4 TIME: Time stamp for console Nov 4 00:45:01 lola-4 CROND[27149]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 00:45:01 lola-4 TIME: Time stamp for console Nov 4 00:50:01 lola-4 CROND[27183]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 00:50:01 lola-4 CROND[27184]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 00:50:01 lola-4 TIME: Time stamp for console Nov 4 00:55:01 lola-4 CROND[27220]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 00:55:01 lola-4 TIME: Time stamp for console Nov 4 01:00:01 lola-4 CROND[27254]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 01:00:01 lola-4 CROND[27255]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 01:00:01 lola-4 TIME: Time stamp for console Nov 4 01:01:01 lola-4 CROND[27268]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 01:01:01 lola-4 run-parts(/etc/cron.hourly)[2726 starting 0anacron Nov 4 01:01:01 lola-4 anacron[27279]: Anacron started on 2016-11-04 Nov 4 01:01:01 lola-4 anacron[27279]: Jobs will be executed sequentially Nov 4 01:01:01 lola-4 anacron[27279]: Normal exit (0 jobs run) Nov 4 01:01:01 lola-4 run-parts(/etc/cron.hourly)[2728 finished 0anacron Nov 4 01:05:01 lola-4 CROND[27309]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 01:05:01 lola-4 TIME: Time stamp for console Nov 4 01:10:01 lola-4 CROND[27343]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 01:10:01 lola-4 CROND[27344]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 01:10:01 lola-4 TIME: Time stamp for console Nov 4 01:15:01 lola-4 CROND[27380]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 01:15:01 lola-4 TIME: Time stamp for console Nov 4 01:20:01 lola-4 CROND[27414]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 01:20:01 lola-4 CROND[27415]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 01:20:01 lola-4 TIME: Time stamp for console Nov 4 01:25:02 lola-4 CROND[27451]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 01:25:02 lola-4 TIME: Time stamp for console Nov 4 01:30:01 lola-4 CROND[27485]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 01:30:01 lola-4 CROND[27486]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 01:30:01 lola-4 TIME: Time stamp for console Nov 4 01:35:01 lola-4 CROND[27522]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 01:35:01 lola-4 TIME: Time stamp for console Nov 4 01:40:01 lola-4 CROND[27557]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 01:40:01 lola-4 CROND[27558]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 01:40:01 lola-4 TIME: Time stamp for console Nov 4 01:45:01 lola-4 CROND[27594]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 01:45:01 lola-4 TIME: Time stamp for console Nov 4 01:50:01 lola-4 CROND[27628]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 01:50:01 lola-4 CROND[27629]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 01:50:01 lola-4 TIME: Time stamp for console Nov 4 01:55:01 lola-4 CROND[27665]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 01:55:01 lola-4 TIME: Time stamp for console Nov 4 02:00:01 lola-4 CROND[27699]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 02:00:01 lola-4 TIME: Time stamp for console Nov 4 02:00:01 lola-4 CROND[27700]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 02:01:01 lola-4 CROND[27713]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 02:01:01 lola-4 run-parts(/etc/cron.hourly)[2771 starting 0anacron Nov 4 02:01:01 lola-4 anacron[27724]: Anacron started on 2016-11-04 Nov 4 02:01:01 lola-4 anacron[27724]: Jobs will be executed sequentially Nov 4 02:01:01 lola-4 anacron[27724]: Normal exit (0 jobs run) Nov 4 02:01:01 lola-4 run-parts(/etc/cron.hourly)[2772 finished 0anacron Nov 4 02:05:01 lola-4 CROND[27753]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 02:05:01 lola-4 TIME: Time stamp for console Nov 4 02:10:01 lola-4 CROND[27787]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 02:10:01 lola-4 CROND[27788]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 02:10:01 lola-4 TIME: Time stamp for console Nov 4 02:15:01 lola-4 CROND[27825]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 02:15:01 lola-4 TIME: Time stamp for console Nov 4 02:20:01 lola-4 CROND[27859]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 02:20:01 lola-4 CROND[27860]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 02:20:01 lola-4 TIME: Time stamp for console Nov 4 02:25:01 lola-4 CROND[27896]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 02:25:01 lola-4 TIME: Time stamp for console Nov 4 02:30:01 lola-4 CROND[27930]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 02:30:01 lola-4 CROND[27931]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 02:30:01 lola-4 TIME: Time stamp for console Nov 4 02:35:01 lola-4 CROND[27967]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 02:35:01 lola-4 TIME: Time stamp for console Nov 4 02:40:01 lola-4 CROND[28001]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 02:40:01 lola-4 CROND[28002]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 02:40:01 lola-4 TIME: Time stamp for console Nov 4 02:45:01 lola-4 CROND[28039]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 02:45:01 lola-4 TIME: Time stamp for console Nov 4 02:50:01 lola-4 CROND[28073]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 02:50:01 lola-4 CROND[28074]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 02:50:01 lola-4 TIME: Time stamp for console Nov 4 02:55:01 lola-4 CROND[28110]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 02:55:01 lola-4 TIME: Time stamp for console Nov 4 03:00:01 lola-4 CROND[28144]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 03:00:01 lola-4 TIME: Time stamp for console Nov 4 03:00:01 lola-4 CROND[28145]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 03:01:01 lola-4 CROND[28158]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 03:01:01 lola-4 run-parts(/etc/cron.hourly)[2815 starting 0anacron Nov 4 03:01:01 lola-4 anacron[28169]: Anacron started on 2016-11-04 Nov 4 03:01:01 lola-4 anacron[28169]: Will run job `cron.daily' in 22 min. Nov 4 03:01:01 lola-4 anacron[28169]: Jobs will be executed sequentially Nov 4 03:01:01 lola-4 run-parts(/etc/cron.hourly)[2817 finished 0anacron Nov 4 03:05:01 lola-4 CROND[28198]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 03:05:01 lola-4 TIME: Time stamp for console Nov 4 03:10:01 lola-4 CROND[28232]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 03:10:01 lola-4 CROND[28233]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 03:10:01 lola-4 TIME: Time stamp for console Nov 4 03:15:01 lola-4 CROND[28269]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 03:15:01 lola-4 TIME: Time stamp for console Nov 4 03:20:01 lola-4 CROND[28304]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 03:20:01 lola-4 CROND[28305]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 03:20:01 lola-4 TIME: Time stamp for console Nov 4 03:23:01 lola-4 anacron[28169]: Job `cron.daily' started Nov 4 03:23:01 lola-4 run-parts(/etc/cron.daily)[28327 starting logrotate Nov 4 03:23:02 lola-4 run-parts(/etc/cron.daily)[28336 finished logrotate Nov 4 03:23:02 lola-4 run-parts(/etc/cron.daily)[28327 starting makewhatis.cron Nov 4 03:23:04 lola-4 run-parts(/etc/cron.daily)[28847 finished makewhatis.cron Nov 4 03:23:04 lola-4 run-parts(/etc/cron.daily)[28327 starting prelink Nov 4 03:23:33 lola-4 run-parts(/etc/cron.daily)[30535 finished prelink Nov 4 03:23:33 lola-4 run-parts(/etc/cron.daily)[28327 starting readahead.cron Nov 4 03:23:33 lola-4 run-parts(/etc/cron.daily)[30547 finished readahead.cron Nov 4 03:23:33 lola-4 run-parts(/etc/cron.daily)[28327 starting tmpwatch Nov 4 03:23:33 lola-4 run-parts(/etc/cron.daily)[30585 finished tmpwatch Nov 4 03:23:33 lola-4 anacron[28169]: Job `cron.daily' terminated Nov 4 03:23:33 lola-4 anacron[28169]: Normal exit (1 job run) Nov 4 03:25:01 lola-4 CROND[30598]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 03:25:01 lola-4 TIME: Time stamp for console Nov 4 03:30:01 lola-4 CROND[30632]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 03:30:01 lola-4 CROND[30633]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 03:30:01 lola-4 TIME: Time stamp for console Nov 4 03:35:01 lola-4 CROND[30669]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 03:35:01 lola-4 TIME: Time stamp for console Nov 4 03:40:01 lola-4 CROND[30703]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 03:40:01 lola-4 CROND[30704]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 03:40:01 lola-4 TIME: Time stamp for console Nov 4 03:45:01 lola-4 CROND[30740]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 03:45:01 lola-4 TIME: Time stamp for console Nov 4 03:50:01 lola-4 CROND[30774]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 03:50:01 lola-4 TIME: Time stamp for console Nov 4 03:50:01 lola-4 CROND[30775]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 03:55:01 lola-4 CROND[30812]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 03:55:01 lola-4 TIME: Time stamp for console Nov 4 04:00:01 lola-4 CROND[30846]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 04:00:01 lola-4 CROND[30847]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 04:00:01 lola-4 TIME: Time stamp for console Nov 4 04:01:01 lola-4 CROND[30860]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 04:01:01 lola-4 run-parts(/etc/cron.hourly)[3086 starting 0anacron Nov 4 04:01:01 lola-4 run-parts(/etc/cron.hourly)[3086 finished 0anacron Nov 4 04:05:01 lola-4 CROND[30896]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 04:05:01 lola-4 TIME: Time stamp for console Nov 4 04:10:01 lola-4 CROND[30930]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 04:10:01 lola-4 CROND[30931]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 04:10:01 lola-4 TIME: Time stamp for console Nov 4 04:15:01 lola-4 CROND[30967]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 04:15:01 lola-4 TIME: Time stamp for console Nov 4 04:20:01 lola-4 CROND[31001]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 04:20:01 lola-4 CROND[31002]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 04:20:01 lola-4 TIME: Time stamp for console Nov 4 04:25:01 lola-4 CROND[31039]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 04:25:01 lola-4 TIME: Time stamp for console Nov 4 04:30:01 lola-4 CROND[31073]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 04:30:01 lola-4 CROND[31074]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 04:30:01 lola-4 TIME: Time stamp for console Nov 4 04:35:01 lola-4 CROND[31110]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 04:35:01 lola-4 TIME: Time stamp for console Nov 4 04:40:01 lola-4 CROND[31144]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 04:40:01 lola-4 CROND[31145]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 04:40:01 lola-4 TIME: Time stamp for console Nov 4 04:45:02 lola-4 CROND[31181]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 04:45:02 lola-4 TIME: Time stamp for console Nov 4 04:50:01 lola-4 CROND[31215]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 04:50:01 lola-4 CROND[31216]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 04:50:01 lola-4 TIME: Time stamp for console Nov 4 04:55:01 lola-4 CROND[31252]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 04:55:01 lola-4 TIME: Time stamp for console Nov 4 05:00:01 lola-4 CROND[31287]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 05:00:01 lola-4 CROND[31288]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 05:00:01 lola-4 TIME: Time stamp for console Nov 4 05:01:01 lola-4 CROND[31301]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 05:01:01 lola-4 run-parts(/etc/cron.hourly)[3130 starting 0anacron Nov 4 05:01:01 lola-4 run-parts(/etc/cron.hourly)[3131 finished 0anacron Nov 4 05:05:01 lola-4 CROND[31337]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 05:05:01 lola-4 TIME: Time stamp for console Nov 4 05:10:01 lola-4 CROND[31371]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 05:10:01 lola-4 CROND[31372]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 05:10:01 lola-4 TIME: Time stamp for console Nov 4 05:15:01 lola-4 CROND[31408]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 05:15:01 lola-4 TIME: Time stamp for console Nov 4 05:20:01 lola-4 CROND[31442]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 05:20:01 lola-4 CROND[31443]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 05:20:01 lola-4 TIME: Time stamp for console Nov 4 05:25:01 lola-4 CROND[31479]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 05:25:01 lola-4 TIME: Time stamp for console Nov 4 05:30:01 lola-4 CROND[31513]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 05:30:01 lola-4 CROND[31514]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 05:30:01 lola-4 TIME: Time stamp for console Nov 4 05:35:01 lola-4 CROND[31551]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 05:35:01 lola-4 TIME: Time stamp for console Nov 4 05:40:01 lola-4 CROND[31585]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 05:40:01 lola-4 CROND[31586]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 05:40:01 lola-4 TIME: Time stamp for console Nov 4 05:45:01 lola-4 CROND[31622]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 05:45:01 lola-4 TIME: Time stamp for console Nov 4 05:50:01 lola-4 CROND[31656]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 05:50:01 lola-4 CROND[31657]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 05:50:01 lola-4 TIME: Time stamp for console Nov 4 05:55:01 lola-4 CROND[31693]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 05:55:01 lola-4 TIME: Time stamp for console Nov 4 06:00:01 lola-4 CROND[31727]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 06:00:01 lola-4 CROND[31728]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 06:00:01 lola-4 TIME: Time stamp for console Nov 4 06:01:01 lola-4 CROND[31741]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 06:01:01 lola-4 run-parts(/etc/cron.hourly)[3174 starting 0anacron Nov 4 06:01:01 lola-4 run-parts(/etc/cron.hourly)[3175 finished 0anacron Nov 4 06:05:01 lola-4 CROND[31778]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 06:05:01 lola-4 TIME: Time stamp for console Nov 4 06:10:01 lola-4 CROND[31812]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 06:10:01 lola-4 CROND[31813]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 06:10:01 lola-4 TIME: Time stamp for console Nov 4 06:15:01 lola-4 CROND[31849]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 06:15:01 lola-4 TIME: Time stamp for console Nov 4 06:20:01 lola-4 CROND[31883]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 06:20:01 lola-4 TIME: Time stamp for console Nov 4 06:20:01 lola-4 CROND[31884]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 06:25:01 lola-4 CROND[31920]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 06:25:01 lola-4 TIME: Time stamp for console Nov 4 06:30:01 lola-4 CROND[31954]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 06:30:01 lola-4 CROND[31955]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 06:30:01 lola-4 TIME: Time stamp for console Nov 4 06:35:01 lola-4 CROND[31991]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 06:35:01 lola-4 TIME: Time stamp for console Nov 4 06:40:01 lola-4 CROND[32026]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 06:40:01 lola-4 CROND[32027]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 06:40:01 lola-4 TIME: Time stamp for console Nov 4 06:45:01 lola-4 CROND[32063]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 06:45:01 lola-4 TIME: Time stamp for console Nov 4 06:50:01 lola-4 CROND[32097]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 06:50:01 lola-4 CROND[32098]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 06:50:01 lola-4 TIME: Time stamp for console Nov 4 06:55:01 lola-4 CROND[32134]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 06:55:01 lola-4 TIME: Time stamp for console Nov 4 07:00:01 lola-4 CROND[32168]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 07:00:01 lola-4 CROND[32170]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 07:00:01 lola-4 TIME: Time stamp for console Nov 4 07:01:01 lola-4 CROND[32182]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 07:01:01 lola-4 run-parts(/etc/cron.hourly)[3218 starting 0anacron Nov 4 07:01:01 lola-4 run-parts(/etc/cron.hourly)[3219 finished 0anacron Nov 4 07:05:01 lola-4 CROND[32218]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 07:05:01 lola-4 TIME: Time stamp for console Nov 4 07:10:01 lola-4 CROND[32252]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 07:10:01 lola-4 TIME: Time stamp for console Nov 4 07:10:01 lola-4 CROND[32253]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 07:15:01 lola-4 CROND[32290]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 07:15:01 lola-4 TIME: Time stamp for console Nov 4 07:20:01 lola-4 CROND[32324]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 07:20:01 lola-4 CROND[32325]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 07:20:01 lola-4 TIME: Time stamp for console Nov 4 07:25:01 lola-4 CROND[32361]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 07:25:01 lola-4 TIME: Time stamp for console Nov 4 07:30:01 lola-4 CROND[32395]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 07:30:01 lola-4 CROND[32396]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 07:30:01 lola-4 TIME: Time stamp for console Nov 4 07:35:01 lola-4 CROND[32432]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 07:35:01 lola-4 TIME: Time stamp for console Nov 4 07:40:01 lola-4 CROND[32466]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 07:40:01 lola-4 CROND[32467]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 07:40:01 lola-4 TIME: Time stamp for console Nov 4 07:45:01 lola-4 CROND[32504]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 07:45:01 lola-4 TIME: Time stamp for console Nov 4 07:50:01 lola-4 CROND[32538]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 07:50:01 lola-4 CROND[32539]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 07:50:01 lola-4 TIME: Time stamp for console Nov 4 07:55:01 lola-4 CROND[32575]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 07:55:01 lola-4 TIME: Time stamp for console Nov 4 08:00:01 lola-4 CROND[32609]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 08:00:01 lola-4 CROND[32610]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 08:00:01 lola-4 TIME: Time stamp for console Nov 4 08:01:01 lola-4 CROND[32623]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 08:01:01 lola-4 run-parts(/etc/cron.hourly)[3262 starting 0anacron Nov 4 08:01:01 lola-4 run-parts(/etc/cron.hourly)[3263 finished 0anacron Nov 4 08:05:01 lola-4 CROND[32659]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 08:05:01 lola-4 TIME: Time stamp for console Nov 4 08:10:01 lola-4 CROND[32693]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 08:10:01 lola-4 CROND[32694]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 08:10:01 lola-4 TIME: Time stamp for console Nov 4 08:15:02 lola-4 CROND[32730]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 08:15:02 lola-4 TIME: Time stamp for console Nov 4 08:20:01 lola-4 CROND[32765]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 08:20:01 lola-4 CROND[32766]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 08:20:01 lola-4 TIME: Time stamp for console Nov 4 08:25:01 lola-4 CROND[32802]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 08:25:01 lola-4 TIME: Time stamp for console Nov 4 08:30:01 lola-4 CROND[32836]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 08:30:01 lola-4 CROND[32837]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 08:30:01 lola-4 TIME: Time stamp for console Nov 4 08:35:01 lola-4 CROND[32873]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 08:35:01 lola-4 TIME: Time stamp for console Nov 4 08:40:01 lola-4 CROND[32907]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 08:40:01 lola-4 CROND[32908]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 08:40:01 lola-4 TIME: Time stamp for console Nov 4 08:45:01 lola-4 CROND[32944]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 08:45:01 lola-4 TIME: Time stamp for console Nov 4 08:50:01 lola-4 CROND[32978]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 08:50:01 lola-4 CROND[32979]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 08:50:01 lola-4 TIME: Time stamp for console Nov 4 08:55:01 lola-4 CROND[33016]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 08:55:01 lola-4 TIME: Time stamp for console Nov 4 09:00:01 lola-4 CROND[33061]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 09:00:01 lola-4 CROND[33062]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 09:00:01 lola-4 TIME: Time stamp for console Nov 4 09:01:01 lola-4 CROND[33075]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 09:01:01 lola-4 run-parts(/etc/cron.hourly)[3307 starting 0anacron Nov 4 09:01:01 lola-4 run-parts(/etc/cron.hourly)[3308 finished 0anacron Nov 4 09:05:01 lola-4 CROND[33111]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 09:05:01 lola-4 TIME: Time stamp for console Nov 4 09:10:01 lola-4 CROND[33147]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 09:10:01 lola-4 CROND[33148]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 09:10:01 lola-4 TIME: Time stamp for console Nov 4 09:15:01 lola-4 CROND[33184]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 09:15:01 lola-4 TIME: Time stamp for console Nov 4 09:16:06 lola-4 kernel: Lustre: 9436:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1478276142/real 1478276166] req@ffff88027fc7e9c0 x1550004701422144/t0(0) o400->soaked-MDT0001-lwp-OST000e@192.168.1.109@o2ib10:12/10 lens 224/224 e 0 to 1 dl 1478276194 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Nov 4 09:16:06 lola-4 kernel: Lustre: soaked-MDT0001-lwp-OST0008: Connection to soaked-MDT0001 (at 192.168.1.109@o2ib10) was lost; in progress operations using this service will wait for recovery to complete Nov 4 09:16:06 lola-4 kernel: Lustre: Skipped 1 previous similar message Nov 4 09:16:06 lola-4 kernel: Lustre: 9436:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Nov 4 09:16:56 lola-4 kernel: Lustre: 9424:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1478276216/real 1478276216] req@ffff8803728b7680 x1550004701424896/t0(0) o38->soaked-MDT0001-lwp-OST0008@192.168.1.109@o2ib10:12/10 lens 520/544 e 0 to 1 dl 1478276227 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Nov 4 09:16:56 lola-4 kernel: Lustre: 9424:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 1 previous similar message Nov 4 09:17:46 lola-4 kernel: Lustre: 9424:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1478276266/real 1478276266] req@ffff8803aad5f680 x1550004701425568/t0(0) o38->soaked-MDT0001-lwp-OST0014@192.168.1.109@o2ib10:12/10 lens 520/544 e 0 to 1 dl 1478276282 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Nov 4 09:17:46 lola-4 kernel: Lustre: 9424:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Nov 4 09:18:57 lola-4 kernel: Lustre: 9424:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1478276316/real 0] req@ffff88040ebda380 x1550004701426272/t0(0) o38->soaked-MDT0001-lwp-OST0014@192.168.1.109@o2ib10:12/10 lens 520/544 e 0 to 1 dl 1478276337 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Nov 4 09:18:57 lola-4 kernel: Lustre: 9424:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Nov 4 09:19:12 lola-4 kernel: Lustre: soaked-OST0002: haven't heard from client soaked-MDT0001-mdtlov_UUID (at 192.168.1.109@o2ib10) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880825110800, cur 1478276352 expire 1478276202 last 1478276125 Nov 4 09:19:12 lola-4 kernel: Lustre: soaked-OST0014: haven't heard from client soaked-MDT0001-mdtlov_UUID (at 192.168.1.109@o2ib10) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880809964400, cur 1478276352 expire 1478276202 last 1478276125 Nov 4 09:19:14 lola-4 kernel: Lustre: soaked-OST0008: haven't heard from client soaked-MDT0001-mdtlov_UUID (at 192.168.1.109@o2ib10) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88082055e800, cur 1478276354 expire 1478276204 last 1478276127 Nov 4 09:19:16 lola-4 kernel: Lustre: soaked-OST000e: haven't heard from client soaked-MDT0001-mdtlov_UUID (at 192.168.1.109@o2ib10) in 228 seconds. I think it's dead, and I am evicting it. exp ffff8808236b0c00, cur 1478276356 expire 1478276206 last 1478276128 Nov 4 09:20:01 lola-4 CROND[33218]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 09:20:01 lola-4 CROND[33219]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 09:20:01 lola-4 TIME: Time stamp for console Nov 4 09:21:05 lola-4 kernel: Lustre: 9424:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1478276441/real 1478276465] req@ffff8803039cc6c0 x1550004701554000/t0(0) o38->soaked-MDT0001-lwp-OST0014@192.168.1.109@o2ib10:12/10 lens 520/544 e 0 to 1 dl 1478276467 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Nov 4 09:21:05 lola-4 kernel: Lustre: 9424:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Nov 4 09:21:31 lola-4 kernel: Lustre: 9424:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1478276491/real 1478276491] req@ffff88034f6c3680 x1550004702032912/t0(0) o38->soaked-MDT0001-lwp-OST0008@192.168.1.109@o2ib10:12/10 lens 520/544 e 0 to 1 dl 1478276522 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Nov 4 09:21:31 lola-4 kernel: Lustre: 9424:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Nov 4 09:22:21 lola-4 kernel: Lustre: 9424:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1478276541/real 1478276541] req@ffff8802e8ea70c0 x1550004702653904/t0(0) o38->soaked-MDT0001-lwp-OST0002@192.168.1.109@o2ib10:12/10 lens 520/544 e 0 to 1 dl 1478276577 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Nov 4 09:22:21 lola-4 kernel: Lustre: 9424:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Nov 4 09:22:57 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0001-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 09:22:57 lola-4 kernel: Lustre: Skipped 5 previous similar messages Nov 4 09:23:11 lola-4 kernel: Lustre: 9424:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1478276591/real 1478276591] req@ffff88039c4269c0 x1550004703334224/t0(0) o38->soaked-MDT0001-lwp-OST0008@192.168.1.109@o2ib10:12/10 lens 520/544 e 0 to 1 dl 1478276632 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Nov 4 09:23:11 lola-4 kernel: Lustre: 9424:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Nov 4 09:23:36 lola-4 kernel: Lustre: soaked-OST0002: deleting orphan objects from 0x380000401:601544 to 0x380000401:608913 Nov 4 09:23:36 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x500000400:598151 to 0x500000400:603329 Nov 4 09:23:36 lola-4 kernel: Lustre: soaked-OST000e: deleting orphan objects from 0x680000401:594262 to 0x680000401:598497 Nov 4 09:23:36 lola-4 kernel: Lustre: soaked-OST0014: deleting orphan objects from 0x800000402:608123 to 0x800000402:615489 Nov 4 09:23:36 lola-4 kernel: LustreError: 167-0: soaked-MDT0001-lwp-OST0008: This client was evicted by soaked-MDT0001; in progress operations using this service will fail. Nov 4 09:23:36 lola-4 kernel: Lustre: soaked-MDT0001-lwp-OST0008: Connection restored to 192.168.1.108@o2ib10 (at 192.168.1.108@o2ib10) Nov 4 09:23:36 lola-4 kernel: Lustre: Skipped 5 previous similar messages Nov 4 09:24:01 lola-4 kernel: LustreError: 11-0: soaked-MDT0001-lwp-OST0008: operation obd_ping to node 192.168.1.108@o2ib10 failed: rc = -107 Nov 4 09:24:01 lola-4 kernel: Lustre: soaked-MDT0001-lwp-OST0002: Connection to soaked-MDT0001 (at 192.168.1.108@o2ib10) was lost; in progress operations using this service will wait for recovery to complete Nov 4 09:24:01 lola-4 kernel: Lustre: Skipped 1 previous similar message Nov 4 09:24:01 lola-4 kernel: LustreError: Skipped 3 previous similar messages Nov 4 09:24:26 lola-4 kernel: LustreError: 167-0: soaked-MDT0001-lwp-OST0002: This client was evicted by soaked-MDT0001; in progress operations using this service will fail. Nov 4 09:24:26 lola-4 kernel: LustreError: Skipped 3 previous similar messages Nov 4 09:24:49 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x500000400:612445 to 0x500000400:615745 Nov 4 09:24:49 lola-4 kernel: Lustre: soaked-OST000e: deleting orphan objects from 0x680000401:607523 to 0x680000401:610881 Nov 4 09:24:49 lola-4 kernel: Lustre: soaked-OST0002: deleting orphan objects from 0x380000401:618021 to 0x380000401:621329 Nov 4 09:24:49 lola-4 kernel: Lustre: soaked-OST0014: deleting orphan objects from 0x800000402:624618 to 0x800000402:627873 Nov 4 09:24:51 lola-4 kernel: LustreError: 167-0: soaked-MDT0001-lwp-OST0008: This client was evicted by soaked-MDT0001; in progress operations using this service will fail. Nov 4 09:24:51 lola-4 kernel: LustreError: Skipped 2 previous similar messages Nov 4 09:24:51 lola-4 kernel: Lustre: soaked-MDT0001-lwp-OST0008: Connection restored to 192.168.1.109@o2ib10 (at 192.168.1.109@o2ib10) Nov 4 09:24:51 lola-4 kernel: Lustre: Skipped 7 previous similar messages Nov 4 09:25:01 lola-4 CROND[33273]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 09:25:01 lola-4 TIME: Time stamp for console Nov 4 09:30:01 lola-4 CROND[33312]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 09:30:01 lola-4 CROND[33313]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 09:30:01 lola-4 TIME: Time stamp for console Nov 4 09:30:37 lola-4 kernel: LustreError: 0:0:(ldlm_lockd.c:338:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.1.113@o2ib100 ns: filter-soaked-OST0002_UUID lock: ffff8802deb14780/0xb6d84a885bb70cc5 lrc: 3/0,0 mode: PW/PW res: [0x1a8db5c:0x0:0x0].0x0 rrc: 6 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x60000400000020 nid: 192.168.1.113@o2ib100 remote: 0x4228a8485a52b257 expref: 806 pid: 21596 timeout: 4372137576 lvb_type: 0 Nov 4 09:31:32 lola-4 kernel: INFO: task txg_quiesce:12998 blocked for more than 120 seconds. Nov 4 09:31:32 lola-4 kernel: Tainted: P -- ------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Nov 4 09:31:32 lola-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 4 09:31:32 lola-4 kernel: txg_quiesce D 000000000000000c 0 12998 2 0x00000080 Nov 4 09:31:32 lola-4 kernel: ffff88040ae23d20 0000000000000046 0000000000000000 ffffffff810674be Nov 4 09:31:32 lola-4 kernel: 0000465576d5e718 ffff880400000000 0000465576dd3cd4 000000010497fc9b Nov 4 09:31:32 lola-4 kernel: ffff88040e8725f8 000000010497fc9d ffff88040e8725f8 ffff88040ae23fd8 Nov 4 09:31:32 lola-4 kernel: Call Trace: Nov 4 09:31:32 lola-4 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Nov 4 09:31:32 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:31:32 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:31:32 lola-4 kernel: [] ? __bitmap_weight+0x8c/0xb0 Nov 4 09:31:32 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:31:32 lola-4 kernel: [] txg_quiesce_thread+0x32b/0x470 [zfs] Nov 4 09:31:32 lola-4 kernel: [] ? txg_quiesce_thread+0x0/0x470 [zfs] Nov 4 09:31:32 lola-4 kernel: [] ? txg_quiesce_thread+0x0/0x470 [zfs] Nov 4 09:31:32 lola-4 kernel: [] thread_generic_wrapper+0x68/0x80 [spl] Nov 4 09:31:32 lola-4 kernel: [] ? thread_generic_wrapper+0x0/0x80 [spl] Nov 4 09:31:32 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:31:32 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:31:32 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:31:32 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:31:32 lola-4 kernel: INFO: task ll_ost01_000:13077 blocked for more than 120 seconds. Nov 4 09:31:32 lola-4 kernel: Tainted: P -- ------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Nov 4 09:31:32 lola-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 4 09:31:32 lola-4 kernel: ll_ost01_000 D 000000000000000d 0 13077 2 0x00000080 Nov 4 09:31:32 lola-4 kernel: ffff880409d6b940 0000000000000046 0000000000000080 0000000000000000 Nov 4 09:31:32 lola-4 kernel: ffff880409d6b8b0 ffffffffa03744f0 ffff88040c3643b0 0000000000000011 Nov 4 09:31:32 lola-4 kernel: ffff880409d6ba10 ffffffffa038d583 ffff88040a34e5f8 ffff880409d6bfd8 Nov 4 09:31:32 lola-4 kernel: Call Trace: Nov 4 09:31:32 lola-4 kernel: [] ? dbuf_rele+0x40/0x50 [zfs] Nov 4 09:31:32 lola-4 kernel: [] ? dmu_tx_count_write+0x1d3/0x6f0 [zfs] Nov 4 09:31:32 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 09:31:32 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:31:32 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:31:32 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:31:32 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 09:31:32 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 09:31:32 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 09:31:32 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 09:31:32 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 09:31:32 lola-4 kernel: [] ofd_object_destroy+0x3d0/0x8e0 [ofd] Nov 4 09:31:32 lola-4 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Nov 4 09:31:32 lola-4 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Nov 4 09:31:32 lola-4 kernel: [] ? ldlm_completion_ast+0x0/0x9a0 [ptlrpc] Nov 4 09:31:32 lola-4 kernel: [] ofd_destroy_hdl+0x2f2/0xb60 [ofd] Nov 4 09:31:32 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:31:32 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:31:32 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 09:31:32 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 09:31:32 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:31:32 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:31:32 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:31:32 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:31:32 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:31:32 lola-4 kernel: INFO: task ll_ost_io00_002:13086 blocked for more than 120 seconds. Nov 4 09:31:32 lola-4 kernel: Tainted: P -- ------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Nov 4 09:31:32 lola-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 4 09:31:32 lola-4 kernel: ll_ost_io00_0 D 0000000000000012 0 13086 2 0x00000080 Nov 4 09:31:32 lola-4 kernel: ffff8804090137e0 0000000000000046 ffff8804090137a8 ffff8804090137a4 Nov 4 09:31:32 lola-4 kernel: ffff880409013740 ffff88043fe84200 0000465586358ec6 ffff88044e415a00 Nov 4 09:31:32 lola-4 kernel: 000000000000e87b 000000010497fdc4 ffff88040979a5f8 ffff880409013fd8 Nov 4 09:31:32 lola-4 kernel: Call Trace: Nov 4 09:31:32 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:31:32 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:31:32 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:31:32 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 09:31:32 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 09:31:32 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 09:31:32 lola-4 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Nov 4 09:31:32 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 09:31:32 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 09:31:32 lola-4 kernel: [] ofd_commitrw_write+0x683/0x10a0 [ofd] Nov 4 09:31:32 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 09:31:32 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 09:31:32 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 09:31:32 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 09:31:32 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 09:31:32 lola-4 kernel: [] ? enqueue_task+0x66/0x80 Nov 4 09:31:32 lola-4 kernel: [] ? check_preempt_curr+0x6d/0x90 Nov 4 09:31:32 lola-4 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Nov 4 09:31:32 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 09:31:32 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:31:32 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:31:32 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 09:31:32 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:31:32 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:31:32 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:31:32 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:31:32 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:31:33 lola-4 kernel: INFO: task ll_ost_io01_002:13089 blocked for more than 120 seconds. Nov 4 09:31:33 lola-4 kernel: Tainted: P -- ------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Nov 4 09:31:33 lola-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 4 09:31:33 lola-4 kernel: ll_ost_io01_0 D 0000000000000008 0 13089 2 0x00000080 Nov 4 09:31:33 lola-4 kernel: ffff880409027790 0000000000000046 0000000000000000 0000000000000000 Nov 4 09:31:33 lola-4 kernel: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 Nov 4 09:31:33 lola-4 kernel: 0000000000000000 0000000000000000 ffff8804090025f8 ffff880409027fd8 Nov 4 09:31:33 lola-4 kernel: Call Trace: Nov 4 09:31:33 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 09:31:33 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:31:33 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:31:33 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:31:33 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 09:31:33 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 09:31:33 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 09:31:33 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 09:31:33 lola-4 kernel: [] ofd_write_attr_set+0x2c5/0x9e0 [ofd] Nov 4 09:31:33 lola-4 kernel: [] ofd_commitrw_write+0x23f/0x10a0 [ofd] Nov 4 09:31:33 lola-4 kernel: [] ? ofd_fmd_find_nolock+0xad/0xd0 [ofd] Nov 4 09:31:33 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 09:31:33 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 09:31:33 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 09:31:33 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] ? enqueue_task+0x66/0x80 Nov 4 09:31:33 lola-4 kernel: [] ? check_preempt_curr+0x6d/0x90 Nov 4 09:31:33 lola-4 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Nov 4 09:31:33 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 09:31:33 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:31:33 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:31:33 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:31:33 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:31:33 lola-4 kernel: INFO: task ll_ost_out01_00:13096 blocked for more than 120 seconds. Nov 4 09:31:33 lola-4 kernel: Tainted: P -- ------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Nov 4 09:31:33 lola-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 4 09:31:33 lola-4 kernel: ll_ost_out01_ D 000000000000000d 0 13096 2 0x00000080 Nov 4 09:31:33 lola-4 kernel: ffff88040933f990 0000000000000046 ffff88040933f958 ffff88040933f954 Nov 4 09:31:33 lola-4 kernel: ffff88040933f900 ffff88043fe83800 000046558e4e460f ffff8800386d5a00 Nov 4 09:31:33 lola-4 kernel: 000000000000af73 000000010497fe4d ffff88040932bad8 ffff88040933ffd8 Nov 4 09:31:33 lola-4 kernel: Call Trace: Nov 4 09:31:33 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:31:33 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:31:33 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:31:33 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 09:31:33 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 09:31:33 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 09:31:33 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 09:31:33 lola-4 kernel: [] out_tx_end+0x9c/0x5d0 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] out_handle+0x11d9/0x18d0 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] ? lc_watchdog_touch+0x7a/0x190 [libcfs] Nov 4 09:31:33 lola-4 kernel: [] ? mutex_lock+0x1e/0x50 Nov 4 09:31:33 lola-4 kernel: [] ? req_can_reconstruct+0x6a/0x120 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 09:31:33 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 09:31:33 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:31:33 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:31:33 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:31:33 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:31:33 lola-4 kernel: INFO: task ll_ost_out01_00:13097 blocked for more than 120 seconds. Nov 4 09:31:33 lola-4 kernel: Tainted: P -- ------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Nov 4 09:31:33 lola-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 4 09:31:33 lola-4 kernel: ll_ost_out01_ D 000000000000001b 0 13097 2 0x00000080 Nov 4 09:31:33 lola-4 kernel: ffff880409347990 0000000000000046 ffff880409347958 ffff880409347954 Nov 4 09:31:33 lola-4 kernel: ffff880409347900 ffff88043fe85400 00004655954987ab ffff8800386d5a00 Nov 4 09:31:33 lola-4 kernel: 000000000000b573 000000010497feb8 ffff88040932b068 ffff880409347fd8 Nov 4 09:31:33 lola-4 kernel: Call Trace: Nov 4 09:31:33 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:31:33 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:31:33 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:31:33 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 09:31:33 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 09:31:33 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 09:31:33 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 09:31:33 lola-4 kernel: [] out_tx_end+0x9c/0x5d0 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] out_handle+0x11d9/0x18d0 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] ? enqueue_task+0x66/0x80 Nov 4 09:31:33 lola-4 kernel: [] ? check_preempt_curr+0x6d/0x90 Nov 4 09:31:33 lola-4 kernel: [] ? lc_watchdog_touch+0x7a/0x190 [libcfs] Nov 4 09:31:33 lola-4 kernel: [] ? mutex_lock+0x1e/0x50 Nov 4 09:31:33 lola-4 kernel: [] ? req_can_reconstruct+0x6a/0x120 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 09:31:33 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 09:31:33 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:31:33 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:31:33 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:31:33 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:31:33 lola-4 kernel: INFO: task ll_ost01_005:13364 blocked for more than 120 seconds. Nov 4 09:31:33 lola-4 kernel: Tainted: P -- ------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Nov 4 09:31:33 lola-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 4 09:31:33 lola-4 kernel: ll_ost01_005 D 000000000000000b 0 13364 2 0x00000080 Nov 4 09:31:33 lola-4 kernel: ffff880404b17940 0000000000000046 0000000000000080 0000000000000000 Nov 4 09:31:33 lola-4 kernel: ffff880404b178b0 ffffffffa03744f0 ffff88040c3643b0 0000000000000011 Nov 4 09:31:33 lola-4 kernel: ffff880404b17a10 ffffffffa038d583 ffff88040e9a9ad8 ffff880404b17fd8 Nov 4 09:31:33 lola-4 kernel: Call Trace: Nov 4 09:31:33 lola-4 kernel: [] ? dbuf_rele+0x40/0x50 [zfs] Nov 4 09:31:33 lola-4 kernel: [] ? dmu_tx_count_write+0x1d3/0x6f0 [zfs] Nov 4 09:31:33 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 09:31:33 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:31:33 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:31:33 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:31:33 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 09:31:33 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 09:31:33 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 09:31:33 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 09:31:33 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 09:31:33 lola-4 kernel: [] ofd_object_destroy+0x3d0/0x8e0 [ofd] Nov 4 09:31:33 lola-4 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Nov 4 09:31:33 lola-4 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] ? ldlm_completion_ast+0x0/0x9a0 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] ? libcfs_debug_msg+0x41/0x50 [libcfs] Nov 4 09:31:33 lola-4 kernel: [] ofd_destroy_hdl+0x2f2/0xb60 [ofd] Nov 4 09:31:33 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 09:31:33 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 09:31:33 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:31:33 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:31:33 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:31:33 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:31:33 lola-4 kernel: INFO: task ll_ost01_009:21164 blocked for more than 120 seconds. Nov 4 09:31:33 lola-4 kernel: Tainted: P -- ------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Nov 4 09:31:33 lola-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 4 09:31:33 lola-4 kernel: ll_ost01_009 D 000000000000001c 0 21164 2 0x00000080 Nov 4 09:31:33 lola-4 kernel: ffff8803fb237940 0000000000000046 0000000000000000 0000000000000000 Nov 4 09:31:33 lola-4 kernel: ffff8803fb2378b0 ffffffffa03744f0 000046563c125b6d 0000000000000011 Nov 4 09:31:33 lola-4 kernel: ffff8803fb237a10 00000001049809b3 ffff8803fdd205f8 ffff8803fb237fd8 Nov 4 09:31:33 lola-4 kernel: Call Trace: Nov 4 09:31:33 lola-4 kernel: [] ? dbuf_rele+0x40/0x50 [zfs] Nov 4 09:31:33 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:31:33 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:31:33 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:31:33 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 09:31:33 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 09:31:33 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 09:31:33 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 09:31:33 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 09:31:33 lola-4 kernel: [] ofd_object_destroy+0x3d0/0x8e0 [ofd] Nov 4 09:31:33 lola-4 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Nov 4 09:31:33 lola-4 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] ? ldlm_completion_ast+0x0/0x9a0 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] ? libcfs_debug_msg+0x41/0x50 [libcfs] Nov 4 09:31:33 lola-4 kernel: [] ofd_destroy_hdl+0x2f2/0xb60 [ofd] Nov 4 09:31:33 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 09:31:33 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:31:33 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:31:33 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:31:33 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:31:33 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:31:33 lola-4 kernel: INFO: task ll_ost01_016:21171 blocked for more than 120 seconds. Nov 4 09:31:33 lola-4 kernel: Tainted: P -- ------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Nov 4 09:31:33 lola-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 4 09:31:33 lola-4 kernel: ll_ost01_016 D 000000000000000b 0 21171 2 0x00000080 Nov 4 09:31:33 lola-4 kernel: ffff880403deb940 0000000000000046 0000000000000080 0000000000000000 Nov 4 09:31:33 lola-4 kernel: ffff880403deb8b0 ffffffffa03744f0 ffff88040c3643b0 0000000000000011 Nov 4 09:31:33 lola-4 kernel: ffff880403deba10 ffffffffa038d583 ffff88041bf7dad8 ffff880403debfd8 Nov 4 09:31:33 lola-4 kernel: Call Trace: Nov 4 09:31:33 lola-4 kernel: [] ? dbuf_rele+0x40/0x50 [zfs] Nov 4 09:31:33 lola-4 kernel: [] ? dmu_tx_count_write+0x1d3/0x6f0 [zfs] Nov 4 09:31:33 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 09:31:33 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:31:33 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:31:33 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:31:33 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 09:31:33 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 09:31:33 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 09:31:33 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 09:31:34 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 09:31:34 lola-4 kernel: [] ofd_object_destroy+0x3d0/0x8e0 [ofd] Nov 4 09:31:34 lola-4 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Nov 4 09:31:34 lola-4 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Nov 4 09:31:34 lola-4 kernel: [] ? ldlm_completion_ast+0x0/0x9a0 [ptlrpc] Nov 4 09:31:34 lola-4 kernel: [] ? libcfs_debug_msg+0x41/0x50 [libcfs] Nov 4 09:31:34 lola-4 kernel: [] ofd_destroy_hdl+0x2f2/0xb60 [ofd] Nov 4 09:31:34 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:31:34 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:31:34 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 09:31:34 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:31:34 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:31:34 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:31:34 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:31:34 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:31:34 lola-4 kernel: INFO: task ll_ost_io01_004:21243 blocked for more than 120 seconds. Nov 4 09:31:34 lola-4 kernel: Tainted: P -- ------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 Nov 4 09:31:34 lola-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 4 09:31:34 lola-4 kernel: ll_ost_io01_0 D 0000000000000019 0 21243 2 0x00000080 Nov 4 09:31:34 lola-4 kernel: ffff8808099b77e0 0000000000000046 ffff8808099b77a8 00000000000003aa Nov 4 09:31:34 lola-4 kernel: ffff8808099b7740 00000001a030daeb ffff880800000000 0000000000000000 Nov 4 09:31:34 lola-4 kernel: ffff8808099b78b0 ffff88057cbd6000 ffff880836ac3ad8 ffff8808099b7fd8 Nov 4 09:31:34 lola-4 kernel: Call Trace: Nov 4 09:31:34 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 09:31:34 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:31:34 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:31:34 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:31:34 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 09:31:34 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 09:31:34 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 09:31:34 lola-4 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Nov 4 09:31:34 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 09:31:34 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 09:31:34 lola-4 kernel: [] ofd_commitrw_write+0x683/0x10a0 [ofd] Nov 4 09:31:34 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 09:31:34 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 09:31:34 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 09:31:34 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 09:31:34 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 09:31:34 lola-4 kernel: [] ? mod_timer+0x144/0x220 Nov 4 09:31:34 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 09:31:34 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:31:34 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:31:34 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 09:31:34 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:31:34 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:31:34 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:31:34 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:31:34 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:32:11 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8803f3f82000 already connecting from 192.168.1.113@o2ib100 Nov 4 09:32:12 lola-4 kernel: LNet: Service thread pid 21730 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 09:32:12 lola-4 kernel: Pid: 21730, comm: ll_ost01_097 Nov 4 09:32:12 lola-4 kernel: Nov 4 09:32:12 lola-4 kernel: Call Trace: Nov 4 09:32:12 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:32:12 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:32:12 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:32:12 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 09:32:12 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 09:32:12 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 09:32:12 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 09:32:12 lola-4 kernel: [] ofd_precreate_objects+0xa2c/0x1910 [ofd] Nov 4 09:32:12 lola-4 kernel: [] ofd_create_hdl+0x56e/0x2530 [ofd] Nov 4 09:32:12 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 09:32:12 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:32:12 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:32:12 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 09:32:12 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:32:12 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:32:12 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:32:12 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:32:12 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:32:12 lola-4 kernel: Nov 4 09:32:12 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478277132.21730 Nov 4 09:32:15 lola-4 kernel: LNet: Service thread pid 33274 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 09:32:15 lola-4 kernel: Pid: 33274, comm: ll_ost_out01_00 Nov 4 09:32:15 lola-4 kernel: Nov 4 09:32:15 lola-4 kernel: Call Trace: Nov 4 09:32:15 lola-4 kernel: [] ? dsl_dir_tempreserve_impl+0x2ae/0x330 [zfs] Nov 4 09:32:15 lola-4 kernel: [] rwsem_down_failed_common+0x95/0x1d0 Nov 4 09:32:15 lola-4 kernel: [] rwsem_down_write_failed+0x23/0x30 Nov 4 09:32:15 lola-4 kernel: [] call_rwsem_down_write_failed+0x13/0x20 Nov 4 09:32:15 lola-4 kernel: [] ? down_write+0x32/0x40 Nov 4 09:32:15 lola-4 kernel: [] osd_object_write_lock+0x61/0x70 [osd_zfs] Nov 4 09:32:15 lola-4 kernel: [] out_tx_attr_set_exec+0x69/0x480 [ptlrpc] Nov 4 09:32:15 lola-4 kernel: [] out_tx_end+0xda/0x5d0 [ptlrpc] Nov 4 09:32:15 lola-4 kernel: [] out_handle+0x11d9/0x18d0 [ptlrpc] Nov 4 09:32:15 lola-4 kernel: [] ? enqueue_task+0x66/0x80 Nov 4 09:32:15 lola-4 kernel: [] ? check_preempt_curr+0x6d/0x90 Nov 4 09:32:16 lola-4 kernel: [] ? lc_watchdog_touch+0x7a/0x190 [libcfs] Nov 4 09:32:16 lola-4 kernel: [] ? mutex_lock+0x1e/0x50 Nov 4 09:32:16 lola-4 kernel: [] ? req_can_reconstruct+0x6a/0x120 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 09:32:16 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:32:16 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:32:16 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:32:16 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:32:16 lola-4 kernel: Nov 4 09:32:16 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478277136.33274 Nov 4 09:32:16 lola-4 kernel: Pid: 21255, comm: ll_ost_io01_008 Nov 4 09:32:16 lola-4 kernel: Nov 4 09:32:16 lola-4 kernel: Call Trace: Nov 4 09:32:16 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 09:32:16 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:32:16 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:32:16 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:32:16 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 09:32:16 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 09:32:16 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 09:32:16 lola-4 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Nov 4 09:32:16 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 09:32:16 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 09:32:16 lola-4 kernel: [] ofd_commitrw_write+0x683/0x10a0 [ofd] Nov 4 09:32:16 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 09:32:16 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 09:32:16 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 09:32:16 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] ? enqueue_task+0x66/0x80 Nov 4 09:32:16 lola-4 kernel: [] ? check_preempt_curr+0x6d/0x90 Nov 4 09:32:16 lola-4 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Nov 4 09:32:16 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 09:32:16 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:32:16 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:32:16 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:32:16 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:32:16 lola-4 kernel: Nov 4 09:32:16 lola-4 kernel: Pid: 21318, comm: ll_ost_io01_035 Nov 4 09:32:16 lola-4 kernel: Nov 4 09:32:16 lola-4 kernel: Call Trace: Nov 4 09:32:16 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 09:32:16 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:32:16 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:32:16 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:32:16 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 09:32:16 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 09:32:16 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 09:32:16 lola-4 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Nov 4 09:32:16 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 09:32:16 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 09:32:16 lola-4 kernel: [] ofd_commitrw_write+0x683/0x10a0 [ofd] Nov 4 09:32:16 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 09:32:16 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 09:32:16 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 09:32:16 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] ? enqueue_task+0x66/0x80 Nov 4 09:32:16 lola-4 kernel: [] ? check_preempt_curr+0x6d/0x90 Nov 4 09:32:16 lola-4 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Nov 4 09:32:16 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 09:32:16 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 09:32:16 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:32:16 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:32:16 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:32:16 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:32:16 lola-4 kernel: Nov 4 09:32:16 lola-4 kernel: Pid: 13086, comm: ll_ost_io00_002 Nov 4 09:32:16 lola-4 kernel: Nov 4 09:32:16 lola-4 kernel: Call Trace: Nov 4 09:32:16 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:32:16 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:32:16 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:32:16 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 09:32:16 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 09:32:16 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 09:32:16 lola-4 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Nov 4 09:32:16 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 09:32:16 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 09:32:16 lola-4 kernel: [] ofd_commitrw_write+0x683/0x10a0 [ofd] Nov 4 09:32:16 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 09:32:16 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 09:32:16 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 09:32:16 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] ? enqueue_task+0x66/0x80 Nov 4 09:32:16 lola-4 kernel: [] ? check_preempt_curr+0x6d/0x90 Nov 4 09:32:16 lola-4 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Nov 4 09:32:16 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 09:32:16 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:32:16 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:32:16 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:32:16 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:32:16 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:32:16 lola-4 kernel: Nov 4 09:32:16 lola-4 kernel: LNet: Service thread pid 33279 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 4 09:32:17 lola-4 kernel: LNet: Service thread pid 21686 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 4 09:32:17 lola-4 kernel: LNet: Skipped 90 previous similar messages Nov 4 09:32:17 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478277137.21686 Nov 4 09:32:18 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478277138.21727 Nov 4 09:32:18 lola-4 kernel: LNet: Service thread pid 21419 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 4 09:32:18 lola-4 kernel: LNet: Skipped 9 previous similar messages Nov 4 09:32:19 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478277139.21669 Nov 4 09:33:01 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8803f3f82000 already connecting from 192.168.1.113@o2ib100 Nov 4 09:33:51 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8803f3f82000 already connecting from 192.168.1.113@o2ib100 Nov 4 09:34:41 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8803f3f82000 already connecting from 192.168.1.113@o2ib100 Nov 4 09:34:44 lola-4 kernel: Lustre: soaked-OST0002: haven't heard from client ad6dda67-e2c9-e03f-696c-99e5c7b54833 (at (no nid)) in 228 seconds. I think it's dead, and I am evicting it. exp ffff8803f3f82000, cur 1478277284 expire 1478277134 last 1478277056 Nov 4 09:35:01 lola-4 CROND[33359]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 09:35:01 lola-4 TIME: Time stamp for console Nov 4 09:35:43 lola-4 kernel: LNet: Service thread pid 21485 was inactive for 286.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 4 09:35:43 lola-4 kernel: LNet: Skipped 17 previous similar messages Nov 4 09:35:43 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478277343.21485 Nov 4 09:36:46 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 09:37:36 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 09:38:26 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 09:38:47 lola-4 kernel: Lustre: 21167:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 09:38:47 lola-4 kernel: req@ffff880681361850 x1550005401006992/t0(0) o5->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 432/432 e 23 to 0 dl 1478277532 ref 2 fl Interpret:/0/0 rc 0/0 Nov 4 09:38:50 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 09:38:50 lola-4 kernel: req@ffff8807fea5d450 x1550005401079776/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/4320 e 23 to 0 dl 1478277535 ref 2 fl Interpret:/0/0 rc 0/0 Nov 4 09:38:51 lola-4 kernel: Lustre: 21342:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 09:38:51 lola-4 kernel: req@ffff8807dfc04050 x1550005006963264/t0(0) o4->d9e9d3a3-2cdb-3592-6ba7-0fa4ebdd2aa8@192.168.1.117@o2ib100:-1/-1 lens 608/448 e 24 to 0 dl 1478277536 ref 2 fl Interpret:/0/0 rc 0/0 Nov 4 09:38:51 lola-4 kernel: Lustre: 21342:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 40 previous similar messages Nov 4 09:38:53 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 09:38:53 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 09:38:54 lola-4 kernel: Lustre: 21674:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 09:38:54 lola-4 kernel: req@ffff8808146bfc50 x1550085428398016/t0(0) o6->soaked-MDT0001-mdtlov_UUID@192.168.1.109@o2ib10:-1/-1 lens 664/432 e 23 to 0 dl 1478277539 ref 2 fl Interpret:/0/0 rc 0/0 Nov 4 09:38:54 lola-4 kernel: Lustre: 21674:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 64 previous similar messages Nov 4 09:38:57 lola-4 kernel: Lustre: soaked-OST0002: Client a9a24c56-16c3-bde7-190d-18fb77af2167 (at 192.168.1.119@o2ib100) reconnecting Nov 4 09:38:57 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27885754 to 0x0:27891153 Nov 4 09:38:59 lola-4 kernel: Lustre: soaked-OST0002: Client fa1cc277-d053-b449-5830-2859ee0b4cdb (at 192.168.1.120@o2ib100) reconnecting Nov 4 09:38:59 lola-4 kernel: Lustre: Skipped 9 previous similar messages Nov 4 09:39:00 lola-4 kernel: Lustre: soaked-OST0002: deleting orphan objects from 0x380000401:652713 to 0x380000401:654641 Nov 4 09:39:16 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 09:40:01 lola-4 CROND[33395]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 09:40:01 lola-4 CROND[33396]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 09:40:01 lola-4 TIME: Time stamp for console Nov 4 09:40:56 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 09:40:56 lola-4 kernel: Lustre: Skipped 1 previous similar message Nov 4 09:43:26 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 09:43:26 lola-4 kernel: Lustre: Skipped 2 previous similar messages Nov 4 09:45:01 lola-4 CROND[33432]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 09:45:01 lola-4 TIME: Time stamp for console Nov 4 09:48:26 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 09:48:26 lola-4 kernel: Lustre: Skipped 5 previous similar messages Nov 4 09:48:48 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 09:48:48 lola-4 kernel: req@ffff880804021c50 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478278133 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 09:48:48 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 7 previous similar messages Nov 4 09:48:54 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 09:48:54 lola-4 kernel: Lustre: Skipped 1 previous similar message Nov 4 09:48:54 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 09:48:54 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 09:48:58 lola-4 kernel: Lustre: soaked-OST0002: Client e093a05c-49f1-1f4e-3cd0-0381aa4e6173 (at 192.168.1.130@o2ib100) reconnecting Nov 4 09:48:58 lola-4 kernel: Lustre: Skipped 2 previous similar messages Nov 4 09:48:58 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891162 to 0x0:27891185 Nov 4 09:49:27 lola-4 kernel: LNet: Service thread pid 21472 was inactive for 836.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 09:49:27 lola-4 kernel: LNet: Skipped 3 previous similar messages Nov 4 09:49:27 lola-4 kernel: Pid: 21472, comm: ll_ost00_010 Nov 4 09:49:27 lola-4 kernel: Nov 4 09:49:27 lola-4 kernel: Call Trace: Nov 4 09:49:27 lola-4 kernel: [] ? dbuf_rele_and_unlock+0x268/0x390 [zfs] Nov 4 09:49:27 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 09:49:28 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:49:28 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:49:28 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:49:28 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 09:49:28 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 09:49:28 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 09:49:28 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 09:49:28 lola-4 kernel: [] tgt_client_data_update+0x29d/0x6a0 [ptlrpc] Nov 4 09:49:28 lola-4 kernel: [] tgt_client_new+0x3c8/0x690 [ptlrpc] Nov 4 09:49:28 lola-4 kernel: [] ofd_obd_connect+0x3f3/0x4c0 [ofd] Nov 4 09:49:28 lola-4 kernel: [] target_handle_connect+0xf00/0x2e40 [ptlrpc] Nov 4 09:49:28 lola-4 kernel: [] ? thread_return+0x40/0x7d0 Nov 4 09:49:28 lola-4 kernel: [] ? tgt_ses_key_fini+0xb8/0x100 [ptlrpc] Nov 4 09:49:28 lola-4 kernel: [] ? lc_watchdog_touch+0x7a/0x190 [libcfs] Nov 4 09:49:28 lola-4 kernel: [] tgt_request_handle+0x487/0x1440 [ptlrpc] Nov 4 09:49:28 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:49:28 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 09:49:28 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:49:28 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:49:28 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:49:28 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:49:28 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:49:28 lola-4 kernel: Nov 4 09:49:28 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478278168.21472 Nov 4 09:50:01 lola-4 CROND[33468]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 09:50:01 lola-4 CROND[33469]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 09:50:01 lola-4 TIME: Time stamp for console Nov 4 09:51:23 lola-4 kernel: Lustre: 21674:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Nov 4 09:51:23 lola-4 kernel: req@ffff880809787850 x1550005401189552/t0(0) o5->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 432/432 e 0 to 0 dl 1478278288 ref 2 fl Interpret:/0/0 rc 0/0 Nov 4 09:51:23 lola-4 kernel: Lustre: 21674:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 93 previous similar messages Nov 4 09:55:01 lola-4 CROND[33505]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 09:55:01 lola-4 TIME: Time stamp for console Nov 4 09:57:36 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 09:57:36 lola-4 kernel: Lustre: Skipped 10 previous similar messages Nov 4 09:58:49 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 09:58:49 lola-4 kernel: req@ffff8804f2559450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478278734 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 09:58:49 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Nov 4 09:58:53 lola-4 kernel: LNet: Service thread pid 21167 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 09:58:53 lola-4 kernel: Pid: 21167, comm: ll_ost01_012 Nov 4 09:58:53 lola-4 kernel: Nov 4 09:58:53 lola-4 kernel: Call Trace: Nov 4 09:58:53 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 09:58:53 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 09:58:53 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 09:58:53 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 09:58:53 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 09:58:53 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 09:58:53 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 09:58:53 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:58:53 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:58:53 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 09:58:53 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:58:53 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:58:53 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:58:53 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:58:53 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:58:53 lola-4 kernel: Nov 4 09:58:53 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478278733.21167 Nov 4 09:58:55 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 09:58:55 lola-4 kernel: Lustre: Skipped 9 previous similar messages Nov 4 09:58:55 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 09:58:55 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 09:58:57 lola-4 kernel: LNet: Service thread pid 21276 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 09:58:57 lola-4 kernel: Pid: 21276, comm: ll_ost_io01_017 Nov 4 09:58:57 lola-4 kernel: Nov 4 09:58:57 lola-4 kernel: Call Trace: Nov 4 09:58:57 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 09:58:57 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:58:57 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:58:57 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:58:57 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 09:58:57 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 09:58:57 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 09:58:57 lola-4 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Nov 4 09:58:57 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 09:58:57 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 09:58:57 lola-4 kernel: [] ofd_commitrw_write+0x683/0x10a0 [ofd] Nov 4 09:58:57 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 09:58:57 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 09:58:57 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 09:58:57 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] ? enqueue_task+0x66/0x80 Nov 4 09:58:57 lola-4 kernel: [] ? check_preempt_curr+0x6d/0x90 Nov 4 09:58:57 lola-4 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Nov 4 09:58:57 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 09:58:57 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:58:57 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:58:57 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:58:57 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:58:57 lola-4 kernel: Nov 4 09:58:57 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478278737.21276 Nov 4 09:58:57 lola-4 kernel: Pid: 21309, comm: ll_ost_io01_031 Nov 4 09:58:57 lola-4 kernel: Nov 4 09:58:57 lola-4 kernel: Call Trace: Nov 4 09:58:57 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:58:57 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:58:57 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:58:57 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 09:58:57 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 09:58:57 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 09:58:57 lola-4 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Nov 4 09:58:57 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 09:58:57 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 09:58:57 lola-4 kernel: [] ofd_commitrw_write+0x683/0x10a0 [ofd] Nov 4 09:58:57 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 09:58:57 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 09:58:57 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 09:58:57 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] ? check_preempt_curr+0x7c/0x90 Nov 4 09:58:57 lola-4 kernel: [] ? mod_timer+0x144/0x220 Nov 4 09:58:57 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 09:58:57 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:58:57 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:58:57 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:58:57 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:58:57 lola-4 kernel: Nov 4 09:58:57 lola-4 kernel: Pid: 21431, comm: ll_ost_io00_091 Nov 4 09:58:57 lola-4 kernel: Nov 4 09:58:57 lola-4 kernel: Call Trace: Nov 4 09:58:57 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 09:58:57 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:58:57 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:58:57 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:58:57 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 09:58:57 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 09:58:57 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 09:58:57 lola-4 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Nov 4 09:58:57 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 09:58:57 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 09:58:57 lola-4 kernel: [] ofd_commitrw_write+0x683/0x10a0 [ofd] Nov 4 09:58:57 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 09:58:57 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 09:58:57 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 09:58:57 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] ? enqueue_task+0x66/0x80 Nov 4 09:58:57 lola-4 kernel: [] ? check_preempt_curr+0x6d/0x90 Nov 4 09:58:57 lola-4 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Nov 4 09:58:57 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 09:58:57 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:58:57 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:58:57 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:58:57 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:58:57 lola-4 kernel: Nov 4 09:58:57 lola-4 kernel: Pid: 21310, comm: ll_ost_io00_044 Nov 4 09:58:57 lola-4 kernel: Nov 4 09:58:57 lola-4 kernel: Call Trace: Nov 4 09:58:57 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 09:58:57 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 09:58:57 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 09:58:57 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 09:58:57 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 09:58:57 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 09:58:57 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 09:58:57 lola-4 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Nov 4 09:58:57 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 09:58:57 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 09:58:57 lola-4 kernel: [] ofd_commitrw_write+0x683/0x10a0 [ofd] Nov 4 09:58:57 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 09:58:57 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 09:58:57 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 09:58:57 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] ? enqueue_task+0x66/0x80 Nov 4 09:58:57 lola-4 kernel: [] ? check_preempt_curr+0x6d/0x90 Nov 4 09:58:57 lola-4 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Nov 4 09:58:57 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 09:58:57 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 09:58:57 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 09:58:57 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 09:58:57 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 09:58:57 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 09:58:57 lola-4 kernel: Nov 4 09:58:57 lola-4 kernel: LNet: Service thread pid 21408 was inactive for 1200.05s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 4 09:58:59 lola-4 kernel: LNet: Service thread pid 21250 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 4 09:58:59 lola-4 kernel: LNet: Skipped 76 previous similar messages Nov 4 09:58:59 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478278739.21250 Nov 4 09:58:59 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891194 to 0x0:27891217 Nov 4 09:59:00 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478278740.21668 Nov 4 10:00:01 lola-4 CROND[33548]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 10:00:01 lola-4 TIME: Time stamp for console Nov 4 10:00:01 lola-4 CROND[33549]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 10:01:01 lola-4 CROND[33562]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 10:01:01 lola-4 run-parts(/etc/cron.hourly)[3356 starting 0anacron Nov 4 10:01:01 lola-4 run-parts(/etc/cron.hourly)[3357 finished 0anacron Nov 4 10:03:59 lola-4 kernel: Lustre: 21183:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Nov 4 10:03:59 lola-4 kernel: req@ffff880809607450 x1550005401248736/t0(0) o5->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 432/432 e 0 to 0 dl 1478279044 ref 2 fl Interpret:/0/0 rc 0/0 Nov 4 10:03:59 lola-4 kernel: Lustre: 21183:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 93 previous similar messages Nov 4 10:05:01 lola-4 CROND[33598]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 10:05:01 lola-4 TIME: Time stamp for console Nov 4 10:08:26 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 10:08:26 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 10:08:50 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 10:08:50 lola-4 kernel: req@ffff880833a8a050 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478279335 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 10:08:50 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message Nov 4 10:08:56 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 10:08:56 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 10:08:56 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 10:08:56 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 10:08:58 lola-4 kernel: LNet: Service thread pid 21831 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 10:08:58 lola-4 kernel: LNet: Skipped 3 previous similar messages Nov 4 10:08:58 lola-4 kernel: Pid: 21831, comm: ll_ost_io01_109 Nov 4 10:08:58 lola-4 kernel: Nov 4 10:08:58 lola-4 kernel: Call Trace: Nov 4 10:08:58 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 10:08:58 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 10:08:58 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 10:08:58 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 10:08:58 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 10:08:58 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 10:08:58 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 10:08:58 lola-4 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Nov 4 10:08:58 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 10:08:58 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 10:08:58 lola-4 kernel: [] ofd_commitrw_write+0x683/0x10a0 [ofd] Nov 4 10:08:58 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 10:08:58 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 10:08:58 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 10:08:58 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] ? enqueue_task+0x66/0x80 Nov 4 10:08:58 lola-4 kernel: [] ? check_preempt_curr+0x6d/0x90 Nov 4 10:08:58 lola-4 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Nov 4 10:08:58 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 10:08:58 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 10:08:58 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 10:08:58 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 10:08:58 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 10:08:58 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 10:08:58 lola-4 kernel: Nov 4 10:08:58 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478279338.21831 Nov 4 10:08:58 lola-4 kernel: Pid: 13087, comm: ll_ost_io01_000 Nov 4 10:08:58 lola-4 kernel: Nov 4 10:08:58 lola-4 kernel: Call Trace: Nov 4 10:08:58 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 10:08:58 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 10:08:58 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 10:08:58 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 10:08:58 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 10:08:58 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 10:08:58 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 10:08:58 lola-4 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Nov 4 10:08:58 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 10:08:58 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 10:08:58 lola-4 kernel: [] ofd_commitrw_write+0x683/0x10a0 [ofd] Nov 4 10:08:58 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 10:08:58 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 10:08:58 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 10:08:58 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] ? cache_alloc_refill+0x15b/0x240 Nov 4 10:08:58 lola-4 kernel: [] ? class_handle2object+0x95/0x190 [obdclass] Nov 4 10:08:58 lola-4 kernel: [] ? ofd_prolong_extent_locks+0x17f/0x360 [ofd] Nov 4 10:08:58 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 10:08:58 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 10:08:58 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 10:08:58 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 10:08:58 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 10:08:58 lola-4 kernel: Nov 4 10:08:58 lola-4 kernel: Pid: 21348, comm: ll_ost_io01_061 Nov 4 10:08:58 lola-4 kernel: Nov 4 10:08:58 lola-4 kernel: Call Trace: Nov 4 10:08:58 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 10:08:58 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 10:08:58 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 10:08:58 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 10:08:58 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 10:08:58 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 10:08:58 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 10:08:58 lola-4 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Nov 4 10:08:58 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 10:08:58 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 10:08:58 lola-4 kernel: [] ofd_commitrw_write+0x683/0x10a0 [ofd] Nov 4 10:08:58 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 10:08:58 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 10:08:58 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 10:08:58 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] ? enqueue_task+0x66/0x80 Nov 4 10:08:58 lola-4 kernel: [] ? check_preempt_curr+0x6d/0x90 Nov 4 10:08:58 lola-4 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Nov 4 10:08:58 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 10:08:58 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 10:08:58 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 10:08:58 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 10:08:58 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 10:08:58 lola-4 kernel: Nov 4 10:08:58 lola-4 kernel: Pid: 21372, comm: ll_ost_io00_065 Nov 4 10:08:58 lola-4 kernel: Nov 4 10:08:58 lola-4 kernel: Call Trace: Nov 4 10:08:58 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 10:08:58 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 10:08:58 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 10:08:58 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 10:08:58 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 10:08:58 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 10:08:58 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 10:08:58 lola-4 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Nov 4 10:08:58 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 10:08:58 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 10:08:58 lola-4 kernel: [] ofd_commitrw_write+0x683/0x10a0 [ofd] Nov 4 10:08:58 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 10:08:58 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 10:08:58 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 10:08:58 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] ? enqueue_task+0x66/0x80 Nov 4 10:08:58 lola-4 kernel: [] ? check_preempt_curr+0x6d/0x90 Nov 4 10:08:58 lola-4 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Nov 4 10:08:58 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 10:08:58 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 10:08:58 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 10:08:58 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 10:08:58 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 10:08:58 lola-4 kernel: Nov 4 10:08:58 lola-4 kernel: Pid: 33045, comm: ll_ost_io00_117 Nov 4 10:08:58 lola-4 kernel: Nov 4 10:08:58 lola-4 kernel: Call Trace: Nov 4 10:08:58 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 10:08:58 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 10:08:58 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 10:08:58 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 10:08:58 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 10:08:58 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 10:08:58 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 10:08:58 lola-4 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Nov 4 10:08:58 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 10:08:58 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 10:08:58 lola-4 kernel: [] ofd_commitrw_write+0x683/0x10a0 [ofd] Nov 4 10:08:58 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 10:08:58 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 10:08:58 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 10:08:58 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] ? enqueue_task+0x66/0x80 Nov 4 10:08:58 lola-4 kernel: [] ? check_preempt_curr+0x6d/0x90 Nov 4 10:08:58 lola-4 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Nov 4 10:08:58 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 10:08:58 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 10:08:58 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 10:08:58 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 10:08:58 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 10:08:58 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 10:08:58 lola-4 kernel: Nov 4 10:08:58 lola-4 kernel: LNet: Service thread pid 21373 was inactive for 1200.09s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 4 10:08:58 lola-4 kernel: LNet: Skipped 1 previous similar message Nov 4 10:09:00 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478279340.21294 Nov 4 10:09:00 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891228 to 0x0:27891249 Nov 4 10:10:01 lola-4 CROND[33636]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 10:10:01 lola-4 CROND[33637]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 10:10:01 lola-4 TIME: Time stamp for console Nov 4 10:11:29 lola-4 kernel: LNet: Service thread pid 21732 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 4 10:11:29 lola-4 kernel: LNet: Skipped 32 previous similar messages Nov 4 10:11:29 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478279489.21732 Nov 4 10:11:36 lola-4 kernel: LNet: Service thread pid 21762 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 4 10:11:36 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478279496.21762 Nov 4 10:15:01 lola-4 CROND[33677]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 10:15:01 lola-4 TIME: Time stamp for console Nov 4 10:16:35 lola-4 kernel: Lustre: 21183:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Nov 4 10:16:35 lola-4 kernel: req@ffff8808100a0c50 x1550005401307360/t0(0) o5->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 432/432 e 0 to 0 dl 1478279800 ref 2 fl Interpret:/0/0 rc 0/0 Nov 4 10:16:35 lola-4 kernel: Lustre: 21183:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 93 previous similar messages Nov 4 10:18:57 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 10:18:57 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 10:18:57 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 10:18:57 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 10:18:59 lola-4 kernel: LNet: Service thread pid 21321 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 10:18:59 lola-4 kernel: LNet: Skipped 4 previous similar messages Nov 4 10:18:59 lola-4 kernel: Pid: 21321, comm: ll_ost_io00_049 Nov 4 10:18:59 lola-4 kernel: Nov 4 10:18:59 lola-4 kernel: Call Trace: Nov 4 10:18:59 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 10:18:59 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 10:18:59 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 10:18:59 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 10:18:59 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 10:18:59 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 10:18:59 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 10:18:59 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 10:18:59 lola-4 kernel: [] ofd_write_attr_set+0x2c5/0x9e0 [ofd] Nov 4 10:18:59 lola-4 kernel: [] ofd_commitrw_write+0x23f/0x10a0 [ofd] Nov 4 10:18:59 lola-4 kernel: [] ? ofd_fmd_find_nolock+0xad/0xd0 [ofd] Nov 4 10:18:59 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 10:18:59 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 10:18:59 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 10:18:59 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 10:18:59 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 10:18:59 lola-4 kernel: [] ? mod_timer+0x144/0x220 Nov 4 10:18:59 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 10:18:59 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 10:18:59 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 10:18:59 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 10:18:59 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 10:18:59 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 10:18:59 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 10:18:59 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 10:18:59 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 10:18:59 lola-4 kernel: Nov 4 10:18:59 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478279939.21321 Nov 4 10:18:59 lola-4 kernel: Pid: 21283, comm: ll_ost_io00_029 Nov 4 10:18:59 lola-4 kernel: Nov 4 10:18:59 lola-4 kernel: Call Trace: Nov 4 10:18:59 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 10:18:59 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 10:18:59 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 10:18:59 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 10:18:59 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 10:18:59 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 10:18:59 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 10:18:59 lola-4 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Nov 4 10:18:59 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 10:18:59 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 10:18:59 lola-4 kernel: [] ofd_commitrw_write+0x683/0x10a0 [ofd] Nov 4 10:18:59 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 10:18:59 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 10:18:59 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 10:18:59 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 10:18:59 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 10:18:59 lola-4 kernel: [] ? enqueue_task+0x66/0x80 Nov 4 10:18:59 lola-4 kernel: [] ? check_preempt_curr+0x6d/0x90 Nov 4 10:18:59 lola-4 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Nov 4 10:18:59 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 10:18:59 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 10:18:59 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 10:18:59 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 10:18:59 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 10:18:59 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 10:18:59 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 10:18:59 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 10:18:59 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 10:18:59 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 10:18:59 lola-4 kernel: Nov 4 10:18:59 lola-4 kernel: Pid: 21444, comm: ll_ost_io00_099 Nov 4 10:18:59 lola-4 kernel: Nov 4 10:18:59 lola-4 kernel: Call Trace: Nov 4 10:18:59 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 10:18:59 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 10:18:59 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 10:18:59 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 10:18:59 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 10:18:59 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 10:18:59 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 10:18:59 lola-4 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Nov 4 10:18:59 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 10:18:59 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 10:18:59 lola-4 kernel: [] ofd_commitrw_write+0x683/0x10a0 [ofd] Nov 4 10:18:59 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 10:18:59 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 10:18:59 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 10:18:59 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 10:18:59 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 10:18:59 lola-4 kernel: [] ? class_handle2object+0x95/0x190 [obdclass] Nov 4 10:18:59 lola-4 kernel: [] ? task_rq_lock+0x5d/0xa0 Nov 4 10:18:59 lola-4 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Nov 4 10:18:59 lola-4 kernel: [] ? ofd_prolong_extent_locks+0x17f/0x360 [ofd] Nov 4 10:19:00 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 10:19:00 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 10:19:00 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 10:19:00 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 10:19:00 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 10:19:00 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 10:19:00 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 10:19:00 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 10:19:00 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 10:19:00 lola-4 kernel: Nov 4 10:19:00 lola-4 kernel: Pid: 13085, comm: ll_ost_io00_001 Nov 4 10:19:00 lola-4 kernel: Nov 4 10:19:00 lola-4 kernel: Call Trace: Nov 4 10:19:00 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 10:19:00 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 10:19:00 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 10:19:00 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 10:19:00 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 10:19:00 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 10:19:00 lola-4 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Nov 4 10:19:00 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 10:19:00 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 10:19:00 lola-4 kernel: [] ofd_commitrw_write+0x683/0x10a0 [ofd] Nov 4 10:19:00 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 10:19:00 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 10:19:00 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 10:19:00 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 10:19:00 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 10:19:00 lola-4 kernel: [] ? class_handle2object+0x95/0x190 [obdclass] Nov 4 10:19:00 lola-4 kernel: [] ? task_rq_lock+0x5d/0xa0 Nov 4 10:19:00 lola-4 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Nov 4 10:19:00 lola-4 kernel: [] ? ofd_prolong_extent_locks+0x17f/0x360 [ofd] Nov 4 10:19:00 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 10:19:00 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 10:19:00 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 10:19:00 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 10:19:00 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 10:19:00 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 10:19:00 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 10:19:00 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 10:19:00 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 10:19:00 lola-4 kernel: Nov 4 10:19:00 lola-4 kernel: Pid: 21281, comm: ll_ost_io00_028 Nov 4 10:19:00 lola-4 kernel: Nov 4 10:19:00 lola-4 kernel: Call Trace: Nov 4 10:19:00 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 10:19:00 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 10:19:00 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 10:19:00 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 10:19:00 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 10:19:00 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 10:19:00 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 10:19:00 lola-4 kernel: [] ? dt_declare_version_set+0x9f/0x1a0 [obdclass] Nov 4 10:19:00 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 10:19:00 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 10:19:00 lola-4 kernel: [] ofd_commitrw_write+0x683/0x10a0 [ofd] Nov 4 10:19:00 lola-4 kernel: [] ofd_commitrw+0x5bf/0xb10 [ofd] Nov 4 10:19:00 lola-4 kernel: [] ? kzfree+0x31/0x40 Nov 4 10:19:00 lola-4 kernel: [] ? lprocfs_counter_add+0x151/0x1c0 [obdclass] Nov 4 10:19:00 lola-4 kernel: [] obd_commitrw+0x114/0x380 [ptlrpc] Nov 4 10:19:00 lola-4 kernel: [] tgt_brw_write+0xc70/0x1530 [ptlrpc] Nov 4 10:19:00 lola-4 kernel: [] ? class_handle2object+0x95/0x190 [obdclass] Nov 4 10:19:00 lola-4 kernel: [] ? task_rq_lock+0x5d/0xa0 Nov 4 10:19:00 lola-4 kernel: [] ? try_to_wake_up+0x24e/0x3e0 Nov 4 10:19:00 lola-4 kernel: [] ? ofd_prolong_extent_locks+0x17f/0x360 [ofd] Nov 4 10:19:00 lola-4 kernel: [] ? __mutex_lock_slowpath+0x1d4/0x210 Nov 4 10:19:00 lola-4 kernel: [] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] Nov 4 10:19:00 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 10:19:00 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 10:19:00 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 10:19:00 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 10:19:00 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 10:19:00 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 10:19:00 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 10:19:00 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 10:19:00 lola-4 kernel: Nov 4 10:19:00 lola-4 kernel: LNet: Service thread pid 21241 was inactive for 1200.54s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 4 10:19:01 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891259 to 0x0:27891281 Nov 4 10:19:16 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 10:19:16 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 10:20:01 lola-4 CROND[33713]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 10:20:01 lola-4 CROND[33714]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 10:20:01 lola-4 TIME: Time stamp for console Nov 4 10:24:05 lola-4 kernel: LNet: Service thread pid 21738 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 10:24:05 lola-4 kernel: LNet: Skipped 4 previous similar messages Nov 4 10:24:05 lola-4 kernel: Pid: 21738, comm: ll_ost01_103 Nov 4 10:24:05 lola-4 kernel: Nov 4 10:24:05 lola-4 kernel: Call Trace: Nov 4 10:24:05 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 10:24:05 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 10:24:05 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 10:24:05 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 10:24:05 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 10:24:05 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 10:24:05 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 10:24:05 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 10:24:05 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 10:24:05 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 10:24:05 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 10:24:05 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 10:24:05 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 10:24:05 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 10:24:05 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 10:24:05 lola-4 kernel: Nov 4 10:24:05 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478280245.21738 Nov 4 10:24:12 lola-4 kernel: Pid: 21388, comm: ll_ost01_027 Nov 4 10:24:12 lola-4 kernel: Nov 4 10:24:12 lola-4 kernel: Call Trace: Nov 4 10:24:12 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 10:24:12 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 10:24:12 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 10:24:12 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 10:24:12 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 10:24:12 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 10:24:12 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 10:24:12 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 10:24:12 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 10:24:12 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 10:24:12 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 10:24:12 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 10:24:12 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 10:24:12 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 10:24:12 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 10:24:12 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 10:24:12 lola-4 kernel: Nov 4 10:24:12 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478280252.21388 Nov 4 10:25:01 lola-4 CROND[33754]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 10:25:01 lola-4 TIME: Time stamp for console Nov 4 10:26:36 lola-4 kernel: Lustre: 21674:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 10:26:36 lola-4 kernel: req@ffff880809bdbc50 x1550005401366048/t0(0) o5->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 432/432 e 1 to 0 dl 1478280401 ref 2 fl Interpret:/0/0 rc 0/0 Nov 4 10:26:36 lola-4 kernel: Lustre: 21674:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 95 previous similar messages Nov 4 10:28:58 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 10:28:58 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 10:28:58 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 10:28:58 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 10:29:02 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891291 to 0x0:27891313 Nov 4 10:29:16 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 10:29:16 lola-4 kernel: Lustre: Skipped 11 previous similar messages Nov 4 10:30:01 lola-4 CROND[33788]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 10:30:01 lola-4 CROND[33789]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 10:30:01 lola-4 TIME: Time stamp for console Nov 4 10:35:01 lola-4 CROND[33826]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 10:35:01 lola-4 TIME: Time stamp for console Nov 4 10:36:41 lola-4 kernel: LNet: Service thread pid 21183 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 10:36:41 lola-4 kernel: LNet: Skipped 1 previous similar message Nov 4 10:36:41 lola-4 kernel: Pid: 21183, comm: ll_ost01_025 Nov 4 10:36:41 lola-4 kernel: Nov 4 10:36:41 lola-4 kernel: Call Trace: Nov 4 10:36:41 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 10:36:41 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 10:36:41 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 10:36:41 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 10:36:41 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 10:36:41 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 10:36:41 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 10:36:41 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 10:36:41 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 10:36:41 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 10:36:41 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 10:36:41 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 10:36:41 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 10:36:41 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 10:36:41 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 10:36:41 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 10:36:41 lola-4 kernel: Nov 4 10:36:41 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478281001.21183 Nov 4 10:36:48 lola-4 kernel: LNet: Service thread pid 13115 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 10:36:48 lola-4 kernel: Pid: 13115, comm: ll_ost01_004 Nov 4 10:36:48 lola-4 kernel: Nov 4 10:36:48 lola-4 kernel: Call Trace: Nov 4 10:36:48 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 10:36:48 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 10:36:48 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 10:36:48 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 10:36:48 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 10:36:48 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 10:36:48 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 10:36:48 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 10:36:48 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 10:36:48 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 10:36:48 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 10:36:48 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 10:36:48 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 10:36:48 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 10:36:48 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 10:36:48 lola-4 kernel: Nov 4 10:36:48 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478281008.13115 Nov 4 10:38:53 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 10:38:53 lola-4 kernel: req@ffff880811626450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478281138 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 10:38:53 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 95 previous similar messages Nov 4 10:38:59 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 10:38:59 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 10:38:59 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 10:38:59 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 10:39:03 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891323 to 0x0:27891345 Nov 4 10:40:01 lola-4 CROND[33864]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 10:40:01 lola-4 CROND[33865]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 10:40:01 lola-4 TIME: Time stamp for console Nov 4 10:40:06 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 10:40:06 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 10:45:01 lola-4 CROND[33901]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 10:45:01 lola-4 TIME: Time stamp for console Nov 4 10:46:42 lola-4 kernel: LNet: Service thread pid 21674 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 10:46:42 lola-4 kernel: Pid: 21674, comm: ll_ost01_058 Nov 4 10:46:42 lola-4 kernel: Nov 4 10:46:42 lola-4 kernel: Call Trace: Nov 4 10:46:42 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 10:46:42 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 10:46:42 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 10:46:42 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 10:46:42 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 10:46:42 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 10:46:42 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 10:46:42 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 10:46:42 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 10:46:42 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 10:46:42 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 10:46:42 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 10:46:42 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 10:46:42 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 10:46:42 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 10:46:42 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 10:46:42 lola-4 kernel: Nov 4 10:46:42 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478281602.21674 Nov 4 10:46:49 lola-4 kernel: Pid: 21740, comm: ll_ost01_105 Nov 4 10:46:49 lola-4 kernel: Nov 4 10:46:49 lola-4 kernel: Call Trace: Nov 4 10:46:49 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 10:46:49 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 10:46:49 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 10:46:49 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 10:46:49 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 10:46:49 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 10:46:49 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 10:46:49 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 10:46:49 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 10:46:49 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 10:46:49 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 10:46:49 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 10:46:49 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 10:46:49 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 10:46:49 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 10:46:49 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 10:46:49 lola-4 kernel: Nov 4 10:46:49 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478281609.21740 Nov 4 10:48:54 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 10:48:54 lola-4 kernel: req@ffff880836d66050 x1550005401097056/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478281739 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 10:48:54 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 95 previous similar messages Nov 4 10:49:00 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 10:49:00 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 10:49:00 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 10:49:00 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 10:49:04 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891355 to 0x0:27891377 Nov 4 10:50:01 lola-4 CROND[33939]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 10:50:01 lola-4 CROND[33940]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 10:50:01 lola-4 TIME: Time stamp for console Nov 4 10:50:56 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 10:50:56 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 10:55:01 lola-4 CROND[33976]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 10:55:01 lola-4 TIME: Time stamp for console Nov 4 10:58:55 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 10:58:55 lola-4 kernel: req@ffff88061e3bb450 x1550005401097056/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478282340 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 10:58:55 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 95 previous similar messages Nov 4 10:59:01 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 10:59:01 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 10:59:01 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 10:59:01 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 10:59:05 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891386 to 0x0:27891409 Nov 4 10:59:18 lola-4 kernel: LNet: Service thread pid 21173 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 10:59:18 lola-4 kernel: LNet: Skipped 1 previous similar message Nov 4 10:59:18 lola-4 kernel: Pid: 21173, comm: ll_ost01_018 Nov 4 10:59:18 lola-4 kernel: Nov 4 10:59:18 lola-4 kernel: Call Trace: Nov 4 10:59:18 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 10:59:18 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 10:59:18 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 10:59:18 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 10:59:18 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 10:59:18 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 10:59:18 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 10:59:18 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 10:59:18 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 10:59:18 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 10:59:18 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 10:59:18 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 10:59:18 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 10:59:18 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 10:59:18 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 10:59:18 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 10:59:18 lola-4 kernel: Nov 4 10:59:18 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478282358.21173 Nov 4 10:59:25 lola-4 kernel: LNet: Service thread pid 21739 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 10:59:25 lola-4 kernel: Pid: 21739, comm: ll_ost01_104 Nov 4 10:59:25 lola-4 kernel: Nov 4 10:59:25 lola-4 kernel: Call Trace: Nov 4 10:59:25 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 10:59:25 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 10:59:25 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 10:59:25 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 10:59:25 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 10:59:25 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 10:59:25 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 10:59:25 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 10:59:25 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 10:59:25 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 10:59:25 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 10:59:25 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 10:59:25 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 10:59:25 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 10:59:25 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 10:59:25 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 10:59:25 lola-4 kernel: Nov 4 10:59:25 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478282365.21739 Nov 4 11:00:01 lola-4 CROND[34014]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 11:00:01 lola-4 CROND[34015]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 11:00:01 lola-4 TIME: Time stamp for console Nov 4 11:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 4 11:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 4 11:01:01 lola-4 CROND[34026]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 11:01:01 lola-4 run-parts(/etc/cron.hourly)[3402 starting 0anacron Nov 4 11:01:01 lola-4 run-parts(/etc/cron.hourly)[3403 finished 0anacron Nov 4 11:01:46 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 11:01:46 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 11:05:01 lola-4 CROND[34063]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 11:05:01 lola-4 TIME: Time stamp for console Nov 4 11:08:56 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 11:08:56 lola-4 kernel: req@ffff8808333f5050 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478282941 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 11:08:56 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 95 previous similar messages Nov 4 11:09:02 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 11:09:02 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 11:09:02 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 11:09:02 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 11:09:06 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891419 to 0x0:27891441 Nov 4 11:09:19 lola-4 kernel: LNet: Service thread pid 21465 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 11:09:19 lola-4 kernel: Pid: 21465, comm: ll_ost01_035 Nov 4 11:09:19 lola-4 kernel: Nov 4 11:09:19 lola-4 kernel: Call Trace: Nov 4 11:09:19 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 11:09:19 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 11:09:19 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 11:09:19 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 11:09:19 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 11:09:19 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 11:09:19 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 11:09:19 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 11:09:19 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 11:09:19 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 11:09:19 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 11:09:19 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 11:09:19 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 11:09:19 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 11:09:19 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 11:09:19 lola-4 kernel: Nov 4 11:09:19 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478282959.21465 Nov 4 11:09:26 lola-4 kernel: LNet: Service thread pid 21742 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 11:09:26 lola-4 kernel: Pid: 21742, comm: ll_ost01_106 Nov 4 11:09:26 lola-4 kernel: Nov 4 11:09:26 lola-4 kernel: Call Trace: Nov 4 11:09:26 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 11:09:26 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 11:09:26 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 11:09:26 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 11:09:26 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 11:09:26 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 11:09:26 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 11:09:26 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 11:09:26 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 11:09:26 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 11:09:26 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 11:09:26 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 11:09:26 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 11:09:26 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 11:09:26 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 11:09:26 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 11:09:26 lola-4 kernel: Nov 4 11:09:26 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478282966.21742 Nov 4 11:10:01 lola-4 CROND[34101]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 11:10:01 lola-4 CROND[34102]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 11:10:01 lola-4 TIME: Time stamp for console Nov 4 11:12:36 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 11:12:36 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 11:15:01 lola-4 CROND[34138]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 11:15:01 lola-4 TIME: Time stamp for console Nov 4 11:18:57 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 11:18:57 lola-4 kernel: req@ffff8807f3c4f450 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478283542 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 11:18:57 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 95 previous similar messages Nov 4 11:19:03 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 11:19:03 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 11:19:03 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 11:19:03 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 11:19:07 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891451 to 0x0:27891473 Nov 4 11:20:01 lola-4 CROND[34172]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 11:20:01 lola-4 CROND[34173]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 11:20:01 lola-4 TIME: Time stamp for console Nov 4 11:21:55 lola-4 kernel: LNet: Service thread pid 21670 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 11:21:55 lola-4 kernel: Pid: 21670, comm: ll_ost01_054 Nov 4 11:21:55 lola-4 kernel: Nov 4 11:21:55 lola-4 kernel: Call Trace: Nov 4 11:21:55 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 11:21:55 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 11:21:55 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 11:21:55 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 11:21:55 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 11:21:55 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 11:21:55 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 11:21:55 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 11:21:55 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 11:21:55 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 11:21:55 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 11:21:55 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 11:21:55 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 11:21:55 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 11:21:55 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 11:21:55 lola-4 kernel: Nov 4 11:21:55 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478283715.21670 Nov 4 11:22:02 lola-4 kernel: LNet: Service thread pid 21703 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 11:22:02 lola-4 kernel: Pid: 21703, comm: ll_ost01_079 Nov 4 11:22:02 lola-4 kernel: Nov 4 11:22:02 lola-4 kernel: Call Trace: Nov 4 11:22:02 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 11:22:02 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 11:22:02 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 11:22:02 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 11:22:02 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 11:22:02 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 11:22:02 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 11:22:02 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 11:22:02 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 11:22:02 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 11:22:02 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 11:22:02 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 11:22:02 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 11:22:02 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 11:22:02 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 11:22:02 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 11:22:02 lola-4 kernel: Nov 4 11:22:02 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478283722.21703 Nov 4 11:23:26 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 11:23:26 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 11:25:01 lola-4 CROND[34213]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 11:25:01 lola-4 TIME: Time stamp for console Nov 4 11:28:58 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 11:28:58 lola-4 kernel: req@ffff88054c57e850 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478284143 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 11:28:58 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 95 previous similar messages Nov 4 11:29:04 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 11:29:04 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 11:29:04 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 11:29:04 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 11:29:08 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891484 to 0x0:27891505 Nov 4 11:30:01 lola-4 CROND[34247]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 11:30:01 lola-4 CROND[34248]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 11:30:01 lola-4 TIME: Time stamp for console Nov 4 11:31:56 lola-4 kernel: LNet: Service thread pid 21716 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 11:31:56 lola-4 kernel: Pid: 21716, comm: ll_ost01_089 Nov 4 11:31:56 lola-4 kernel: Nov 4 11:31:56 lola-4 kernel: Call Trace: Nov 4 11:31:56 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 11:31:56 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 11:31:56 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 11:31:56 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 11:31:56 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 11:31:56 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 11:31:56 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 11:31:56 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 11:31:56 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 11:31:56 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 11:31:56 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 11:31:56 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 11:31:56 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 11:31:56 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 11:31:56 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 11:31:56 lola-4 kernel: Nov 4 11:31:56 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478284316.21716 Nov 4 11:32:03 lola-4 kernel: LNet: Service thread pid 21744 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 11:32:03 lola-4 kernel: Pid: 21744, comm: ll_ost01_108 Nov 4 11:32:03 lola-4 kernel: Nov 4 11:32:03 lola-4 kernel: Call Trace: Nov 4 11:32:03 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 11:32:03 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 11:32:03 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 11:32:03 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 11:32:03 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 11:32:03 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 11:32:03 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 11:32:03 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 11:32:03 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 11:32:03 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 11:32:03 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 11:32:03 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 11:32:03 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 11:32:03 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 11:32:03 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 11:32:03 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 11:32:03 lola-4 kernel: Nov 4 11:32:03 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478284323.21744 Nov 4 11:34:16 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 11:34:16 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 11:35:01 lola-4 CROND[34288]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 11:35:01 lola-4 TIME: Time stamp for console Nov 4 11:38:59 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 11:38:59 lola-4 kernel: req@ffff8804de3fe050 x1550005401079776/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478284744 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 11:38:59 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 95 previous similar messages Nov 4 11:39:05 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 11:39:05 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 11:39:05 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 11:39:05 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 11:39:09 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891515 to 0x0:27891537 Nov 4 11:40:01 lola-4 CROND[34323]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 11:40:01 lola-4 CROND[34324]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 11:40:01 lola-4 TIME: Time stamp for console Nov 4 11:44:32 lola-4 kernel: LNet: Service thread pid 21169 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 11:44:32 lola-4 kernel: Pid: 21169, comm: ll_ost01_014 Nov 4 11:44:32 lola-4 kernel: Nov 4 11:44:32 lola-4 kernel: Call Trace: Nov 4 11:44:32 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 11:44:32 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 11:44:32 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 11:44:32 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 11:44:32 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 11:44:32 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 11:44:32 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 11:44:32 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 11:44:32 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 11:44:32 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 11:44:32 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 11:44:32 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 11:44:32 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 11:44:32 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 11:44:32 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 11:44:32 lola-4 kernel: Nov 4 11:44:32 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478285072.21169 Nov 4 11:44:39 lola-4 kernel: LNet: Service thread pid 21707 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 11:44:39 lola-4 kernel: Pid: 21707, comm: ll_ost01_083 Nov 4 11:44:39 lola-4 kernel: Nov 4 11:44:39 lola-4 kernel: Call Trace: Nov 4 11:44:39 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 11:44:39 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 11:44:39 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 11:44:39 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 11:44:39 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 11:44:39 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 11:44:39 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 11:44:39 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 11:44:39 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 11:44:39 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 11:44:39 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 11:44:39 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 11:44:39 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 11:44:39 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 11:44:39 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 11:44:39 lola-4 kernel: Nov 4 11:44:39 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478285079.21707 Nov 4 11:45:01 lola-4 CROND[34364]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 11:45:01 lola-4 TIME: Time stamp for console Nov 4 11:45:06 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 11:45:06 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 11:49:00 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 11:49:00 lola-4 kernel: req@ffff8806ce5e5c50 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478285345 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 11:49:00 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 94 previous similar messages Nov 4 11:49:06 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 11:49:06 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 11:49:06 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 11:49:06 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 11:49:10 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891547 to 0x0:27891569 Nov 4 11:50:01 lola-4 CROND[34398]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 11:50:01 lola-4 CROND[34399]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 11:50:01 lola-4 TIME: Time stamp for console Nov 4 11:54:33 lola-4 kernel: LNet: Service thread pid 21658 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 11:54:33 lola-4 kernel: Pid: 21658, comm: ll_ost01_042 Nov 4 11:54:33 lola-4 kernel: Nov 4 11:54:33 lola-4 kernel: Call Trace: Nov 4 11:54:33 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 11:54:33 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 11:54:33 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 11:54:33 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 11:54:33 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 11:54:33 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 11:54:33 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 11:54:33 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 11:54:33 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 11:54:33 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 11:54:33 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 11:54:33 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 11:54:33 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 11:54:33 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 11:54:33 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 11:54:33 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 11:54:33 lola-4 kernel: Nov 4 11:54:33 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478285673.21658 Nov 4 11:55:01 lola-4 CROND[34437]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 11:55:01 lola-4 TIME: Time stamp for console Nov 4 11:55:56 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 11:55:56 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 11:57:15 lola-4 kernel: LNet: Service thread pid 21391 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 11:57:15 lola-4 kernel: Pid: 21391, comm: ll_ost01_030 Nov 4 11:57:15 lola-4 kernel: Nov 4 11:57:15 lola-4 kernel: Call Trace: Nov 4 11:57:15 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 11:57:15 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 11:57:15 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 11:57:15 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 11:57:15 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 11:57:15 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 11:57:15 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 11:57:15 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 11:57:15 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 11:57:15 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 11:57:15 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 11:57:15 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 11:57:15 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 11:57:15 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 11:57:15 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 11:57:15 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 11:57:15 lola-4 kernel: Nov 4 11:57:15 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478285835.21391 Nov 4 11:59:01 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 11:59:01 lola-4 kernel: req@ffff880670467450 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478285946 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 11:59:01 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 94 previous similar messages Nov 4 11:59:07 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 11:59:07 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 11:59:07 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 11:59:07 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 11:59:11 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891579 to 0x0:27891601 Nov 4 12:00:01 lola-4 CROND[34473]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 12:00:01 lola-4 CROND[34474]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 12:00:01 lola-4 TIME: Time stamp for console Nov 4 12:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 4 12:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 4 12:01:01 lola-4 CROND[34575]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 12:01:01 lola-4 run-parts(/etc/cron.hourly)[3457 starting 0anacron Nov 4 12:01:01 lola-4 run-parts(/etc/cron.hourly)[3458 finished 0anacron Nov 4 12:05:01 lola-4 CROND[34611]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 12:05:01 lola-4 TIME: Time stamp for console Nov 4 12:06:46 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 12:06:46 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 12:07:09 lola-4 kernel: LNet: Service thread pid 21695 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 12:07:09 lola-4 kernel: Pid: 21695, comm: ll_ost01_071 Nov 4 12:07:09 lola-4 kernel: Nov 4 12:07:09 lola-4 kernel: Call Trace: Nov 4 12:07:09 lola-4 kernel: [] ? account_entity_enqueue+0x7e/0x90 Nov 4 12:07:09 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 12:07:09 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 12:07:09 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 12:07:09 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 12:07:09 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 12:07:09 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 12:07:09 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 12:07:09 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 12:07:09 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 12:07:09 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 12:07:09 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 12:07:09 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 12:07:09 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 12:07:09 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 12:07:09 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 12:07:09 lola-4 kernel: Nov 4 12:07:09 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478286429.21695 Nov 4 12:09:02 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 12:09:02 lola-4 kernel: req@ffff8804fef76450 x1550005401097056/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478286547 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 12:09:02 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 95 previous similar messages Nov 4 12:09:08 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 12:09:08 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 12:09:08 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 12:09:08 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 12:09:12 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891613 to 0x0:27891633 Nov 4 12:09:51 lola-4 kernel: LNet: Service thread pid 21726 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 12:09:51 lola-4 kernel: Pid: 21726, comm: ll_ost01_094 Nov 4 12:09:51 lola-4 kernel: Nov 4 12:09:51 lola-4 kernel: Call Trace: Nov 4 12:09:51 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 12:09:51 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 12:09:51 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 12:09:51 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 12:09:51 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 12:09:51 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 12:09:51 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 12:09:51 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 12:09:51 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 12:09:51 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 12:09:51 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 12:09:51 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 12:09:51 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 12:09:51 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 12:09:51 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 12:09:51 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 12:09:51 lola-4 kernel: Nov 4 12:09:51 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478286591.21726 Nov 4 12:10:01 lola-4 CROND[34649]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 12:10:01 lola-4 CROND[34650]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 12:10:01 lola-4 TIME: Time stamp for console Nov 4 12:15:01 lola-4 CROND[34687]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 12:15:01 lola-4 TIME: Time stamp for console Nov 4 12:17:36 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 12:17:36 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 12:19:03 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 12:19:03 lola-4 kernel: req@ffff8807fcb6e850 x1550005401093744/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478287148 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 12:19:03 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 95 previous similar messages Nov 4 12:19:09 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 12:19:09 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 12:19:09 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 12:19:09 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 12:19:13 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891643 to 0x0:27891665 Nov 4 12:19:45 lola-4 kernel: LNet: Service thread pid 21694 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 12:19:45 lola-4 kernel: Pid: 21694, comm: ll_ost01_070 Nov 4 12:19:45 lola-4 kernel: Nov 4 12:19:45 lola-4 kernel: Call Trace: Nov 4 12:19:45 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 12:19:45 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 12:19:45 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 12:19:45 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 12:19:45 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 12:19:45 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 12:19:45 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 12:19:45 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 12:19:45 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 12:19:45 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 12:19:45 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 12:19:45 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 12:19:45 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 12:19:45 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 12:19:45 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 12:19:45 lola-4 kernel: Nov 4 12:19:45 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478287185.21694 Nov 4 12:20:01 lola-4 CROND[34723]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 12:20:01 lola-4 CROND[34724]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 12:20:01 lola-4 TIME: Time stamp for console Nov 4 12:22:27 lola-4 kernel: LNet: Service thread pid 21710 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 12:22:27 lola-4 kernel: Pid: 21710, comm: ll_ost01_085 Nov 4 12:22:27 lola-4 kernel: Nov 4 12:22:27 lola-4 kernel: Call Trace: Nov 4 12:22:27 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 12:22:27 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 12:22:27 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 12:22:27 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 12:22:27 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 12:22:27 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 12:22:27 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 12:22:27 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 12:22:27 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 12:22:27 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 12:22:27 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 12:22:27 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 12:22:27 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 12:22:27 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 12:22:27 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 12:22:27 lola-4 kernel: Nov 4 12:22:27 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478287347.21710 Nov 4 12:25:01 lola-4 CROND[34762]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 12:25:01 lola-4 TIME: Time stamp for console Nov 4 12:28:26 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 12:28:26 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 12:29:04 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 12:29:04 lola-4 kernel: req@ffff8807f9e15050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478287749 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 12:29:04 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 95 previous similar messages Nov 4 12:29:10 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 12:29:10 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 12:29:10 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 12:29:10 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 12:29:14 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891675 to 0x0:27891697 Nov 4 12:29:46 lola-4 kernel: LNet: Service thread pid 13078 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 12:29:46 lola-4 kernel: Pid: 13078, comm: ll_ost01_001 Nov 4 12:29:46 lola-4 kernel: Nov 4 12:29:46 lola-4 kernel: Call Trace: Nov 4 12:29:46 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 12:29:46 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 12:29:46 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 12:29:46 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 12:29:46 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 12:29:46 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 12:29:46 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 12:29:46 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 12:29:46 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 12:29:46 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 12:29:46 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 12:29:46 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 12:29:46 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 12:29:46 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 12:29:46 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 12:29:46 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 12:29:46 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 12:29:46 lola-4 kernel: Nov 4 12:29:46 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478287786.13078 Nov 4 12:30:01 lola-4 CROND[34798]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 12:30:01 lola-4 CROND[34799]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 12:30:01 lola-4 TIME: Time stamp for console Nov 4 12:35:01 lola-4 CROND[34835]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 12:35:01 lola-4 TIME: Time stamp for console Nov 4 12:35:03 lola-4 kernel: LNet: Service thread pid 21681 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 12:35:03 lola-4 kernel: Pid: 21681, comm: ll_ost01_065 Nov 4 12:35:03 lola-4 kernel: Nov 4 12:35:03 lola-4 kernel: Call Trace: Nov 4 12:35:03 lola-4 kernel: [] ? account_entity_enqueue+0x7e/0x90 Nov 4 12:35:03 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 12:35:03 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 12:35:03 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 12:35:03 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 12:35:03 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 12:35:03 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 12:35:03 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 12:35:03 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 12:35:03 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 12:35:03 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 12:35:03 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 12:35:03 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 12:35:03 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 12:35:03 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 12:35:03 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 12:35:03 lola-4 kernel: Nov 4 12:35:03 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478288103.21681 Nov 4 12:39:05 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 12:39:05 lola-4 kernel: req@ffff8807aed1e450 x1550005401079776/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478288350 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 12:39:05 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 94 previous similar messages Nov 4 12:39:11 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 12:39:11 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 12:39:11 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 12:39:11 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 12:39:15 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891706 to 0x0:27891729 Nov 4 12:39:16 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 12:39:16 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 12:40:01 lola-4 CROND[34871]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 12:40:01 lola-4 CROND[34872]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 12:40:01 lola-4 TIME: Time stamp for console Nov 4 12:42:22 lola-4 kernel: Pid: 21662, comm: ll_ost01_046 Nov 4 12:42:22 lola-4 kernel: Nov 4 12:42:22 lola-4 kernel: Call Trace: Nov 4 12:42:22 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 12:42:22 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 12:42:22 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 12:42:22 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 12:42:22 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 12:42:22 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 12:42:22 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 12:42:22 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 12:42:22 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 12:42:22 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 12:42:22 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 12:42:22 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 12:42:22 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 12:42:22 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 12:42:22 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 12:42:22 lola-4 kernel: Nov 4 12:42:22 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478288542.21662 Nov 4 12:45:01 lola-4 CROND[34911]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 12:45:01 lola-4 TIME: Time stamp for console Nov 4 12:47:39 lola-4 kernel: LNet: Service thread pid 21664 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 12:47:39 lola-4 kernel: LNet: Skipped 1 previous similar message Nov 4 12:47:39 lola-4 kernel: Pid: 21664, comm: ll_ost01_048 Nov 4 12:47:39 lola-4 kernel: Nov 4 12:47:39 lola-4 kernel: Call Trace: Nov 4 12:47:39 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 12:47:39 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 12:47:39 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 12:47:39 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 12:47:39 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 12:47:39 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 12:47:39 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 12:47:39 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 12:47:39 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 12:47:39 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 12:47:39 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 12:47:39 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 12:47:39 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 12:47:39 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 12:47:39 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 12:47:39 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 12:47:39 lola-4 kernel: Nov 4 12:47:39 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478288859.21664 Nov 4 12:49:06 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 12:49:06 lola-4 kernel: req@ffff88080da1f450 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478288951 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 12:49:06 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 95 previous similar messages Nov 4 12:49:12 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 12:49:12 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 12:49:12 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 12:49:12 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 12:49:16 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891739 to 0x0:27891761 Nov 4 12:50:01 lola-4 CROND[34947]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 12:50:01 lola-4 CROND[34948]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 12:50:01 lola-4 TIME: Time stamp for console Nov 4 12:50:06 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 12:50:06 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 12:54:58 lola-4 kernel: Pid: 21696, comm: ll_ost01_072 Nov 4 12:54:58 lola-4 kernel: Nov 4 12:54:58 lola-4 kernel: Call Trace: Nov 4 12:54:58 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 12:54:58 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 12:54:58 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 12:54:58 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 12:54:58 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 12:54:58 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 12:54:58 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 12:54:58 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 12:54:58 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 12:54:58 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 12:54:58 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 12:54:58 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 12:54:58 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 12:54:58 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 12:54:58 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 12:54:58 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 12:54:58 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 12:54:58 lola-4 kernel: Nov 4 12:54:58 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478289298.21696 Nov 4 12:55:01 lola-4 CROND[34986]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 12:55:01 lola-4 TIME: Time stamp for console Nov 4 12:59:07 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 12:59:07 lola-4 kernel: req@ffff8804e16ee050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478289552 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 12:59:07 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 94 previous similar messages Nov 4 12:59:13 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 12:59:13 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 12:59:13 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 12:59:13 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 12:59:17 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891770 to 0x0:27891793 Nov 4 13:00:01 lola-4 CROND[35020]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 13:00:01 lola-4 CROND[35021]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 13:00:01 lola-4 TIME: Time stamp for console Nov 4 13:00:15 lola-4 kernel: LNet: Service thread pid 21748 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 13:00:15 lola-4 kernel: LNet: Skipped 1 previous similar message Nov 4 13:00:15 lola-4 kernel: Pid: 21748, comm: ll_ost01_111 Nov 4 13:00:15 lola-4 kernel: Nov 4 13:00:15 lola-4 kernel: Call Trace: Nov 4 13:00:15 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 13:00:15 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 13:00:15 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 13:00:15 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 13:00:15 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 13:00:15 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 13:00:15 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 13:00:15 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 13:00:15 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 13:00:15 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 13:00:15 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 13:00:15 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 13:00:15 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 13:00:15 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 13:00:15 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 13:00:15 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 13:00:15 lola-4 kernel: Nov 4 13:00:15 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478289615.21748 Nov 4 13:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 4 13:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 4 13:00:56 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 13:00:56 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 13:01:01 lola-4 CROND[35034]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 13:01:01 lola-4 run-parts(/etc/cron.hourly)[3503 starting 0anacron Nov 4 13:01:01 lola-4 run-parts(/etc/cron.hourly)[3504 finished 0anacron Nov 4 13:05:01 lola-4 CROND[35070]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 13:05:01 lola-4 TIME: Time stamp for console Nov 4 13:07:34 lola-4 kernel: Pid: 21706, comm: ll_ost01_082 Nov 4 13:07:34 lola-4 kernel: Nov 4 13:07:34 lola-4 kernel: Call Trace: Nov 4 13:07:34 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 13:07:34 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 13:07:34 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 13:07:34 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 13:07:34 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 13:07:34 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 13:07:34 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 13:07:34 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 13:07:34 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 13:07:34 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 13:07:34 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 13:07:34 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 13:07:34 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 13:07:34 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 13:07:34 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 13:07:34 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 13:07:34 lola-4 kernel: Nov 4 13:07:34 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478290054.21706 Nov 4 13:09:08 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 13:09:08 lola-4 kernel: req@ffff8804e27fac50 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478290153 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 13:09:08 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 95 previous similar messages Nov 4 13:09:14 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 13:09:14 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 13:09:14 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 13:09:14 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 13:09:18 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891805 to 0x0:27891825 Nov 4 13:10:01 lola-4 CROND[35106]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 13:10:01 lola-4 CROND[35107]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 13:10:01 lola-4 TIME: Time stamp for console Nov 4 13:11:46 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 13:11:46 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 13:12:51 lola-4 kernel: LNet: Service thread pid 21714 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 13:12:51 lola-4 kernel: LNet: Skipped 1 previous similar message Nov 4 13:12:51 lola-4 kernel: Pid: 21714, comm: ll_ost01_088 Nov 4 13:12:51 lola-4 kernel: Nov 4 13:12:51 lola-4 kernel: Call Trace: Nov 4 13:12:51 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 13:12:51 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 13:12:51 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 13:12:51 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 13:12:51 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 13:12:51 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 13:12:51 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 13:12:51 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 13:12:51 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 13:12:51 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 13:12:51 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 13:12:51 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 13:12:51 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 13:12:51 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 13:12:51 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 13:12:51 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 13:12:51 lola-4 kernel: Nov 4 13:12:51 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478290371.21714 Nov 4 13:15:01 lola-4 CROND[35145]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 13:15:01 lola-4 TIME: Time stamp for console Nov 4 13:19:09 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 13:19:09 lola-4 kernel: req@ffff88058a177c50 x1550005401094688/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478290754 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 13:19:09 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 95 previous similar messages Nov 4 13:19:15 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 13:19:15 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 13:19:15 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 13:19:15 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 13:20:01 lola-4 CROND[35180]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 13:20:01 lola-4 CROND[35181]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 13:20:01 lola-4 TIME: Time stamp for console Nov 4 13:20:10 lola-4 kernel: Pid: 21166, comm: ll_ost01_011 Nov 4 13:20:10 lola-4 kernel: Nov 4 13:20:10 lola-4 kernel: Call Trace: Nov 4 13:20:10 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 13:20:10 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 13:20:10 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 13:20:10 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 13:20:10 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 13:20:10 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 13:20:10 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 13:20:10 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 13:20:10 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 13:20:10 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 13:20:10 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 13:20:10 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 13:20:10 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 13:20:10 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 13:20:10 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 13:20:10 lola-4 kernel: Nov 4 13:20:10 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478290810.21166 Nov 4 13:22:36 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 13:22:36 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 13:22:38 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891840 to 0x0:27891857 Nov 4 13:25:01 lola-4 CROND[35219]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 13:25:01 lola-4 TIME: Time stamp for console Nov 4 13:25:27 lola-4 kernel: LNet: Service thread pid 21181 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 13:25:27 lola-4 kernel: LNet: Skipped 1 previous similar message Nov 4 13:25:27 lola-4 kernel: Pid: 21181, comm: ll_ost01_023 Nov 4 13:25:27 lola-4 kernel: Nov 4 13:25:27 lola-4 kernel: Call Trace: Nov 4 13:25:27 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 13:25:27 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 13:25:27 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 13:25:27 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 13:25:27 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 13:25:27 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 13:25:27 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 13:25:27 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 13:25:27 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 13:25:27 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 13:25:27 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 13:25:27 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 13:25:27 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 13:25:27 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 13:25:27 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 13:25:27 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 13:25:27 lola-4 kernel: Nov 4 13:25:27 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478291127.21181 Nov 4 13:29:10 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 13:29:10 lola-4 kernel: req@ffff8807de3a9c50 x1550005401097056/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478291355 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 13:29:10 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 94 previous similar messages Nov 4 13:29:16 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 13:29:16 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 13:29:16 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 13:29:16 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 13:29:20 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891873 to 0x0:27891889 Nov 4 13:30:01 lola-4 CROND[35256]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 13:30:01 lola-4 CROND[35257]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 13:30:01 lola-4 TIME: Time stamp for console Nov 4 13:32:46 lola-4 kernel: Pid: 21396, comm: ll_ost01_034 Nov 4 13:32:46 lola-4 kernel: Nov 4 13:32:46 lola-4 kernel: Call Trace: Nov 4 13:32:46 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 13:32:46 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 13:32:46 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 13:32:46 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 13:32:46 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 13:32:46 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 13:32:46 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 13:32:46 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 13:32:46 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 13:32:46 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 13:32:46 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 13:32:46 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 13:32:46 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 13:32:46 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 13:32:46 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 13:32:46 lola-4 kernel: Nov 4 13:32:46 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478291566.21396 Nov 4 13:33:26 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 13:33:26 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 13:35:01 lola-4 CROND[35295]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 13:35:01 lola-4 TIME: Time stamp for console Nov 4 13:38:03 lola-4 kernel: LNet: Service thread pid 21737 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 13:38:03 lola-4 kernel: LNet: Skipped 1 previous similar message Nov 4 13:38:03 lola-4 kernel: Pid: 21737, comm: ll_ost01_102 Nov 4 13:38:03 lola-4 kernel: Nov 4 13:38:03 lola-4 kernel: Call Trace: Nov 4 13:38:03 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 13:38:03 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 13:38:03 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 13:38:03 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 13:38:03 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 13:38:03 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 13:38:03 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 13:38:03 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 13:38:03 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 13:38:03 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 13:38:03 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 13:38:03 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 13:38:03 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 13:38:03 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 13:38:03 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 13:38:03 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 13:38:03 lola-4 kernel: Nov 4 13:38:03 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478291883.21737 Nov 4 13:39:11 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 13:39:11 lola-4 kernel: req@ffff8804e5cd5850 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478291956 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 13:39:11 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 95 previous similar messages Nov 4 13:39:17 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 13:39:17 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 13:39:17 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 13:39:17 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 13:39:21 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891905 to 0x0:27891921 Nov 4 13:40:01 lola-4 CROND[35331]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 13:40:01 lola-4 CROND[35332]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 13:40:01 lola-4 TIME: Time stamp for console Nov 4 13:44:16 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 13:44:16 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 13:45:01 lola-4 CROND[35368]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 13:45:01 lola-4 TIME: Time stamp for console Nov 4 13:45:22 lola-4 kernel: Pid: 21701, comm: ll_ost01_077 Nov 4 13:45:22 lola-4 kernel: Nov 4 13:45:22 lola-4 kernel: Call Trace: Nov 4 13:45:22 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 13:45:22 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 13:45:22 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 13:45:22 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 13:45:22 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 13:45:22 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 13:45:22 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 13:45:22 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 13:45:22 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 13:45:22 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 13:45:22 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 13:45:22 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 13:45:22 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 13:45:22 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 13:45:22 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 13:45:22 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 13:45:22 lola-4 kernel: Nov 4 13:45:22 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478292322.21701 Nov 4 13:49:12 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 13:49:12 lola-4 kernel: req@ffff8807ff403850 x1550005401094688/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478292557 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 13:49:12 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 94 previous similar messages Nov 4 13:49:18 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 13:49:18 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 13:49:18 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 13:49:18 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 13:49:22 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27891947 to 0x0:27891985 Nov 4 13:50:01 lola-4 CROND[35404]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 13:50:01 lola-4 CROND[35405]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 13:50:01 lola-4 TIME: Time stamp for console Nov 4 13:50:39 lola-4 kernel: LNet: Service thread pid 21392 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 13:50:39 lola-4 kernel: LNet: Skipped 1 previous similar message Nov 4 13:50:39 lola-4 kernel: Pid: 21392, comm: ll_ost01_031 Nov 4 13:50:39 lola-4 kernel: Nov 4 13:50:39 lola-4 kernel: Call Trace: Nov 4 13:50:39 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 13:50:39 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 13:50:39 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 13:50:39 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 13:50:39 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 13:50:39 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 13:50:39 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 13:50:39 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 13:50:39 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 13:50:39 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 13:50:39 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 13:50:39 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 13:50:39 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 13:50:39 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 13:50:39 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 13:50:39 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 13:50:39 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 13:50:39 lola-4 kernel: Nov 4 13:50:39 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478292639.21392 Nov 4 13:55:01 lola-4 CROND[35444]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 13:55:01 lola-4 TIME: Time stamp for console Nov 4 13:55:06 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 13:55:06 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 13:57:58 lola-4 kernel: Pid: 21175, comm: ll_ost01_020 Nov 4 13:57:58 lola-4 kernel: Nov 4 13:57:58 lola-4 kernel: Call Trace: Nov 4 13:57:58 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 13:57:58 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 13:57:58 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 13:57:58 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 13:57:58 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 13:57:58 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 13:57:58 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 13:57:58 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 13:57:58 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 13:57:58 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 13:57:58 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 13:57:58 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 13:57:58 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 13:57:58 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 13:57:58 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 13:57:58 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 13:57:58 lola-4 kernel: Nov 4 13:57:58 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478293078.21175 Nov 4 13:59:13 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 13:59:13 lola-4 kernel: req@ffff8806b5190850 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478293158 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 13:59:13 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 95 previous similar messages Nov 4 13:59:19 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 13:59:19 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 13:59:19 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 13:59:19 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 13:59:23 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892002 to 0x0:27892017 Nov 4 14:00:01 lola-4 CROND[35480]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 14:00:01 lola-4 TIME: Time stamp for console Nov 4 14:00:01 lola-4 CROND[35481]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 14:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 4 14:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 4 14:01:01 lola-4 CROND[35492]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 14:01:01 lola-4 run-parts(/etc/cron.hourly)[3549 starting 0anacron Nov 4 14:01:01 lola-4 run-parts(/etc/cron.hourly)[3550 finished 0anacron Nov 4 14:03:15 lola-4 kernel: LNet: Service thread pid 13114 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 14:03:15 lola-4 kernel: LNet: Skipped 1 previous similar message Nov 4 14:03:15 lola-4 kernel: Pid: 13114, comm: ll_ost01_003 Nov 4 14:03:15 lola-4 kernel: Nov 4 14:03:15 lola-4 kernel: Call Trace: Nov 4 14:03:15 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 14:03:15 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 14:03:15 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 14:03:15 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 14:03:15 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 14:03:15 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 14:03:15 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 14:03:15 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 14:03:15 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 14:03:15 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 14:03:15 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 14:03:15 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 14:03:15 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 14:03:15 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 14:03:15 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 14:03:15 lola-4 kernel: Nov 4 14:03:15 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478293395.13114 Nov 4 14:05:01 lola-4 CROND[35530]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 14:05:01 lola-4 TIME: Time stamp for console Nov 4 14:05:56 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 14:05:56 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 14:09:14 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 14:09:14 lola-4 kernel: req@ffff8804ef623850 x1550005401097056/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478293759 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 14:09:14 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 95 previous similar messages Nov 4 14:09:20 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 14:09:20 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 14:09:20 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 14:09:20 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 14:10:01 lola-4 CROND[35564]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 14:10:01 lola-4 CROND[35565]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 14:10:01 lola-4 TIME: Time stamp for console Nov 4 14:10:34 lola-4 kernel: Pid: 21180, comm: ll_ost01_022 Nov 4 14:10:34 lola-4 kernel: Nov 4 14:10:34 lola-4 kernel: Call Trace: Nov 4 14:10:34 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 14:10:34 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 14:10:34 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 14:10:34 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 14:10:34 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 14:10:34 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 14:10:34 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 14:10:34 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 14:10:34 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 14:10:34 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 14:10:34 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 14:10:34 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 14:10:34 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 14:10:34 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 14:10:34 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 14:10:34 lola-4 kernel: Nov 4 14:10:34 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478293834.21180 Nov 4 14:14:18 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892037 to 0x0:27892081 Nov 4 14:15:01 lola-4 CROND[35603]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 14:15:01 lola-4 TIME: Time stamp for console Nov 4 14:15:51 lola-4 kernel: LNet: Service thread pid 21733 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 14:15:51 lola-4 kernel: LNet: Skipped 1 previous similar message Nov 4 14:15:51 lola-4 kernel: Pid: 21733, comm: ll_ost01_100 Nov 4 14:15:51 lola-4 kernel: Nov 4 14:15:51 lola-4 kernel: Call Trace: Nov 4 14:15:51 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 14:15:51 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 14:15:51 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 14:15:51 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 14:15:51 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 14:15:51 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 14:15:51 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 14:15:51 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 14:15:51 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 14:15:51 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 14:15:51 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 14:15:51 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 14:15:51 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 14:15:51 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 14:15:51 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 14:15:51 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 14:15:51 lola-4 kernel: Nov 4 14:15:51 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478294151.21733 Nov 4 14:16:30 lola-4 kernel: Pid: 21635, comm: ll_ost00_087 Nov 4 14:16:30 lola-4 kernel: Nov 4 14:16:30 lola-4 kernel: Call Trace: Nov 4 14:16:30 lola-4 kernel: [] ? dbuf_rele+0x40/0x50 [zfs] Nov 4 14:16:30 lola-4 kernel: [] ? dmu_tx_count_write+0x1d3/0x6f0 [zfs] Nov 4 14:16:30 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 14:16:30 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 14:16:30 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 14:16:30 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 14:16:30 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 14:16:30 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 14:16:30 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 14:16:30 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 14:16:30 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 14:16:30 lola-4 kernel: [] ofd_object_destroy+0x3d0/0x8e0 [ofd] Nov 4 14:16:30 lola-4 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Nov 4 14:16:30 lola-4 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Nov 4 14:16:30 lola-4 kernel: [] ? ldlm_completion_ast+0x0/0x9a0 [ptlrpc] Nov 4 14:16:30 lola-4 kernel: [] ? libcfs_debug_msg+0x41/0x50 [libcfs] Nov 4 14:16:30 lola-4 kernel: [] ofd_destroy_hdl+0x2f2/0xb60 [ofd] Nov 4 14:16:30 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 14:16:30 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 14:16:30 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 14:16:30 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 14:16:30 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 14:16:30 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 14:16:30 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 14:16:30 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 14:16:30 lola-4 kernel: Nov 4 14:16:30 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478294190.21635 Nov 4 14:16:30 lola-4 kernel: Pid: 21632, comm: ll_ost00_084 Nov 4 14:16:30 lola-4 kernel: Nov 4 14:16:30 lola-4 kernel: Call Trace: Nov 4 14:16:30 lola-4 kernel: [] ? dbuf_rele+0x40/0x50 [zfs] Nov 4 14:16:30 lola-4 kernel: [] ? dmu_tx_count_write+0x1d3/0x6f0 [zfs] Nov 4 14:16:30 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 14:16:30 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 14:16:30 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 14:16:30 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 14:16:30 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 14:16:30 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 14:16:30 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 14:16:30 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 14:16:30 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 14:16:30 lola-4 kernel: [] ofd_object_destroy+0x3d0/0x8e0 [ofd] Nov 4 14:16:30 lola-4 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Nov 4 14:16:30 lola-4 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Nov 4 14:16:30 lola-4 kernel: [] ? ldlm_completion_ast+0x0/0x9a0 [ptlrpc] Nov 4 14:16:30 lola-4 kernel: [] ? libcfs_debug_msg+0x41/0x50 [libcfs] Nov 4 14:16:30 lola-4 kernel: [] ofd_destroy_hdl+0x2f2/0xb60 [ofd] Nov 4 14:16:30 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 14:16:30 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 14:16:30 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 14:16:30 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 14:16:30 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 14:16:30 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 14:16:30 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 14:16:30 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 14:16:30 lola-4 kernel: Nov 4 14:16:30 lola-4 kernel: Pid: 21600, comm: ll_ost00_052 Nov 4 14:16:30 lola-4 kernel: Nov 4 14:16:30 lola-4 kernel: Call Trace: Nov 4 14:16:30 lola-4 kernel: [] ? dbuf_rele+0x40/0x50 [zfs] Nov 4 14:16:30 lola-4 kernel: [] ? dmu_tx_count_write+0x1d3/0x6f0 [zfs] Nov 4 14:16:30 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 14:16:30 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 14:16:30 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 14:16:30 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 14:16:30 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 14:16:30 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 14:16:30 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 14:16:30 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 14:16:30 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 14:16:30 lola-4 kernel: [] ofd_object_destroy+0x3d0/0x8e0 [ofd] Nov 4 14:16:30 lola-4 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Nov 4 14:16:30 lola-4 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Nov 4 14:16:30 lola-4 kernel: [] ? ldlm_completion_ast+0x0/0x9a0 [ptlrpc] Nov 4 14:16:30 lola-4 kernel: [] ? libcfs_debug_msg+0x41/0x50 [libcfs] Nov 4 14:16:30 lola-4 kernel: [] ofd_destroy_hdl+0x2f2/0xb60 [ofd] Nov 4 14:16:30 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 14:16:30 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 14:16:30 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 14:16:30 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 14:16:30 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 14:16:30 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 14:16:30 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 14:16:30 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 14:16:30 lola-4 kernel: Nov 4 14:16:30 lola-4 kernel: Pid: 21495, comm: ll_ost00_032 Nov 4 14:16:30 lola-4 kernel: Nov 4 14:16:30 lola-4 kernel: Call Trace: Nov 4 14:16:30 lola-4 kernel: [] ? dbuf_rele+0x40/0x50 [zfs] Nov 4 14:16:30 lola-4 kernel: [] ? dmu_tx_count_write+0x1d3/0x6f0 [zfs] Nov 4 14:16:30 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 14:16:30 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 14:16:30 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 14:16:30 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 14:16:30 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 14:16:30 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 14:16:30 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 14:16:30 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 14:16:30 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 14:16:30 lola-4 kernel: [] ofd_object_destroy+0x3d0/0x8e0 [ofd] Nov 4 14:16:30 lola-4 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Nov 4 14:16:30 lola-4 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Nov 4 14:16:30 lola-4 kernel: [] ? ldlm_completion_ast+0x0/0x9a0 [ptlrpc] Nov 4 14:16:30 lola-4 kernel: [] ? libcfs_debug_msg+0x41/0x50 [libcfs] Nov 4 14:16:30 lola-4 kernel: [] ofd_destroy_hdl+0x2f2/0xb60 [ofd] Nov 4 14:16:30 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 14:16:30 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 14:16:31 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 14:16:31 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 14:16:31 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 14:16:31 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 14:16:31 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 14:16:31 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 14:16:31 lola-4 kernel: Nov 4 14:16:31 lola-4 kernel: LNet: Service thread pid 21590 was inactive for 200.60s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 4 14:16:31 lola-4 kernel: LNet: Skipped 20 previous similar messages Nov 4 14:16:46 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 14:16:46 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 14:19:15 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 14:19:15 lola-4 kernel: req@ffff880803a10050 x1550005401094688/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478294360 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 14:19:15 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 94 previous similar messages Nov 4 14:19:21 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 14:19:21 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 14:19:21 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 14:19:21 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 14:19:25 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892098 to 0x0:27892113 Nov 4 14:20:01 lola-4 CROND[35641]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 14:20:01 lola-4 CROND[35642]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 14:20:01 lola-4 TIME: Time stamp for console Nov 4 14:23:10 lola-4 kernel: Pid: 21163, comm: ll_ost01_008 Nov 4 14:23:10 lola-4 kernel: Nov 4 14:23:10 lola-4 kernel: Call Trace: Nov 4 14:23:10 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 14:23:10 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 14:23:10 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 14:23:10 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 14:23:10 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 14:23:10 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 14:23:10 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 14:23:10 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 14:23:10 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 14:23:10 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 14:23:10 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 14:23:10 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 14:23:10 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 14:23:10 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 14:23:10 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 14:23:10 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 14:23:10 lola-4 kernel: Nov 4 14:23:10 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478294590.21163 Nov 4 14:23:11 lola-4 kernel: Lustre: soaked-OST0002: deleting orphan objects from 0x380000400:641331 to 0x380000400:646561 Nov 4 14:25:01 lola-4 CROND[35681]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 14:25:01 lola-4 TIME: Time stamp for console Nov 4 14:27:36 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 14:27:36 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 14:28:27 lola-4 kernel: LNet: Service thread pid 21758 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 14:28:27 lola-4 kernel: LNet: Skipped 5 previous similar messages Nov 4 14:28:27 lola-4 kernel: Pid: 21758, comm: ll_ost01_115 Nov 4 14:28:27 lola-4 kernel: Nov 4 14:28:27 lola-4 kernel: Call Trace: Nov 4 14:28:27 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 14:28:27 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 14:28:27 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 14:28:27 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 14:28:27 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 14:28:27 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 14:28:27 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 14:28:27 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 14:28:27 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 14:28:27 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 14:28:27 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 14:28:27 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 14:28:27 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 14:28:27 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 14:28:27 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 14:28:27 lola-4 kernel: Nov 4 14:28:27 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478294907.21758 Nov 4 14:29:16 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 14:29:16 lola-4 kernel: req@ffff8804db2afc50 x1550005401094688/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478294961 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 14:29:16 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 103 previous similar messages Nov 4 14:29:22 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 14:29:22 lola-4 kernel: Lustre: Skipped 13 previous similar messages Nov 4 14:29:22 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 14:29:22 lola-4 kernel: Lustre: Skipped 13 previous similar messages Nov 4 14:29:26 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892125 to 0x0:27892145 Nov 4 14:30:01 lola-4 CROND[35717]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 14:30:01 lola-4 CROND[35718]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 14:30:01 lola-4 TIME: Time stamp for console Nov 4 14:35:01 lola-4 CROND[35754]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 14:35:01 lola-4 TIME: Time stamp for console Nov 4 14:35:46 lola-4 kernel: Pid: 21673, comm: ll_ost01_057 Nov 4 14:35:46 lola-4 kernel: Nov 4 14:35:46 lola-4 kernel: Call Trace: Nov 4 14:35:46 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 14:35:46 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 14:35:46 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 14:35:46 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 14:35:46 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 14:35:46 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 14:35:46 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 14:35:46 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 14:35:46 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 14:35:46 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 14:35:46 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 14:35:46 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 14:35:46 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 14:35:46 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 14:35:46 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 14:35:46 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 14:35:46 lola-4 kernel: Nov 4 14:35:46 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478295346.21673 Nov 4 14:38:26 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 14:38:26 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 14:39:17 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 14:39:17 lola-4 kernel: req@ffff88052335ec50 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478295562 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 14:39:17 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 95 previous similar messages Nov 4 14:39:23 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 14:39:23 lola-4 kernel: Lustre: Skipped 13 previous similar messages Nov 4 14:39:23 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 14:39:23 lola-4 kernel: Lustre: Skipped 13 previous similar messages Nov 4 14:39:27 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892167 to 0x0:27892209 Nov 4 14:40:01 lola-4 CROND[35790]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 14:40:01 lola-4 CROND[35791]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 14:40:01 lola-4 TIME: Time stamp for console Nov 4 14:41:03 lola-4 kernel: LNet: Service thread pid 21760 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 14:41:03 lola-4 kernel: LNet: Skipped 1 previous similar message Nov 4 14:41:03 lola-4 kernel: Pid: 21760, comm: ll_ost01_117 Nov 4 14:41:03 lola-4 kernel: Nov 4 14:41:03 lola-4 kernel: Call Trace: Nov 4 14:41:03 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 14:41:03 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 14:41:03 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 14:41:03 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 14:41:03 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 14:41:03 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 14:41:03 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 14:41:03 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 14:41:03 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 14:41:03 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 14:41:03 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 14:41:03 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 14:41:03 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 14:41:03 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 14:41:03 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 14:41:03 lola-4 kernel: Nov 4 14:41:03 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478295663.21760 Nov 4 14:43:11 lola-4 kernel: Pid: 21636, comm: ll_ost00_088 Nov 4 14:43:11 lola-4 kernel: Nov 4 14:43:11 lola-4 kernel: Call Trace: Nov 4 14:43:11 lola-4 kernel: [] ? RW_WRITE_HELD+0x4c/0x70 [zfs] Nov 4 14:43:11 lola-4 kernel: [] ? dbuf_rele_and_unlock+0x268/0x390 [zfs] Nov 4 14:43:11 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 14:43:11 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 14:43:11 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 14:43:11 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 14:43:11 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 14:43:11 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 14:43:11 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 14:43:11 lola-4 kernel: [] ofd_object_destroy+0x280/0x8e0 [ofd] Nov 4 14:43:11 lola-4 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Nov 4 14:43:11 lola-4 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Nov 4 14:43:11 lola-4 kernel: [] ? ldlm_completion_ast+0x0/0x9a0 [ptlrpc] Nov 4 14:43:11 lola-4 kernel: [] ? libcfs_debug_msg+0x41/0x50 [libcfs] Nov 4 14:43:11 lola-4 kernel: [] ofd_create_hdl+0x1973/0x2530 [ofd] Nov 4 14:43:11 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 14:43:11 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 14:43:11 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 14:43:11 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 14:43:11 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 14:43:11 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 14:43:11 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 14:43:11 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 14:43:11 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 14:43:11 lola-4 kernel: Nov 4 14:43:11 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478295791.21636 Nov 4 14:45:01 lola-4 CROND[35831]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 14:45:01 lola-4 TIME: Time stamp for console Nov 4 14:48:22 lola-4 kernel: Pid: 21680, comm: ll_ost01_064 Nov 4 14:48:22 lola-4 kernel: Nov 4 14:48:22 lola-4 kernel: Call Trace: Nov 4 14:48:22 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 14:48:22 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 14:48:22 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 14:48:22 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 14:48:22 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 14:48:22 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 14:48:22 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 14:48:22 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 14:48:22 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 14:48:22 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 14:48:22 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 14:48:22 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 14:48:22 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 14:48:22 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 14:48:22 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 14:48:22 lola-4 kernel: Nov 4 14:48:22 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478296102.21680 Nov 4 14:49:16 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 14:49:16 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 14:49:18 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 14:49:18 lola-4 kernel: req@ffff88050f9c2450 x1550005401098800/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478296163 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 14:49:18 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 96 previous similar messages Nov 4 14:49:24 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 14:49:24 lola-4 kernel: Lustre: Skipped 13 previous similar messages Nov 4 14:49:24 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 14:49:24 lola-4 kernel: Lustre: Skipped 13 previous similar messages Nov 4 14:49:28 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892228 to 0x0:27892273 Nov 4 14:50:01 lola-4 CROND[35867]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 14:50:01 lola-4 CROND[35868]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 14:50:01 lola-4 TIME: Time stamp for console Nov 4 14:53:39 lola-4 kernel: LNet: Service thread pid 21743 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 14:53:39 lola-4 kernel: LNet: Skipped 2 previous similar messages Nov 4 14:53:39 lola-4 kernel: Pid: 21743, comm: ll_ost01_107 Nov 4 14:53:39 lola-4 kernel: Nov 4 14:53:39 lola-4 kernel: Call Trace: Nov 4 14:53:39 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 14:53:39 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 14:53:39 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 14:53:39 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 14:53:39 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 14:53:39 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 14:53:39 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 14:53:39 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 14:53:39 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 14:53:39 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 14:53:39 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 14:53:39 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 14:53:39 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 14:53:39 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 14:53:39 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 14:53:39 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 14:53:39 lola-4 kernel: Nov 4 14:53:39 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478296419.21743 Nov 4 14:55:01 lola-4 CROND[35906]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 14:55:01 lola-4 TIME: Time stamp for console Nov 4 14:55:47 lola-4 kernel: Pid: 21481, comm: ll_ost00_019 Nov 4 14:55:47 lola-4 kernel: Nov 4 14:55:47 lola-4 kernel: Call Trace: Nov 4 14:55:47 lola-4 kernel: [] ? account_entity_enqueue+0x7e/0x90 Nov 4 14:55:47 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 14:55:47 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 14:55:47 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 14:55:47 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 14:55:47 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 14:55:47 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 14:55:47 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 14:55:47 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 14:55:47 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 14:55:47 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 14:55:47 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 14:55:47 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 14:55:47 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 14:55:47 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 14:55:47 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 14:55:47 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 14:55:47 lola-4 kernel: Nov 4 14:55:47 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478296547.21481 Nov 4 14:59:19 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 14:59:19 lola-4 kernel: req@ffff8804f0b56850 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478296764 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 14:59:19 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 4 14:59:25 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 14:59:25 lola-4 kernel: Lustre: Skipped 21 previous similar messages Nov 4 14:59:25 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 14:59:25 lola-4 kernel: Lustre: Skipped 21 previous similar messages Nov 4 15:00:01 lola-4 CROND[35943]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 15:00:01 lola-4 CROND[35944]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 15:00:01 lola-4 TIME: Time stamp for console Nov 4 15:00:06 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 15:00:06 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 15:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 4 15:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 4 15:00:58 lola-4 kernel: Pid: 21731, comm: ll_ost01_098 Nov 4 15:00:58 lola-4 kernel: Nov 4 15:00:58 lola-4 kernel: Call Trace: Nov 4 15:00:58 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:00:58 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:00:58 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:00:58 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:00:58 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:00:58 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:00:58 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:00:58 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:00:58 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:00:58 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 15:00:58 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:00:58 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:00:58 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:00:58 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:00:58 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:00:58 lola-4 kernel: Nov 4 15:00:58 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478296858.21731 Nov 4 15:00:58 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892296 to 0x0:27892337 Nov 4 15:01:01 lola-4 CROND[35957]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 15:01:01 lola-4 run-parts(/etc/cron.hourly)[3595 starting 0anacron Nov 4 15:01:01 lola-4 run-parts(/etc/cron.hourly)[3596 finished 0anacron Nov 4 15:05:01 lola-4 CROND[35993]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 15:05:01 lola-4 TIME: Time stamp for console Nov 4 15:06:15 lola-4 kernel: LNet: Service thread pid 21165 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 15:06:15 lola-4 kernel: LNet: Skipped 2 previous similar messages Nov 4 15:06:15 lola-4 kernel: Pid: 21165, comm: ll_ost01_010 Nov 4 15:06:15 lola-4 kernel: Nov 4 15:06:15 lola-4 kernel: Call Trace: Nov 4 15:06:15 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 15:06:15 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:06:15 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:06:15 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:06:15 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:06:15 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:06:15 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:06:15 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:06:15 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:06:15 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:06:15 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 15:06:15 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:06:15 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:06:15 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:06:15 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:06:16 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:06:16 lola-4 kernel: Nov 4 15:06:16 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478297176.21165 Nov 4 15:06:25 lola-4 kernel: Pid: 21168, comm: ll_ost01_013 Nov 4 15:06:25 lola-4 kernel: Nov 4 15:06:25 lola-4 kernel: Call Trace: Nov 4 15:06:25 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 15:06:25 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 15:06:25 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 15:06:25 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 15:06:25 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 15:06:25 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 15:06:25 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 15:06:25 lola-4 kernel: [] ofd_precreate_objects+0xa2c/0x1910 [ofd] Nov 4 15:06:25 lola-4 kernel: [] ? ofd_grant_create+0x168/0x4c0 [ofd] Nov 4 15:06:25 lola-4 kernel: [] ofd_create_hdl+0x56e/0x2530 [ofd] Nov 4 15:06:25 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:06:25 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:06:25 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:06:25 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 15:06:25 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:06:25 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:06:25 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:06:25 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:06:25 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:06:25 lola-4 kernel: Nov 4 15:06:25 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478297185.21168 Nov 4 15:06:29 lola-4 kernel: Pid: 21759, comm: ll_ost01_116 Nov 4 15:06:29 lola-4 kernel: Nov 4 15:06:29 lola-4 kernel: Call Trace: Nov 4 15:06:29 lola-4 kernel: [] ? dbuf_rele+0x40/0x50 [zfs] Nov 4 15:06:29 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 15:06:29 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 15:06:29 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 15:06:29 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 15:06:29 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 15:06:29 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 15:06:29 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 15:06:29 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 15:06:29 lola-4 kernel: [] ofd_object_destroy+0x3d0/0x8e0 [ofd] Nov 4 15:06:29 lola-4 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Nov 4 15:06:29 lola-4 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Nov 4 15:06:29 lola-4 kernel: [] ? ldlm_completion_ast+0x0/0x9a0 [ptlrpc] Nov 4 15:06:29 lola-4 kernel: [] ? libcfs_debug_msg+0x41/0x50 [libcfs] Nov 4 15:06:29 lola-4 kernel: [] ofd_destroy_hdl+0x2f2/0xb60 [ofd] Nov 4 15:06:29 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:06:29 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:06:29 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 15:06:29 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 15:06:29 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:06:29 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:06:29 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:06:29 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:06:29 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:06:29 lola-4 kernel: Nov 4 15:06:29 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478297189.21759 Nov 4 15:06:29 lola-4 kernel: Pid: 21692, comm: ll_ost01_069 Nov 4 15:06:29 lola-4 kernel: Nov 4 15:06:29 lola-4 kernel: Call Trace: Nov 4 15:06:29 lola-4 kernel: [] ? dbuf_rele+0x40/0x50 [zfs] Nov 4 15:06:29 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 15:06:29 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 15:06:29 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 15:06:30 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 15:06:30 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 15:06:30 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 15:06:30 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 15:06:30 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 15:06:30 lola-4 kernel: [] ofd_object_destroy+0x3d0/0x8e0 [ofd] Nov 4 15:06:30 lola-4 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Nov 4 15:06:30 lola-4 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Nov 4 15:06:30 lola-4 kernel: [] ? ldlm_completion_ast+0x0/0x9a0 [ptlrpc] Nov 4 15:06:30 lola-4 kernel: [] ? libcfs_debug_msg+0x41/0x50 [libcfs] Nov 4 15:06:30 lola-4 kernel: [] ofd_destroy_hdl+0x2f2/0xb60 [ofd] Nov 4 15:06:30 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:06:30 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:06:30 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 15:06:30 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:06:30 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:06:30 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:06:30 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:06:30 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:06:30 lola-4 kernel: Nov 4 15:06:30 lola-4 kernel: Pid: 21468, comm: ll_ost01_038 Nov 4 15:06:30 lola-4 kernel: Nov 4 15:06:30 lola-4 kernel: Call Trace: Nov 4 15:06:30 lola-4 kernel: [] ? dbuf_rele+0x40/0x50 [zfs] Nov 4 15:06:30 lola-4 kernel: [] ? dmu_tx_count_write+0x1d3/0x6f0 [zfs] Nov 4 15:06:30 lola-4 kernel: [] ? prepare_to_wait_exclusive+0x4e/0x80 Nov 4 15:06:30 lola-4 kernel: [] cv_wait_common+0x11d/0x130 [spl] Nov 4 15:06:30 lola-4 kernel: [] ? autoremove_wake_function+0x0/0x40 Nov 4 15:06:30 lola-4 kernel: [] __cv_wait+0x15/0x20 [spl] Nov 4 15:06:30 lola-4 kernel: [] dmu_tx_wait+0x21d/0x400 [zfs] Nov 4 15:06:30 lola-4 kernel: [] ? txg_kick+0x92/0xd0 [zfs] Nov 4 15:06:30 lola-4 kernel: [] dmu_tx_assign+0xa1/0x570 [zfs] Nov 4 15:06:30 lola-4 kernel: [] osd_trans_start+0xed/0x430 [osd_zfs] Nov 4 15:06:30 lola-4 kernel: [] ofd_trans_start+0x7c/0x100 [ofd] Nov 4 15:06:30 lola-4 kernel: [] ofd_object_destroy+0x3d0/0x8e0 [ofd] Nov 4 15:06:30 lola-4 kernel: [] ofd_destroy_by_fid+0x35d/0x620 [ofd] Nov 4 15:06:30 lola-4 kernel: [] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc] Nov 4 15:06:30 lola-4 kernel: [] ? ldlm_completion_ast+0x0/0x9a0 [ptlrpc] Nov 4 15:06:30 lola-4 kernel: [] ? libcfs_debug_msg+0x41/0x50 [libcfs] Nov 4 15:06:30 lola-4 kernel: [] ofd_destroy_hdl+0x2f2/0xb60 [ofd] Nov 4 15:06:30 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:06:30 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:06:30 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 15:06:30 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 15:06:30 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:06:30 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:06:30 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:06:30 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:06:30 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:06:30 lola-4 kernel: Nov 4 15:06:30 lola-4 kernel: LNet: Service thread pid 13079 was inactive for 1200.44s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 4 15:06:30 lola-4 kernel: LNet: Skipped 3 previous similar messages Nov 4 15:08:23 lola-4 kernel: LNet: Service thread pid 21750 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Nov 4 15:08:23 lola-4 kernel: LNet: Skipped 4 previous similar messages Nov 4 15:08:23 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478297303.21750 Nov 4 15:09:20 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 15:09:20 lola-4 kernel: req@ffff880510f5d850 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478297365 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 15:09:20 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 103 previous similar messages Nov 4 15:09:26 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 15:09:26 lola-4 kernel: Lustre: Skipped 21 previous similar messages Nov 4 15:09:26 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 15:09:26 lola-4 kernel: Lustre: Skipped 21 previous similar messages Nov 4 15:09:30 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892361 to 0x0:27892401 Nov 4 15:10:01 lola-4 CROND[36035]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 15:10:01 lola-4 CROND[36036]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 15:10:01 lola-4 TIME: Time stamp for console Nov 4 15:10:56 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 15:10:56 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 15:13:34 lola-4 kernel: Pid: 21708, comm: ll_ost01_084 Nov 4 15:13:34 lola-4 kernel: Nov 4 15:13:34 lola-4 kernel: Call Trace: Nov 4 15:13:34 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 15:13:34 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:13:34 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:13:34 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:13:34 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:13:34 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:13:34 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:13:34 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:13:34 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:13:34 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:13:34 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 15:13:34 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 15:13:34 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:13:34 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:13:34 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:13:34 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:13:34 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:13:34 lola-4 kernel: Nov 4 15:13:34 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478297614.21708 Nov 4 15:15:01 lola-4 CROND[36074]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 15:15:01 lola-4 TIME: Time stamp for console Nov 4 15:16:26 lola-4 kernel: LNet: Service thread pid 21172 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 15:16:26 lola-4 kernel: LNet: Skipped 5 previous similar messages Nov 4 15:16:26 lola-4 kernel: Pid: 21172, comm: ll_ost01_017 Nov 4 15:16:26 lola-4 kernel: Nov 4 15:16:26 lola-4 kernel: Call Trace: Nov 4 15:16:26 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 15:16:26 lola-4 kernel: [] ? cache_alloc_refill+0x9e/0x240 Nov 4 15:16:26 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:16:26 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:16:26 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:16:26 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:16:26 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:16:26 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:16:26 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:16:26 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:16:26 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:16:26 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 15:16:26 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 15:16:26 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:16:26 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:16:26 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:16:26 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:16:26 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:16:26 lola-4 kernel: Nov 4 15:16:26 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478297786.21172 Nov 4 15:18:24 lola-4 kernel: Pid: 21492, comm: ll_ost00_029 Nov 4 15:18:24 lola-4 kernel: Nov 4 15:18:24 lola-4 kernel: Call Trace: Nov 4 15:18:24 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:18:24 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:18:24 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:18:24 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:18:24 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:18:24 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:18:24 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:18:24 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:18:24 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:18:24 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 15:18:24 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 15:18:24 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:18:24 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:18:24 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:18:24 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:18:24 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:18:24 lola-4 kernel: Nov 4 15:18:24 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478297904.21492 Nov 4 15:18:51 lola-4 kernel: Pid: 21665, comm: ll_ost01_049 Nov 4 15:18:51 lola-4 kernel: Nov 4 15:18:51 lola-4 kernel: Call Trace: Nov 4 15:18:51 lola-4 kernel: [] ? account_entity_enqueue+0x7e/0x90 Nov 4 15:18:51 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:18:51 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:18:51 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:18:51 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:18:51 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:18:51 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:18:51 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:18:51 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:18:51 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:18:51 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 15:18:51 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:18:51 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:18:51 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:18:51 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:18:51 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:18:51 lola-4 kernel: Nov 4 15:18:51 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478297931.21665 Nov 4 15:19:21 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 15:19:21 lola-4 kernel: req@ffff88055b991450 x1550005401093744/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478297966 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 15:19:21 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 103 previous similar messages Nov 4 15:19:27 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 15:19:27 lola-4 kernel: Lustre: Skipped 21 previous similar messages Nov 4 15:19:27 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 15:19:27 lola-4 kernel: Lustre: Skipped 21 previous similar messages Nov 4 15:20:01 lola-4 CROND[36114]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 15:20:01 lola-4 CROND[36115]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 15:20:01 lola-4 TIME: Time stamp for console Nov 4 15:21:35 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 15:21:35 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 15:25:02 lola-4 CROND[36151]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 15:25:02 lola-4 TIME: Time stamp for console Nov 4 15:25:22 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892419 to 0x0:27892465 Nov 4 15:26:10 lola-4 kernel: Pid: 21704, comm: ll_ost01_080 Nov 4 15:26:10 lola-4 kernel: Nov 4 15:26:10 lola-4 kernel: Call Trace: Nov 4 15:26:10 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 15:26:10 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:26:10 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:26:10 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:26:10 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:26:10 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:26:10 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:26:10 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:26:10 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:26:10 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:26:10 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 15:26:10 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:26:10 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:26:10 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:26:10 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:26:10 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:26:10 lola-4 kernel: Nov 4 15:26:10 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478298370.21704 Nov 4 15:28:25 lola-4 kernel: LNet: Service thread pid 21629 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 15:28:25 lola-4 kernel: LNet: Skipped 3 previous similar messages Nov 4 15:28:25 lola-4 kernel: Pid: 21629, comm: ll_ost00_081 Nov 4 15:28:25 lola-4 kernel: Nov 4 15:28:25 lola-4 kernel: Call Trace: Nov 4 15:28:25 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:28:25 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:28:25 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:28:25 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:28:25 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:28:25 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:28:25 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:28:25 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:28:25 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:28:25 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 15:28:25 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:28:25 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:28:25 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:28:25 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:28:25 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:28:25 lola-4 kernel: Nov 4 15:28:25 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478298505.21629 Nov 4 15:29:02 lola-4 kernel: Pid: 21162, comm: ll_ost01_007 Nov 4 15:29:02 lola-4 kernel: Nov 4 15:29:02 lola-4 kernel: Call Trace: Nov 4 15:29:02 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 15:29:02 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:29:02 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:29:02 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:29:02 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:29:02 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:29:02 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:29:02 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:29:02 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:29:02 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:29:02 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 15:29:02 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:29:02 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:29:02 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:29:02 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:29:02 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:29:02 lola-4 kernel: Nov 4 15:29:02 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478298542.21162 Nov 4 15:29:22 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 15:29:22 lola-4 kernel: req@ffff8804ec0e5450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478298567 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 15:29:22 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 103 previous similar messages Nov 4 15:29:28 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 15:29:28 lola-4 kernel: Lustre: Skipped 21 previous similar messages Nov 4 15:29:28 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 15:29:28 lola-4 kernel: Lustre: Skipped 21 previous similar messages Nov 4 15:29:32 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892477 to 0x0:27892497 Nov 4 15:30:01 lola-4 CROND[36191]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 15:30:01 lola-4 CROND[36192]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 15:30:01 lola-4 TIME: Time stamp for console Nov 4 15:31:27 lola-4 kernel: Pid: 21467, comm: ll_ost01_037 Nov 4 15:31:27 lola-4 kernel: Nov 4 15:31:27 lola-4 kernel: Call Trace: Nov 4 15:31:27 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:31:27 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:31:27 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:31:27 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:31:27 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:31:27 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:31:27 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:31:27 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:31:27 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:31:27 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 15:31:27 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 15:31:27 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:31:27 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:31:27 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:31:28 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:31:28 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:31:28 lola-4 kernel: Nov 4 15:31:28 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478298688.21467 Nov 4 15:31:35 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 15:31:35 lola-4 kernel: Lustre: Skipped 11 previous similar messages Nov 4 15:35:01 lola-4 CROND[36231]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 15:35:01 lola-4 TIME: Time stamp for console Nov 4 15:38:26 lola-4 kernel: LNet: Service thread pid 21630 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 15:38:26 lola-4 kernel: LNet: Skipped 2 previous similar messages Nov 4 15:38:26 lola-4 kernel: Pid: 21630, comm: ll_ost00_082 Nov 4 15:38:26 lola-4 kernel: Nov 4 15:38:26 lola-4 kernel: Call Trace: Nov 4 15:38:26 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:38:26 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:38:26 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:38:26 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:38:26 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:38:26 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:38:26 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:38:26 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:38:26 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:38:26 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 15:38:26 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:38:26 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:38:26 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:38:26 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:38:26 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:38:26 lola-4 kernel: Nov 4 15:38:26 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478299106.21630 Nov 4 15:38:46 lola-4 kernel: Pid: 20338, comm: ll_ost01_006 Nov 4 15:38:46 lola-4 kernel: Nov 4 15:38:46 lola-4 kernel: Call Trace: Nov 4 15:38:46 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 15:38:46 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:38:46 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:38:46 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:38:46 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:38:46 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:38:46 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:38:46 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:38:46 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:38:46 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:38:46 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 15:38:46 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:38:46 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:38:46 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:38:46 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:38:46 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:38:46 lola-4 kernel: Nov 4 15:38:46 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478299126.20338 Nov 4 15:39:23 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 15:39:23 lola-4 kernel: req@ffff88052e622c50 x1550005401094688/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478299168 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 15:39:23 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 104 previous similar messages Nov 4 15:39:29 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 15:39:29 lola-4 kernel: Lustre: Skipped 21 previous similar messages Nov 4 15:39:29 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 15:39:29 lola-4 kernel: Lustre: Skipped 21 previous similar messages Nov 4 15:39:33 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892522 to 0x0:27892561 Nov 4 15:40:01 lola-4 CROND[36269]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 15:40:01 lola-4 CROND[36270]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 15:40:01 lola-4 TIME: Time stamp for console Nov 4 15:41:38 lola-4 kernel: Pid: 21390, comm: ll_ost01_029 Nov 4 15:41:38 lola-4 kernel: Nov 4 15:41:38 lola-4 kernel: Call Trace: Nov 4 15:41:38 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 15:41:38 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:41:38 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:41:38 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:41:38 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:41:38 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:41:38 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:41:38 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:41:38 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:41:38 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:41:38 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 15:41:38 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:41:38 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:41:38 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:41:38 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:41:38 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:41:38 lola-4 kernel: Nov 4 15:41:38 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478299298.21390 Nov 4 15:42:25 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 15:42:25 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 15:44:03 lola-4 kernel: Pid: 21176, comm: ll_ost01_021 Nov 4 15:44:03 lola-4 kernel: Nov 4 15:44:03 lola-4 kernel: Call Trace: Nov 4 15:44:03 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 15:44:03 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:44:03 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:44:03 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:44:03 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:44:03 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:44:03 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:44:03 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:44:03 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:44:03 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:44:03 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 15:44:04 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:44:04 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:44:04 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:44:04 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:44:04 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:44:04 lola-4 kernel: Nov 4 15:44:04 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478299444.21176 Nov 4 15:45:01 lola-4 CROND[36310]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 15:45:01 lola-4 TIME: Time stamp for console Nov 4 15:48:27 lola-4 kernel: LNet: Service thread pid 21597 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 15:48:27 lola-4 kernel: LNet: Skipped 3 previous similar messages Nov 4 15:48:27 lola-4 kernel: Pid: 21597, comm: ll_ost00_049 Nov 4 15:48:27 lola-4 kernel: Nov 4 15:48:27 lola-4 kernel: Call Trace: Nov 4 15:48:27 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:48:27 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:48:27 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:48:27 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:48:27 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:48:27 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:48:27 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:48:27 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:48:27 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:48:27 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 15:48:27 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:48:27 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:48:27 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:48:27 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:48:27 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:48:27 lola-4 kernel: Nov 4 15:48:27 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478299707.21597 Nov 4 15:49:24 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 15:49:24 lola-4 kernel: req@ffff880813956c50 x1550005401079776/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478299769 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 15:49:24 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 104 previous similar messages Nov 4 15:49:30 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 15:49:30 lola-4 kernel: Lustre: Skipped 21 previous similar messages Nov 4 15:49:30 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 15:49:30 lola-4 kernel: Lustre: Skipped 21 previous similar messages Nov 4 15:50:01 lola-4 CROND[36346]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 15:50:01 lola-4 CROND[36347]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 15:50:01 lola-4 TIME: Time stamp for console Nov 4 15:50:30 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892582 to 0x0:27892625 Nov 4 15:51:22 lola-4 kernel: Pid: 21389, comm: ll_ost01_028 Nov 4 15:51:22 lola-4 kernel: Nov 4 15:51:22 lola-4 kernel: Call Trace: Nov 4 15:51:22 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 15:51:22 lola-4 kernel: [] ? cache_alloc_refill+0x9e/0x240 Nov 4 15:51:22 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:51:22 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:51:22 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:51:22 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:51:22 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:51:22 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:51:22 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:51:22 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:51:22 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:51:22 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 15:51:22 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:51:22 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:51:22 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:51:22 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:51:22 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:51:22 lola-4 kernel: Nov 4 15:51:22 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478299882.21389 Nov 4 15:52:25 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 15:52:25 lola-4 kernel: Lustre: Skipped 11 previous similar messages Nov 4 15:54:14 lola-4 kernel: Pid: 21761, comm: ll_ost01_118 Nov 4 15:54:14 lola-4 kernel: Nov 4 15:54:14 lola-4 kernel: Call Trace: Nov 4 15:54:14 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 15:54:14 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:54:14 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:54:14 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:54:14 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:54:14 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:54:14 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:54:14 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:54:14 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:54:14 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:54:14 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 15:54:14 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 15:54:14 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:54:14 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:54:14 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:54:14 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:54:14 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:54:14 lola-4 kernel: Nov 4 15:54:14 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478300054.21761 Nov 4 15:55:01 lola-4 CROND[36387]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 15:55:01 lola-4 TIME: Time stamp for console Nov 4 15:56:39 lola-4 kernel: Pid: 21717, comm: ll_ost01_090 Nov 4 15:56:39 lola-4 kernel: Nov 4 15:56:39 lola-4 kernel: Call Trace: Nov 4 15:56:39 lola-4 kernel: [] ? vmstat_update+0x0/0x40 Nov 4 15:56:39 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:56:39 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:56:39 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:56:39 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:56:39 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:56:39 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:56:39 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:56:40 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:56:40 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:56:40 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 15:56:40 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:56:40 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:56:40 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:56:40 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:56:40 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:56:40 lola-4 kernel: Nov 4 15:56:40 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478300200.21717 Nov 4 15:58:28 lola-4 kernel: LNet: Service thread pid 21689 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 15:58:28 lola-4 kernel: LNet: Skipped 3 previous similar messages Nov 4 15:58:28 lola-4 kernel: Pid: 21689, comm: ll_ost00_098 Nov 4 15:58:28 lola-4 kernel: Nov 4 15:58:28 lola-4 kernel: Call Trace: Nov 4 15:58:28 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 15:58:28 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 15:58:28 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 15:58:28 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 15:58:28 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 15:58:28 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 15:58:28 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 15:58:28 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 15:58:28 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 15:58:28 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 15:58:28 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 15:58:28 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 15:58:28 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 15:58:28 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 15:58:28 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 15:58:28 lola-4 kernel: Nov 4 15:58:28 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478300308.21689 Nov 4 15:59:25 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 15:59:25 lola-4 kernel: req@ffff880809569850 x1550005401094688/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478300370 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 15:59:25 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 103 previous similar messages Nov 4 15:59:31 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 15:59:31 lola-4 kernel: Lustre: Skipped 21 previous similar messages Nov 4 15:59:31 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 15:59:31 lola-4 kernel: Lustre: Skipped 21 previous similar messages Nov 4 15:59:35 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892641 to 0x0:27892657 Nov 4 16:00:01 lola-4 CROND[36425]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 16:00:01 lola-4 CROND[36426]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 16:00:01 lola-4 TIME: Time stamp for console Nov 4 16:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 4 16:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 4 16:01:01 lola-4 CROND[36437]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 16:01:01 lola-4 run-parts(/etc/cron.hourly)[3643 starting 0anacron Nov 4 16:01:01 lola-4 run-parts(/etc/cron.hourly)[3644 finished 0anacron Nov 4 16:03:15 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 16:03:15 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 16:03:58 lola-4 kernel: Pid: 21675, comm: ll_ost01_059 Nov 4 16:03:58 lola-4 kernel: Nov 4 16:03:58 lola-4 kernel: Call Trace: Nov 4 16:03:58 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 16:03:58 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:03:58 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:03:58 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:03:58 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:03:58 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:03:58 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:03:58 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:03:58 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:03:58 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:03:58 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 16:03:58 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:03:58 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:03:58 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:03:58 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:03:58 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:03:58 lola-4 kernel: Nov 4 16:03:58 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478300638.21675 Nov 4 16:05:01 lola-4 CROND[36476]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 16:05:01 lola-4 TIME: Time stamp for console Nov 4 16:06:50 lola-4 kernel: Pid: 21659, comm: ll_ost01_043 Nov 4 16:06:50 lola-4 kernel: Nov 4 16:06:50 lola-4 kernel: Call Trace: Nov 4 16:06:50 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 16:06:50 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:06:50 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:06:50 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:06:50 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:06:50 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:06:50 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:06:50 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:06:50 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:06:50 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:06:50 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 16:06:50 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:06:50 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:06:50 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:06:50 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:06:50 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:06:50 lola-4 kernel: Nov 4 16:06:50 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478300810.21659 Nov 4 16:08:29 lola-4 kernel: LNet: Service thread pid 21623 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 16:08:29 lola-4 kernel: LNet: Skipped 2 previous similar messages Nov 4 16:08:29 lola-4 kernel: Pid: 21623, comm: ll_ost00_075 Nov 4 16:08:29 lola-4 kernel: Nov 4 16:08:29 lola-4 kernel: Call Trace: Nov 4 16:08:29 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:08:29 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:08:29 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:08:29 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:08:29 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:08:29 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:08:29 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:08:29 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:08:29 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:08:29 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 16:08:29 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:08:29 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:08:29 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:08:29 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:08:29 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:08:29 lola-4 kernel: Nov 4 16:08:29 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478300909.21623 Nov 4 16:09:15 lola-4 kernel: Pid: 21170, comm: ll_ost01_015 Nov 4 16:09:15 lola-4 kernel: Nov 4 16:09:15 lola-4 kernel: Call Trace: Nov 4 16:09:15 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:09:15 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:09:15 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:09:15 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:09:15 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:09:15 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:09:16 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:09:16 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:09:16 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:09:16 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 16:09:16 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:09:16 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:09:16 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:09:16 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:09:16 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:09:16 lola-4 kernel: Nov 4 16:09:16 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478300956.21170 Nov 4 16:09:26 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 16:09:26 lola-4 kernel: req@ffff880560929c50 x1550005401093744/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478300971 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 16:09:26 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 103 previous similar messages Nov 4 16:09:32 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 16:09:32 lola-4 kernel: Lustre: Skipped 21 previous similar messages Nov 4 16:09:32 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 16:09:32 lola-4 kernel: Lustre: Skipped 21 previous similar messages Nov 4 16:09:36 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892673 to 0x0:27892689 Nov 4 16:10:01 lola-4 CROND[36516]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 16:10:01 lola-4 CROND[36517]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 16:10:01 lola-4 TIME: Time stamp for console Nov 4 16:14:05 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 16:14:05 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 16:15:01 lola-4 CROND[36553]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 16:15:01 lola-4 TIME: Time stamp for console Nov 4 16:16:34 lola-4 kernel: Pid: 21713, comm: ll_ost01_087 Nov 4 16:16:34 lola-4 kernel: Nov 4 16:16:34 lola-4 kernel: Call Trace: Nov 4 16:16:34 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:16:34 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:16:34 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:16:34 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:16:34 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:16:34 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:16:34 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:16:34 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:16:34 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:16:34 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 16:16:34 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:16:34 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:16:34 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:16:34 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:16:34 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:16:34 lola-4 kernel: Nov 4 16:16:34 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478301394.21713 Nov 4 16:18:30 lola-4 kernel: LNet: Service thread pid 21755 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 16:18:30 lola-4 kernel: LNet: Skipped 2 previous similar messages Nov 4 16:18:30 lola-4 kernel: Pid: 21755, comm: ll_ost00_119 Nov 4 16:18:30 lola-4 kernel: Nov 4 16:18:30 lola-4 kernel: Call Trace: Nov 4 16:18:30 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:18:30 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:18:30 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:18:30 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:18:30 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:18:30 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:18:30 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:18:30 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:18:30 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:18:30 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 16:18:30 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 16:18:30 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:18:30 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:18:30 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:18:30 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:18:30 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:18:30 lola-4 kernel: Nov 4 16:18:30 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478301510.21755 Nov 4 16:19:26 lola-4 kernel: Pid: 21676, comm: ll_ost01_060 Nov 4 16:19:26 lola-4 kernel: Nov 4 16:19:26 lola-4 kernel: Call Trace: Nov 4 16:19:26 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:19:26 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:19:26 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:19:26 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:19:26 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:19:26 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:19:26 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:19:26 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:19:26 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:19:26 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 16:19:26 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:19:26 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:19:26 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:19:26 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:19:26 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:19:26 lola-4 kernel: Nov 4 16:19:26 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478301566.21676 Nov 4 16:19:27 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 16:19:27 lola-4 kernel: req@ffff8807fbf59450 x1550005401093744/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478301572 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 16:19:27 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 103 previous similar messages Nov 4 16:19:33 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 16:19:33 lola-4 kernel: Lustre: Skipped 21 previous similar messages Nov 4 16:19:33 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 16:19:33 lola-4 kernel: Lustre: Skipped 21 previous similar messages Nov 4 16:19:37 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892706 to 0x0:27892721 Nov 4 16:20:01 lola-4 CROND[36593]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 16:20:01 lola-4 CROND[36594]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 16:20:01 lola-4 TIME: Time stamp for console Nov 4 16:21:51 lola-4 kernel: Pid: 21719, comm: ll_ost01_091 Nov 4 16:21:51 lola-4 kernel: Nov 4 16:21:51 lola-4 kernel: Call Trace: Nov 4 16:21:51 lola-4 kernel: [] ? cache_alloc_refill+0x15b/0x240 Nov 4 16:21:51 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:21:51 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:21:51 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:21:51 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:21:52 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:21:52 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:21:52 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:21:52 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:21:52 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:21:52 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 16:21:52 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 16:21:52 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:21:52 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:21:52 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:21:52 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:21:52 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:21:52 lola-4 kernel: Nov 4 16:21:52 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478301712.21719 Nov 4 16:24:55 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 16:24:55 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 16:25:01 lola-4 CROND[36632]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 16:25:01 lola-4 TIME: Time stamp for console Nov 4 16:28:31 lola-4 kernel: LNet: Service thread pid 21476 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 16:28:31 lola-4 kernel: LNet: Skipped 2 previous similar messages Nov 4 16:28:31 lola-4 kernel: Pid: 21476, comm: ll_ost00_014 Nov 4 16:28:31 lola-4 kernel: Nov 4 16:28:31 lola-4 kernel: Call Trace: Nov 4 16:28:31 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:28:31 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:28:31 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:28:31 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:28:31 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:28:31 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:28:31 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:28:31 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:28:31 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:28:31 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 16:28:31 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 16:28:31 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:28:31 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:28:31 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:28:31 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:28:31 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:28:31 lola-4 kernel: Nov 4 16:28:31 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478302111.21476 Nov 4 16:29:10 lola-4 kernel: Pid: 21671, comm: ll_ost01_055 Nov 4 16:29:10 lola-4 kernel: Nov 4 16:29:10 lola-4 kernel: Call Trace: Nov 4 16:29:10 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 16:29:10 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:29:10 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:29:10 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:29:10 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:29:10 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:29:10 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:29:10 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:29:10 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:29:10 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:29:10 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 16:29:10 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:29:10 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:29:10 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:29:10 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:29:10 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:29:10 lola-4 kernel: Nov 4 16:29:10 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478302150.21671 Nov 4 16:29:28 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 16:29:28 lola-4 kernel: req@ffff880528979450 x1550005401097056/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478302173 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 16:29:28 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 4 16:29:34 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 16:29:34 lola-4 kernel: Lustre: Skipped 22 previous similar messages Nov 4 16:29:34 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 16:29:34 lola-4 kernel: Lustre: Skipped 22 previous similar messages Nov 4 16:29:38 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892738 to 0x0:27892753 Nov 4 16:30:01 lola-4 CROND[36670]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 16:30:01 lola-4 CROND[36671]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 16:30:01 lola-4 TIME: Time stamp for console Nov 4 16:32:02 lola-4 kernel: Pid: 21702, comm: ll_ost01_078 Nov 4 16:32:02 lola-4 kernel: Nov 4 16:32:02 lola-4 kernel: Call Trace: Nov 4 16:32:02 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 16:32:02 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:32:02 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:32:02 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:32:02 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:32:02 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:32:02 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:32:02 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:32:02 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:32:02 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:32:02 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 16:32:02 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 16:32:02 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:32:02 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:32:02 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:32:02 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:32:02 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:32:02 lola-4 kernel: Nov 4 16:32:02 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478302322.21702 Nov 4 16:34:27 lola-4 kernel: Pid: 21666, comm: ll_ost01_050 Nov 4 16:34:27 lola-4 kernel: Nov 4 16:34:27 lola-4 kernel: Call Trace: Nov 4 16:34:27 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 16:34:27 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:34:28 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:34:28 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:34:28 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:34:28 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:34:28 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:34:28 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:34:28 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:34:28 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:34:28 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 16:34:28 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:34:28 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:34:28 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:34:28 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:34:28 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:34:28 lola-4 kernel: Nov 4 16:34:28 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478302468.21666 Nov 4 16:35:01 lola-4 CROND[36711]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 16:35:01 lola-4 TIME: Time stamp for console Nov 4 16:35:45 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 16:35:45 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 16:38:32 lola-4 kernel: LNet: Service thread pid 21486 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 16:38:32 lola-4 kernel: LNet: Skipped 3 previous similar messages Nov 4 16:38:32 lola-4 kernel: Pid: 21486, comm: ll_ost00_023 Nov 4 16:38:32 lola-4 kernel: Nov 4 16:38:32 lola-4 kernel: Call Trace: Nov 4 16:38:32 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:38:32 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:38:32 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:38:32 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:38:32 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:38:32 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:38:32 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:38:32 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:38:32 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:38:32 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 16:38:32 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 16:38:32 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:38:32 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:38:32 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:38:32 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:38:32 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:38:32 lola-4 kernel: Nov 4 16:38:32 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478302712.21486 Nov 4 16:38:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 384 Nov 4 16:39:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 384 Nov 4 16:39:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 384 Nov 4 16:39:29 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 16:39:29 lola-4 kernel: req@ffff88067808e850 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478302774 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 16:39:29 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 104 previous similar messages Nov 4 16:39:35 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 16:39:35 lola-4 kernel: Lustre: Skipped 22 previous similar messages Nov 4 16:39:35 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 16:39:35 lola-4 kernel: Lustre: Skipped 22 previous similar messages Nov 4 16:39:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 384 Nov 4 16:39:39 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892769 to 0x0:27892785 Nov 4 16:39:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 384 Nov 4 16:40:02 lola-4 CROND[36748]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 16:40:02 lola-4 CROND[36749]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 16:40:02 lola-4 TIME: Time stamp for console Nov 4 16:40:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 385 Nov 4 16:40:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 385 Nov 4 16:40:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 385 Nov 4 16:40:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 385 Nov 4 16:41:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 385 Nov 4 16:41:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 385 Nov 4 16:41:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 385 Nov 4 16:41:46 lola-4 kernel: Pid: 21721, comm: ll_ost01_092 Nov 4 16:41:46 lola-4 kernel: Nov 4 16:41:46 lola-4 kernel: Call Trace: Nov 4 16:41:46 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:41:46 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:41:46 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:41:46 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:41:46 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:41:46 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:41:46 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:41:46 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:41:46 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:41:46 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 16:41:46 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 16:41:46 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:41:46 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:41:46 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:41:46 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:41:46 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:41:46 lola-4 kernel: Nov 4 16:41:46 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478302906.21721 Nov 4 16:41:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 385 Nov 4 16:42:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 385 Nov 4 16:42:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 386 Nov 4 16:42:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 386 Nov 4 16:42:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 386 Nov 4 16:43:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 386 Nov 4 16:43:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 386 Nov 4 16:43:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 386 Nov 4 16:43:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 386 Nov 4 16:44:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 386 Nov 4 16:44:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 385 Nov 4 16:44:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 385 Nov 4 16:44:38 lola-4 kernel: Pid: 21747, comm: ll_ost01_110 Nov 4 16:44:38 lola-4 kernel: Nov 4 16:44:38 lola-4 kernel: Call Trace: Nov 4 16:44:38 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 16:44:38 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:44:38 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:44:38 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:44:38 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:44:38 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:44:38 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:44:38 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:44:38 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:44:38 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:44:38 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 16:44:38 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 16:44:38 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:44:38 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:44:38 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:44:38 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:44:38 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:44:38 lola-4 kernel: Nov 4 16:44:38 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478303078.21747 Nov 4 16:44:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 385 Nov 4 16:45:01 lola-4 CROND[36789]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 16:45:01 lola-4 TIME: Time stamp for console Nov 4 16:45:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 385 Nov 4 16:45:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 385 Nov 4 16:45:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 385 Nov 4 16:45:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 385 Nov 4 16:46:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 385 Nov 4 16:46:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 386 Nov 4 16:46:35 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 16:46:35 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 16:46:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 386 Nov 4 16:46:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 386 Nov 4 16:47:04 lola-4 kernel: Pid: 21677, comm: ll_ost01_061 Nov 4 16:47:04 lola-4 kernel: Nov 4 16:47:04 lola-4 kernel: Call Trace: Nov 4 16:47:04 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:47:04 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:47:04 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:47:04 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:47:04 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:47:04 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:47:04 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:47:04 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:47:04 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:47:04 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 16:47:04 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:47:04 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:47:04 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:47:04 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:47:04 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:47:04 lola-4 kernel: Nov 4 16:47:04 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478303224.21677 Nov 4 16:47:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 386 Nov 4 16:47:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 386 Nov 4 16:47:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 386 Nov 4 16:47:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 386 Nov 4 16:48:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 386 Nov 4 16:48:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 386 Nov 4 16:48:33 lola-4 kernel: LNet: Service thread pid 21478 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 16:48:33 lola-4 kernel: LNet: Skipped 3 previous similar messages Nov 4 16:48:33 lola-4 kernel: Pid: 21478, comm: ll_ost00_016 Nov 4 16:48:33 lola-4 kernel: Nov 4 16:48:33 lola-4 kernel: Call Trace: Nov 4 16:48:33 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 16:48:33 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:48:33 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:48:33 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:48:33 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:48:33 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:48:33 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:48:33 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:48:33 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:48:33 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:48:33 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 16:48:33 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 16:48:33 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:48:33 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:48:33 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:48:33 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:48:33 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:48:33 lola-4 kernel: Nov 4 16:48:33 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478303313.21478 Nov 4 16:48:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 386 Nov 4 16:48:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 387 Nov 4 16:49:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 387 Nov 4 16:49:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 387 Nov 4 16:49:30 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 16:49:30 lola-4 kernel: req@ffff8807ee85d450 x1550005401079776/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478303375 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 16:49:30 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 104 previous similar messages Nov 4 16:49:36 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 16:49:36 lola-4 kernel: Lustre: Skipped 22 previous similar messages Nov 4 16:49:36 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 16:49:36 lola-4 kernel: Lustre: Skipped 22 previous similar messages Nov 4 16:49:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 387 Nov 4 16:49:40 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892801 to 0x0:27892817 Nov 4 16:49:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 387 Nov 4 16:50:01 lola-4 CROND[36827]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 16:50:01 lola-4 CROND[36828]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 16:50:01 lola-4 TIME: Time stamp for console Nov 4 16:50:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 388 Nov 4 16:50:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 388 Nov 4 16:50:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 388 Nov 4 16:50:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 388 Nov 4 16:51:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 388 Nov 4 16:51:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 388 Nov 4 16:51:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 388 Nov 4 16:51:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 388 Nov 4 16:52:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 388 Nov 4 16:52:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 388 Nov 4 16:52:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 389 Nov 4 16:52:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 389 Nov 4 16:53:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 389 Nov 4 16:53:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 389 Nov 4 16:53:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 389 Nov 4 16:53:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 389 Nov 4 16:54:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 389 Nov 4 16:54:22 lola-4 kernel: Pid: 21679, comm: ll_ost01_063 Nov 4 16:54:22 lola-4 kernel: Nov 4 16:54:22 lola-4 kernel: Call Trace: Nov 4 16:54:22 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 16:54:22 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:54:22 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:54:22 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:54:22 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:54:22 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:54:22 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:54:22 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:54:22 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:54:22 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:54:22 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 16:54:22 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 16:54:22 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:54:22 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:54:22 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:54:22 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:54:22 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:54:22 lola-4 kernel: Nov 4 16:54:22 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478303662.21679 Nov 4 16:54:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 389 Nov 4 16:54:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 389 Nov 4 16:54:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 389 Nov 4 16:55:01 lola-4 CROND[36866]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 16:55:01 lola-4 TIME: Time stamp for console Nov 4 16:55:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 389 Nov 4 16:55:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 389 Nov 4 16:55:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 389 Nov 4 16:55:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 390 Nov 4 16:56:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 390 Nov 4 16:56:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 390 Nov 4 16:56:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 390 Nov 4 16:56:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 390 Nov 4 16:57:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 390 Nov 4 16:57:14 lola-4 kernel: Pid: 21184, comm: ll_ost01_026 Nov 4 16:57:14 lola-4 kernel: Nov 4 16:57:14 lola-4 kernel: Call Trace: Nov 4 16:57:14 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:57:14 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:57:14 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:57:14 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:57:14 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:57:14 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:57:14 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:57:14 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:57:14 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:57:14 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 16:57:14 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:57:14 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:57:14 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:57:14 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:57:14 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:57:14 lola-4 kernel: Nov 4 16:57:14 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478303834.21184 Nov 4 16:57:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 390 Nov 4 16:57:25 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 16:57:25 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 16:57:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 390 Nov 4 16:57:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 390 Nov 4 16:58:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 390 Nov 4 16:58:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 390 Nov 4 16:58:34 lola-4 kernel: LNet: Service thread pid 21490 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 16:58:34 lola-4 kernel: LNet: Skipped 2 previous similar messages Nov 4 16:58:34 lola-4 kernel: Pid: 21490, comm: ll_ost00_027 Nov 4 16:58:34 lola-4 kernel: Nov 4 16:58:34 lola-4 kernel: Call Trace: Nov 4 16:58:34 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 16:58:34 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:58:34 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:58:34 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:58:34 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:58:34 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:58:34 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:58:34 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:58:35 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:58:35 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:58:35 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 16:58:35 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:58:35 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:58:35 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:58:35 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:58:35 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:58:35 lola-4 kernel: Nov 4 16:58:35 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478303915.21490 Nov 4 16:58:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 390 Nov 4 16:58:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 390 Nov 4 16:59:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 390 Nov 4 16:59:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 390 Nov 4 16:59:31 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 16:59:31 lola-4 kernel: req@ffff8806b572f050 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478303976 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 16:59:31 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 4 16:59:37 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 16:59:37 lola-4 kernel: Lustre: Skipped 22 previous similar messages Nov 4 16:59:37 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 16:59:37 lola-4 kernel: Lustre: Skipped 22 previous similar messages Nov 4 16:59:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 390 Nov 4 16:59:40 lola-4 kernel: Pid: 21466, comm: ll_ost01_036 Nov 4 16:59:40 lola-4 kernel: Nov 4 16:59:40 lola-4 kernel: Call Trace: Nov 4 16:59:40 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 16:59:40 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 16:59:40 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 16:59:40 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 16:59:40 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 16:59:40 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 16:59:40 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 16:59:40 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 16:59:40 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 16:59:40 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 16:59:40 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 16:59:40 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 16:59:40 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 16:59:40 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 16:59:40 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 16:59:40 lola-4 kernel: Nov 4 16:59:40 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478303980.21466 Nov 4 16:59:41 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892828 to 0x0:27892849 Nov 4 16:59:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 391 Nov 4 17:00:01 lola-4 CROND[36906]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 17:00:01 lola-4 CROND[36907]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 17:00:01 lola-4 TIME: Time stamp for console Nov 4 17:00:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 391 Nov 4 17:00:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 391 Nov 4 17:00:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 391 Nov 4 17:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 4 17:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 4 17:00:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 391 Nov 4 17:01:01 lola-4 CROND[36918]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 17:01:01 lola-4 run-parts(/etc/cron.hourly)[3691 starting 0anacron Nov 4 17:01:01 lola-4 run-parts(/etc/cron.hourly)[3692 finished 0anacron Nov 4 17:01:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 391 Nov 4 17:01:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 391 Nov 4 17:01:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 391 Nov 4 17:01:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 391 Nov 4 17:02:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 391 Nov 4 17:02:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 391 Nov 4 17:02:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 391 Nov 4 17:02:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 392 Nov 4 17:03:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 392 Nov 4 17:03:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 392 Nov 4 17:03:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 392 Nov 4 17:03:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 392 Nov 4 17:04:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 392 Nov 4 17:04:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 392 Nov 4 17:04:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 392 Nov 4 17:04:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 392 Nov 4 17:05:01 lola-4 CROND[36954]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 17:05:01 lola-4 TIME: Time stamp for console Nov 4 17:05:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 392 Nov 4 17:05:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 392 Nov 4 17:05:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 393 Nov 4 17:05:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 393 Nov 4 17:06:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 393 Nov 4 17:06:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 393 Nov 4 17:06:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 393 Nov 4 17:06:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 393 Nov 4 17:06:58 lola-4 kernel: Pid: 21682, comm: ll_ost01_066 Nov 4 17:06:58 lola-4 kernel: Nov 4 17:06:58 lola-4 kernel: Call Trace: Nov 4 17:06:58 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 17:06:58 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 17:06:58 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 17:06:58 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 17:06:58 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 17:06:58 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 17:06:58 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 17:06:58 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 17:06:58 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 17:06:58 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 17:06:58 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 17:06:58 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 17:06:58 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 17:06:58 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 17:06:58 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 17:06:58 lola-4 kernel: Nov 4 17:06:58 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478304418.21682 Nov 4 17:07:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 393 Nov 4 17:07:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 393 Nov 4 17:07:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 393 Nov 4 17:07:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 393 Nov 4 17:08:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 393 Nov 4 17:08:15 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 17:08:15 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 17:08:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 393 Nov 4 17:08:35 lola-4 kernel: LNet: Service thread pid 13363 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 17:08:35 lola-4 kernel: LNet: Skipped 2 previous similar messages Nov 4 17:08:35 lola-4 kernel: Pid: 13363, comm: ll_ost00_004 Nov 4 17:08:36 lola-4 kernel: Nov 4 17:08:36 lola-4 kernel: Call Trace: Nov 4 17:08:36 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 17:08:36 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 17:08:36 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 17:08:36 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 17:08:36 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 17:08:36 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 17:08:36 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 17:08:36 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 17:08:36 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 17:08:36 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 17:08:36 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 17:08:36 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 17:08:36 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 17:08:36 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 17:08:36 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 17:08:36 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 17:08:36 lola-4 kernel: Nov 4 17:08:36 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478304516.13363 Nov 4 17:08:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 393 Nov 4 17:08:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 394 Nov 4 17:09:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 394 Nov 4 17:09:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 394 Nov 4 17:09:32 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 17:09:32 lola-4 kernel: req@ffff8805d76b2050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478304577 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 17:09:32 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 104 previous similar messages Nov 4 17:09:38 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 17:09:38 lola-4 kernel: Lustre: Skipped 22 previous similar messages Nov 4 17:09:38 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 17:09:38 lola-4 kernel: Lustre: Skipped 22 previous similar messages Nov 4 17:09:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 394 Nov 4 17:09:42 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892864 to 0x0:27892881 Nov 4 17:09:50 lola-4 kernel: Pid: 21657, comm: ll_ost01_041 Nov 4 17:09:50 lola-4 kernel: Nov 4 17:09:50 lola-4 kernel: Call Trace: Nov 4 17:09:50 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 17:09:50 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 17:09:50 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 17:09:50 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 17:09:50 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 17:09:50 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 17:09:50 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 17:09:50 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 17:09:51 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 17:09:51 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 17:09:51 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 17:09:51 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 17:09:51 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 17:09:51 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 17:09:51 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 17:09:51 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 17:09:51 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 17:09:51 lola-4 kernel: Nov 4 17:09:51 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478304591.21657 Nov 4 17:09:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 394 Nov 4 17:10:01 lola-4 CROND[36994]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 17:10:01 lola-4 CROND[36995]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 17:10:01 lola-4 TIME: Time stamp for console Nov 4 17:10:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 394 Nov 4 17:10:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 394 Nov 4 17:10:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 394 Nov 4 17:10:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 395 Nov 4 17:11:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 395 Nov 4 17:11:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 395 Nov 4 17:11:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 395 Nov 4 17:11:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 395 Nov 4 17:12:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 395 Nov 4 17:12:16 lola-4 kernel: Pid: 21729, comm: ll_ost01_096 Nov 4 17:12:16 lola-4 kernel: Nov 4 17:12:16 lola-4 kernel: Call Trace: Nov 4 17:12:16 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 17:12:16 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 17:12:16 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 17:12:16 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 17:12:16 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 17:12:16 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 17:12:16 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 17:12:16 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 17:12:16 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 17:12:16 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 17:12:16 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 17:12:16 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 17:12:16 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 17:12:16 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 17:12:16 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 17:12:16 lola-4 kernel: Nov 4 17:12:16 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478304736.21729 Nov 4 17:12:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 395 Nov 4 17:12:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 395 Nov 4 17:12:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 395 Nov 4 17:13:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 395 Nov 4 17:13:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 395 Nov 4 17:13:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 395 Nov 4 17:13:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 395 Nov 4 17:14:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 395 Nov 4 17:14:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 396 Nov 4 17:14:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 396 Nov 4 17:14:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 396 Nov 4 17:15:01 lola-4 CROND[37034]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 17:15:01 lola-4 TIME: Time stamp for console Nov 4 17:15:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 396 Nov 4 17:15:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 396 Nov 4 17:15:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 396 Nov 4 17:15:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 396 Nov 4 17:16:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 396 Nov 4 17:16:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 396 Nov 4 17:16:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 396 Nov 4 17:16:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 396 Nov 4 17:17:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 396 Nov 4 17:17:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 396 Nov 4 17:17:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 397 Nov 4 17:17:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 397 Nov 4 17:18:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 397 Nov 4 17:18:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 397 Nov 4 17:18:37 lola-4 kernel: LNet: Service thread pid 21591 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 17:18:37 lola-4 kernel: LNet: Skipped 2 previous similar messages Nov 4 17:18:37 lola-4 kernel: Pid: 21591, comm: ll_ost00_043 Nov 4 17:18:37 lola-4 kernel: Nov 4 17:18:37 lola-4 kernel: Call Trace: Nov 4 17:18:37 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 17:18:37 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 17:18:37 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 17:18:37 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 17:18:37 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 17:18:37 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 17:18:37 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 17:18:37 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 17:18:37 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 17:18:37 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 17:18:37 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 17:18:37 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 17:18:37 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 17:18:37 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 17:18:37 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 17:18:37 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 17:18:37 lola-4 kernel: Nov 4 17:18:37 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478305117.21591 Nov 4 17:18:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 397 Nov 4 17:18:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:19:05 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 17:19:05 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 17:19:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:19:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:19:33 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 17:19:33 lola-4 kernel: req@ffff8807bf625050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478305178 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 17:19:33 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 4 17:19:34 lola-4 kernel: Pid: 21756, comm: ll_ost01_113 Nov 4 17:19:34 lola-4 kernel: Nov 4 17:19:34 lola-4 kernel: Call Trace: Nov 4 17:19:34 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 17:19:34 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 17:19:34 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 17:19:34 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 17:19:34 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 17:19:34 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 17:19:34 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 17:19:34 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 17:19:34 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 17:19:34 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 17:19:34 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 17:19:34 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 17:19:34 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 17:19:34 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 17:19:34 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 17:19:34 lola-4 kernel: Nov 4 17:19:34 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478305174.21756 Nov 4 17:19:39 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 17:19:39 lola-4 kernel: Lustre: Skipped 22 previous similar messages Nov 4 17:19:39 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 17:19:39 lola-4 kernel: Lustre: Skipped 22 previous similar messages Nov 4 17:19:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:19:43 lola-4 kernel: Lustre: soaked-OST0008: deleting orphan objects from 0x0:27892898 to 0x0:27892913 Nov 4 17:19:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:20:01 lola-4 CROND[37072]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 17:20:01 lola-4 CROND[37073]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 17:20:01 lola-4 TIME: Time stamp for console Nov 4 17:20:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:20:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:20:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:20:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:21:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:21:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:21:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:21:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:22:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:22:27 lola-4 kernel: Pid: 21722, comm: ll_ost01_093 Nov 4 17:22:27 lola-4 kernel: Nov 4 17:22:27 lola-4 kernel: Call Trace: Nov 4 17:22:27 lola-4 kernel: [] ? account_entity_enqueue+0x7e/0x90 Nov 4 17:22:27 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 17:22:27 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 17:22:27 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 17:22:27 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 17:22:27 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 17:22:27 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 17:22:27 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 17:22:27 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 17:22:27 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 17:22:27 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 17:22:27 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 17:22:27 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 17:22:27 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 17:22:27 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 17:22:27 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 17:22:27 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 17:22:27 lola-4 kernel: Nov 4 17:22:27 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478305347.21722 Nov 4 17:22:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:22:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:22:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:23:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 399 Nov 4 17:23:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 399 Nov 4 17:23:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:23:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 399 Nov 4 17:24:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:24:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 399 Nov 4 17:24:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 398 Nov 4 17:24:52 lola-4 kernel: Pid: 21174, comm: ll_ost01_019 Nov 4 17:24:52 lola-4 kernel: Nov 4 17:24:52 lola-4 kernel: Call Trace: Nov 4 17:24:52 lola-4 kernel: [] ? select_idle_sibling+0x95/0x150 Nov 4 17:24:52 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 17:24:52 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 17:24:52 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 17:24:52 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 17:24:52 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 17:24:52 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 17:24:52 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 17:24:52 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 17:24:52 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 17:24:52 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 17:24:52 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 17:24:52 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 17:24:52 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 17:24:52 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 17:24:52 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 17:24:52 lola-4 kernel: Nov 4 17:24:52 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478305492.21174 Nov 4 17:24:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 399 Nov 4 17:25:01 lola-4 CROND[37113]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 17:25:01 lola-4 TIME: Time stamp for console Nov 4 17:25:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 399 Nov 4 17:25:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 399 Nov 4 17:25:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 399 Nov 4 17:25:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 399 Nov 4 17:26:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 399 Nov 4 17:26:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 399 Nov 4 17:26:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 399 Nov 4 17:26:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 399 Nov 4 17:27:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 399 Nov 4 17:27:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 399 Nov 4 17:27:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 399 Nov 4 17:27:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 400 Nov 4 17:28:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 400 Nov 4 17:28:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 400 Nov 4 17:28:38 lola-4 kernel: LNet: Service thread pid 21589 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 17:28:38 lola-4 kernel: LNet: Skipped 3 previous similar messages Nov 4 17:28:38 lola-4 kernel: Pid: 21589, comm: ll_ost00_041 Nov 4 17:28:38 lola-4 kernel: Nov 4 17:28:38 lola-4 kernel: Call Trace: Nov 4 17:28:38 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 17:28:38 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 17:28:38 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 17:28:38 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 17:28:38 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 17:28:38 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 17:28:38 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 17:28:38 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 17:28:38 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 17:28:38 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 17:28:38 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 17:28:38 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 17:28:38 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 17:28:38 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 17:28:38 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 17:28:38 lola-4 kernel: Nov 4 17:28:38 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478305718.21589 Nov 4 17:28:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 400 Nov 4 17:28:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:29:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:29:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:29:34 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 17:29:34 lola-4 kernel: req@ffff8807f87cb050 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478305779 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 17:29:34 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 104 previous similar messages Nov 4 17:29:40 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 17:29:40 lola-4 kernel: Lustre: Skipped 22 previous similar messages Nov 4 17:29:40 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 17:29:40 lola-4 kernel: Lustre: Skipped 22 previous similar messages Nov 4 17:29:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:29:55 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 17:29:55 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 17:29:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:30:01 lola-4 CROND[37149]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 17:30:01 lola-4 CROND[37150]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 17:30:01 lola-4 TIME: Time stamp for console Nov 4 17:30:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:30:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:30:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:30:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:31:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:31:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:31:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:31:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:32:10 lola-4 kernel: Pid: 21705, comm: ll_ost01_081 Nov 4 17:32:10 lola-4 kernel: Nov 4 17:32:10 lola-4 kernel: Call Trace: Nov 4 17:32:10 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 17:32:10 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 17:32:10 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 17:32:10 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 17:32:10 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 17:32:10 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 17:32:10 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 17:32:10 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 17:32:10 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 17:32:10 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 17:32:10 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 17:32:10 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 17:32:10 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 17:32:10 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 17:32:10 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 17:32:10 lola-4 kernel: Nov 4 17:32:10 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478305930.21705 Nov 4 17:32:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:32:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:32:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:32:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:33:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:33:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:33:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:33:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:34:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:34:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:34:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:34:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:35:01 lola-4 CROND[37188]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 17:35:01 lola-4 TIME: Time stamp for console Nov 4 17:35:03 lola-4 kernel: Pid: 21699, comm: ll_ost01_075 Nov 4 17:35:03 lola-4 kernel: Nov 4 17:35:03 lola-4 kernel: Call Trace: Nov 4 17:35:03 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 17:35:03 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 17:35:03 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 17:35:03 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 17:35:03 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 17:35:03 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 17:35:03 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 17:35:03 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 17:35:03 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 17:35:03 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 17:35:03 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 17:35:03 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 17:35:03 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 17:35:03 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 17:35:03 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 17:35:03 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 17:35:03 lola-4 kernel: Nov 4 17:35:03 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478306103.21699 Nov 4 17:35:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:35:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:35:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:35:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:36:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:36:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:36:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:36:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:37:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:37:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:37:28 lola-4 kernel: Pid: 21736, comm: ll_ost01_101 Nov 4 17:37:28 lola-4 kernel: Nov 4 17:37:28 lola-4 kernel: Call Trace: Nov 4 17:37:28 lola-4 kernel: [] ? account_entity_enqueue+0x7e/0x90 Nov 4 17:37:28 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 17:37:28 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 17:37:28 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 17:37:28 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 17:37:28 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 17:37:28 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 17:37:28 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 17:37:28 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 17:37:28 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 17:37:28 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 17:37:28 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 17:37:28 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 17:37:28 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 17:37:28 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 17:37:28 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 17:37:28 lola-4 kernel: Nov 4 17:37:28 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478306248.21736 Nov 4 17:37:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:37:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:38:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:38:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:38:39 lola-4 kernel: LNet: Service thread pid 21718 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 17:38:39 lola-4 kernel: LNet: Skipped 3 previous similar messages Nov 4 17:38:39 lola-4 kernel: Pid: 21718, comm: ll_ost00_104 Nov 4 17:38:39 lola-4 kernel: Nov 4 17:38:39 lola-4 kernel: Call Trace: Nov 4 17:38:39 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 17:38:39 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 17:38:39 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 17:38:39 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 17:38:39 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 17:38:39 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 17:38:39 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 17:38:39 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 17:38:39 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 17:38:39 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 17:38:39 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 17:38:39 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 17:38:39 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 17:38:39 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 17:38:39 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 17:38:39 lola-4 kernel: Nov 4 17:38:39 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478306319.21718 Nov 4 17:38:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 401 Nov 4 17:38:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:39:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:39:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:39:35 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 17:39:35 lola-4 kernel: req@ffff88071ca79050 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478306380 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 17:39:35 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 104 previous similar messages Nov 4 17:39:41 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 17:39:41 lola-4 kernel: Lustre: Skipped 22 previous similar messages Nov 4 17:39:41 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 17:39:41 lola-4 kernel: Lustre: Skipped 22 previous similar messages Nov 4 17:39:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:39:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:40:01 lola-4 CROND[37228]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 17:40:01 lola-4 CROND[37229]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 17:40:01 lola-4 TIME: Time stamp for console Nov 4 17:40:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:40:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:40:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:40:45 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 17:40:45 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 17:40:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:41:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:41:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:41:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:41:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:42:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:42:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:42:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:42:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:43:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:43:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:43:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:43:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:44:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:44:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:44:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:44:46 lola-4 kernel: Pid: 21678, comm: ll_ost01_062 Nov 4 17:44:46 lola-4 kernel: Nov 4 17:44:46 lola-4 kernel: Call Trace: Nov 4 17:44:46 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 17:44:46 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 17:44:46 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 17:44:46 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 17:44:46 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 17:44:46 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 17:44:46 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 17:44:46 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 17:44:46 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 17:44:46 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 17:44:46 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 17:44:46 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 17:44:46 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 17:44:46 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 17:44:46 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 17:44:46 lola-4 kernel: Nov 4 17:44:46 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478306686.21678 Nov 4 17:44:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:45:01 lola-4 CROND[37268]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 17:45:01 lola-4 TIME: Time stamp for console Nov 4 17:45:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:45:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:45:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:45:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 402 Nov 4 17:46:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:46:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:46:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:46:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:47:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:47:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:47:39 lola-4 kernel: Pid: 21182, comm: ll_ost01_024 Nov 4 17:47:39 lola-4 kernel: Nov 4 17:47:39 lola-4 kernel: Call Trace: Nov 4 17:47:39 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 17:47:39 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 17:47:39 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 17:47:39 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 17:47:39 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 17:47:39 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 17:47:39 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 17:47:39 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 17:47:39 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 17:47:39 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 17:47:39 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 17:47:39 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 17:47:39 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 17:47:39 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 17:47:39 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 17:47:39 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 17:47:39 lola-4 kernel: Nov 4 17:47:39 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478306859.21182 Nov 4 17:47:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:47:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:48:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:48:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:48:40 lola-4 kernel: LNet: Service thread pid 21595 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 17:48:40 lola-4 kernel: LNet: Skipped 2 previous similar messages Nov 4 17:48:40 lola-4 kernel: Pid: 21595, comm: ll_ost00_047 Nov 4 17:48:40 lola-4 kernel: Nov 4 17:48:40 lola-4 kernel: Call Trace: Nov 4 17:48:40 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 17:48:40 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 17:48:40 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 17:48:40 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 17:48:40 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 17:48:40 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 17:48:40 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 17:48:40 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 17:48:40 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 17:48:40 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 17:48:40 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 17:48:40 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 17:48:40 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 17:48:40 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 17:48:40 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 17:48:40 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 17:48:40 lola-4 kernel: Nov 4 17:48:40 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478306920.21595 Nov 4 17:48:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:48:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:49:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:49:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:49:36 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 17:49:36 lola-4 kernel: req@ffff8806385a9450 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478306981 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 17:49:36 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 107 previous similar messages Nov 4 17:49:42 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 17:49:42 lola-4 kernel: Lustre: Skipped 24 previous similar messages Nov 4 17:49:42 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 17:49:42 lola-4 kernel: Lustre: Skipped 24 previous similar messages Nov 4 17:49:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:49:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:50:01 lola-4 CROND[37306]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 17:50:01 lola-4 CROND[37307]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 17:50:01 lola-4 TIME: Time stamp for console Nov 4 17:50:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:50:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:50:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:50:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:51:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:51:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:51:35 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 17:51:35 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 17:51:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:51:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:52:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:52:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:52:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:52:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:53:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:53:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:53:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:53:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:54:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:54:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:54:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:54:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:55:01 lola-4 CROND[37343]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 17:55:01 lola-4 TIME: Time stamp for console Nov 4 17:55:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:55:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:55:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:55:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:56:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:56:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:56:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:56:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:57:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:57:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:57:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:57:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:58:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:58:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:58:41 lola-4 kernel: LNet: Service thread pid 21639 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 17:58:41 lola-4 kernel: Pid: 21639, comm: ll_ost00_091 Nov 4 17:58:41 lola-4 kernel: Nov 4 17:58:41 lola-4 kernel: Call Trace: Nov 4 17:58:41 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 17:58:41 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 17:58:41 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 17:58:41 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 17:58:41 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 17:58:41 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 17:58:41 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 17:58:41 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 17:58:41 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 17:58:41 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 17:58:41 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 17:58:41 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 17:58:41 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 17:58:41 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 17:58:41 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 17:58:41 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 17:58:41 lola-4 kernel: Nov 4 17:58:41 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478307521.21639 Nov 4 17:58:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:58:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 403 Nov 4 17:59:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:59:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:59:37 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 17:59:37 lola-4 kernel: req@ffff8807b66c3450 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478307582 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 17:59:37 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 108 previous similar messages Nov 4 17:59:43 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 17:59:43 lola-4 kernel: Lustre: Skipped 24 previous similar messages Nov 4 17:59:43 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 17:59:43 lola-4 kernel: Lustre: Skipped 24 previous similar messages Nov 4 17:59:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 17:59:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:00:01 lola-4 CROND[37379]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 18:00:01 lola-4 CROND[37380]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 18:00:01 lola-4 TIME: Time stamp for console Nov 4 18:00:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:00:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 4 18:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 4 18:00:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:01:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:01:01 lola-4 CROND[37391]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 18:01:01 lola-4 run-parts(/etc/cron.hourly)[3739 starting 0anacron Nov 4 18:01:01 lola-4 run-parts(/etc/cron.hourly)[3740 finished 0anacron Nov 4 18:01:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:01:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:01:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:02:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:02:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:02:25 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 18:02:25 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 18:02:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:02:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:03:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:03:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:03:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:03:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:04:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:04:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:04:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:04:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:05:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:05:01 lola-4 CROND[37427]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 18:05:01 lola-4 TIME: Time stamp for console Nov 4 18:05:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:05:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:05:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:06:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:06:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:06:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:06:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:07:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:07:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:07:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:07:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:08:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:08:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:08:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:08:42 lola-4 kernel: LNet: Service thread pid 21691 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 18:08:42 lola-4 kernel: Pid: 21691, comm: ll_ost00_099 Nov 4 18:08:42 lola-4 kernel: Nov 4 18:08:42 lola-4 kernel: Call Trace: Nov 4 18:08:42 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 18:08:42 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 18:08:42 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 18:08:42 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 18:08:42 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 18:08:42 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 18:08:42 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 18:08:42 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 18:08:42 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 18:08:42 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 18:08:42 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 18:08:42 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 18:08:42 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 18:08:42 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 18:08:42 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 18:08:42 lola-4 kernel: Nov 4 18:08:42 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478308122.21691 Nov 4 18:08:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 404 Nov 4 18:09:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:09:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:09:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:09:38 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 18:09:38 lola-4 kernel: req@ffff8805d0e26450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478308183 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 18:09:38 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 108 previous similar messages Nov 4 18:09:44 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 18:09:44 lola-4 kernel: Lustre: Skipped 24 previous similar messages Nov 4 18:09:44 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 18:09:44 lola-4 kernel: Lustre: Skipped 24 previous similar messages Nov 4 18:09:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:10:01 lola-4 CROND[37463]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 18:10:01 lola-4 CROND[37464]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 18:10:01 lola-4 TIME: Time stamp for console Nov 4 18:10:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:10:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:10:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:10:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:11:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:11:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:11:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:11:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:12:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:12:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:12:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:12:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:13:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:13:15 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 18:13:15 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 18:13:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:13:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:13:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:14:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:14:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:14:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:14:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:15:01 lola-4 CROND[37500]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 18:15:01 lola-4 TIME: Time stamp for console Nov 4 18:15:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:15:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:15:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:15:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:16:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:16:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:16:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:16:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:17:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:17:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:17:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:17:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:18:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:18:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:18:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:18:43 lola-4 kernel: LNet: Service thread pid 21596 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 18:18:43 lola-4 kernel: Pid: 21596, comm: ll_ost00_048 Nov 4 18:18:43 lola-4 kernel: Nov 4 18:18:43 lola-4 kernel: Call Trace: Nov 4 18:18:43 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 18:18:43 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 18:18:43 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 18:18:43 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 18:18:43 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 18:18:43 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 18:18:43 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 18:18:43 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 18:18:43 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 18:18:43 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 18:18:43 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 18:18:43 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 18:18:43 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 18:18:43 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 18:18:43 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 18:18:43 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 18:18:43 lola-4 kernel: Nov 4 18:18:43 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478308723.21596 Nov 4 18:18:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 405 Nov 4 18:19:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:19:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:19:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:19:39 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 18:19:39 lola-4 kernel: req@ffff880719b2e450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478308784 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 18:19:39 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 106 previous similar messages Nov 4 18:19:45 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 18:19:45 lola-4 kernel: Lustre: Skipped 24 previous similar messages Nov 4 18:19:45 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 18:19:45 lola-4 kernel: Lustre: Skipped 24 previous similar messages Nov 4 18:19:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:20:01 lola-4 CROND[37537]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 18:20:01 lola-4 CROND[37538]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 18:20:01 lola-4 TIME: Time stamp for console Nov 4 18:20:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:20:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:20:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:20:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:21:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:21:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:21:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:21:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:22:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:22:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:22:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:22:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:23:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:23:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:23:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:23:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:24:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:24:05 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 18:24:05 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 18:24:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:24:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:24:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:25:01 lola-4 CROND[37574]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 18:25:01 lola-4 TIME: Time stamp for console Nov 4 18:25:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:25:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:25:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 406 Nov 4 18:25:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:26:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:26:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:26:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:26:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:27:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:27:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:27:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:27:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:28:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:28:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:28:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:28:44 lola-4 kernel: LNet: Service thread pid 21606 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 18:28:44 lola-4 kernel: Pid: 21606, comm: ll_ost00_058 Nov 4 18:28:44 lola-4 kernel: Nov 4 18:28:44 lola-4 kernel: Call Trace: Nov 4 18:28:44 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 18:28:44 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 18:28:44 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 18:28:44 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 18:28:44 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 18:28:44 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 18:28:44 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 18:28:44 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 18:28:44 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 18:28:44 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 18:28:44 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 18:28:44 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 18:28:44 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 18:28:44 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 18:28:44 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 18:28:44 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 18:28:44 lola-4 kernel: Nov 4 18:28:44 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478309324.21606 Nov 4 18:28:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:29:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:29:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:29:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:29:40 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 18:29:40 lola-4 kernel: req@ffff8807cfc9c450 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478309385 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 18:29:40 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 107 previous similar messages Nov 4 18:29:46 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 18:29:46 lola-4 kernel: Lustre: Skipped 24 previous similar messages Nov 4 18:29:46 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 18:29:46 lola-4 kernel: Lustre: Skipped 24 previous similar messages Nov 4 18:29:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:30:01 lola-4 CROND[37610]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 18:30:01 lola-4 CROND[37611]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 18:30:01 lola-4 TIME: Time stamp for console Nov 4 18:30:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:30:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:30:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:30:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:31:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:31:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:31:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:31:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:32:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:32:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:32:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:32:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:33:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:33:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:33:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:33:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:34:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:34:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:34:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:34:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:34:55 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 18:34:55 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 18:35:01 lola-4 CROND[37647]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 18:35:01 lola-4 TIME: Time stamp for console Nov 4 18:35:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:35:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:35:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:35:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:36:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:36:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:36:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:36:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:37:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:37:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:37:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:37:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:38:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:38:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:38:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:38:45 lola-4 kernel: LNet: Service thread pid 21620 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 18:38:45 lola-4 kernel: Pid: 21620, comm: ll_ost00_072 Nov 4 18:38:45 lola-4 kernel: Nov 4 18:38:45 lola-4 kernel: Call Trace: Nov 4 18:38:45 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 18:38:45 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 18:38:45 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 18:38:45 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 18:38:45 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 18:38:45 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 18:38:45 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 18:38:45 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 18:38:45 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 18:38:45 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 18:38:45 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 18:38:45 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 18:38:45 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 18:38:45 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 18:38:45 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 18:38:45 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 18:38:45 lola-4 kernel: Nov 4 18:38:45 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478309925.21620 Nov 4 18:38:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 407 Nov 4 18:39:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:39:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:39:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:39:41 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 18:39:41 lola-4 kernel: req@ffff880766c3bc50 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478309986 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 18:39:41 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 109 previous similar messages Nov 4 18:39:47 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 18:39:47 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 4 18:39:47 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 18:39:47 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 4 18:39:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:40:01 lola-4 CROND[37683]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 18:40:01 lola-4 CROND[37684]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 18:40:01 lola-4 TIME: Time stamp for console Nov 4 18:40:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:40:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:40:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:40:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:41:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:41:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:41:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:41:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:42:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:42:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:42:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:42:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:43:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:43:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:43:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:43:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:44:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:44:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:44:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:44:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:45:01 lola-4 CROND[37720]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 18:45:01 lola-4 TIME: Time stamp for console Nov 4 18:45:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:45:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:45:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:45:45 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 18:45:45 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 18:45:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:46:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:46:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:46:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:46:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:47:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:47:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:47:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:47:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:48:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:48:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:48:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:48:46 lola-4 kernel: LNet: Service thread pid 21612 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 18:48:46 lola-4 kernel: Pid: 21612, comm: ll_ost00_064 Nov 4 18:48:46 lola-4 kernel: Nov 4 18:48:46 lola-4 kernel: Call Trace: Nov 4 18:48:46 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 18:48:46 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 18:48:46 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 18:48:46 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 18:48:46 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 18:48:46 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 18:48:46 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 18:48:46 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 18:48:46 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 18:48:46 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 18:48:46 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 18:48:46 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 18:48:46 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 18:48:46 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 18:48:46 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 18:48:46 lola-4 kernel: Nov 4 18:48:46 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478310526.21612 Nov 4 18:48:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:49:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 408 Nov 4 18:49:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:49:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:49:42 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 18:49:42 lola-4 kernel: req@ffff880506791850 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478310587 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 18:49:42 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 110 previous similar messages Nov 4 18:49:48 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 18:49:48 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 4 18:49:48 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 18:49:48 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 4 18:49:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:50:01 lola-4 CROND[37756]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 18:50:01 lola-4 CROND[37757]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 18:50:01 lola-4 TIME: Time stamp for console Nov 4 18:50:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:50:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:50:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:50:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:51:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:51:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:51:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:51:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:52:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:52:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:52:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:52:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:53:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:53:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:53:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:53:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:54:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:54:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:54:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:54:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:55:01 lola-4 CROND[37794]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 18:55:01 lola-4 TIME: Time stamp for console Nov 4 18:55:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:55:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 409 Nov 4 18:55:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 18:55:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 18:56:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 18:56:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 18:56:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 18:56:35 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 18:56:35 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 18:56:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 18:57:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 18:57:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 18:57:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 18:57:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 18:58:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 18:58:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 18:58:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 18:58:47 lola-4 kernel: LNet: Service thread pid 21594 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 18:58:47 lola-4 kernel: Pid: 21594, comm: ll_ost00_046 Nov 4 18:58:47 lola-4 kernel: Nov 4 18:58:47 lola-4 kernel: Call Trace: Nov 4 18:58:47 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 18:58:47 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 18:58:47 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 18:58:47 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 18:58:47 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 18:58:47 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 18:58:47 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 18:58:47 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 18:58:47 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 18:58:47 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 18:58:47 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 18:58:47 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 18:58:47 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 18:58:47 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 18:58:47 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 18:58:47 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 18:58:47 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 18:58:47 lola-4 kernel: Nov 4 18:58:47 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478311127.21594 Nov 4 18:58:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 18:59:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 18:59:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 18:59:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 18:59:43 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 18:59:43 lola-4 kernel: req@ffff88051acc0c50 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478311188 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 18:59:43 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 110 previous similar messages Nov 4 18:59:49 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 18:59:49 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 4 18:59:49 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 18:59:49 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 4 18:59:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 19:00:01 lola-4 CROND[37830]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 19:00:01 lola-4 CROND[37831]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 19:00:01 lola-4 TIME: Time stamp for console Nov 4 19:00:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 19:00:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 19:00:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 19:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 4 19:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 4 19:00:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:01:01 lola-4 CROND[37842]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 19:01:01 lola-4 run-parts(/etc/cron.hourly)[3784 starting 0anacron Nov 4 19:01:01 lola-4 run-parts(/etc/cron.hourly)[3785 finished 0anacron Nov 4 19:01:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:01:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:01:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:01:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:02:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:02:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:02:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:02:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:03:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:03:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:03:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:03:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:04:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:04:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:04:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:04:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:05:01 lola-4 CROND[37878]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 19:05:01 lola-4 TIME: Time stamp for console Nov 4 19:05:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:05:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:05:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:05:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 19:06:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 19:06:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 19:06:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:06:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 19:07:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 19:07:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 19:07:25 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 19:07:25 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 19:07:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 19:07:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 19:08:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 19:08:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:08:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 410 Nov 4 19:08:48 lola-4 kernel: LNet: Service thread pid 21617 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 19:08:48 lola-4 kernel: Pid: 21617, comm: ll_ost00_069 Nov 4 19:08:48 lola-4 kernel: Nov 4 19:08:48 lola-4 kernel: Call Trace: Nov 4 19:08:48 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 19:08:48 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 19:08:48 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 19:08:48 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 19:08:48 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 19:08:48 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 19:08:48 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 19:08:48 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 19:08:48 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 19:08:48 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 19:08:48 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 19:08:48 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 19:08:48 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 19:08:48 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 19:08:48 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 19:08:48 lola-4 kernel: Nov 4 19:08:48 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478311728.21617 Nov 4 19:08:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:09:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:09:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:09:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:09:44 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 19:09:44 lola-4 kernel: req@ffff88051ff81850 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478311789 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 19:09:44 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 109 previous similar messages Nov 4 19:09:50 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 19:09:50 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 4 19:09:50 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 19:09:50 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 4 19:09:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:10:01 lola-4 CROND[37914]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 19:10:01 lola-4 CROND[37915]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 19:10:01 lola-4 TIME: Time stamp for console Nov 4 19:10:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:10:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:10:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:10:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:11:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:11:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:11:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:11:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:12:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:12:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:12:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:12:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:13:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:13:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:13:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:13:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:14:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:14:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:14:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:14:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:15:01 lola-4 CROND[37951]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 19:15:01 lola-4 TIME: Time stamp for console Nov 4 19:15:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:15:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:15:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:15:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:16:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:16:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:16:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:16:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:17:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:17:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:17:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:17:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:18:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:18:15 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 19:18:15 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 19:18:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:18:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:18:49 lola-4 kernel: LNet: Service thread pid 21501 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 19:18:49 lola-4 kernel: Pid: 21501, comm: ll_ost00_038 Nov 4 19:18:49 lola-4 kernel: Nov 4 19:18:49 lola-4 kernel: Call Trace: Nov 4 19:18:49 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 19:18:49 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 19:18:49 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 19:18:49 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 19:18:49 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 19:18:49 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 19:18:49 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 19:18:49 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 19:18:49 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 19:18:49 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 19:18:49 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 19:18:49 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 19:18:49 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 19:18:49 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 19:18:49 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 19:18:49 lola-4 kernel: Nov 4 19:18:49 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478312329.21501 Nov 4 19:18:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:19:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 411 Nov 4 19:19:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:19:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:19:45 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 19:19:45 lola-4 kernel: req@ffff88060a51d050 x1550005401093744/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478312390 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 19:19:45 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 108 previous similar messages Nov 4 19:19:51 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 19:19:51 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 4 19:19:51 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 19:19:51 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 4 19:19:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:20:01 lola-4 CROND[37987]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 19:20:01 lola-4 CROND[37988]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 19:20:01 lola-4 TIME: Time stamp for console Nov 4 19:20:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:20:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:20:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:20:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:21:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:21:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:21:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:21:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:22:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:22:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:22:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:22:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:23:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:23:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:23:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:23:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:24:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:24:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:24:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:24:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:25:01 lola-4 CROND[38025]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 19:25:01 lola-4 TIME: Time stamp for console Nov 4 19:25:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:25:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:25:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:25:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:26:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:26:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:26:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 412 Nov 4 19:26:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:27:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:27:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:27:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:27:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:28:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:28:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:28:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:28:50 lola-4 kernel: LNet: Service thread pid 21715 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 19:28:50 lola-4 kernel: Pid: 21715, comm: ll_ost00_103 Nov 4 19:28:50 lola-4 kernel: Nov 4 19:28:50 lola-4 kernel: Call Trace: Nov 4 19:28:50 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 19:28:50 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 19:28:50 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 19:28:50 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 19:28:50 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 19:28:50 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 19:28:50 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 19:28:50 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 19:28:50 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 19:28:50 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 19:28:50 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 19:28:50 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 19:28:50 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 19:28:50 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 19:28:50 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 19:28:50 lola-4 kernel: Nov 4 19:28:50 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478312930.21715 Nov 4 19:28:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:29:05 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 19:29:05 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 19:29:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:29:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:29:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:29:46 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 19:29:46 lola-4 kernel: req@ffff880600de3050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478312991 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 19:29:46 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 4 19:29:52 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 19:29:52 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 19:29:52 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 19:29:52 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 19:29:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:30:01 lola-4 CROND[38061]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 19:30:01 lola-4 CROND[38062]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 19:30:01 lola-4 TIME: Time stamp for console Nov 4 19:30:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:30:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:30:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:30:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:31:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:31:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:31:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:31:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:32:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:32:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:32:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:32:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:33:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 414 Nov 4 19:33:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:33:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:33:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:34:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:34:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:34:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:34:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 414 Nov 4 19:35:01 lola-4 CROND[38098]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 19:35:01 lola-4 TIME: Time stamp for console Nov 4 19:35:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 414 Nov 4 19:35:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 414 Nov 4 19:35:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 414 Nov 4 19:35:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:36:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:36:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:36:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:36:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:37:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:37:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:37:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:37:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:38:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 414 Nov 4 19:38:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:38:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 414 Nov 4 19:38:51 lola-4 kernel: LNet: Service thread pid 13357 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 19:38:51 lola-4 kernel: Pid: 13357, comm: ll_ost00_003 Nov 4 19:38:51 lola-4 kernel: Nov 4 19:38:51 lola-4 kernel: Call Trace: Nov 4 19:38:51 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 19:38:51 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 19:38:51 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 19:38:51 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 19:38:51 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 19:38:51 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 19:38:51 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 19:38:51 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 19:38:51 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 19:38:51 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 19:38:51 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 19:38:51 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 19:38:51 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 19:38:51 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 19:38:51 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 19:38:51 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 19:38:51 lola-4 kernel: Nov 4 19:38:51 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478313531.13357 Nov 4 19:38:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 413 Nov 4 19:39:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 414 Nov 4 19:39:08 lola-4 sendmail[10625]: have been rejecting connections on daemon MTA for 03:00:15 Nov 4 19:39:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 414 Nov 4 19:39:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 414 Nov 4 19:39:47 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 19:39:47 lola-4 kernel: req@ffff88052f099050 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478313592 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 19:39:47 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 4 19:39:53 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 19:39:53 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 4 19:39:53 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 19:39:53 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 4 19:39:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 414 Nov 4 19:39:55 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 19:39:55 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 19:40:01 lola-4 CROND[38134]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 19:40:01 lola-4 CROND[38135]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 19:40:01 lola-4 TIME: Time stamp for console Nov 4 19:40:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 414 Nov 4 19:40:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 414 Nov 4 19:40:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 414 Nov 4 19:40:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 414 Nov 4 19:41:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 414 Nov 4 19:41:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:41:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:41:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 414 Nov 4 19:42:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:42:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 414 Nov 4 19:42:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:42:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:43:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:43:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:43:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:43:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:44:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:44:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:44:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:44:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:45:01 lola-4 CROND[38171]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 19:45:01 lola-4 TIME: Time stamp for console Nov 4 19:45:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:45:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:45:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:45:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:46:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:46:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:46:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:46:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:47:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:47:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:47:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:47:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:48:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:48:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:48:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:48:52 lola-4 kernel: LNet: Service thread pid 21709 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 19:48:52 lola-4 kernel: Pid: 21709, comm: ll_ost00_101 Nov 4 19:48:52 lola-4 kernel: Nov 4 19:48:52 lola-4 kernel: Call Trace: Nov 4 19:48:52 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 19:48:52 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 19:48:52 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 19:48:52 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 19:48:52 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 19:48:52 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 19:48:52 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 19:48:52 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 19:48:52 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 19:48:52 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 19:48:52 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 19:48:52 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 19:48:52 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 19:48:52 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 19:48:52 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 19:48:52 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 19:48:52 lola-4 kernel: Nov 4 19:48:52 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478314132.21709 Nov 4 19:48:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:49:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:49:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:49:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:49:48 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 19:49:48 lola-4 kernel: req@ffff88050e622450 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478314193 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 19:49:48 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 4 19:49:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:49:54 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 19:49:54 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 19:49:54 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 19:49:54 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 19:50:01 lola-4 CROND[38207]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 19:50:01 lola-4 CROND[38208]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 19:50:01 lola-4 TIME: Time stamp for console Nov 4 19:50:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:50:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:50:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:50:45 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 19:50:45 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 19:50:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:51:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:51:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:51:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:51:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 19:52:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 19:52:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 19:52:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 19:52:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 19:53:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 19:53:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:53:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:53:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:54:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:54:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:54:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:54:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:55:01 lola-4 CROND[38244]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 19:55:01 lola-4 TIME: Time stamp for console Nov 4 19:55:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:55:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:55:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:55:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:56:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:56:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:56:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:56:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 415 Nov 4 19:57:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 19:57:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 19:57:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 19:57:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 19:58:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 19:58:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 19:58:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 19:58:53 lola-4 kernel: LNet: Service thread pid 21634 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 19:58:53 lola-4 kernel: Pid: 21634, comm: ll_ost00_086 Nov 4 19:58:53 lola-4 kernel: Nov 4 19:58:53 lola-4 kernel: Call Trace: Nov 4 19:58:53 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 19:58:53 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 19:58:53 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 19:58:53 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 19:58:53 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 19:58:53 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 19:58:53 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 19:58:53 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 19:58:53 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 19:58:53 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 19:58:53 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 19:58:53 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 19:58:53 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 19:58:53 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 19:58:53 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 19:58:53 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 19:58:53 lola-4 kernel: Nov 4 19:58:53 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478314733.21634 Nov 4 19:58:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 19:59:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 19:59:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 19:59:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 19:59:49 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 19:59:49 lola-4 kernel: req@ffff8805f8066450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478314794 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 19:59:49 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 4 19:59:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 19:59:55 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 19:59:55 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 4 19:59:55 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 19:59:55 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 4 20:00:01 lola-4 CROND[38281]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 20:00:01 lola-4 CROND[38282]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 20:00:01 lola-4 TIME: Time stamp for console Nov 4 20:00:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 20:00:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 20:00:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 20:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 4 20:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 4 20:00:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 20:01:02 lola-4 CROND[38293]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 20:01:02 lola-4 run-parts(/etc/cron.hourly)[3829 starting 0anacron Nov 4 20:01:02 lola-4 run-parts(/etc/cron.hourly)[3830 finished 0anacron Nov 4 20:01:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:01:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:01:35 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 20:01:35 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 20:01:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:01:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:02:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:02:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:02:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:02:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:03:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 20:03:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 20:03:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 20:03:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 20:04:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 20:04:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:04:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 20:04:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:05:01 lola-4 CROND[38329]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 20:05:01 lola-4 TIME: Time stamp for console Nov 4 20:05:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:05:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:05:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 20:05:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 20:06:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:06:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:06:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 20:06:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 20:07:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 20:07:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 416 Nov 4 20:07:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:07:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:08:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:08:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:08:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:08:54 lola-4 kernel: LNet: Service thread pid 21498 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 20:08:54 lola-4 kernel: Pid: 21498, comm: ll_ost00_035 Nov 4 20:08:54 lola-4 kernel: Nov 4 20:08:54 lola-4 kernel: Call Trace: Nov 4 20:08:54 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 20:08:54 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 20:08:54 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 20:08:54 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 20:08:54 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 20:08:54 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 20:08:54 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 20:08:55 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 20:08:55 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 20:08:55 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 20:08:55 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 20:08:55 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 20:08:55 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 20:08:55 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 20:08:55 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 20:08:55 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 20:08:55 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 20:08:55 lola-4 kernel: Nov 4 20:08:55 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478315335.21498 Nov 4 20:08:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:09:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:09:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:09:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:09:50 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 20:09:50 lola-4 kernel: req@ffff880728636450 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478315395 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 20:09:50 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 4 20:09:56 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 20:09:56 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 20:09:56 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 20:09:56 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 20:09:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:10:01 lola-4 CROND[38365]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 20:10:01 lola-4 CROND[38366]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 20:10:01 lola-4 TIME: Time stamp for console Nov 4 20:10:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:10:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:10:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:10:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 417 Nov 4 20:11:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:11:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:11:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:11:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:12:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:12:25 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 20:12:25 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 20:12:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:12:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:12:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:13:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:13:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:13:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:13:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:14:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:14:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:14:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:14:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:15:01 lola-4 CROND[38402]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 20:15:01 lola-4 TIME: Time stamp for console Nov 4 20:15:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:15:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:15:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:15:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:16:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:16:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:16:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:16:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:17:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:17:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:17:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:17:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:18:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:18:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:18:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:18:56 lola-4 kernel: LNet: Service thread pid 21637 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 20:18:56 lola-4 kernel: Pid: 21637, comm: ll_ost00_089 Nov 4 20:18:56 lola-4 kernel: Nov 4 20:18:56 lola-4 kernel: Call Trace: Nov 4 20:18:56 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 20:18:56 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 20:18:56 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 20:18:56 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 20:18:56 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 20:18:56 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 20:18:56 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 20:18:56 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 20:18:56 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 20:18:56 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 20:18:56 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 20:18:56 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 20:18:56 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 20:18:56 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 20:18:56 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 20:18:56 lola-4 kernel: Nov 4 20:18:56 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478315936.21637 Nov 4 20:18:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:19:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:19:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:19:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:19:51 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 20:19:51 lola-4 kernel: req@ffff8804ea6d5c50 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478315996 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 20:19:51 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 4 20:19:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:19:57 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 20:19:57 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 20:19:57 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 20:19:57 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 20:20:01 lola-4 CROND[38438]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 20:20:01 lola-4 CROND[38439]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 20:20:01 lola-4 TIME: Time stamp for console Nov 4 20:20:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:20:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 418 Nov 4 20:20:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:20:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:21:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:21:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:21:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:21:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:22:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:22:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:22:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:22:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:23:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:23:15 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 20:23:15 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 20:23:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:23:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:23:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:24:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:24:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:24:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:24:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:25:01 lola-4 CROND[38475]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 20:25:01 lola-4 TIME: Time stamp for console Nov 4 20:25:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:25:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:25:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:25:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:26:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:26:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:26:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:26:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:27:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:27:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:27:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:27:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:28:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:28:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:28:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:28:57 lola-4 kernel: LNet: Service thread pid 21601 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 20:28:57 lola-4 kernel: Pid: 21601, comm: ll_ost00_053 Nov 4 20:28:57 lola-4 kernel: Nov 4 20:28:57 lola-4 kernel: Call Trace: Nov 4 20:28:57 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 20:28:57 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 20:28:57 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 20:28:57 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 20:28:57 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 20:28:57 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 20:28:57 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 20:28:57 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 20:28:57 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 20:28:57 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 20:28:57 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 20:28:57 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 20:28:57 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 20:28:57 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 20:28:57 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 20:28:57 lola-4 kernel: Nov 4 20:28:57 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478316537.21601 Nov 4 20:28:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:29:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 419 Nov 4 20:29:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:29:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:29:52 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 20:29:52 lola-4 kernel: req@ffff88052f99b050 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478316597 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 20:29:52 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 4 20:29:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:29:58 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 20:29:58 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 4 20:29:58 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 20:29:58 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 4 20:30:01 lola-4 CROND[38511]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 20:30:01 lola-4 CROND[38512]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 20:30:01 lola-4 TIME: Time stamp for console Nov 4 20:30:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:30:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:30:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:30:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:31:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:31:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:31:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:31:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:32:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:32:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:32:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:32:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:33:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:33:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:33:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:33:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:34:05 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 20:34:05 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 20:34:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:34:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:34:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:34:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:35:01 lola-4 CROND[38549]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 20:35:01 lola-4 TIME: Time stamp for console Nov 4 20:35:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:35:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:35:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:35:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:36:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:36:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:36:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:36:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:37:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:37:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:37:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:37:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:38:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:38:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:38:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:38:58 lola-4 kernel: LNet: Service thread pid 21477 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 20:38:58 lola-4 kernel: Pid: 21477, comm: ll_ost00_015 Nov 4 20:38:58 lola-4 kernel: Nov 4 20:38:58 lola-4 kernel: Call Trace: Nov 4 20:38:58 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 20:38:58 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 20:38:58 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 20:38:58 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 20:38:58 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 20:38:58 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 20:38:58 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 20:38:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:38:58 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 20:38:58 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 20:38:58 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 20:38:58 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 20:38:58 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 20:38:58 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 20:38:58 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 20:38:58 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 20:38:58 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 20:38:58 lola-4 kernel: Nov 4 20:38:58 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478317138.21477 Nov 4 20:39:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:39:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:39:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:39:53 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 20:39:53 lola-4 kernel: req@ffff88079b2bec50 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478317198 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 20:39:53 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 4 20:39:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:39:59 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 20:39:59 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 20:39:59 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 20:39:59 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 20:40:01 lola-4 CROND[38585]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 20:40:01 lola-4 CROND[38586]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 20:40:01 lola-4 TIME: Time stamp for console Nov 4 20:40:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:40:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:40:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:40:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:41:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:41:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:41:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:41:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:42:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:42:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:42:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:42:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:43:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:43:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:43:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:43:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:44:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:44:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:44:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:44:55 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 20:44:55 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 20:44:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:45:01 lola-4 CROND[38622]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 20:45:01 lola-4 TIME: Time stamp for console Nov 4 20:45:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:45:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:45:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:45:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:46:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:46:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:46:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:46:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:47:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:47:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:47:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:47:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:48:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:48:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:48:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:48:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 420 Nov 4 20:48:59 lola-4 kernel: LNet: Service thread pid 21711 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 20:48:59 lola-4 kernel: Pid: 21711, comm: ll_ost00_102 Nov 4 20:48:59 lola-4 kernel: Nov 4 20:48:59 lola-4 kernel: Call Trace: Nov 4 20:48:59 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 20:48:59 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 20:48:59 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 20:48:59 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 20:48:59 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 20:48:59 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 20:48:59 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 20:48:59 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 20:48:59 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 20:48:59 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 20:48:59 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 20:48:59 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 20:48:59 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 20:48:59 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 20:48:59 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 20:48:59 lola-4 kernel: Nov 4 20:48:59 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478317739.21711 Nov 4 20:49:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:49:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:49:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:49:54 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 20:49:54 lola-4 kernel: req@ffff8807f11ee450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478317799 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 20:49:54 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 4 20:49:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:50:00 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 20:50:00 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 4 20:50:00 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 20:50:00 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 4 20:50:01 lola-4 CROND[38658]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 20:50:01 lola-4 CROND[38659]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 20:50:01 lola-4 TIME: Time stamp for console Nov 4 20:50:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:50:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:50:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:50:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:51:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:51:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:51:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:51:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 421 Nov 4 20:52:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:52:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:52:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:52:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:53:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:53:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:53:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:53:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:54:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:54:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:54:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:54:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:55:01 lola-4 CROND[38695]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 20:55:01 lola-4 TIME: Time stamp for console Nov 4 20:55:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:55:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:55:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:55:45 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 20:55:45 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 20:55:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:56:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:56:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:56:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:56:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:57:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:57:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:57:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:57:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:58:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:58:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:58:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:58:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:59:00 lola-4 kernel: LNet: Service thread pid 21487 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 20:59:00 lola-4 kernel: Pid: 21487, comm: ll_ost00_024 Nov 4 20:59:00 lola-4 kernel: Nov 4 20:59:00 lola-4 kernel: Call Trace: Nov 4 20:59:00 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 20:59:00 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 20:59:00 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 20:59:00 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 20:59:00 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 20:59:00 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 20:59:00 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 20:59:00 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 20:59:00 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 20:59:00 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 20:59:00 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 20:59:00 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 20:59:00 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 20:59:00 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 20:59:00 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 20:59:00 lola-4 kernel: Nov 4 20:59:00 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478318340.21487 Nov 4 20:59:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:59:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 422 Nov 4 20:59:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 20:59:55 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 20:59:55 lola-4 kernel: req@ffff880551c8dc50 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478318400 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 20:59:55 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 4 20:59:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:00:01 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 21:00:01 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 21:00:01 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 21:00:01 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 21:00:01 lola-4 CROND[38731]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 21:00:01 lola-4 CROND[38732]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 21:00:01 lola-4 TIME: Time stamp for console Nov 4 21:00:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:00:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 4 21:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 4 21:00:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:01:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:01:01 lola-4 CROND[38743]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 21:01:01 lola-4 run-parts(/etc/cron.hourly)[3874 starting 0anacron Nov 4 21:01:01 lola-4 run-parts(/etc/cron.hourly)[3875 finished 0anacron Nov 4 21:01:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:01:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:01:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:02:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:02:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:02:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:02:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:03:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:03:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:03:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:03:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:04:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:04:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:04:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:04:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:05:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:05:01 lola-4 CROND[38780]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 21:05:01 lola-4 TIME: Time stamp for console Nov 4 21:05:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:05:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:05:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:06:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:06:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:06:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:06:35 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 21:06:35 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 21:06:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:07:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:07:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:07:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:07:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:08:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:08:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:08:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:08:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:09:01 lola-4 kernel: LNet: Service thread pid 21624 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 21:09:01 lola-4 kernel: Pid: 21624, comm: ll_ost00_076 Nov 4 21:09:01 lola-4 kernel: Nov 4 21:09:01 lola-4 kernel: Call Trace: Nov 4 21:09:01 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 21:09:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:09:01 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 21:09:01 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 21:09:01 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 21:09:01 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 21:09:01 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 21:09:01 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 21:09:01 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 21:09:01 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 21:09:01 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 21:09:01 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 21:09:01 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 21:09:01 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 21:09:01 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 21:09:01 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 21:09:01 lola-4 kernel: Nov 4 21:09:01 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478318941.21624 Nov 4 21:09:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:09:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:09:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:09:56 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 21:09:56 lola-4 kernel: req@ffff88068d8f7450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478319001 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 21:09:56 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 4 21:10:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:10:01 lola-4 CROND[38816]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 21:10:01 lola-4 CROND[38817]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 21:10:01 lola-4 TIME: Time stamp for console Nov 4 21:10:02 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 21:10:02 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 21:10:02 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 21:10:02 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 21:10:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:10:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:10:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:11:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:11:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:11:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 423 Nov 4 21:11:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:12:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:12:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:12:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:12:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:13:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:13:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:13:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:13:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:14:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:14:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:14:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:14:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:15:01 lola-4 CROND[38853]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 21:15:01 lola-4 TIME: Time stamp for console Nov 4 21:15:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:15:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:15:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:15:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:16:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:16:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:16:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:16:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:17:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:17:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:17:25 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 21:17:25 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 21:17:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:17:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:18:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:18:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:18:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:18:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:19:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:19:02 lola-4 kernel: LNet: Service thread pid 21610 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 21:19:02 lola-4 kernel: Pid: 21610, comm: ll_ost00_062 Nov 4 21:19:02 lola-4 kernel: Nov 4 21:19:02 lola-4 kernel: Call Trace: Nov 4 21:19:02 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 21:19:02 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 21:19:02 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 21:19:02 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 21:19:02 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 21:19:02 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 21:19:02 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 21:19:02 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 21:19:02 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 21:19:02 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 21:19:02 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 21:19:02 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 21:19:02 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 21:19:02 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 21:19:02 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 21:19:02 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 21:19:02 lola-4 kernel: Nov 4 21:19:02 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478319542.21610 Nov 4 21:19:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:19:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:19:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:19:57 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 21:19:57 lola-4 kernel: req@ffff88055940fc50 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478319602 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 21:19:57 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 4 21:20:01 lola-4 CROND[38889]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 21:20:01 lola-4 CROND[38890]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 21:20:01 lola-4 TIME: Time stamp for console Nov 4 21:20:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:20:03 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 21:20:03 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 4 21:20:03 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 21:20:03 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 4 21:20:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:20:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:20:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:21:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:21:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:21:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:21:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:22:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:22:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 424 Nov 4 21:22:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:22:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:23:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:23:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:23:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:23:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:24:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:24:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:24:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:24:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:25:01 lola-4 CROND[38926]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 21:25:01 lola-4 TIME: Time stamp for console Nov 4 21:25:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:25:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:25:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:25:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:26:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:26:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:26:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:26:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:27:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:27:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:27:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:27:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:28:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:28:15 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 21:28:15 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 21:28:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:28:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:28:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:29:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:29:03 lola-4 kernel: LNet: Service thread pid 21500 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 21:29:03 lola-4 kernel: Pid: 21500, comm: ll_ost00_037 Nov 4 21:29:03 lola-4 kernel: Nov 4 21:29:03 lola-4 kernel: Call Trace: Nov 4 21:29:03 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 21:29:03 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 21:29:03 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 21:29:03 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 21:29:03 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 21:29:03 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 21:29:03 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 21:29:03 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 21:29:03 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 21:29:03 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 21:29:03 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 21:29:03 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 21:29:03 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 21:29:03 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 21:29:03 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 21:29:03 lola-4 kernel: Nov 4 21:29:03 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478320143.21500 Nov 4 21:29:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:29:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:29:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 425 Nov 4 21:29:58 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 21:29:58 lola-4 kernel: req@ffff8807dd334850 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478320203 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 21:29:58 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 4 21:30:01 lola-4 CROND[38962]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 21:30:01 lola-4 CROND[38963]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 21:30:01 lola-4 TIME: Time stamp for console Nov 4 21:30:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:30:04 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 21:30:04 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 21:30:04 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 21:30:04 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 21:30:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:30:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:30:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:31:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:31:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:31:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:31:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:32:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:32:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:32:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:32:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:33:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:33:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:33:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:33:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:34:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:34:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:34:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:34:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:35:01 lola-4 CROND[38999]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 21:35:01 lola-4 TIME: Time stamp for console Nov 4 21:35:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:35:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:35:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:35:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:36:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:36:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:36:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:36:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:37:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:37:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:37:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:37:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:38:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:38:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:38:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:38:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:39:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:39:04 lola-4 kernel: LNet: Service thread pid 21598 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 21:39:04 lola-4 kernel: Pid: 21598, comm: ll_ost00_050 Nov 4 21:39:04 lola-4 kernel: Nov 4 21:39:04 lola-4 kernel: Call Trace: Nov 4 21:39:04 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 21:39:04 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 21:39:04 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 21:39:04 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 21:39:04 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 21:39:04 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 21:39:04 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 21:39:04 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 21:39:04 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 21:39:04 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 21:39:04 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 21:39:04 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 21:39:04 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 21:39:04 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 21:39:04 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 21:39:04 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 21:39:04 lola-4 kernel: Nov 4 21:39:04 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478320744.21598 Nov 4 21:39:05 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 21:39:05 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 21:39:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:39:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:39:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:39:59 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 21:39:59 lola-4 kernel: req@ffff88072c613850 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478320804 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 21:39:59 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 4 21:40:01 lola-4 CROND[39036]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 21:40:01 lola-4 CROND[39037]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 21:40:01 lola-4 TIME: Time stamp for console Nov 4 21:40:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:40:05 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 21:40:05 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 4 21:40:05 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 21:40:05 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 4 21:40:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:40:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:40:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:41:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:41:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:41:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 426 Nov 4 21:41:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:42:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:42:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:42:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:42:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:43:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:43:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:43:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:43:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:44:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:44:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:44:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:44:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:45:01 lola-4 CROND[39073]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 21:45:01 lola-4 TIME: Time stamp for console Nov 4 21:45:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:45:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:45:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:45:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:46:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:46:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:46:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:46:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:47:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:47:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:47:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:47:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:48:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:48:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:48:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:48:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:49:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:49:05 lola-4 kernel: LNet: Service thread pid 21627 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 21:49:05 lola-4 kernel: Pid: 21627, comm: ll_ost00_079 Nov 4 21:49:05 lola-4 kernel: Nov 4 21:49:05 lola-4 kernel: Call Trace: Nov 4 21:49:05 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 21:49:05 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 21:49:05 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 21:49:05 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 21:49:05 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 21:49:05 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 21:49:05 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 21:49:05 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 21:49:05 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 21:49:05 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 21:49:05 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 21:49:05 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 21:49:05 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 21:49:05 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 21:49:05 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 21:49:05 lola-4 kernel: Nov 4 21:49:05 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478321345.21627 Nov 4 21:49:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:49:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:49:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:49:55 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 21:49:55 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 21:50:00 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 21:50:00 lola-4 kernel: req@ffff8806bcb31850 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478321405 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 21:50:00 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 4 21:50:01 lola-4 CROND[39109]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 21:50:01 lola-4 CROND[39110]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 21:50:01 lola-4 TIME: Time stamp for console Nov 4 21:50:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:50:06 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 21:50:06 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 21:50:06 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 21:50:06 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 21:50:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:50:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:50:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:51:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:51:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:51:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:51:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:52:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:52:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:52:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:52:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:53:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:53:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 427 Nov 4 21:53:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:53:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:54:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:54:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:54:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:54:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:55:01 lola-4 CROND[39146]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 21:55:01 lola-4 TIME: Time stamp for console Nov 4 21:55:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:55:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:55:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:55:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:56:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:56:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:56:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:56:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:57:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:57:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:57:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:57:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:58:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:58:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:58:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:58:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:59:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:59:06 lola-4 kernel: LNet: Service thread pid 21480 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 21:59:06 lola-4 kernel: Pid: 21480, comm: ll_ost00_018 Nov 4 21:59:06 lola-4 kernel: Nov 4 21:59:06 lola-4 kernel: Call Trace: Nov 4 21:59:06 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 21:59:06 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 21:59:06 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 21:59:06 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 21:59:06 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 21:59:06 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 21:59:06 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 21:59:06 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 21:59:06 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 21:59:06 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 21:59:06 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 21:59:06 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 21:59:06 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 21:59:06 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 21:59:06 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 21:59:06 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 21:59:06 lola-4 kernel: Nov 4 21:59:06 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478321946.21480 Nov 4 21:59:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:59:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 21:59:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 428 Nov 4 22:00:01 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 22:00:01 lola-4 kernel: req@ffff880520c50c50 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478322006 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 22:00:01 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 4 22:00:01 lola-4 CROND[39182]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 22:00:01 lola-4 CROND[39183]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 22:00:01 lola-4 TIME: Time stamp for console Nov 4 22:00:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:00:07 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 22:00:07 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 22:00:07 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 22:00:07 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 22:00:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:00:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 4 22:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 4 22:00:45 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 22:00:45 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 22:00:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:01:01 lola-4 CROND[39194]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 22:01:01 lola-4 run-parts(/etc/cron.hourly)[3919 starting 0anacron Nov 4 22:01:01 lola-4 run-parts(/etc/cron.hourly)[3920 finished 0anacron Nov 4 22:01:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:01:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:01:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:01:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:02:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:02:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:02:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:02:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:03:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:03:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:03:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:03:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:04:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:04:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:04:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:04:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:05:01 lola-4 CROND[39230]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 22:05:01 lola-4 TIME: Time stamp for console Nov 4 22:05:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:05:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:05:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:05:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:06:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:06:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:06:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:06:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:07:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:07:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:07:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:07:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:08:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:08:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:08:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:08:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:09:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:09:07 lola-4 kernel: LNet: Service thread pid 21752 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 22:09:07 lola-4 kernel: Pid: 21752, comm: ll_ost00_116 Nov 4 22:09:07 lola-4 kernel: Nov 4 22:09:07 lola-4 kernel: Call Trace: Nov 4 22:09:07 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 22:09:07 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 22:09:07 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 22:09:07 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 22:09:07 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 22:09:07 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 22:09:07 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 22:09:07 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 22:09:07 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 22:09:07 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 22:09:07 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 22:09:07 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 22:09:07 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 22:09:07 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 22:09:07 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 22:09:07 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 22:09:07 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 22:09:07 lola-4 kernel: Nov 4 22:09:07 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478322547.21752 Nov 4 22:09:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:09:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:09:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:10:01 lola-4 CROND[39266]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 22:10:01 lola-4 CROND[39267]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 22:10:01 lola-4 TIME: Time stamp for console Nov 4 22:10:02 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 22:10:02 lola-4 kernel: req@ffff8806721b4c50 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478322607 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 22:10:02 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 4 22:10:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:10:08 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 22:10:08 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 4 22:10:08 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 22:10:08 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 4 22:10:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:10:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 429 Nov 4 22:10:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:11:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:11:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:11:35 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 22:11:35 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 22:11:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:11:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:12:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:12:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:12:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:12:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:13:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:13:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:13:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:13:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:14:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:14:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:14:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:14:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:15:01 lola-4 CROND[39304]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 22:15:01 lola-4 TIME: Time stamp for console Nov 4 22:15:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:15:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:15:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:15:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:16:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:16:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:16:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:16:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:17:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:17:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:17:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:17:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:18:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:18:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:18:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:18:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:19:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:19:08 lola-4 kernel: LNet: Service thread pid 21631 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 22:19:08 lola-4 kernel: Pid: 21631, comm: ll_ost00_083 Nov 4 22:19:08 lola-4 kernel: Nov 4 22:19:08 lola-4 kernel: Call Trace: Nov 4 22:19:08 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 22:19:08 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 22:19:08 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 22:19:08 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 22:19:08 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 22:19:08 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 22:19:08 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 22:19:08 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 22:19:08 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 22:19:08 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 22:19:08 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 22:19:08 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 22:19:08 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 22:19:08 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 22:19:08 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 22:19:08 lola-4 kernel: Nov 4 22:19:08 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478323148.21631 Nov 4 22:19:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:19:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:19:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:20:01 lola-4 CROND[39340]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 22:20:01 lola-4 CROND[39341]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 22:20:01 lola-4 TIME: Time stamp for console Nov 4 22:20:03 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 22:20:03 lola-4 kernel: req@ffff88051dc8b450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478323208 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 22:20:03 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 4 22:20:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:20:09 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 22:20:09 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 22:20:09 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 22:20:09 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 22:20:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:20:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:20:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:21:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:21:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:21:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:21:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:22:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:22:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:22:25 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 22:22:25 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 22:22:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:22:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:23:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:23:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:23:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:23:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:24:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:24:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:24:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 430 Nov 4 22:24:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:25:01 lola-4 CROND[39377]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 22:25:01 lola-4 TIME: Time stamp for console Nov 4 22:25:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:25:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:25:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:25:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:26:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:26:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:26:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:26:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:27:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:27:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:27:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:27:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:28:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:28:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:28:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:28:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:29:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:29:09 lola-4 kernel: LNet: Service thread pid 21473 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 22:29:09 lola-4 kernel: Pid: 21473, comm: ll_ost00_011 Nov 4 22:29:09 lola-4 kernel: Nov 4 22:29:09 lola-4 kernel: Call Trace: Nov 4 22:29:09 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 22:29:09 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 22:29:09 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 22:29:09 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 22:29:09 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 22:29:09 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 22:29:09 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 22:29:09 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 22:29:09 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 22:29:09 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 22:29:09 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 22:29:09 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 22:29:09 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 22:29:09 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 22:29:09 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 22:29:09 lola-4 kernel: Nov 4 22:29:09 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478323749.21473 Nov 4 22:29:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:29:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:29:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 431 Nov 4 22:30:01 lola-4 CROND[39413]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 22:30:01 lola-4 CROND[39414]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 22:30:01 lola-4 TIME: Time stamp for console Nov 4 22:30:04 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 22:30:04 lola-4 kernel: req@ffff88078c71a450 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478323809 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 22:30:04 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 4 22:30:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:30:10 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 22:30:10 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 4 22:30:10 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 22:30:10 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 4 22:30:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:30:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:30:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:31:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:31:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:31:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:31:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:32:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:32:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:32:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:32:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:33:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:33:15 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 22:33:15 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 22:33:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:33:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:33:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:34:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:34:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:34:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:34:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:35:01 lola-4 CROND[39450]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 22:35:01 lola-4 TIME: Time stamp for console Nov 4 22:35:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:35:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:35:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:35:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:36:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:36:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:36:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:36:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:37:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:37:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:37:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:37:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:38:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:38:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:38:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:38:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:39:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:39:10 lola-4 kernel: LNet: Service thread pid 21494 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 22:39:10 lola-4 kernel: Pid: 21494, comm: ll_ost00_031 Nov 4 22:39:10 lola-4 kernel: Nov 4 22:39:10 lola-4 kernel: Call Trace: Nov 4 22:39:10 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 22:39:10 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 22:39:10 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 22:39:10 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 22:39:10 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 22:39:10 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 22:39:10 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 22:39:10 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 22:39:10 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 22:39:10 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 22:39:10 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 22:39:10 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 22:39:10 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 22:39:10 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 22:39:10 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 22:39:10 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 22:39:10 lola-4 kernel: Nov 4 22:39:10 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478324350.21494 Nov 4 22:39:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:39:23 lola-4 sendmail[10625]: have been rejecting connections on daemon MTA for 06:00:30 Nov 4 22:39:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:39:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:40:01 lola-4 CROND[39486]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 22:40:01 lola-4 TIME: Time stamp for console Nov 4 22:40:01 lola-4 CROND[39487]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 22:40:05 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 22:40:05 lola-4 kernel: req@ffff8807abf81450 x1550005401094688/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478324410 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 22:40:05 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 4 22:40:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:40:11 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 22:40:11 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 22:40:11 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 22:40:11 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 22:40:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:40:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:40:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:41:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:41:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:41:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:41:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:42:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:42:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:42:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:42:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:43:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:43:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:43:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:43:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:44:05 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 22:44:05 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 22:44:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:44:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:44:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:44:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:45:01 lola-4 CROND[39524]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 22:45:01 lola-4 TIME: Time stamp for console Nov 4 22:45:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:45:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:45:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:45:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:46:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:46:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:46:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:46:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:47:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:47:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:47:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:47:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:48:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:48:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:48:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:48:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:49:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:49:11 lola-4 kernel: LNet: Service thread pid 21599 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 22:49:11 lola-4 kernel: Pid: 21599, comm: ll_ost00_051 Nov 4 22:49:11 lola-4 kernel: Nov 4 22:49:11 lola-4 kernel: Call Trace: Nov 4 22:49:11 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 22:49:11 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 22:49:11 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 22:49:11 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 22:49:11 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 22:49:11 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 22:49:11 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 22:49:11 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 22:49:11 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 22:49:11 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 22:49:11 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 22:49:11 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 22:49:11 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 22:49:11 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 22:49:11 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 22:49:11 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 22:49:11 lola-4 kernel: Nov 4 22:49:11 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478324951.21599 Nov 4 22:49:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 432 Nov 4 22:49:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:49:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:50:01 lola-4 CROND[39560]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 22:50:01 lola-4 CROND[39561]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 22:50:01 lola-4 TIME: Time stamp for console Nov 4 22:50:06 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 22:50:06 lola-4 kernel: req@ffff8807f37c0050 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478325011 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 22:50:06 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 4 22:50:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:50:12 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 22:50:12 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 22:50:12 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 22:50:12 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 22:50:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:50:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:50:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:51:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:51:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:51:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:51:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:52:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:52:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:52:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:52:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:53:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 22:53:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 22:53:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 22:53:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 22:54:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 22:54:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 22:54:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:54:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 22:54:55 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 22:54:55 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 22:55:01 lola-4 CROND[39597]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 22:55:01 lola-4 TIME: Time stamp for console Nov 4 22:55:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 22:55:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 22:55:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 22:55:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 22:56:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 22:56:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 22:56:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 22:56:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 22:57:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 22:57:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:57:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:57:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:58:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:58:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:58:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:58:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:59:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:59:12 lola-4 kernel: LNet: Service thread pid 21499 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 22:59:12 lola-4 kernel: Pid: 21499, comm: ll_ost00_036 Nov 4 22:59:12 lola-4 kernel: Nov 4 22:59:12 lola-4 kernel: Call Trace: Nov 4 22:59:12 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 22:59:12 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 22:59:12 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 22:59:12 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 22:59:12 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 22:59:12 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 22:59:12 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 22:59:12 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 22:59:12 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 22:59:12 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 22:59:12 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 22:59:12 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 22:59:12 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 22:59:12 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 22:59:12 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 22:59:12 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 22:59:12 lola-4 kernel: Nov 4 22:59:12 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478325552.21499 Nov 4 22:59:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 433 Nov 4 22:59:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 22:59:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 23:00:01 lola-4 CROND[39633]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 23:00:01 lola-4 CROND[39634]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 23:00:01 lola-4 TIME: Time stamp for console Nov 4 23:00:07 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 23:00:07 lola-4 kernel: req@ffff8806d6faf850 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478325612 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 23:00:07 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 4 23:00:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 23:00:13 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 23:00:13 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 4 23:00:13 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 23:00:13 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 4 23:00:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 23:00:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 23:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 4 23:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 4 23:00:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 23:01:01 lola-4 CROND[39645]: (root) CMD (run-parts /etc/cron.hourly) Nov 4 23:01:01 lola-4 run-parts(/etc/cron.hourly)[3964 starting 0anacron Nov 4 23:01:01 lola-4 run-parts(/etc/cron.hourly)[3965 finished 0anacron Nov 4 23:01:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 23:01:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:01:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:01:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:02:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:02:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 23:02:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:02:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:03:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:03:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:03:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:03:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:04:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:04:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:04:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:04:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:05:01 lola-4 CROND[39681]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 23:05:01 lola-4 TIME: Time stamp for console Nov 4 23:05:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:05:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:05:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:05:45 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 23:05:45 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 23:05:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:06:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:06:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:06:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:06:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:07:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 23:07:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:07:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:07:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:08:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:08:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:08:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 23:08:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 434 Nov 4 23:09:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:09:13 lola-4 kernel: LNet: Service thread pid 21607 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 23:09:13 lola-4 kernel: Pid: 21607, comm: ll_ost00_059 Nov 4 23:09:13 lola-4 kernel: Nov 4 23:09:13 lola-4 kernel: Call Trace: Nov 4 23:09:13 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 23:09:13 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 23:09:13 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 23:09:13 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 23:09:13 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 23:09:13 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 23:09:13 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 23:09:13 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 23:09:13 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 23:09:13 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 23:09:13 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 23:09:13 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 23:09:13 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 23:09:13 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 23:09:13 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 23:09:13 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 23:09:13 lola-4 kernel: Nov 4 23:09:13 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478326153.21607 Nov 4 23:09:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:09:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:09:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:10:01 lola-4 CROND[39717]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 23:10:01 lola-4 CROND[39718]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 23:10:01 lola-4 TIME: Time stamp for console Nov 4 23:10:08 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 23:10:08 lola-4 kernel: req@ffff880525842450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478326213 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 23:10:08 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 4 23:10:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:10:14 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 23:10:14 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 23:10:14 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 23:10:14 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 23:10:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:10:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:10:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:11:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:11:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:11:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:11:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:12:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:12:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:12:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:12:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:13:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:13:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:13:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:13:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:14:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:14:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:14:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:14:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 435 Nov 4 23:15:01 lola-4 CROND[39754]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 23:15:01 lola-4 TIME: Time stamp for console Nov 4 23:15:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:15:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:15:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:15:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:16:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:16:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:16:35 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 23:16:35 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 23:16:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:16:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:17:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:17:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:17:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:17:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:18:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:18:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:18:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:18:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:19:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:19:14 lola-4 kernel: LNet: Service thread pid 21603 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 23:19:14 lola-4 kernel: Pid: 21603, comm: ll_ost00_055 Nov 4 23:19:14 lola-4 kernel: Nov 4 23:19:14 lola-4 kernel: Call Trace: Nov 4 23:19:14 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 23:19:14 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 23:19:14 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 23:19:14 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 23:19:14 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 23:19:14 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 23:19:15 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 23:19:15 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 23:19:15 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 23:19:15 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 23:19:15 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 23:19:15 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 23:19:15 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 23:19:15 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 23:19:15 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 23:19:15 lola-4 kernel: Nov 4 23:19:15 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478326755.21603 Nov 4 23:19:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:19:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:19:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:20:01 lola-4 CROND[39791]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 23:20:01 lola-4 CROND[39792]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 23:20:01 lola-4 TIME: Time stamp for console Nov 4 23:20:09 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 23:20:09 lola-4 kernel: req@ffff88051b29f450 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478326814 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 23:20:09 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 4 23:20:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:20:15 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 23:20:15 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 4 23:20:15 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 23:20:15 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 4 23:20:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:20:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:20:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:21:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:21:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:21:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:21:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:22:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:22:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:22:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:22:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:23:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:23:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:23:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:23:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:24:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:24:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:24:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:24:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:25:01 lola-4 CROND[39828]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 23:25:01 lola-4 TIME: Time stamp for console Nov 4 23:25:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:25:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:25:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:25:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:26:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:26:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:26:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:26:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:27:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:27:25 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 23:27:25 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 23:27:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:27:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:27:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 436 Nov 4 23:28:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:28:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:28:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:28:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:29:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:29:16 lola-4 kernel: LNet: Service thread pid 21608 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 23:29:16 lola-4 kernel: Pid: 21608, comm: ll_ost00_060 Nov 4 23:29:16 lola-4 kernel: Nov 4 23:29:16 lola-4 kernel: Call Trace: Nov 4 23:29:16 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 23:29:16 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 23:29:16 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 23:29:16 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 23:29:16 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 23:29:16 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 23:29:16 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 23:29:16 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 23:29:16 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 23:29:16 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 23:29:16 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 4 23:29:16 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 4 23:29:16 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 23:29:16 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 23:29:16 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 23:29:16 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 23:29:16 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 23:29:16 lola-4 kernel: Nov 4 23:29:16 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478327356.21608 Nov 4 23:29:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:29:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:29:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:30:02 lola-4 CROND[39864]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 23:30:02 lola-4 CROND[39865]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 23:30:02 lola-4 TIME: Time stamp for console Nov 4 23:30:10 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 23:30:10 lola-4 kernel: req@ffff880802956c50 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478327415 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 23:30:10 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 4 23:30:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:30:16 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 23:30:16 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 23:30:16 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 23:30:16 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 23:30:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:30:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:30:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:31:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:31:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:31:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:31:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:32:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:32:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:32:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:32:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:33:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:33:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:33:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:33:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:34:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:34:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:34:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:34:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:35:01 lola-4 CROND[39901]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 23:35:01 lola-4 TIME: Time stamp for console Nov 4 23:35:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:35:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:35:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:35:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:36:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:36:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:36:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:36:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:37:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:37:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:37:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:37:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:38:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:38:15 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 23:38:15 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 23:38:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:38:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:38:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:39:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:39:17 lola-4 kernel: LNet: Service thread pid 21625 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 23:39:17 lola-4 kernel: Pid: 21625, comm: ll_ost00_077 Nov 4 23:39:17 lola-4 kernel: Nov 4 23:39:17 lola-4 kernel: Call Trace: Nov 4 23:39:17 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 23:39:17 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 23:39:17 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 23:39:17 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 23:39:17 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 23:39:17 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 23:39:17 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 23:39:17 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 23:39:17 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 23:39:17 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 23:39:17 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 23:39:17 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 23:39:17 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 23:39:17 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 23:39:17 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 23:39:17 lola-4 kernel: Nov 4 23:39:17 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478327957.21625 Nov 4 23:39:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 437 Nov 4 23:39:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:39:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:40:01 lola-4 CROND[39937]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 23:40:01 lola-4 CROND[39938]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 23:40:01 lola-4 TIME: Time stamp for console Nov 4 23:40:11 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 23:40:11 lola-4 kernel: req@ffff8807cb7fc050 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478328016 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 23:40:11 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 110 previous similar messages Nov 4 23:40:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:40:17 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 23:40:17 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 4 23:40:17 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 23:40:17 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 4 23:40:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:40:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:40:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:41:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:41:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:41:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:41:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:42:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:42:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:42:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:42:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:43:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:43:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:43:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:43:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:44:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:44:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:44:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:44:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:45:01 lola-4 CROND[39974]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 23:45:01 lola-4 TIME: Time stamp for console Nov 4 23:45:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:45:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:45:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:45:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:46:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:46:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:46:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:46:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:47:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:47:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:47:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:47:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:48:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:48:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:48:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 438 Nov 4 23:48:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:49:05 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 23:49:05 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 23:49:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:49:18 lola-4 kernel: LNet: Service thread pid 21497 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 23:49:18 lola-4 kernel: Pid: 21497, comm: ll_ost00_034 Nov 4 23:49:18 lola-4 kernel: Nov 4 23:49:18 lola-4 kernel: Call Trace: Nov 4 23:49:18 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 23:49:18 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 23:49:18 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 23:49:18 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 23:49:18 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 23:49:18 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 23:49:18 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 23:49:18 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 23:49:18 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 23:49:18 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 23:49:18 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 23:49:18 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 23:49:18 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 23:49:18 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 23:49:18 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 23:49:18 lola-4 kernel: Nov 4 23:49:18 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478328558.21497 Nov 4 23:49:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:49:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:49:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:50:01 lola-4 CROND[40010]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 23:50:01 lola-4 CROND[40011]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 4 23:50:01 lola-4 TIME: Time stamp for console Nov 4 23:50:12 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 4 23:50:12 lola-4 kernel: req@ffff8806f17bd050 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478328617 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 4 23:50:12 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 4 23:50:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:50:18 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 4 23:50:18 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 23:50:18 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 4 23:50:18 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 4 23:50:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:50:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:50:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:51:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:51:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:51:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:51:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:52:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:52:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:52:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:52:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:53:01 lola-4 CROND[40035]: (root) CMD (/usr/lib64/sa/sa2 -A) Nov 4 23:53:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:53:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:53:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:53:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:54:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:54:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:54:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:54:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:55:01 lola-4 CROND[40057]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 4 23:55:01 lola-4 TIME: Time stamp for console Nov 4 23:55:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:55:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:55:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:55:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:56:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:56:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:56:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:56:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:57:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:57:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:57:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:57:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:58:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:58:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 439 Nov 4 23:58:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:58:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:59:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:59:19 lola-4 kernel: LNet: Service thread pid 21745 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 4 23:59:19 lola-4 kernel: Pid: 21745, comm: ll_ost00_113 Nov 4 23:59:19 lola-4 kernel: Nov 4 23:59:19 lola-4 kernel: Call Trace: Nov 4 23:59:19 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 4 23:59:19 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 4 23:59:19 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 4 23:59:19 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 4 23:59:19 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 4 23:59:19 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 4 23:59:19 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 4 23:59:19 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 4 23:59:19 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 4 23:59:19 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 4 23:59:19 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 4 23:59:19 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 4 23:59:19 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 4 23:59:19 lola-4 kernel: [] child_rip+0xa/0x20 Nov 4 23:59:19 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 4 23:59:19 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 4 23:59:19 lola-4 kernel: Nov 4 23:59:19 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478329159.21745 Nov 4 23:59:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:59:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 4 23:59:55 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 4 23:59:55 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 4 23:59:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:00:01 lola-4 CROND[40091]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 00:00:01 lola-4 CROND[40092]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 00:00:01 lola-4 TIME: Time stamp for console Nov 5 00:00:13 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 00:00:13 lola-4 kernel: req@ffff880519175050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478329218 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 00:00:13 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 110 previous similar messages Nov 5 00:00:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:00:19 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 00:00:19 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 00:00:19 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 00:00:19 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 00:00:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 00:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 00:00:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:01:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:01:01 lola-4 CROND[40105]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 00:01:01 lola-4 run-parts(/etc/cron.hourly)[4010 starting 0anacron Nov 5 00:01:01 lola-4 anacron[40116]: Anacron started on 2016-11-05 Nov 5 00:01:01 lola-4 anacron[40116]: Jobs will be executed sequentially Nov 5 00:01:01 lola-4 anacron[40116]: Normal exit (0 jobs run) Nov 5 00:01:01 lola-4 run-parts(/etc/cron.hourly)[4011 finished 0anacron Nov 5 00:01:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:01:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:01:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:02:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:02:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:02:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:02:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:03:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:03:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:03:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:03:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:04:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:04:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:04:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:04:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:05:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:05:01 lola-4 CROND[40145]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 00:05:01 lola-4 TIME: Time stamp for console Nov 5 00:05:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:05:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:05:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:06:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:06:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:06:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:06:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:07:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:07:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:07:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:07:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:08:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:08:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:08:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:08:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:09:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:09:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 440 Nov 5 00:09:20 lola-4 kernel: LNet: Service thread pid 13074 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 00:09:20 lola-4 kernel: Pid: 13074, comm: ll_ost00_000 Nov 5 00:09:20 lola-4 kernel: Nov 5 00:09:20 lola-4 kernel: Call Trace: Nov 5 00:09:20 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 00:09:20 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 00:09:20 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 00:09:20 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 00:09:20 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 00:09:20 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 00:09:20 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 00:09:20 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 00:09:20 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 00:09:20 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 5 00:09:20 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 5 00:09:20 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 00:09:20 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 00:09:20 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 00:09:20 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 00:09:20 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 00:09:20 lola-4 kernel: Nov 5 00:09:20 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478329760.13074 Nov 5 00:09:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:09:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:10:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:10:01 lola-4 CROND[40181]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 00:10:01 lola-4 CROND[40182]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 00:10:01 lola-4 TIME: Time stamp for console Nov 5 00:10:14 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 00:10:14 lola-4 kernel: req@ffff8806c76c5c50 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478329819 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 00:10:14 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 106 previous similar messages Nov 5 00:10:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:10:20 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 00:10:20 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 00:10:20 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 00:10:20 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 00:10:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:10:45 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 00:10:45 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 00:10:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:11:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:11:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:11:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:11:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:12:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:12:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:12:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:12:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:13:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:13:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:13:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:13:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:14:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:14:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:14:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:14:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:15:01 lola-4 CROND[40218]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 00:15:01 lola-4 TIME: Time stamp for console Nov 5 00:15:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:15:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:15:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:15:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:16:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:16:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:16:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:16:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:17:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:17:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:17:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:17:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:18:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:18:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:18:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:18:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:19:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:19:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:19:21 lola-4 kernel: LNet: Service thread pid 21626 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 00:19:21 lola-4 kernel: Pid: 21626, comm: ll_ost00_078 Nov 5 00:19:21 lola-4 kernel: Nov 5 00:19:21 lola-4 kernel: Call Trace: Nov 5 00:19:21 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 5 00:19:21 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 00:19:21 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 00:19:21 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 00:19:21 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 00:19:21 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 00:19:21 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 00:19:21 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 00:19:21 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 00:19:21 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 00:19:21 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 00:19:21 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 00:19:21 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 00:19:21 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 00:19:21 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 00:19:21 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 00:19:21 lola-4 kernel: Nov 5 00:19:21 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478330361.21626 Nov 5 00:19:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 441 Nov 5 00:19:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:20:01 lola-4 CROND[40254]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 00:20:01 lola-4 TIME: Time stamp for console Nov 5 00:20:01 lola-4 CROND[40255]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 00:20:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:20:15 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 00:20:15 lola-4 kernel: req@ffff8805249ebc50 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478330420 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 00:20:15 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 00:20:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:20:21 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 00:20:21 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 00:20:21 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 00:20:21 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 00:20:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:20:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:21:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:21:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:21:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:21:35 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 00:21:35 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 00:21:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:22:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:22:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:22:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:22:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:23:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:23:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:23:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:23:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:24:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:24:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:24:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:24:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:25:01 lola-4 CROND[40292]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 00:25:01 lola-4 TIME: Time stamp for console Nov 5 00:25:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:25:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:25:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:25:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:26:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:26:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:26:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:26:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:27:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:27:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:27:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:27:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:28:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:28:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:28:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:28:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:29:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:29:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:29:22 lola-4 kernel: LNet: Service thread pid 21475 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 00:29:22 lola-4 kernel: Pid: 21475, comm: ll_ost00_013 Nov 5 00:29:22 lola-4 kernel: Nov 5 00:29:22 lola-4 kernel: Call Trace: Nov 5 00:29:22 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 00:29:22 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 00:29:22 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 00:29:22 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 00:29:22 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 00:29:22 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 00:29:22 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 00:29:22 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 00:29:22 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 00:29:22 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 5 00:29:22 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 5 00:29:22 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 00:29:22 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 00:29:22 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 00:29:22 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 00:29:22 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 00:29:22 lola-4 kernel: Nov 5 00:29:22 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478330962.21475 Nov 5 00:29:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:29:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:30:01 lola-4 CROND[40328]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 00:30:01 lola-4 CROND[40329]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 00:30:01 lola-4 TIME: Time stamp for console Nov 5 00:30:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:30:16 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 00:30:16 lola-4 kernel: req@ffff880572d9bc50 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478331021 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 00:30:16 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 110 previous similar messages Nov 5 00:30:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:30:22 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 00:30:22 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 00:30:22 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 00:30:22 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 00:30:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:30:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:31:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:31:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:31:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:31:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:32:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 442 Nov 5 00:32:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:32:25 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 00:32:25 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 00:32:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:32:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:33:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:33:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:33:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:33:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:34:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:34:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:34:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:34:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:35:01 lola-4 CROND[40365]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 00:35:01 lola-4 TIME: Time stamp for console Nov 5 00:35:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:35:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:35:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:35:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:36:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:36:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:36:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:36:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:37:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:37:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:37:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:37:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:38:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:38:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:38:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:38:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:39:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:39:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:39:23 lola-4 kernel: LNet: Service thread pid 13076 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 00:39:23 lola-4 kernel: Pid: 13076, comm: ll_ost00_002 Nov 5 00:39:23 lola-4 kernel: Nov 5 00:39:23 lola-4 kernel: Call Trace: Nov 5 00:39:23 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 00:39:23 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 00:39:23 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 00:39:23 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 00:39:23 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 00:39:23 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 00:39:23 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 00:39:23 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 00:39:23 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 00:39:23 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 5 00:39:23 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 5 00:39:23 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 00:39:23 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 00:39:23 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 00:39:23 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 00:39:23 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 00:39:23 lola-4 kernel: Nov 5 00:39:23 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478331563.13076 Nov 5 00:39:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:39:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:40:01 lola-4 CROND[40401]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 00:40:01 lola-4 CROND[40402]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 00:40:01 lola-4 TIME: Time stamp for console Nov 5 00:40:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:40:17 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 00:40:17 lola-4 kernel: req@ffff88052c49dc50 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478331622 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 00:40:17 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 00:40:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:40:23 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 00:40:23 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 00:40:23 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 00:40:23 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 00:40:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:40:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 443 Nov 5 00:41:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:41:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:41:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:41:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:42:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:42:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:42:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:42:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:43:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:43:15 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 00:43:15 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 00:43:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:43:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:43:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:44:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:44:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:44:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:44:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:45:01 lola-4 CROND[40438]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 00:45:01 lola-4 TIME: Time stamp for console Nov 5 00:45:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:45:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:45:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:45:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:46:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:46:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:46:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:46:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:47:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:47:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:47:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:47:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 444 Nov 5 00:48:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:48:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:48:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:48:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:49:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:49:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:49:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:49:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:50:01 lola-4 CROND[40472]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 00:50:01 lola-4 CROND[40473]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 00:50:01 lola-4 TIME: Time stamp for console Nov 5 00:50:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:50:17 lola-4 kernel: Lustre: 21660:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Nov 5 00:50:17 lola-4 kernel: req@ffff8805edf25050 x1550004935577520/t0(0) o101->edf622b2-6dfa-7746-932c-dadfb996df24@192.168.1.121@o2ib100:-1/-1 lens 328/0 e 0 to 0 dl 1478332222 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 00:50:17 lola-4 kernel: Lustre: 21660:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 108 previous similar messages Nov 5 00:50:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:50:23 lola-4 kernel: Lustre: soaked-OST0008: Client edf622b2-6dfa-7746-932c-dadfb996df24 (at 192.168.1.121@o2ib100) reconnecting Nov 5 00:50:23 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 00:50:23 lola-4 kernel: Lustre: soaked-OST0008: Connection restored to edf622b2-6dfa-7746-932c-dadfb996df24 (at 192.168.1.121@o2ib100) Nov 5 00:50:23 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 00:50:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:50:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:51:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:51:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:51:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:51:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:51:59 lola-4 kernel: LNet: Service thread pid 21640 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 00:51:59 lola-4 kernel: Pid: 21640, comm: ll_ost00_092 Nov 5 00:51:59 lola-4 kernel: Nov 5 00:51:59 lola-4 kernel: Call Trace: Nov 5 00:51:59 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 00:51:59 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 00:51:59 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 00:51:59 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 00:51:59 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 00:51:59 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 00:51:59 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 00:51:59 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 00:51:59 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 00:51:59 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 00:51:59 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 00:51:59 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 00:51:59 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 00:51:59 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 00:51:59 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 00:51:59 lola-4 kernel: Nov 5 00:51:59 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478332319.21640 Nov 5 00:52:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:52:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:52:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:52:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:53:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:53:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:53:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:53:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:54:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:54:05 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 00:54:05 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 00:54:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:54:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:54:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:55:01 lola-4 CROND[40511]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 00:55:01 lola-4 TIME: Time stamp for console Nov 5 00:55:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:55:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:55:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:55:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:56:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:56:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:56:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:56:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:57:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:57:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:57:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:57:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:58:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:58:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:58:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:58:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:59:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:59:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 00:59:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 00:59:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 01:00:01 lola-4 CROND[40546]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 01:00:01 lola-4 CROND[40547]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 01:00:01 lola-4 TIME: Time stamp for console Nov 5 01:00:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 01:00:19 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 01:00:19 lola-4 kernel: req@ffff880820927850 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478332824 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 01:00:19 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 108 previous similar messages Nov 5 01:00:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:00:25 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 01:00:25 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 01:00:25 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 01:00:25 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 01:00:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 01:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 01:00:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:01:01 lola-4 CROND[40558]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 01:01:01 lola-4 run-parts(/etc/cron.hourly)[4055 starting 0anacron Nov 5 01:01:01 lola-4 anacron[40569]: Anacron started on 2016-11-05 Nov 5 01:01:01 lola-4 anacron[40569]: Jobs will be executed sequentially Nov 5 01:01:01 lola-4 anacron[40569]: Normal exit (0 jobs run) Nov 5 01:01:01 lola-4 run-parts(/etc/cron.hourly)[4057 finished 0anacron Nov 5 01:01:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:01:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:01:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:01:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:02:00 lola-4 kernel: LNet: Service thread pid 21496 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 01:02:00 lola-4 kernel: Pid: 21496, comm: ll_ost00_033 Nov 5 01:02:00 lola-4 kernel: Nov 5 01:02:00 lola-4 kernel: Call Trace: Nov 5 01:02:00 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 01:02:00 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 01:02:00 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 01:02:00 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 01:02:00 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 01:02:00 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 01:02:00 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 01:02:00 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 01:02:00 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 01:02:00 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 01:02:00 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 01:02:00 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 01:02:00 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 01:02:00 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 01:02:00 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 01:02:00 lola-4 kernel: Nov 5 01:02:00 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478332920.21496 Nov 5 01:02:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:02:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:02:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:02:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:03:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:03:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:03:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 01:03:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 01:04:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 01:04:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 01:04:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 445 Nov 5 01:04:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:04:55 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 01:04:55 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 01:05:01 lola-4 CROND[40600]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 01:05:01 lola-4 TIME: Time stamp for console Nov 5 01:05:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:05:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:05:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:05:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:06:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:06:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:06:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:06:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:07:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:07:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:07:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:07:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:08:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:08:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:08:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:08:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:09:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:09:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:09:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:09:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:10:01 lola-4 CROND[40634]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 01:10:01 lola-4 CROND[40635]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 01:10:01 lola-4 TIME: Time stamp for console Nov 5 01:10:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:10:20 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 01:10:20 lola-4 kernel: req@ffff88062d76b450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478333425 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 01:10:20 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 01:10:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:10:26 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 01:10:26 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 01:10:26 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 01:10:26 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 01:10:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:10:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:11:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:11:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:11:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:11:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:12:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:12:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:12:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:12:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:13:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:13:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:13:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:13:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:14:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:14:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:14:36 lola-4 kernel: LNet: Service thread pid 21720 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 01:14:36 lola-4 kernel: Pid: 21720, comm: ll_ost00_105 Nov 5 01:14:36 lola-4 kernel: Nov 5 01:14:36 lola-4 kernel: Call Trace: Nov 5 01:14:36 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 5 01:14:36 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 01:14:36 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 01:14:36 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 01:14:36 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 01:14:36 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 01:14:36 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 01:14:36 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 01:14:36 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 01:14:36 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 01:14:36 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 01:14:36 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 01:14:36 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 01:14:36 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 01:14:36 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 01:14:36 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 01:14:36 lola-4 kernel: Nov 5 01:14:36 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478333676.21720 Nov 5 01:14:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:14:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:15:01 lola-4 CROND[40673]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 01:15:01 lola-4 TIME: Time stamp for console Nov 5 01:15:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:15:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:15:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:15:45 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 01:15:45 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 01:15:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:16:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:16:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:16:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:16:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:17:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:17:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 446 Nov 5 01:17:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:17:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:18:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:18:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:18:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:18:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:19:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:19:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:19:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:19:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:20:01 lola-4 CROND[40707]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 01:20:01 lola-4 CROND[40708]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 01:20:01 lola-4 TIME: Time stamp for console Nov 5 01:20:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:20:21 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 01:20:21 lola-4 kernel: req@ffff8807b891e050 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478334026 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 01:20:21 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 01:20:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:20:27 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 01:20:27 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 01:20:27 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 01:20:27 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 01:20:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:20:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:21:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:21:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:21:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:21:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:22:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 447 Nov 5 01:22:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:22:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:22:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:23:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:23:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:23:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:23:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:24:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:24:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:24:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:24:37 lola-4 kernel: LNet: Service thread pid 21489 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 01:24:37 lola-4 kernel: Pid: 21489, comm: ll_ost00_026 Nov 5 01:24:37 lola-4 kernel: Nov 5 01:24:37 lola-4 kernel: Call Trace: Nov 5 01:24:37 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 01:24:37 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 01:24:37 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 01:24:37 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 01:24:37 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 01:24:37 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 01:24:37 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 01:24:37 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 01:24:37 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 01:24:37 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 5 01:24:37 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 5 01:24:37 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 01:24:37 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 01:24:37 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 01:24:37 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 01:24:37 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 01:24:37 lola-4 kernel: Nov 5 01:24:37 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478334277.21489 Nov 5 01:24:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:25:01 lola-4 CROND[40746]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 01:25:01 lola-4 TIME: Time stamp for console Nov 5 01:25:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:25:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:25:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:25:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:26:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:26:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:26:35 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 01:26:35 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 01:26:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:26:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:27:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:27:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:27:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:27:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:28:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:28:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:28:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:28:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:29:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:29:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:29:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:29:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:30:01 lola-4 CROND[40780]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 01:30:01 lola-4 CROND[40781]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 01:30:01 lola-4 TIME: Time stamp for console Nov 5 01:30:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:30:22 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 01:30:22 lola-4 kernel: req@ffff880642bcfc50 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478334627 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 01:30:22 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 01:30:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:30:28 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 01:30:28 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 01:30:28 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 01:30:28 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 01:30:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:30:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:31:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:31:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:31:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:31:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:32:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:32:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:32:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:32:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:33:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:33:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:33:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:33:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:34:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:34:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:34:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:34:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:35:01 lola-4 CROND[40818]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 01:35:01 lola-4 TIME: Time stamp for console Nov 5 01:35:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:35:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:35:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:35:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:36:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:36:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:36:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:36:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 448 Nov 5 01:37:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:37:13 lola-4 kernel: LNet: Service thread pid 21638 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 01:37:13 lola-4 kernel: Pid: 21638, comm: ll_ost00_090 Nov 5 01:37:13 lola-4 kernel: Nov 5 01:37:13 lola-4 kernel: Call Trace: Nov 5 01:37:13 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 01:37:13 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 01:37:13 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 01:37:13 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 01:37:13 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 01:37:13 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 01:37:13 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 01:37:13 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 01:37:13 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 01:37:13 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 01:37:13 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 01:37:13 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 01:37:13 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 01:37:13 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 01:37:13 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 01:37:13 lola-4 kernel: Nov 5 01:37:13 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478335033.21638 Nov 5 01:37:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:37:25 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 01:37:25 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 01:37:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:37:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:38:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:38:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:38:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:38:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:39:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:39:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:39:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:39:38 lola-4 sendmail[10625]: have been rejecting connections on daemon MTA for 09:00:45 Nov 5 01:39:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:40:01 lola-4 CROND[40854]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 01:40:01 lola-4 CROND[40855]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 01:40:01 lola-4 TIME: Time stamp for console Nov 5 01:40:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:40:23 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 01:40:23 lola-4 kernel: req@ffff8806385aa450 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478335228 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 01:40:23 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 108 previous similar messages Nov 5 01:40:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:40:29 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 01:40:29 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 01:40:29 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 01:40:29 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 01:40:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:40:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:41:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:41:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:41:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:41:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:42:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:42:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:42:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:42:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:43:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:43:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:43:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:43:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:44:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:44:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:44:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:44:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:45:01 lola-4 CROND[40891]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 01:45:01 lola-4 TIME: Time stamp for console Nov 5 01:45:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:45:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:45:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:45:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:46:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:46:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:46:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:46:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:47:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:47:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:47:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:47:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:48:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:48:15 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 01:48:15 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 01:48:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:48:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:48:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:49:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:49:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:49:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 449 Nov 5 01:49:49 lola-4 kernel: LNet: Service thread pid 21611 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 01:49:49 lola-4 kernel: Pid: 21611, comm: ll_ost00_063 Nov 5 01:49:49 lola-4 kernel: Nov 5 01:49:49 lola-4 kernel: Call Trace: Nov 5 01:49:49 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 01:49:49 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 01:49:49 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 01:49:49 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 01:49:49 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 01:49:49 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 01:49:49 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 01:49:49 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 01:49:49 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 01:49:49 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 5 01:49:49 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 5 01:49:49 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 01:49:49 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 01:49:49 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 01:49:49 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 01:49:49 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 01:49:49 lola-4 kernel: Nov 5 01:49:49 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478335789.21611 Nov 5 01:49:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:50:01 lola-4 CROND[40927]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 01:50:01 lola-4 CROND[40928]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 01:50:01 lola-4 TIME: Time stamp for console Nov 5 01:50:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:50:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:50:24 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 01:50:24 lola-4 kernel: req@ffff880808a89450 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478335829 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 01:50:24 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 108 previous similar messages Nov 5 01:50:30 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 01:50:30 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 01:50:30 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 01:50:30 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 01:50:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:50:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:51:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:51:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:51:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:51:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 450 Nov 5 01:52:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:52:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:52:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:52:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:53:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:53:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:53:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:53:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:54:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:54:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:54:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:54:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:55:01 lola-4 CROND[40964]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 01:55:01 lola-4 TIME: Time stamp for console Nov 5 01:55:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:55:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:55:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:55:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:56:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:56:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:56:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:56:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:57:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:57:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:57:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:57:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:58:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:58:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:58:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:58:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:59:05 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 01:59:05 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 01:59:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:59:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:59:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 01:59:50 lola-4 kernel: LNet: Service thread pid 21616 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 01:59:50 lola-4 kernel: Pid: 21616, comm: ll_ost00_068 Nov 5 01:59:50 lola-4 kernel: Nov 5 01:59:50 lola-4 kernel: Call Trace: Nov 5 01:59:50 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 01:59:50 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 01:59:50 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 01:59:50 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 01:59:50 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 01:59:50 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 01:59:50 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 01:59:50 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 01:59:50 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 01:59:50 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 01:59:50 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 01:59:50 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 01:59:50 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 01:59:50 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 01:59:50 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 01:59:50 lola-4 kernel: Nov 5 01:59:50 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478336390.21616 Nov 5 01:59:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:00:01 lola-4 CROND[41000]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 02:00:01 lola-4 CROND[41001]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 02:00:01 lola-4 TIME: Time stamp for console Nov 5 02:00:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:00:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:00:25 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 02:00:25 lola-4 kernel: req@ffff88070dc87450 x1550005401079776/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478336430 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 02:00:25 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 02:00:31 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 02:00:31 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 02:00:31 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 02:00:31 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 02:00:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 02:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 02:00:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:01:01 lola-4 CROND[41012]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 02:01:01 lola-4 run-parts(/etc/cron.hourly)[4101 starting 0anacron Nov 5 02:01:01 lola-4 anacron[41023]: Anacron started on 2016-11-05 Nov 5 02:01:01 lola-4 anacron[41023]: Jobs will be executed sequentially Nov 5 02:01:01 lola-4 anacron[41023]: Normal exit (0 jobs run) Nov 5 02:01:01 lola-4 run-parts(/etc/cron.hourly)[4102 finished 0anacron Nov 5 02:01:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:01:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:01:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:01:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:02:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:02:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:02:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:02:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:03:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:03:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:03:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:03:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:04:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:04:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:04:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:04:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:05:01 lola-4 CROND[41053]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 02:05:01 lola-4 TIME: Time stamp for console Nov 5 02:05:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:05:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:05:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:05:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:06:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:06:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:06:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:06:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:07:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:07:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:07:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 451 Nov 5 02:07:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:08:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:08:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:08:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:08:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:09:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:09:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:09:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:09:51 lola-4 kernel: LNet: Service thread pid 21471 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 02:09:51 lola-4 kernel: Pid: 21471, comm: ll_ost00_009 Nov 5 02:09:51 lola-4 kernel: Nov 5 02:09:51 lola-4 kernel: Call Trace: Nov 5 02:09:51 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 5 02:09:51 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 02:09:51 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 02:09:51 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 02:09:51 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 02:09:51 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 02:09:51 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 02:09:51 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 02:09:51 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 02:09:51 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 02:09:51 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 02:09:51 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 02:09:51 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 02:09:51 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 02:09:51 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 02:09:51 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 02:09:51 lola-4 kernel: Nov 5 02:09:51 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478336991.21471 Nov 5 02:09:55 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 02:09:55 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 02:09:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:10:01 lola-4 CROND[41089]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 02:10:01 lola-4 CROND[41090]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 02:10:01 lola-4 TIME: Time stamp for console Nov 5 02:10:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:10:26 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 02:10:26 lola-4 kernel: req@ffff88077bed9850 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478337031 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 02:10:26 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 02:10:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:10:32 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 02:10:32 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 02:10:32 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 02:10:32 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 02:10:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:10:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:11:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:11:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:11:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:11:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:12:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:12:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:12:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:12:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:13:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:13:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:13:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:13:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:14:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:14:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:14:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:14:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:15:01 lola-4 CROND[41126]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 02:15:01 lola-4 TIME: Time stamp for console Nov 5 02:15:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:15:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:15:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:15:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:16:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:16:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:16:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:16:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:17:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:17:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:17:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:17:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:18:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:18:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:18:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:18:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:19:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:19:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:19:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 452 Nov 5 02:19:52 lola-4 kernel: LNet: Service thread pid 21602 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 02:19:52 lola-4 kernel: Pid: 21602, comm: ll_ost00_054 Nov 5 02:19:52 lola-4 kernel: Nov 5 02:19:52 lola-4 kernel: Call Trace: Nov 5 02:19:52 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 02:19:52 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 02:19:52 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 02:19:52 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 02:19:52 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 02:19:52 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 02:19:52 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 02:19:52 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 02:19:52 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 02:19:52 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 02:19:52 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 02:19:52 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 02:19:52 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 02:19:52 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 02:19:52 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 02:19:52 lola-4 kernel: Nov 5 02:19:52 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478337592.21602 Nov 5 02:19:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:20:01 lola-4 CROND[41162]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 02:20:01 lola-4 CROND[41163]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 02:20:01 lola-4 TIME: Time stamp for console Nov 5 02:20:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:20:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:20:27 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 02:20:27 lola-4 kernel: req@ffff8806362ddc50 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478337632 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 02:20:27 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 02:20:33 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 02:20:33 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 02:20:33 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 02:20:33 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 02:20:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:20:45 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 02:20:45 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 02:20:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:21:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:21:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:21:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:21:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:22:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:22:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:22:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:22:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:23:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:23:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:23:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:23:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:24:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:24:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:24:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:24:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:25:01 lola-4 CROND[41199]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 02:25:01 lola-4 TIME: Time stamp for console Nov 5 02:25:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:25:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:25:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:25:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:26:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:26:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:26:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:26:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:27:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:27:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:27:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:27:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:28:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:28:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:28:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:28:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:29:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:29:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:29:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:29:53 lola-4 kernel: LNet: Service thread pid 21633 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 02:29:53 lola-4 kernel: Pid: 21633, comm: ll_ost00_085 Nov 5 02:29:53 lola-4 kernel: Nov 5 02:29:53 lola-4 kernel: Call Trace: Nov 5 02:29:53 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 02:29:53 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 02:29:53 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 02:29:53 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 02:29:53 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 02:29:53 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 02:29:53 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 02:29:53 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 02:29:53 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 02:29:53 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 02:29:53 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 02:29:53 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 02:29:53 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 02:29:53 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 02:29:53 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 02:29:53 lola-4 kernel: Nov 5 02:29:53 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478338193.21633 Nov 5 02:29:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 453 Nov 5 02:30:02 lola-4 CROND[41235]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 02:30:02 lola-4 CROND[41236]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 02:30:02 lola-4 TIME: Time stamp for console Nov 5 02:30:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:30:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:30:28 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 02:30:28 lola-4 kernel: req@ffff8806ec281c50 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478338233 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 02:30:28 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 5 02:30:34 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 02:30:34 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 02:30:34 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 02:30:34 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 02:30:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:30:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:31:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:31:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:31:35 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 02:31:35 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 02:31:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:31:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:32:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:32:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:32:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:32:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:33:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:33:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:33:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:33:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:34:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:34:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:34:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:34:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:35:01 lola-4 CROND[41272]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 02:35:01 lola-4 TIME: Time stamp for console Nov 5 02:35:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:35:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:35:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:35:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:36:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:36:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:36:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:36:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 454 Nov 5 02:37:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:37:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:37:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:37:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:38:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:38:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:38:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:38:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:39:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:39:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:39:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:39:54 lola-4 kernel: LNet: Service thread pid 21484 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 02:39:54 lola-4 kernel: Pid: 21484, comm: ll_ost00_021 Nov 5 02:39:54 lola-4 kernel: Nov 5 02:39:54 lola-4 kernel: Call Trace: Nov 5 02:39:54 lola-4 kernel: [] ? account_entity_enqueue+0x7e/0x90 Nov 5 02:39:54 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 02:39:54 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 02:39:54 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 02:39:54 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 02:39:54 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 02:39:54 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 02:39:54 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 02:39:54 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 02:39:54 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 02:39:54 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 5 02:39:54 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 5 02:39:54 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 02:39:54 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 02:39:54 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 02:39:54 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 02:39:54 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 02:39:54 lola-4 kernel: Nov 5 02:39:54 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478338794.21484 Nov 5 02:39:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:40:01 lola-4 CROND[41309]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 02:40:01 lola-4 CROND[41310]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 02:40:01 lola-4 TIME: Time stamp for console Nov 5 02:40:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:40:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:40:29 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 02:40:29 lola-4 kernel: req@ffff8806a5e22850 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478338834 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 02:40:29 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 02:40:35 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 02:40:35 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 02:40:35 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 02:40:35 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 02:40:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:40:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:41:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:41:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:41:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:41:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:42:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:42:25 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 02:42:25 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 02:42:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:42:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:42:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:43:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:43:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:43:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:43:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:44:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:44:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:44:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:44:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:45:01 lola-4 CROND[41346]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 02:45:01 lola-4 TIME: Time stamp for console Nov 5 02:45:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:45:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:45:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:45:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:46:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:46:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:46:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:46:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:47:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:47:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:47:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:47:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:48:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:48:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:48:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:48:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:49:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:49:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:49:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:49:55 lola-4 kernel: LNet: Service thread pid 21685 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 02:49:55 lola-4 kernel: Pid: 21685, comm: ll_ost00_094 Nov 5 02:49:55 lola-4 kernel: Nov 5 02:49:55 lola-4 kernel: Call Trace: Nov 5 02:49:55 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 02:49:55 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 02:49:55 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 02:49:55 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 02:49:55 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 02:49:55 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 02:49:55 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 02:49:55 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 02:49:55 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 02:49:55 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 02:49:55 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 02:49:55 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 02:49:55 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 02:49:55 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 02:49:55 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 02:49:55 lola-4 kernel: Nov 5 02:49:55 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478339395.21685 Nov 5 02:49:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:50:01 lola-4 CROND[41382]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 02:50:01 lola-4 CROND[41383]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 02:50:01 lola-4 TIME: Time stamp for console Nov 5 02:50:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 455 Nov 5 02:50:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:50:30 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 02:50:30 lola-4 kernel: req@ffff88051fa72050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478339435 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 02:50:30 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 02:50:36 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 02:50:36 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 02:50:36 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 02:50:36 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 02:50:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:50:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:51:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:51:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:51:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:51:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:52:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:52:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:52:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:52:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:53:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:53:15 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 02:53:15 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 02:53:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:53:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:53:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:54:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:54:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:54:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:54:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:55:01 lola-4 CROND[41419]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 02:55:01 lola-4 TIME: Time stamp for console Nov 5 02:55:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:55:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:55:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:55:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:56:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:56:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:56:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:56:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:57:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:57:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:57:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:57:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:58:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:58:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:58:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:58:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:59:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:59:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 02:59:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 02:59:56 lola-4 kernel: LNet: Service thread pid 20337 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 02:59:56 lola-4 kernel: Pid: 20337, comm: ll_ost00_005 Nov 5 02:59:56 lola-4 kernel: Nov 5 02:59:56 lola-4 kernel: Call Trace: Nov 5 02:59:56 lola-4 kernel: [] ? cache_alloc_refill+0x1c0/0x240 Nov 5 02:59:56 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 02:59:56 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 02:59:56 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 02:59:56 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 02:59:56 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 02:59:56 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 02:59:56 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 02:59:56 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 02:59:56 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 02:59:56 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 02:59:56 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 02:59:56 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 02:59:56 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 02:59:56 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 02:59:56 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 02:59:56 lola-4 kernel: Nov 5 02:59:56 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478339996.20337 Nov 5 02:59:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 456 Nov 5 03:00:01 lola-4 CROND[41455]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 03:00:01 lola-4 TIME: Time stamp for console Nov 5 03:00:01 lola-4 CROND[41456]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 03:00:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:00:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:00:31 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 03:00:31 lola-4 kernel: req@ffff8806c381b050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478340036 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 03:00:31 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 03:00:37 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 03:00:37 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 03:00:37 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 03:00:37 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 03:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 03:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 03:00:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:01:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:01:01 lola-4 CROND[41467]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 03:01:01 lola-4 run-parts(/etc/cron.hourly)[4146 starting 0anacron Nov 5 03:01:01 lola-4 anacron[41478]: Anacron started on 2016-11-05 Nov 5 03:01:01 lola-4 anacron[41478]: Will run job `cron.daily' in 49 min. Nov 5 03:01:01 lola-4 anacron[41478]: Jobs will be executed sequentially Nov 5 03:01:01 lola-4 run-parts(/etc/cron.hourly)[4148 finished 0anacron Nov 5 03:01:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:01:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:01:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:02:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:02:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:02:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:02:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:03:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:03:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:03:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:03:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:04:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:04:05 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 03:04:05 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 03:04:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:04:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:04:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:05:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:05:01 lola-4 CROND[41507]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 03:05:01 lola-4 TIME: Time stamp for console Nov 5 03:05:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:05:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:05:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:06:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:06:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 457 Nov 5 03:06:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:06:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:07:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:07:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:07:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:07:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:08:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:08:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:08:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:08:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:09:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:09:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:09:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:09:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:09:57 lola-4 kernel: LNet: Service thread pid 21474 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 03:09:57 lola-4 kernel: Pid: 21474, comm: ll_ost00_012 Nov 5 03:09:57 lola-4 kernel: Nov 5 03:09:57 lola-4 kernel: Call Trace: Nov 5 03:09:57 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 03:09:57 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 03:09:57 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 03:09:57 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 03:09:57 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 03:09:57 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 03:09:57 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 03:09:57 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 03:09:57 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 03:09:57 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 03:09:57 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 03:09:57 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 03:09:57 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 03:09:57 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 03:09:57 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 03:09:57 lola-4 kernel: Nov 5 03:09:57 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478340597.21474 Nov 5 03:10:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:10:01 lola-4 CROND[41543]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 03:10:01 lola-4 CROND[41544]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 03:10:01 lola-4 TIME: Time stamp for console Nov 5 03:10:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:10:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:10:32 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 03:10:32 lola-4 kernel: req@ffff88062d75e050 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478340637 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 03:10:32 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 03:10:38 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 03:10:38 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 03:10:38 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 03:10:38 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 03:10:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 458 Nov 5 03:11:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:11:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:11:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:11:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:12:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:12:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:12:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:12:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:13:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:13:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:13:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:13:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:14:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:14:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:14:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:14:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:14:55 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 03:14:55 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 03:15:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:15:01 lola-4 CROND[41581]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 03:15:01 lola-4 TIME: Time stamp for console Nov 5 03:15:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:15:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:15:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:16:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:16:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:16:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:16:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:17:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:17:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:17:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:17:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:18:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:18:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:18:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:18:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:19:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:19:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:19:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:19:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:19:58 lola-4 kernel: LNet: Service thread pid 21621 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 03:19:58 lola-4 kernel: Pid: 21621, comm: ll_ost00_073 Nov 5 03:19:58 lola-4 kernel: Nov 5 03:19:58 lola-4 kernel: Call Trace: Nov 5 03:19:58 lola-4 kernel: [] ? select_idle_sibling+0x95/0x150 Nov 5 03:19:58 lola-4 kernel: [] ? account_entity_enqueue+0x7e/0x90 Nov 5 03:19:58 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 03:19:58 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 03:19:58 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 03:19:58 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 03:19:58 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 03:19:58 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 03:19:58 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 03:19:58 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 03:19:58 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 03:19:58 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 03:19:58 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 03:19:58 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 03:19:58 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 03:19:58 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 03:19:58 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 03:19:58 lola-4 kernel: Nov 5 03:19:58 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478341198.21621 Nov 5 03:20:01 lola-4 CROND[41617]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 03:20:01 lola-4 CROND[41618]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 03:20:01 lola-4 TIME: Time stamp for console Nov 5 03:20:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:20:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:20:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 459 Nov 5 03:20:33 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 03:20:33 lola-4 kernel: req@ffff8805edc61450 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478341238 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 03:20:33 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 5 03:20:39 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 03:20:39 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 03:20:39 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 03:20:39 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 03:20:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:21:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:21:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:21:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:21:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:22:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:22:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:22:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:22:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:23:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:23:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:23:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:23:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:24:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:24:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:24:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:24:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:25:01 lola-4 CROND[41654]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 03:25:01 lola-4 TIME: Time stamp for console Nov 5 03:25:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:25:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:25:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:25:45 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 03:25:45 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 03:25:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:26:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:26:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:26:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:26:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:27:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:27:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:27:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:27:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:28:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:28:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:28:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:28:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:29:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:29:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:29:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:29:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:29:59 lola-4 kernel: LNet: Service thread pid 21684 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 03:29:59 lola-4 kernel: Pid: 21684, comm: ll_ost00_093 Nov 5 03:29:59 lola-4 kernel: Nov 5 03:29:59 lola-4 kernel: Call Trace: Nov 5 03:29:59 lola-4 kernel: [] ? account_entity_enqueue+0x7e/0x90 Nov 5 03:29:59 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 03:29:59 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 03:29:59 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 03:29:59 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 03:29:59 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 03:29:59 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 03:29:59 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 03:29:59 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 03:29:59 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 03:29:59 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 03:29:59 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 03:29:59 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 03:29:59 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 03:29:59 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 03:29:59 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 03:29:59 lola-4 kernel: Nov 5 03:29:59 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478341799.21684 Nov 5 03:30:01 lola-4 CROND[41690]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 03:30:01 lola-4 CROND[41691]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 03:30:01 lola-4 TIME: Time stamp for console Nov 5 03:30:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:30:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:30:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:30:34 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 03:30:34 lola-4 kernel: req@ffff8804e834ec50 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478341839 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 03:30:34 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 03:30:40 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 03:30:40 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 03:30:40 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 03:30:40 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 03:30:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:31:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:31:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:31:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 460 Nov 5 03:31:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:32:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:32:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:32:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:32:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:33:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:33:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:33:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:33:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:34:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:34:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:34:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:34:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:35:01 lola-4 CROND[41727]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 03:35:01 lola-4 TIME: Time stamp for console Nov 5 03:35:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:35:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:35:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:35:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:36:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:36:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:36:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:36:35 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 03:36:35 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 03:36:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:37:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:37:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:37:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:37:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:38:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:38:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:38:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:38:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:39:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:39:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:39:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:39:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:40:00 lola-4 kernel: LNet: Service thread pid 21753 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 03:40:00 lola-4 kernel: Pid: 21753, comm: ll_ost00_117 Nov 5 03:40:00 lola-4 kernel: Nov 5 03:40:00 lola-4 kernel: Call Trace: Nov 5 03:40:00 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 03:40:00 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 03:40:00 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 03:40:00 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 03:40:00 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 03:40:00 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 03:40:00 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 03:40:00 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 03:40:00 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 03:40:00 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 03:40:00 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 03:40:00 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 03:40:00 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 03:40:00 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 03:40:00 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 03:40:00 lola-4 kernel: Nov 5 03:40:00 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478342400.21753 Nov 5 03:40:01 lola-4 CROND[41763]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 03:40:01 lola-4 CROND[41764]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 03:40:01 lola-4 TIME: Time stamp for console Nov 5 03:40:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:40:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:40:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:40:35 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 03:40:35 lola-4 kernel: req@ffff88065091d850 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478342440 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 03:40:35 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 03:40:41 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 03:40:41 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 03:40:41 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 03:40:41 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 03:40:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:41:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:41:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:41:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:41:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:42:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:42:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:42:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:42:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:43:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:43:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:43:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:43:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:44:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:44:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:44:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:44:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:45:01 lola-4 CROND[41801]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 03:45:01 lola-4 TIME: Time stamp for console Nov 5 03:45:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:45:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:45:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:45:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:46:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:46:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:46:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:46:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:47:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:47:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 461 Nov 5 03:47:25 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 03:47:25 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 03:47:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:47:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:48:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:48:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:48:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:48:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:49:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:49:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:49:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:49:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:50:01 lola-4 CROND[41835]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 03:50:01 lola-4 CROND[41836]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 03:50:01 lola-4 TIME: Time stamp for console Nov 5 03:50:01 lola-4 anacron[41478]: Job `cron.daily' started Nov 5 03:50:01 lola-4 run-parts(/etc/cron.daily)[41839 starting logrotate Nov 5 03:50:01 lola-4 run-parts(/etc/cron.daily)[41847 finished logrotate Nov 5 03:50:01 lola-4 run-parts(/etc/cron.daily)[41839 starting makewhatis.cron Nov 5 03:50:01 lola-4 kernel: LNet: Service thread pid 21605 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 03:50:01 lola-4 kernel: Pid: 21605, comm: ll_ost00_057 Nov 5 03:50:01 lola-4 kernel: Nov 5 03:50:01 lola-4 kernel: Call Trace: Nov 5 03:50:01 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 03:50:01 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 03:50:01 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 03:50:01 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 03:50:01 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 03:50:01 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 03:50:01 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 03:50:01 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 03:50:01 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 03:50:01 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 03:50:01 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 03:50:01 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 03:50:01 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 03:50:01 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 03:50:01 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 03:50:01 lola-4 kernel: Nov 5 03:50:01 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478343001.21605 Nov 5 03:50:02 lola-4 run-parts(/etc/cron.daily)[42006 finished makewhatis.cron Nov 5 03:50:02 lola-4 run-parts(/etc/cron.daily)[41839 starting prelink Nov 5 03:50:02 lola-4 run-parts(/etc/cron.daily)[42018 finished prelink Nov 5 03:50:02 lola-4 run-parts(/etc/cron.daily)[41839 starting readahead.cron Nov 5 03:50:02 lola-4 run-parts(/etc/cron.daily)[42030 finished readahead.cron Nov 5 03:50:02 lola-4 run-parts(/etc/cron.daily)[41839 starting tmpwatch Nov 5 03:50:02 lola-4 run-parts(/etc/cron.daily)[42068 finished tmpwatch Nov 5 03:50:02 lola-4 anacron[41478]: Job `cron.daily' terminated Nov 5 03:50:02 lola-4 anacron[41478]: Normal exit (1 job run) Nov 5 03:50:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:50:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:50:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:50:36 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 03:50:36 lola-4 kernel: req@ffff88053af7a050 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478343041 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 03:50:36 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 03:50:42 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 03:50:42 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 03:50:42 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 03:50:42 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 03:50:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 462 Nov 5 03:51:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:51:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:51:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:51:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:52:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:52:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:52:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:52:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:53:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:53:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:53:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:53:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:54:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:54:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:54:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:54:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:55:01 lola-4 CROND[42102]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 03:55:01 lola-4 TIME: Time stamp for console Nov 5 03:55:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:55:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:55:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:55:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:56:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:56:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:56:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:56:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:57:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:57:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:57:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:57:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:58:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:58:15 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 03:58:15 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 03:58:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:58:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:58:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:59:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:59:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:59:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 03:59:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 04:00:01 lola-4 CROND[42136]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 04:00:01 lola-4 CROND[42137]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 04:00:01 lola-4 TIME: Time stamp for console Nov 5 04:00:02 lola-4 kernel: LNet: Service thread pid 21688 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 04:00:02 lola-4 kernel: Pid: 21688, comm: ll_ost00_097 Nov 5 04:00:02 lola-4 kernel: Nov 5 04:00:02 lola-4 kernel: Call Trace: Nov 5 04:00:02 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 04:00:02 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 04:00:02 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 04:00:02 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 04:00:02 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 04:00:02 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 04:00:02 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 04:00:02 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 04:00:02 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 04:00:02 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 04:00:02 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 04:00:02 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 04:00:02 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 04:00:02 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 04:00:02 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 04:00:02 lola-4 kernel: Nov 5 04:00:02 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478343602.21688 Nov 5 04:00:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 04:00:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 04:00:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 04:00:37 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 04:00:37 lola-4 kernel: req@ffff8804cb98bc50 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478343642 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 04:00:37 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 04:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 04:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 04:00:43 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 04:00:43 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 04:00:43 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 04:00:43 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 04:00:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 04:01:01 lola-4 CROND[42150]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 04:01:01 lola-4 run-parts(/etc/cron.hourly)[4215 starting 0anacron Nov 5 04:01:01 lola-4 run-parts(/etc/cron.hourly)[4215 finished 0anacron Nov 5 04:01:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 463 Nov 5 04:01:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:01:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:01:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:02:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:02:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:02:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:02:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:03:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:03:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:03:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:03:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:04:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:04:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:04:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:04:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:05:01 lola-4 CROND[42186]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 04:05:01 lola-4 TIME: Time stamp for console Nov 5 04:05:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:05:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:05:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:05:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:06:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:06:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:06:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:06:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:07:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:07:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:07:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:07:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:08:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:08:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:08:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:08:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:09:05 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 04:09:05 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 04:09:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:09:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:09:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:09:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:10:01 lola-4 CROND[42220]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 04:10:01 lola-4 CROND[42221]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 04:10:01 lola-4 TIME: Time stamp for console Nov 5 04:10:03 lola-4 kernel: LNet: Service thread pid 13075 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 04:10:03 lola-4 kernel: Pid: 13075, comm: ll_ost00_001 Nov 5 04:10:03 lola-4 kernel: Nov 5 04:10:03 lola-4 kernel: Call Trace: Nov 5 04:10:03 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 04:10:03 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 04:10:03 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 04:10:03 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 04:10:03 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 04:10:03 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 04:10:03 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 04:10:03 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 04:10:03 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 04:10:03 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 5 04:10:03 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 5 04:10:03 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 04:10:03 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 04:10:03 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 04:10:03 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 04:10:03 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 04:10:03 lola-4 kernel: Nov 5 04:10:03 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478344203.13075 Nov 5 04:10:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:10:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:10:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:10:38 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 04:10:38 lola-4 kernel: req@ffff88052fef8050 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478344243 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 04:10:38 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 5 04:10:44 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 04:10:44 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 04:10:44 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 04:10:44 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 04:10:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:11:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 464 Nov 5 04:11:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:11:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:11:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:12:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:12:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:12:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:12:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:13:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:13:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:13:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:13:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:14:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:14:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:14:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:14:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:15:01 lola-4 CROND[42259]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 04:15:01 lola-4 TIME: Time stamp for console Nov 5 04:15:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:15:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:15:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:15:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:16:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:16:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:16:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:16:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:17:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:17:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:17:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:17:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:18:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:18:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:18:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:18:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:19:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:19:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:19:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:19:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:19:55 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 04:19:55 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 04:20:01 lola-4 CROND[42294]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 04:20:01 lola-4 CROND[42295]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 04:20:01 lola-4 TIME: Time stamp for console Nov 5 04:20:04 lola-4 kernel: LNet: Service thread pid 21619 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 04:20:04 lola-4 kernel: Pid: 21619, comm: ll_ost00_071 Nov 5 04:20:04 lola-4 kernel: Nov 5 04:20:04 lola-4 kernel: Call Trace: Nov 5 04:20:04 lola-4 kernel: [] ? account_entity_enqueue+0x7e/0x90 Nov 5 04:20:04 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 04:20:04 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 04:20:04 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 04:20:04 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 04:20:04 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 04:20:04 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 04:20:04 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 04:20:04 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 04:20:04 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 04:20:04 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 04:20:04 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 04:20:04 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 04:20:04 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 04:20:04 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 04:20:04 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 04:20:04 lola-4 kernel: Nov 5 04:20:04 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478344804.21619 Nov 5 04:20:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:20:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:20:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:20:39 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 04:20:39 lola-4 kernel: req@ffff8807cb7fb450 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478344844 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 04:20:39 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 04:20:45 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 04:20:45 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 04:20:45 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 04:20:45 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 04:20:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:21:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 465 Nov 5 04:21:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:21:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:21:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:22:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:22:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:22:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:22:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:23:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:23:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:23:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:23:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:24:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:24:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:24:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:24:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:25:01 lola-4 CROND[42333]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 04:25:01 lola-4 TIME: Time stamp for console Nov 5 04:25:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:25:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:25:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:25:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:26:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:26:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:26:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:26:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:27:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:27:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:27:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:27:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:28:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:28:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:28:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:28:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:29:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:29:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:29:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:29:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:30:01 lola-4 CROND[42367]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 04:30:01 lola-4 CROND[42368]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 04:30:01 lola-4 TIME: Time stamp for console Nov 5 04:30:05 lola-4 kernel: LNet: Service thread pid 21186 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 04:30:05 lola-4 kernel: Pid: 21186, comm: ll_ost00_008 Nov 5 04:30:05 lola-4 kernel: Nov 5 04:30:05 lola-4 kernel: Call Trace: Nov 5 04:30:05 lola-4 kernel: [] ? cache_alloc_refill+0x1c0/0x240 Nov 5 04:30:05 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 04:30:05 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 04:30:05 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 04:30:05 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 04:30:05 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 04:30:05 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 04:30:05 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 04:30:05 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 04:30:05 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 04:30:05 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 5 04:30:05 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 5 04:30:05 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 04:30:05 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 04:30:05 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 04:30:05 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 04:30:05 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 04:30:05 lola-4 kernel: Nov 5 04:30:05 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478345405.21186 Nov 5 04:30:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:30:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:30:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:30:40 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 04:30:40 lola-4 kernel: req@ffff8807877f7850 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478345445 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 04:30:40 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 04:30:45 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 04:30:45 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 04:30:46 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 04:30:46 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 04:30:46 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 04:30:46 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 04:30:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:31:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:31:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:31:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:31:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:32:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:32:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:32:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:32:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:33:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:33:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:33:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:33:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:34:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:34:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:34:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:34:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:35:01 lola-4 CROND[42406]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 04:35:01 lola-4 TIME: Time stamp for console Nov 5 04:35:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:35:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:35:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:35:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:36:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:36:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:36:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:36:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 466 Nov 5 04:37:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:37:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:37:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:37:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:38:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:38:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:38:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:38:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:39:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:39:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:39:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:39:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:39:53 lola-4 sendmail[10625]: have been rejecting connections on daemon MTA for 12:01:00 Nov 5 04:40:01 lola-4 CROND[42440]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 04:40:01 lola-4 CROND[42441]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 04:40:01 lola-4 TIME: Time stamp for console Nov 5 04:40:06 lola-4 kernel: LNet: Service thread pid 21491 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 04:40:06 lola-4 kernel: Pid: 21491, comm: ll_ost00_028 Nov 5 04:40:06 lola-4 kernel: Nov 5 04:40:06 lola-4 kernel: Call Trace: Nov 5 04:40:06 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 04:40:06 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 04:40:06 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 04:40:06 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 04:40:06 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 04:40:06 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 04:40:06 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 04:40:06 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 04:40:06 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 04:40:06 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 5 04:40:06 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 5 04:40:06 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 04:40:06 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 04:40:06 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 04:40:06 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 04:40:06 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 04:40:06 lola-4 kernel: Nov 5 04:40:06 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478346006.21491 Nov 5 04:40:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:40:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:40:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:40:41 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 04:40:41 lola-4 kernel: req@ffff8804e294bc50 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478346046 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 04:40:41 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 04:40:47 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 04:40:47 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 04:40:47 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 04:40:47 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 04:40:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:41:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:41:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:41:35 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 04:41:35 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 04:41:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:41:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:42:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:42:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:42:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:42:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:43:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:43:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:43:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:43:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:44:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:44:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:44:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:44:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:45:01 lola-4 CROND[42479]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 04:45:01 lola-4 TIME: Time stamp for console Nov 5 04:45:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:45:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:45:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:45:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:46:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:46:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:46:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:46:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:47:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:47:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:47:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:47:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:48:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:48:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:48:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 467 Nov 5 04:48:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:49:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:49:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:49:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:49:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:50:01 lola-4 CROND[42513]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 04:50:02 lola-4 CROND[42514]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 04:50:02 lola-4 TIME: Time stamp for console Nov 5 04:50:07 lola-4 kernel: LNet: Service thread pid 21628 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 04:50:07 lola-4 kernel: Pid: 21628, comm: ll_ost00_080 Nov 5 04:50:07 lola-4 kernel: Nov 5 04:50:07 lola-4 kernel: Call Trace: Nov 5 04:50:07 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 5 04:50:07 lola-4 kernel: [] ? apic_timer_interrupt+0xe/0x20 Nov 5 04:50:07 lola-4 kernel: [] ? __kmalloc+0xef/0x230 Nov 5 04:50:07 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 04:50:07 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 04:50:07 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 04:50:07 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 04:50:07 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 04:50:07 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 04:50:07 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 04:50:07 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 04:50:07 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 04:50:07 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 04:50:07 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 04:50:07 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 04:50:07 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 04:50:07 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 04:50:07 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 04:50:07 lola-4 kernel: Nov 5 04:50:07 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478346607.21628 Nov 5 04:50:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:50:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:50:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:50:42 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 04:50:42 lola-4 kernel: req@ffff88058f934c50 x1550005401093744/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478346647 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 04:50:42 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 04:50:48 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 04:50:48 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 04:50:48 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 04:50:48 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 04:50:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:51:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:51:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:51:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:51:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:52:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:52:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:52:25 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 04:52:25 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 04:52:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 468 Nov 5 04:52:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:53:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:53:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:53:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:53:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:54:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:54:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:54:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:54:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:55:01 lola-4 CROND[42553]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 04:55:01 lola-4 TIME: Time stamp for console Nov 5 04:55:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:55:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:55:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:55:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:56:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:56:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:56:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:56:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:57:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:57:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:57:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:57:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:58:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:58:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:58:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:58:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:59:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:59:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:59:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 04:59:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 05:00:01 lola-4 CROND[42587]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 05:00:01 lola-4 CROND[42588]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 05:00:01 lola-4 TIME: Time stamp for console Nov 5 05:00:08 lola-4 kernel: LNet: Service thread pid 21735 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 05:00:08 lola-4 kernel: Pid: 21735, comm: ll_ost00_111 Nov 5 05:00:08 lola-4 kernel: Nov 5 05:00:08 lola-4 kernel: Call Trace: Nov 5 05:00:08 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 05:00:08 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 05:00:08 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 05:00:08 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 05:00:08 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 05:00:08 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 05:00:08 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 05:00:08 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 05:00:08 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 05:00:08 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 05:00:08 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 05:00:08 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 05:00:08 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 05:00:08 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 05:00:08 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 05:00:08 lola-4 kernel: Nov 5 05:00:08 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478347208.21735 Nov 5 05:00:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 05:00:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 05:00:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 05:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 05:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 05:00:43 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 05:00:43 lola-4 kernel: req@ffff880761595c50 x1550005401094688/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478347248 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 05:00:43 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 5 05:00:49 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 05:00:49 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 05:00:49 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 05:00:49 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 05:00:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 05:01:01 lola-4 CROND[42601]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 05:01:01 lola-4 run-parts(/etc/cron.hourly)[4260 starting 0anacron Nov 5 05:01:01 lola-4 run-parts(/etc/cron.hourly)[4261 finished 0anacron Nov 5 05:01:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 05:01:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 05:01:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 05:01:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:02:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:02:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:02:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:02:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:03:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:03:15 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 05:03:15 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 05:03:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:03:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:03:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:04:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:04:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:04:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:04:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:05:01 lola-4 CROND[42637]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 05:05:01 lola-4 TIME: Time stamp for console Nov 5 05:05:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:05:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:05:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:05:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:06:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:06:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:06:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 05:06:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 469 Nov 5 05:07:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:07:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:07:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:07:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:08:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:08:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:08:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:08:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:09:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:09:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:09:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:09:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:10:01 lola-4 CROND[42671]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 05:10:01 lola-4 CROND[42672]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 05:10:01 lola-4 TIME: Time stamp for console Nov 5 05:10:09 lola-4 kernel: LNet: Service thread pid 21613 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 05:10:09 lola-4 kernel: Pid: 21613, comm: ll_ost00_065 Nov 5 05:10:09 lola-4 kernel: Nov 5 05:10:09 lola-4 kernel: Call Trace: Nov 5 05:10:09 lola-4 kernel: [] ? select_idle_sibling+0x95/0x150 Nov 5 05:10:09 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 05:10:09 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 05:10:09 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 05:10:09 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 05:10:09 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 05:10:09 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 05:10:09 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 05:10:09 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 05:10:09 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 05:10:09 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 5 05:10:09 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 5 05:10:09 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 05:10:09 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 05:10:09 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 05:10:09 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 05:10:09 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 05:10:09 lola-4 kernel: Nov 5 05:10:09 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478347809.21613 Nov 5 05:10:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:10:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:10:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:10:44 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 05:10:44 lola-4 kernel: req@ffff880649fc0450 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478347849 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 05:10:44 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 05:10:50 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 05:10:50 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 05:10:50 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 05:10:50 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 05:10:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:11:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:11:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 470 Nov 5 05:11:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:11:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:12:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:12:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:12:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:12:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:13:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:13:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:13:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:13:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:14:05 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 05:14:05 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 05:14:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:14:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:14:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:14:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:15:01 lola-4 CROND[42710]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 05:15:01 lola-4 TIME: Time stamp for console Nov 5 05:15:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:15:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:15:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:15:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:16:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:16:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:16:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:16:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:17:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:17:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:17:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:17:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:18:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:18:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:18:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:18:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:19:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:19:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:19:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:19:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:20:01 lola-4 CROND[42744]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 05:20:01 lola-4 CROND[42745]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 05:20:01 lola-4 TIME: Time stamp for console Nov 5 05:20:10 lola-4 kernel: LNet: Service thread pid 21587 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 05:20:10 lola-4 kernel: Pid: 21587, comm: ll_ost00_039 Nov 5 05:20:10 lola-4 kernel: Nov 5 05:20:10 lola-4 kernel: Call Trace: Nov 5 05:20:10 lola-4 kernel: [] ? account_entity_enqueue+0x7e/0x90 Nov 5 05:20:10 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 05:20:10 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 05:20:10 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 05:20:10 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 05:20:10 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 05:20:10 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 05:20:10 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 05:20:10 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 05:20:10 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 05:20:10 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 5 05:20:10 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 5 05:20:10 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 05:20:10 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 05:20:10 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 05:20:10 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 05:20:10 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 05:20:10 lola-4 kernel: Nov 5 05:20:10 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478348410.21587 Nov 5 05:20:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:20:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:20:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:20:45 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 05:20:45 lola-4 kernel: req@ffff8805e9c22450 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478348450 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 05:20:45 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 05:20:51 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 05:20:51 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 05:20:51 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 05:20:51 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 05:20:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 471 Nov 5 05:21:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:21:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:21:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:21:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:22:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:22:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:22:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:22:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:23:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:23:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:23:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:23:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:24:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:24:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:24:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:24:55 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 05:24:55 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 05:24:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:25:01 lola-4 CROND[42784]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 05:25:01 lola-4 TIME: Time stamp for console Nov 5 05:25:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:25:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:25:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:25:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:26:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:26:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:26:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:26:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:27:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:27:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:27:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:27:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:28:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:28:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:28:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:28:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:29:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:29:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:29:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:29:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:30:01 lola-4 CROND[42818]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 05:30:01 lola-4 CROND[42819]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 05:30:01 lola-4 TIME: Time stamp for console Nov 5 05:30:11 lola-4 kernel: LNet: Service thread pid 21618 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 05:30:11 lola-4 kernel: Pid: 21618, comm: ll_ost00_070 Nov 5 05:30:11 lola-4 kernel: Nov 5 05:30:11 lola-4 kernel: Call Trace: Nov 5 05:30:11 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 05:30:11 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 05:30:11 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 05:30:11 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 05:30:11 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 05:30:11 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 05:30:11 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 05:30:11 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 05:30:11 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 05:30:11 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 05:30:11 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 05:30:11 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 05:30:11 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 05:30:11 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 05:30:11 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 05:30:11 lola-4 kernel: Nov 5 05:30:11 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478349011.21618 Nov 5 05:30:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:30:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:30:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:30:46 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 05:30:46 lola-4 kernel: req@ffff88077cb4fc50 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478349051 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 05:30:46 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 05:30:52 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 05:30:52 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 05:30:52 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 05:30:52 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 05:30:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:31:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:31:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:31:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:31:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:32:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:32:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:32:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:32:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:33:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:33:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:33:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:33:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:34:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:34:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:34:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:34:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:35:01 lola-4 CROND[42857]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 05:35:01 lola-4 TIME: Time stamp for console Nov 5 05:35:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:35:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:35:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:35:45 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 05:35:45 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 05:35:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:36:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:36:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:36:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:36:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:37:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:37:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:37:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:37:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:38:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:38:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:38:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:38:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:39:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:39:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:39:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:39:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:40:01 lola-4 CROND[42891]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 05:40:01 lola-4 CROND[42892]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 05:40:01 lola-4 TIME: Time stamp for console Nov 5 05:40:12 lola-4 kernel: LNet: Service thread pid 21479 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 05:40:12 lola-4 kernel: Pid: 21479, comm: ll_ost00_017 Nov 5 05:40:12 lola-4 kernel: Nov 5 05:40:12 lola-4 kernel: Call Trace: Nov 5 05:40:12 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 05:40:12 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 05:40:12 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 05:40:12 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 05:40:12 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 05:40:12 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 05:40:12 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 05:40:12 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 05:40:12 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 05:40:12 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 05:40:12 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 05:40:12 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 05:40:12 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 05:40:12 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 05:40:12 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 05:40:12 lola-4 kernel: Nov 5 05:40:12 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478349612.21479 Nov 5 05:40:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 472 Nov 5 05:40:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:40:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:40:46 lola-4 kernel: Lustre: 21660:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Nov 5 05:40:46 lola-4 kernel: req@ffff8807d3f10c50 x1550085434011504/t0(0) o5->soaked-MDT0001-mdtlov_UUID@192.168.1.109@o2ib10:-1/-1 lens 432/0 e 0 to 0 dl 1478349651 ref 2 fl New:/0/ffffffff rc 0/-1 Nov 5 05:40:46 lola-4 kernel: Lustre: 21660:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 05:40:53 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 05:40:53 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 05:40:53 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 05:40:53 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 05:40:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:41:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:41:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:41:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:41:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:42:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:42:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:42:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:42:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:43:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:43:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:43:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:43:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:44:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:44:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:44:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:44:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:45:01 lola-4 CROND[42930]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 05:45:01 lola-4 TIME: Time stamp for console Nov 5 05:45:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:45:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:45:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:45:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:46:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:46:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:46:35 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 05:46:35 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 05:46:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:46:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:47:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:47:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:47:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:47:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:48:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:48:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:48:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:48:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:49:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:49:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:49:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:49:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:50:01 lola-4 CROND[42964]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 05:50:01 lola-4 CROND[42965]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 05:50:01 lola-4 TIME: Time stamp for console Nov 5 05:50:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 473 Nov 5 05:50:13 lola-4 kernel: LNet: Service thread pid 21734 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 05:50:13 lola-4 kernel: Pid: 21734, comm: ll_ost00_110 Nov 5 05:50:13 lola-4 kernel: Nov 5 05:50:13 lola-4 kernel: Call Trace: Nov 5 05:50:13 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 05:50:13 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 05:50:13 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 05:50:13 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 05:50:13 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 05:50:13 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 05:50:13 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 05:50:13 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 05:50:13 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 05:50:13 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 05:50:13 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 05:50:13 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 05:50:13 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 05:50:13 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 05:50:13 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 05:50:13 lola-4 kernel: Nov 5 05:50:13 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478350213.21734 Nov 5 05:50:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:50:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:50:48 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 05:50:48 lola-4 kernel: req@ffff88060112ac50 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478350253 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 05:50:48 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 5 05:50:54 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 05:50:54 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 05:50:54 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 05:50:54 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 05:50:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:51:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:51:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:51:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:51:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:52:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:52:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:52:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:52:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:53:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:53:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:53:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:53:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:54:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:54:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:54:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:54:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:55:02 lola-4 CROND[43003]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 05:55:02 lola-4 TIME: Time stamp for console Nov 5 05:55:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 05:55:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 05:55:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 05:55:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 05:56:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 05:56:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 05:56:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 05:56:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 05:57:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:57:25 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 05:57:25 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 05:57:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 05:57:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:57:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 05:58:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:58:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 05:58:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:58:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 05:59:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:59:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 05:59:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 05:59:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:00:01 lola-4 CROND[43038]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 06:00:01 lola-4 CROND[43039]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 06:00:01 lola-4 TIME: Time stamp for console Nov 5 06:00:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 474 Nov 5 06:00:14 lola-4 kernel: LNet: Service thread pid 21724 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 06:00:14 lola-4 kernel: Pid: 21724, comm: ll_ost00_107 Nov 5 06:00:14 lola-4 kernel: Nov 5 06:00:14 lola-4 kernel: Call Trace: Nov 5 06:00:14 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 06:00:14 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 06:00:14 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 06:00:14 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 06:00:14 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 06:00:14 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 06:00:14 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 06:00:14 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 06:00:14 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 06:00:14 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 06:00:14 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 06:00:14 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 06:00:14 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 06:00:14 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 06:00:14 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 06:00:14 lola-4 kernel: Nov 5 06:00:14 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478350814.21724 Nov 5 06:00:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 06:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 06:00:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:00:49 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 06:00:49 lola-4 kernel: req@ffff880757ffcc50 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478350854 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 06:00:49 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 06:00:55 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 06:00:55 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 06:00:55 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 06:00:55 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 06:01:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:01:01 lola-4 CROND[43052]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 06:01:01 lola-4 run-parts(/etc/cron.hourly)[4305 starting 0anacron Nov 5 06:01:01 lola-4 run-parts(/etc/cron.hourly)[4306 finished 0anacron Nov 5 06:01:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:01:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:01:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:02:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:02:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:02:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:02:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:03:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:03:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:03:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:03:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:04:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:04:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:04:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:04:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:05:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:05:01 lola-4 CROND[43088]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 06:05:01 lola-4 TIME: Time stamp for console Nov 5 06:05:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:05:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:05:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:06:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:06:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:06:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:06:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:07:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:07:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:07:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:07:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:08:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:08:15 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 06:08:15 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 06:08:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:08:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:08:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:09:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:09:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:09:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:09:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:10:01 lola-4 CROND[43122]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 06:10:01 lola-4 CROND[43123]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 06:10:01 lola-4 TIME: Time stamp for console Nov 5 06:10:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:10:15 lola-4 kernel: LNet: Service thread pid 21749 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 06:10:15 lola-4 kernel: Pid: 21749, comm: ll_ost00_114 Nov 5 06:10:15 lola-4 kernel: Nov 5 06:10:15 lola-4 kernel: Call Trace: Nov 5 06:10:15 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 06:10:15 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 06:10:15 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 06:10:15 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 06:10:15 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 06:10:15 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 06:10:15 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 06:10:15 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 06:10:15 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 06:10:15 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 5 06:10:15 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 5 06:10:15 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 06:10:15 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 06:10:15 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 06:10:15 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 06:10:15 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 06:10:15 lola-4 kernel: Nov 5 06:10:15 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478351415.21749 Nov 5 06:10:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 475 Nov 5 06:10:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:10:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:10:50 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 06:10:50 lola-4 kernel: req@ffff880799582c50 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478351455 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 06:10:50 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 06:10:56 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 06:10:56 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 06:10:56 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 06:10:56 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 06:11:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:11:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:11:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:11:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:12:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:12:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:12:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:12:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:13:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:13:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:13:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:13:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:14:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:14:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:14:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:14:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:15:01 lola-4 CROND[43161]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 06:15:01 lola-4 TIME: Time stamp for console Nov 5 06:15:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:15:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:15:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:15:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:16:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:16:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:16:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:16:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:17:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:17:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:17:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:17:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:18:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:18:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:18:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:18:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:19:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:19:05 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 06:19:05 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 06:19:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:19:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:19:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:20:01 lola-4 CROND[43195]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 06:20:01 lola-4 CROND[43196]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 06:20:01 lola-4 TIME: Time stamp for console Nov 5 06:20:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:20:16 lola-4 kernel: LNet: Service thread pid 21604 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 06:20:16 lola-4 kernel: Pid: 21604, comm: ll_ost00_056 Nov 5 06:20:16 lola-4 kernel: Nov 5 06:20:16 lola-4 kernel: Call Trace: Nov 5 06:20:16 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 06:20:16 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 06:20:16 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 06:20:16 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 06:20:16 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 06:20:16 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 06:20:16 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 06:20:16 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 06:20:16 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 06:20:16 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 06:20:16 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 06:20:16 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 06:20:16 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 06:20:16 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 06:20:16 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 06:20:16 lola-4 kernel: Nov 5 06:20:16 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478352016.21604 Nov 5 06:20:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:20:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 476 Nov 5 06:20:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:20:51 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 06:20:51 lola-4 kernel: req@ffff88053402d450 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478352056 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 06:20:51 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 06:20:57 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 06:20:57 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 06:20:57 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 06:20:57 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 06:21:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:21:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:21:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:21:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:22:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:22:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:22:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:22:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:23:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:23:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:23:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:23:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:24:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:24:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:24:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:24:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:25:01 lola-4 CROND[43234]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 06:25:01 lola-4 TIME: Time stamp for console Nov 5 06:25:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:25:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:25:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:25:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:26:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:26:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:26:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:26:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 477 Nov 5 06:27:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:27:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:27:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:27:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:28:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:28:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:28:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:28:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:29:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:29:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:29:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:29:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:29:55 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 06:29:55 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 06:30:01 lola-4 CROND[43268]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 06:30:01 lola-4 TIME: Time stamp for console Nov 5 06:30:01 lola-4 CROND[43269]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 06:30:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:30:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:30:17 lola-4 kernel: LNet: Service thread pid 21488 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 06:30:17 lola-4 kernel: Pid: 21488, comm: ll_ost00_025 Nov 5 06:30:17 lola-4 kernel: Nov 5 06:30:17 lola-4 kernel: Call Trace: Nov 5 06:30:17 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 06:30:17 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 06:30:17 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 06:30:17 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 06:30:17 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 06:30:17 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 06:30:17 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 06:30:17 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 06:30:17 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 06:30:17 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 5 06:30:17 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 5 06:30:17 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 06:30:17 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 06:30:17 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 06:30:17 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 06:30:17 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 06:30:17 lola-4 kernel: Nov 5 06:30:17 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478352617.21488 Nov 5 06:30:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:30:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:30:52 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 06:30:52 lola-4 kernel: req@ffff88063fb54450 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478352657 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 06:30:52 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 06:30:58 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 06:30:58 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 06:30:58 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 06:30:58 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 06:31:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:31:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:31:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 479 Nov 5 06:31:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:32:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:32:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:32:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:32:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:33:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:33:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:33:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:33:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:34:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:34:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:34:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:34:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:35:01 lola-4 CROND[43308]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 06:35:01 lola-4 TIME: Time stamp for console Nov 5 06:35:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:35:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:35:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:35:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:36:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:36:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:36:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:36:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:37:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:37:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:37:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:37:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:38:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:38:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:38:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:38:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:39:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:39:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 479 Nov 5 06:39:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:39:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 479 Nov 5 06:40:01 lola-4 CROND[43342]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 06:40:01 lola-4 CROND[43343]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 06:40:01 lola-4 TIME: Time stamp for console Nov 5 06:40:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:40:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 478 Nov 5 06:40:18 lola-4 kernel: LNet: Service thread pid 21622 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 06:40:18 lola-4 kernel: Pid: 21622, comm: ll_ost00_074 Nov 5 06:40:18 lola-4 kernel: Nov 5 06:40:18 lola-4 kernel: Call Trace: Nov 5 06:40:18 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 06:40:18 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 06:40:18 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 06:40:18 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 06:40:18 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 06:40:18 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 06:40:18 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 06:40:18 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 06:40:18 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 06:40:18 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 06:40:18 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 06:40:18 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 06:40:18 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 06:40:18 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 06:40:18 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 06:40:19 lola-4 kernel: Nov 5 06:40:19 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478353219.21622 Nov 5 06:40:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 479 Nov 5 06:40:45 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 06:40:45 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 06:40:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 479 Nov 5 06:40:53 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 06:40:53 lola-4 kernel: req@ffff8807b36ec450 x1550005401079776/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478353258 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 06:40:53 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 5 06:40:59 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 06:40:59 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 06:40:59 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 06:40:59 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 06:41:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 479 Nov 5 06:41:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 479 Nov 5 06:41:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 479 Nov 5 06:41:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 479 Nov 5 06:42:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 479 Nov 5 06:42:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 479 Nov 5 06:42:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 479 Nov 5 06:42:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 479 Nov 5 06:43:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 479 Nov 5 06:43:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:43:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:43:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:44:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:44:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:44:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:44:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:45:01 lola-4 CROND[43381]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 06:45:01 lola-4 TIME: Time stamp for console Nov 5 06:45:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:45:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:45:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:45:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:46:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:46:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:46:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:46:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:47:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:47:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:47:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:47:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:48:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:48:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:48:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:48:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:49:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:49:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:49:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:49:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:50:01 lola-4 CROND[43415]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 06:50:01 lola-4 CROND[43416]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 06:50:01 lola-4 TIME: Time stamp for console Nov 5 06:50:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:50:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:50:19 lola-4 kernel: LNet: Service thread pid 21728 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 06:50:19 lola-4 kernel: Pid: 21728, comm: ll_ost00_109 Nov 5 06:50:19 lola-4 kernel: Nov 5 06:50:19 lola-4 kernel: Call Trace: Nov 5 06:50:19 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 06:50:19 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 06:50:19 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 06:50:19 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 06:50:19 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 06:50:19 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 06:50:19 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 06:50:19 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 06:50:19 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 06:50:19 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 06:50:19 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 06:50:19 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 06:50:20 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 06:50:20 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 06:50:20 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 06:50:20 lola-4 kernel: Nov 5 06:50:20 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478353820.21728 Nov 5 06:50:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:50:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:50:54 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 06:50:54 lola-4 kernel: req@ffff88050663f450 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478353859 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 06:50:54 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 06:51:00 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 06:51:00 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 06:51:00 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 06:51:00 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 06:51:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:51:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:51:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:51:35 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 06:51:35 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 06:51:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:52:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:52:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:52:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:52:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:53:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:53:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:53:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:53:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:54:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:54:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:54:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:54:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:55:01 lola-4 CROND[43454]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 06:55:01 lola-4 TIME: Time stamp for console Nov 5 06:55:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:55:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:55:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:55:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:56:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:56:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:56:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:56:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:57:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:57:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:57:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:57:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 480 Nov 5 06:58:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 06:58:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 06:58:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 06:58:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 06:59:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 06:59:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 06:59:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 06:59:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:00:01 lola-4 CROND[43488]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 07:00:01 lola-4 CROND[43489]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 07:00:01 lola-4 TIME: Time stamp for console Nov 5 07:00:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:00:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:00:20 lola-4 kernel: LNet: Service thread pid 20591 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 07:00:20 lola-4 kernel: Pid: 20591, comm: ll_ost00_006 Nov 5 07:00:20 lola-4 kernel: Nov 5 07:00:20 lola-4 kernel: Call Trace: Nov 5 07:00:20 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 07:00:20 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 07:00:20 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 07:00:20 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 07:00:20 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 07:00:20 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 07:00:20 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 07:00:20 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 07:00:20 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 07:00:20 lola-4 kernel: [] ? pick_next_task_fair+0xd0/0x130 Nov 5 07:00:21 lola-4 kernel: [] ? schedule+0x176/0x3a0 Nov 5 07:00:21 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 07:00:21 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 07:00:21 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 07:00:21 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 07:00:21 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 07:00:21 lola-4 kernel: Nov 5 07:00:21 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478354421.20591 Nov 5 07:00:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 07:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 07:00:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:00:55 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 07:00:55 lola-4 kernel: req@ffff880520a4f050 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478354460 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 07:00:55 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 07:01:01 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 07:01:01 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 07:01:01 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 07:01:01 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 07:01:02 lola-4 CROND[43502]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 07:01:02 lola-4 run-parts(/etc/cron.hourly)[4350 starting 0anacron Nov 5 07:01:02 lola-4 run-parts(/etc/cron.hourly)[4351 finished 0anacron Nov 5 07:01:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:01:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:01:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:01:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:02:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:02:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:02:25 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 07:02:25 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 07:02:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:02:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:03:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:03:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:03:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:03:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:04:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:04:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:04:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:04:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:05:01 lola-4 CROND[43539]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 07:05:01 lola-4 TIME: Time stamp for console Nov 5 07:05:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:05:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:05:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:05:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:06:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:06:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:06:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:06:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:07:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:07:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:07:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:07:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:08:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:08:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:08:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:08:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:09:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:09:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:09:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:09:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 481 Nov 5 07:10:01 lola-4 CROND[43573]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 07:10:01 lola-4 CROND[43574]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 07:10:01 lola-4 TIME: Time stamp for console Nov 5 07:10:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:10:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:10:21 lola-4 kernel: LNet: Service thread pid 21741 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 07:10:21 lola-4 kernel: Pid: 21741, comm: ll_ost00_112 Nov 5 07:10:21 lola-4 kernel: Nov 5 07:10:21 lola-4 kernel: Call Trace: Nov 5 07:10:21 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 5 07:10:21 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 07:10:21 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 07:10:21 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 07:10:21 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 07:10:21 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 07:10:22 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 07:10:22 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 07:10:22 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 07:10:22 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 07:10:22 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 07:10:22 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 07:10:22 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 07:10:22 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 07:10:22 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 07:10:22 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 07:10:22 lola-4 kernel: Nov 5 07:10:22 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478355022.21741 Nov 5 07:10:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:10:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:10:56 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 07:10:56 lola-4 kernel: req@ffff880795ff4c50 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478355061 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 07:10:56 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 07:11:02 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 07:11:02 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 07:11:02 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 07:11:02 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 07:11:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:11:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:11:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:11:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:12:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:12:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:12:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:12:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:13:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:13:15 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 07:13:15 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 07:13:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:13:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:13:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:14:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:14:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:14:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:14:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:15:01 lola-4 CROND[43612]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 07:15:01 lola-4 TIME: Time stamp for console Nov 5 07:15:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:15:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:15:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:15:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:16:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:16:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:16:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 482 Nov 5 07:16:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:17:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:17:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:17:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:17:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:18:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:18:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:18:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:18:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:19:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:19:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:19:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:19:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:20:01 lola-4 CROND[43646]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 07:20:01 lola-4 CROND[43647]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 07:20:01 lola-4 TIME: Time stamp for console Nov 5 07:20:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:20:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:20:22 lola-4 kernel: LNet: Service thread pid 21609 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 07:20:22 lola-4 kernel: Pid: 21609, comm: ll_ost00_061 Nov 5 07:20:22 lola-4 kernel: Nov 5 07:20:22 lola-4 kernel: Call Trace: Nov 5 07:20:22 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 07:20:22 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 07:20:22 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 07:20:22 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 07:20:23 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 07:20:23 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 07:20:23 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 07:20:23 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 07:20:23 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 07:20:23 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 07:20:23 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 07:20:23 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 07:20:23 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 07:20:23 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 07:20:23 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 07:20:23 lola-4 kernel: Nov 5 07:20:23 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478355623.21609 Nov 5 07:20:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:20:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:20:57 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 07:20:57 lola-4 kernel: req@ffff8805f0953c50 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478355662 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 07:20:57 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 110 previous similar messages Nov 5 07:21:02 lola-4 kernel: Lustre: soaked-OST0014: Client edf622b2-6dfa-7746-932c-dadfb996df24 (at 192.168.1.121@o2ib100) reconnecting Nov 5 07:21:02 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 07:21:02 lola-4 kernel: Lustre: soaked-OST0014: Connection restored to edf622b2-6dfa-7746-932c-dadfb996df24 (at 192.168.1.121@o2ib100) Nov 5 07:21:02 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 07:21:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:21:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:21:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:21:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:22:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:22:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:22:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:22:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:23:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:23:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:23:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:23:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:24:05 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 07:24:05 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 07:24:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:24:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:24:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:24:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:25:01 lola-4 CROND[43685]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 07:25:01 lola-4 TIME: Time stamp for console Nov 5 07:25:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:25:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:25:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:25:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:26:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:26:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:26:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:26:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:27:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:27:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:27:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:27:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:28:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 483 Nov 5 07:28:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:28:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:28:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:29:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:29:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:29:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:29:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:30:01 lola-4 CROND[43719]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 07:30:01 lola-4 CROND[43720]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 07:30:01 lola-4 TIME: Time stamp for console Nov 5 07:30:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:30:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:30:23 lola-4 kernel: LNet: Service thread pid 21614 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 07:30:23 lola-4 kernel: Pid: 21614, comm: ll_ost00_066 Nov 5 07:30:23 lola-4 kernel: Nov 5 07:30:23 lola-4 kernel: Call Trace: Nov 5 07:30:23 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 5 07:30:23 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 07:30:23 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 07:30:24 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 07:30:24 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 07:30:24 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 07:30:24 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 07:30:24 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 07:30:24 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 07:30:24 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 07:30:24 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 07:30:24 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 07:30:24 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 07:30:24 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 07:30:24 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 07:30:24 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 07:30:24 lola-4 kernel: Nov 5 07:30:24 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478356224.21614 Nov 5 07:30:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:30:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:30:58 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 07:30:58 lola-4 kernel: req@ffff8804d5418050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478356263 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 07:30:58 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 106 previous similar messages Nov 5 07:31:04 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 07:31:04 lola-4 kernel: Lustre: Skipped 24 previous similar messages Nov 5 07:31:04 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 07:31:04 lola-4 kernel: Lustre: Skipped 24 previous similar messages Nov 5 07:31:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:31:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:31:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:31:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:32:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:32:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:32:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:32:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:33:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:33:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:33:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:33:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:34:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:34:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:34:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:34:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:34:55 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 07:34:55 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 07:35:01 lola-4 CROND[43758]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 07:35:01 lola-4 TIME: Time stamp for console Nov 5 07:35:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:35:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:35:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:35:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:36:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:36:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:36:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:36:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:37:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:37:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:37:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:37:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:38:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:38:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:38:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:38:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:39:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:39:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:39:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:39:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:40:01 lola-4 CROND[43793]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 07:40:01 lola-4 CROND[43794]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 07:40:01 lola-4 TIME: Time stamp for console Nov 5 07:40:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:40:08 lola-4 sendmail[10625]: have been rejecting connections on daemon MTA for 15:01:15 Nov 5 07:40:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 484 Nov 5 07:40:24 lola-4 kernel: LNet: Service thread pid 21482 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 07:40:24 lola-4 kernel: Pid: 21482, comm: ll_ost00_020 Nov 5 07:40:24 lola-4 kernel: Nov 5 07:40:24 lola-4 kernel: Call Trace: Nov 5 07:40:24 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 07:40:25 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 07:40:25 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 07:40:25 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 07:40:25 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 07:40:25 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 07:40:25 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 07:40:25 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 07:40:25 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 07:40:25 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 07:40:25 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 07:40:25 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 07:40:25 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 07:40:25 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 07:40:25 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 07:40:25 lola-4 kernel: Nov 5 07:40:25 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478356825.21482 Nov 5 07:40:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:40:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:40:59 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 07:40:59 lola-4 kernel: req@ffff88062a72f050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478356864 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 07:40:59 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 07:41:05 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 07:41:05 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 07:41:05 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 07:41:05 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 07:41:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:41:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:41:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:41:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:42:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:42:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:42:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:42:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:43:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:43:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:43:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:43:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:44:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:44:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:44:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:44:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:45:01 lola-4 CROND[43832]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 07:45:01 lola-4 TIME: Time stamp for console Nov 5 07:45:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:45:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:45:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:45:45 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 07:45:45 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 07:45:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:46:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:46:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:46:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:46:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:47:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:47:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:47:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:47:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:48:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:48:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:48:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:48:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:49:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:49:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:49:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:49:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:50:01 lola-4 CROND[43866]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 07:50:01 lola-4 CROND[43867]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 07:50:01 lola-4 TIME: Time stamp for console Nov 5 07:50:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:50:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 485 Nov 5 07:50:26 lola-4 kernel: LNet: Service thread pid 21693 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 07:50:26 lola-4 kernel: Pid: 21693, comm: ll_ost00_100 Nov 5 07:50:26 lola-4 kernel: Nov 5 07:50:26 lola-4 kernel: Call Trace: Nov 5 07:50:26 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 07:50:26 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 07:50:26 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 07:50:26 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 07:50:26 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 07:50:26 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 07:50:26 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 07:50:26 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 07:50:26 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 07:50:26 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 07:50:26 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 07:50:26 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 07:50:26 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 07:50:26 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 07:50:26 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 07:50:26 lola-4 kernel: Nov 5 07:50:26 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478357426.21693 Nov 5 07:50:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:50:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:51:00 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 07:51:00 lola-4 kernel: req@ffff880629b71c50 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478357465 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 07:51:00 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 07:51:06 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 07:51:06 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 07:51:06 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 07:51:06 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 07:51:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:51:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:51:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:51:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:52:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:52:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:52:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:52:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:53:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:53:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:53:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:53:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:54:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:54:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:54:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:54:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:55:01 lola-4 CROND[43905]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 07:55:01 lola-4 TIME: Time stamp for console Nov 5 07:55:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:55:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:55:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:55:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 07:56:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 07:56:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 07:56:35 lola-4 kernel: Lustre: soaked-OST0002: Export ffff8802e106a800 already connecting from 192.168.1.113@o2ib100 Nov 5 07:56:35 lola-4 kernel: Lustre: Skipped 12 previous similar messages Nov 5 07:56:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 07:56:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 07:57:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 07:57:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 07:57:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 07:57:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 07:58:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 07:58:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 07:58:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 07:58:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 07:59:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 07:59:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 07:59:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 07:59:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:00:01 lola-4 CROND[43939]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 08:00:01 lola-4 CROND[43940]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 08:00:01 lola-4 TIME: Time stamp for console Nov 5 08:00:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 08:00:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 486 Nov 5 08:00:27 lola-4 kernel: LNet: Service thread pid 21185 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 08:00:27 lola-4 kernel: Pid: 21185, comm: ll_ost00_007 Nov 5 08:00:27 lola-4 kernel: Nov 5 08:00:27 lola-4 kernel: Call Trace: Nov 5 08:00:27 lola-4 kernel: [] ? account_entity_enqueue+0x7e/0x90 Nov 5 08:00:27 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 08:00:27 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 08:00:27 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 08:00:27 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 08:00:27 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 08:00:27 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 08:00:27 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 08:00:27 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 08:00:27 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 08:00:27 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 08:00:27 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 08:00:27 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 08:00:27 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 08:00:27 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 08:00:27 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 08:00:27 lola-4 kernel: Nov 5 08:00:27 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478358027.21185 Nov 5 08:00:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 08:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 08:00:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:01:01 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 08:01:01 lola-4 kernel: req@ffff880524073450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478358066 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 08:01:01 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 08:01:01 lola-4 CROND[43953]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 08:01:01 lola-4 run-parts(/etc/cron.hourly)[4395 starting 0anacron Nov 5 08:01:01 lola-4 run-parts(/etc/cron.hourly)[4396 finished 0anacron Nov 5 08:01:07 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 08:01:07 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 08:01:07 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 08:01:07 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 08:01:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:01:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:01:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:01:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:02:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:02:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:02:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:02:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:03:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:03:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:03:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:03:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:04:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:04:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:04:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:04:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:05:01 lola-4 CROND[43989]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 08:05:01 lola-4 TIME: Time stamp for console Nov 5 08:05:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:05:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:05:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:05:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:06:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:06:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:06:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:06:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:07:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:07:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:07:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:07:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:08:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:08:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:08:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:08:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:09:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:09:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:09:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:09:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:10:01 lola-4 CROND[44023]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 08:10:01 lola-4 CROND[44024]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 08:10:01 lola-4 TIME: Time stamp for console Nov 5 08:10:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:10:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:10:28 lola-4 kernel: LNet: Service thread pid 21687 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 08:10:28 lola-4 kernel: Pid: 21687, comm: ll_ost00_096 Nov 5 08:10:28 lola-4 kernel: Nov 5 08:10:28 lola-4 kernel: Call Trace: Nov 5 08:10:28 lola-4 kernel: [] ? account_entity_enqueue+0x7e/0x90 Nov 5 08:10:28 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 08:10:28 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 08:10:28 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 08:10:28 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 08:10:28 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 08:10:28 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 08:10:28 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 08:10:28 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 08:10:28 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 08:10:28 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 08:10:28 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 08:10:28 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 08:10:28 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 08:10:28 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 08:10:28 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 08:10:28 lola-4 kernel: Nov 5 08:10:28 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478358628.21687 Nov 5 08:10:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:10:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:11:02 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 08:11:02 lola-4 kernel: req@ffff8806acaf9c50 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478358667 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 08:11:02 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 108 previous similar messages Nov 5 08:11:08 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 08:11:08 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 08:11:08 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 08:11:08 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 08:11:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:11:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:11:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:11:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:12:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:12:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:12:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:12:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:13:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:13:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:13:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:13:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:14:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:14:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:14:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:14:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:15:01 lola-4 CROND[44063]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 08:15:01 lola-4 TIME: Time stamp for console Nov 5 08:15:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:15:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:15:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:15:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:16:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:16:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:16:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:16:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:17:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:17:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:17:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:17:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:18:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:18:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:18:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:18:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:19:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:19:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:19:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:19:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:20:01 lola-4 CROND[44097]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 08:20:01 lola-4 CROND[44098]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 08:20:01 lola-4 TIME: Time stamp for console Nov 5 08:20:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:20:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:20:29 lola-4 kernel: LNet: Service thread pid 21723 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 5 08:20:29 lola-4 kernel: Pid: 21723, comm: ll_ost00_106 Nov 5 08:20:29 lola-4 kernel: Nov 5 08:20:29 lola-4 kernel: Call Trace: Nov 5 08:20:29 lola-4 kernel: [] ? transfer_objects+0x5c/0x80 Nov 5 08:20:29 lola-4 kernel: [] __mutex_lock_slowpath+0x96/0x210 Nov 5 08:20:29 lola-4 kernel: [] ? ofd_seq_load+0xbb/0x9c0 [ofd] Nov 5 08:20:29 lola-4 kernel: [] mutex_lock+0x2b/0x50 Nov 5 08:20:29 lola-4 kernel: [] ofd_create_hdl+0xba8/0x2530 [ofd] Nov 5 08:20:29 lola-4 kernel: [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] Nov 5 08:20:29 lola-4 kernel: [] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] Nov 5 08:20:29 lola-4 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Nov 5 08:20:29 lola-4 kernel: [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] Nov 5 08:20:29 lola-4 kernel: [] ptlrpc_main+0xd31/0x1800 [ptlrpc] Nov 5 08:20:29 lola-4 kernel: [] ? thread_return+0x4e/0x7d0 Nov 5 08:20:29 lola-4 kernel: [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] Nov 5 08:20:29 lola-4 kernel: [] kthread+0x9e/0xc0 Nov 5 08:20:29 lola-4 kernel: [] child_rip+0xa/0x20 Nov 5 08:20:29 lola-4 kernel: [] ? kthread+0x0/0xc0 Nov 5 08:20:29 lola-4 kernel: [] ? child_rip+0x0/0x20 Nov 5 08:20:29 lola-4 kernel: Nov 5 08:20:29 lola-4 kernel: LustreError: dumping log to /tmp/lustre-log.1478359229.21723 Nov 5 08:20:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:20:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:21:03 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 08:21:03 lola-4 kernel: req@ffff8805e7927c50 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478359268 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 08:21:03 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 108 previous similar messages Nov 5 08:21:09 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 08:21:09 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 08:21:09 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 08:21:09 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 08:21:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:21:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:21:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:21:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:22:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:22:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:22:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:22:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:23:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:23:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:23:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:23:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:24:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:24:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:24:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:24:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:25:01 lola-4 CROND[44136]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 08:25:01 lola-4 TIME: Time stamp for console Nov 5 08:25:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:25:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:25:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:25:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:26:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:26:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:26:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:26:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:27:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:27:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:27:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:27:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:28:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:28:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:28:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:28:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:29:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:29:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:29:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:29:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:30:01 lola-4 CROND[44170]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 08:30:01 lola-4 CROND[44171]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 08:30:01 lola-4 TIME: Time stamp for console Nov 5 08:30:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:30:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:30:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:30:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:31:04 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 08:31:04 lola-4 kernel: req@ffff8806a693cc50 x1550005401093744/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478359869 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 08:31:04 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 08:31:10 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 08:31:10 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 08:31:10 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 08:31:10 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 08:31:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:31:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:31:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:31:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:32:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:32:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:32:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:32:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:33:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:33:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:33:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:33:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:34:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:34:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:34:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:34:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:35:01 lola-4 CROND[44207]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 08:35:01 lola-4 TIME: Time stamp for console Nov 5 08:35:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:35:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:35:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:35:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:36:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:36:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:36:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:36:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:37:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:37:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:37:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:37:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:38:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:38:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:38:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:38:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:39:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:39:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:39:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:39:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:40:01 lola-4 CROND[44241]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 08:40:01 lola-4 CROND[44242]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 08:40:01 lola-4 TIME: Time stamp for console Nov 5 08:40:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:40:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:40:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:40:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:41:05 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 08:41:05 lola-4 kernel: req@ffff880558de1c50 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478360470 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 08:41:05 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 08:41:11 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 08:41:11 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 08:41:11 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 08:41:11 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 08:41:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:41:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:41:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:41:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:42:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:42:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:42:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:42:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:43:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:43:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:43:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:43:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:44:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:44:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:44:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:44:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:45:01 lola-4 CROND[44279]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 08:45:01 lola-4 TIME: Time stamp for console Nov 5 08:45:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:45:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:45:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:45:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:46:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:46:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:46:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:46:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:47:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:47:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:47:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:47:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:48:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:48:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:48:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:48:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:49:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:49:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:49:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:49:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:50:01 lola-4 CROND[44313]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 08:50:01 lola-4 CROND[44314]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 08:50:01 lola-4 TIME: Time stamp for console Nov 5 08:50:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:50:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:50:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:50:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:51:06 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 08:51:06 lola-4 kernel: req@ffff880501340c50 x1550005401079776/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478361071 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 08:51:06 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 08:51:12 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 08:51:12 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 08:51:12 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 08:51:12 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 08:51:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:51:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:51:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:51:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:52:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:52:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:52:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:52:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:53:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:53:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:53:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:53:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:54:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:54:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:54:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:54:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:55:01 lola-4 CROND[44350]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 08:55:01 lola-4 TIME: Time stamp for console Nov 5 08:55:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:55:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:55:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:55:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:56:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:56:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:56:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:56:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:57:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:57:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:57:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:57:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:58:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:58:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:58:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:58:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 08:59:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:59:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:59:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 08:59:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:00:01 lola-4 CROND[44384]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 09:00:01 lola-4 CROND[44385]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 09:00:01 lola-4 TIME: Time stamp for console Nov 5 09:00:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:00:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 09:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 09:00:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:01:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:01:01 lola-4 CROND[44396]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 09:01:01 lola-4 run-parts(/etc/cron.hourly)[4439 starting 0anacron Nov 5 09:01:01 lola-4 run-parts(/etc/cron.hourly)[4440 finished 0anacron Nov 5 09:01:07 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 09:01:07 lola-4 kernel: req@ffff8805875b2450 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478361672 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 09:01:07 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 5 09:01:13 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 09:01:13 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 09:01:13 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 09:01:13 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 09:01:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:01:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:01:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:02:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:02:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:02:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:02:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:03:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:03:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:03:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:03:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:04:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:04:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:04:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:04:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:05:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:05:01 lola-4 CROND[44432]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 09:05:01 lola-4 TIME: Time stamp for console Nov 5 09:05:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:05:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:05:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:06:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:06:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:06:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:06:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:07:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:07:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:07:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:07:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:08:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:08:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:08:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:08:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:09:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:09:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:09:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:09:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:10:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:10:01 lola-4 CROND[44466]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 09:10:01 lola-4 CROND[44467]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 09:10:01 lola-4 TIME: Time stamp for console Nov 5 09:10:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:10:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:10:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:11:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:11:08 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 09:11:08 lola-4 kernel: req@ffff8806e8f7dc50 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478362273 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 09:11:08 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 09:11:14 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 09:11:14 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 09:11:14 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 09:11:14 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 09:11:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:11:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:11:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:12:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:12:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:12:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:12:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:13:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:13:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:13:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:13:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:14:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:14:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:14:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:14:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:15:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:15:01 lola-4 CROND[44503]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 09:15:01 lola-4 TIME: Time stamp for console Nov 5 09:15:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:15:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:15:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:16:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:16:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:16:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:16:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:17:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:17:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:17:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:17:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:18:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:18:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:18:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:18:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:19:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:19:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:19:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:19:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:20:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:20:01 lola-4 CROND[44538]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 09:20:01 lola-4 CROND[44539]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 09:20:01 lola-4 TIME: Time stamp for console Nov 5 09:20:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:20:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:20:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:21:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:21:09 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 09:21:09 lola-4 kernel: req@ffff8806cbcb1c50 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478362874 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 09:21:09 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 09:21:15 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 09:21:15 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 09:21:15 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 09:21:15 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 09:21:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:21:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:21:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:22:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:22:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:22:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:22:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:23:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:23:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:23:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:23:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:24:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:24:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:24:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:24:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:25:01 lola-4 CROND[44575]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 09:25:01 lola-4 TIME: Time stamp for console Nov 5 09:25:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:25:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:25:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:25:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:26:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:26:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:26:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:26:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:27:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:27:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:27:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:27:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:28:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:28:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:28:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:28:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:29:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:29:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:29:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:29:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:30:01 lola-4 CROND[44609]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 09:30:01 lola-4 CROND[44610]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 09:30:01 lola-4 TIME: Time stamp for console Nov 5 09:30:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:30:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:30:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:30:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:31:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:31:10 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 09:31:10 lola-4 kernel: req@ffff880520478050 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478363475 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 09:31:10 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 09:31:16 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 09:31:16 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 09:31:16 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 09:31:16 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 09:31:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:31:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:31:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:32:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:32:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:32:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:32:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:33:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:33:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:33:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:33:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:34:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:34:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:34:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:34:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:35:01 lola-4 CROND[44646]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 09:35:01 lola-4 TIME: Time stamp for console Nov 5 09:35:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:35:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:35:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:35:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:36:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:36:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:36:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:36:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:37:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:37:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:37:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:37:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:38:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:38:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:38:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:38:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:39:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:39:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:39:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:39:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:40:01 lola-4 CROND[44680]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 09:40:01 lola-4 CROND[44681]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 09:40:01 lola-4 TIME: Time stamp for console Nov 5 09:40:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:40:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:40:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:40:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:41:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:41:11 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 09:41:11 lola-4 kernel: req@ffff88052d7c4850 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478364076 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 09:41:11 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 09:41:17 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 09:41:17 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 09:41:17 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 09:41:17 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 09:41:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:41:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:41:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:42:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:42:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:42:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:42:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:43:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:43:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:43:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:43:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:44:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:44:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:44:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:44:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:45:01 lola-4 CROND[44717]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 09:45:01 lola-4 TIME: Time stamp for console Nov 5 09:45:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:45:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:45:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:45:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:46:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:46:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:46:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:46:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:47:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:47:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:47:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:47:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:48:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:48:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:48:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:48:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:49:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:49:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:49:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:49:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:50:01 lola-4 CROND[44751]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 09:50:01 lola-4 CROND[44752]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 09:50:01 lola-4 TIME: Time stamp for console Nov 5 09:50:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:50:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:50:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:50:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:51:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:51:12 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 09:51:12 lola-4 kernel: req@ffff8806ae344450 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478364677 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 09:51:12 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 5 09:51:18 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 09:51:18 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 09:51:18 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 09:51:18 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 09:51:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:51:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:51:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:52:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:52:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:52:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:52:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:53:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:53:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:53:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:53:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:54:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:54:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:54:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:54:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:55:01 lola-4 CROND[44789]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 09:55:01 lola-4 TIME: Time stamp for console Nov 5 09:55:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:55:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:55:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:55:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:56:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:56:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:56:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:56:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:57:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:57:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:57:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:57:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 09:58:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:58:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:58:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:58:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:59:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:59:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:59:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 09:59:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:00:01 lola-4 CROND[44823]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 10:00:01 lola-4 CROND[44824]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 10:00:01 lola-4 TIME: Time stamp for console Nov 5 10:00:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:00:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:00:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 10:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 10:00:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:01:01 lola-4 CROND[44835]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 10:01:01 lola-4 run-parts(/etc/cron.hourly)[4483 starting 0anacron Nov 5 10:01:01 lola-4 run-parts(/etc/cron.hourly)[4484 finished 0anacron Nov 5 10:01:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:01:13 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 10:01:13 lola-4 kernel: req@ffff880656f43c50 x1550005401079776/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478365278 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 10:01:13 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 10:01:19 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 10:01:19 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 10:01:19 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 10:01:19 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 10:01:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:01:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:01:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:02:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:02:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:02:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:02:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:03:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:03:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:03:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:03:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:04:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:04:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:04:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:04:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:05:01 lola-4 CROND[44871]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 10:05:01 lola-4 TIME: Time stamp for console Nov 5 10:05:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:05:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:05:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:05:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:06:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:06:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:06:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:06:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:07:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:07:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:07:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:07:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:08:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:08:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:08:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:08:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:09:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:09:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:09:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:09:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:10:01 lola-4 CROND[44905]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 10:10:01 lola-4 CROND[44906]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 10:10:01 lola-4 TIME: Time stamp for console Nov 5 10:10:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:10:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:10:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:10:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:11:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:11:14 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 10:11:14 lola-4 kernel: req@ffff88078ceba050 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478365879 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 10:11:14 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 10:11:20 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 10:11:20 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 10:11:20 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 10:11:20 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 10:11:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:11:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:11:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:12:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:12:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:12:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:12:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:13:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:13:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:13:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:13:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:14:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:14:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:14:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:14:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:15:01 lola-4 CROND[44942]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 10:15:01 lola-4 TIME: Time stamp for console Nov 5 10:15:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:15:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:15:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:15:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:16:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:16:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:16:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:16:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:17:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:17:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:17:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:17:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:18:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:18:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:18:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:18:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:19:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:19:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:19:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:19:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:20:01 lola-4 CROND[44976]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 10:20:01 lola-4 CROND[44977]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 10:20:01 lola-4 TIME: Time stamp for console Nov 5 10:20:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:20:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:20:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:20:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:21:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:21:15 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 10:21:15 lola-4 kernel: req@ffff88059322d050 x1550005401096032/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478366480 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 10:21:15 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 10:21:21 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 10:21:21 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 10:21:21 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 10:21:21 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 10:21:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 489 Nov 5 10:21:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:21:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:22:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:22:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:22:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:22:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:23:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:23:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:23:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:23:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:24:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:24:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:24:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:24:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:25:01 lola-4 CROND[45014]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 10:25:01 lola-4 TIME: Time stamp for console Nov 5 10:25:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:25:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:25:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:25:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:26:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:26:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:26:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:26:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:27:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:27:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:27:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:27:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:28:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:28:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:28:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:28:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:29:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:29:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:29:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:29:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:30:01 lola-4 CROND[45048]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 10:30:01 lola-4 CROND[45049]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 10:30:01 lola-4 TIME: Time stamp for console Nov 5 10:30:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:30:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:30:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:30:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:31:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:31:16 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 10:31:16 lola-4 kernel: req@ffff88071f147850 x1550005401094688/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478367081 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 10:31:16 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 10:31:22 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 10:31:22 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 10:31:22 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 10:31:22 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 10:31:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:31:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:31:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:32:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:32:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:32:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:32:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:33:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:33:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:33:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:33:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:34:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:34:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:34:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:34:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:35:01 lola-4 CROND[45085]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 10:35:01 lola-4 TIME: Time stamp for console Nov 5 10:35:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:35:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:35:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:35:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:36:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:36:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:36:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:36:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:37:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:37:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:37:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:37:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:38:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:38:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:38:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:38:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:39:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:39:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:39:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:39:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:40:01 lola-4 CROND[45119]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 10:40:01 lola-4 CROND[45120]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 10:40:01 lola-4 TIME: Time stamp for console Nov 5 10:40:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:40:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:40:23 lola-4 sendmail[10625]: have been rejecting connections on daemon MTA for 18:01:30 Nov 5 10:40:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:40:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:41:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:41:17 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 10:41:17 lola-4 kernel: req@ffff8806882b6850 x1550005401079776/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478367682 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 10:41:17 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 5 10:41:23 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 10:41:23 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 10:41:23 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 10:41:23 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 10:41:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:41:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:41:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:42:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:42:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:42:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:42:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:43:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 10:43:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:43:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:43:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:44:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:44:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:44:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:44:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:45:01 lola-4 CROND[45156]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 10:45:01 lola-4 TIME: Time stamp for console Nov 5 10:45:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:45:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:45:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:45:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:46:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:46:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:46:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:46:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:47:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:47:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:47:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:47:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:48:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:48:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:48:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:48:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:49:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:49:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:49:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:49:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:50:01 lola-4 CROND[45190]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 10:50:01 lola-4 CROND[45191]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 10:50:01 lola-4 TIME: Time stamp for console Nov 5 10:50:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:50:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:50:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:50:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:51:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:51:18 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 10:51:18 lola-4 kernel: req@ffff8805404c5450 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478368283 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 10:51:18 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 10:51:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:51:24 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 10:51:24 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 10:51:24 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 10:51:24 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 10:51:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:51:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:52:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:52:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:52:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:52:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:53:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:53:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:53:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:53:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:54:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:54:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:54:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:54:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:55:01 lola-4 CROND[45227]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 10:55:01 lola-4 TIME: Time stamp for console Nov 5 10:55:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:55:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:55:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:55:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:56:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:56:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:56:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:56:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:57:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:57:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:57:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:57:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:58:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:58:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:58:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:58:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:59:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:59:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:59:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 10:59:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:00:01 lola-4 CROND[45262]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 11:00:01 lola-4 CROND[45263]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 11:00:01 lola-4 TIME: Time stamp for console Nov 5 11:00:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:00:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:00:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 11:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 11:00:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:01:01 lola-4 CROND[45274]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 11:01:01 lola-4 run-parts(/etc/cron.hourly)[4527 starting 0anacron Nov 5 11:01:01 lola-4 run-parts(/etc/cron.hourly)[4528 finished 0anacron Nov 5 11:01:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:01:19 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 11:01:19 lola-4 kernel: req@ffff88079cc4b850 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478368884 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 11:01:19 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 11:01:25 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 11:01:25 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 11:01:25 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 11:01:25 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 11:01:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:01:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:01:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:02:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:02:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:02:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:02:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:03:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:03:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:03:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:03:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:04:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:04:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:04:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:04:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:05:01 lola-4 CROND[45310]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 11:05:01 lola-4 TIME: Time stamp for console Nov 5 11:05:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:05:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:05:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:05:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:06:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:06:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:06:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:06:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:07:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:07:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:07:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:07:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:08:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:08:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:08:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:08:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:09:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:09:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:09:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:09:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:10:01 lola-4 CROND[45344]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 11:10:01 lola-4 CROND[45345]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 11:10:01 lola-4 TIME: Time stamp for console Nov 5 11:10:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:10:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:10:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:10:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:11:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:11:20 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 11:11:20 lola-4 kernel: req@ffff8805239b9850 x1550005401097056/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478369485 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 11:11:20 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 11:11:26 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 11:11:26 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 11:11:26 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 11:11:26 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 11:11:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:11:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:11:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:12:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:12:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:12:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:12:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:13:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:13:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:13:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:13:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:14:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:14:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:14:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:14:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:15:01 lola-4 CROND[45381]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 11:15:01 lola-4 TIME: Time stamp for console Nov 5 11:15:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:15:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:15:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:15:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:16:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:16:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:16:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:16:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:17:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:17:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:17:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:17:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:18:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:18:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:18:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:18:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:19:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:19:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:19:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:19:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:20:01 lola-4 CROND[45415]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 11:20:01 lola-4 CROND[45416]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 11:20:01 lola-4 TIME: Time stamp for console Nov 5 11:20:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:20:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:20:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:20:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:21:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:21:21 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 11:21:21 lola-4 kernel: req@ffff8806b05a4850 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478370086 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 11:21:21 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 11:21:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:21:27 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 11:21:27 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 11:21:27 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 11:21:27 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 11:21:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:21:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:22:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:22:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:22:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:22:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:23:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:23:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:23:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:23:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:24:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:24:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:24:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:24:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:25:01 lola-4 CROND[45452]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 11:25:01 lola-4 TIME: Time stamp for console Nov 5 11:25:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:25:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:25:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:25:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:26:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:26:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:26:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:26:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:27:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:27:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:27:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:27:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:28:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:28:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:28:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:28:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:29:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:29:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:29:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:29:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:30:01 lola-4 CROND[45486]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 11:30:01 lola-4 CROND[45487]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 11:30:01 lola-4 TIME: Time stamp for console Nov 5 11:30:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:30:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:30:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:30:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:31:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:31:22 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 11:31:22 lola-4 kernel: req@ffff8804db564850 x1550005401097056/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478370687 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 11:31:22 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 5 11:31:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:31:28 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 11:31:28 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 11:31:28 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 11:31:28 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 11:31:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:31:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:32:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:32:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:32:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:32:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:33:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:33:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:33:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:33:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:34:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:34:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:34:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:34:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:35:01 lola-4 CROND[45524]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 11:35:01 lola-4 TIME: Time stamp for console Nov 5 11:35:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:35:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:35:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:35:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:36:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:36:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:36:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:36:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:37:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:37:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:37:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:37:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:38:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:38:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:38:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:38:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:39:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:39:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:39:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:39:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:40:01 lola-4 CROND[45558]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 11:40:01 lola-4 CROND[45559]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 11:40:01 lola-4 TIME: Time stamp for console Nov 5 11:40:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:40:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:40:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:40:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:41:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:41:23 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 11:41:23 lola-4 kernel: req@ffff8804eac27c50 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478371288 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 11:41:23 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 11:41:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:41:29 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 11:41:29 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 11:41:29 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 11:41:29 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 11:41:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:41:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:42:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:42:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:42:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:42:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:43:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:43:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:43:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 11:43:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:44:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:44:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:44:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:44:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:45:01 lola-4 CROND[45595]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 11:45:01 lola-4 TIME: Time stamp for console Nov 5 11:45:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:45:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:45:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:45:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:46:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:46:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:46:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:46:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:47:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:47:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:47:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:47:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:48:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:48:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:48:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:48:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:49:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:49:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:49:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:49:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:50:01 lola-4 CROND[45629]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 11:50:01 lola-4 CROND[45630]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 11:50:01 lola-4 TIME: Time stamp for console Nov 5 11:50:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:50:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:50:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:50:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:51:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:51:24 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 11:51:24 lola-4 kernel: req@ffff88051c3e6c50 x1550005401097056/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478371889 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 11:51:24 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 11:51:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:51:30 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 11:51:30 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 11:51:30 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 11:51:30 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 11:51:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:51:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:52:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:52:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:52:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:52:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:53:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:53:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:53:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:53:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:54:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:54:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:54:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:54:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:55:01 lola-4 CROND[45666]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 11:55:01 lola-4 TIME: Time stamp for console Nov 5 11:55:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:55:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:55:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:55:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:56:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:56:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:56:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:56:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:57:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:57:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:57:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:57:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:58:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:58:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:58:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:58:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:59:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:59:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:59:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 11:59:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:00:01 lola-4 CROND[45700]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 12:00:01 lola-4 CROND[45701]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 12:00:01 lola-4 TIME: Time stamp for console Nov 5 12:00:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:00:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 12:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 12:00:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:01:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:01:01 lola-4 CROND[45802]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 12:01:01 lola-4 run-parts(/etc/cron.hourly)[4580 starting 0anacron Nov 5 12:01:01 lola-4 run-parts(/etc/cron.hourly)[4581 finished 0anacron Nov 5 12:01:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:01:25 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 12:01:25 lola-4 kernel: req@ffff880526c31450 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478372490 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 12:01:25 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 12:01:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:01:31 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 12:01:31 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 12:01:31 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 12:01:31 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 12:01:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:02:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:02:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:02:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:02:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:03:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:03:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:03:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:03:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:04:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:04:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:04:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:04:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:05:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:05:01 lola-4 CROND[45839]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 12:05:01 lola-4 TIME: Time stamp for console Nov 5 12:05:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:05:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:05:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:06:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:06:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:06:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:06:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:07:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:07:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:07:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:07:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:08:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:08:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:08:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:08:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:09:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:09:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:09:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:09:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:10:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:10:01 lola-4 CROND[45873]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 12:10:01 lola-4 TIME: Time stamp for console Nov 5 12:10:01 lola-4 CROND[45874]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 12:10:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:10:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:10:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:11:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:11:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:11:26 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 12:11:26 lola-4 kernel: req@ffff88051c3e6050 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478373091 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 12:11:26 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 12:11:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:11:32 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 12:11:32 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 12:11:32 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 12:11:32 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 12:11:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:12:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:12:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:12:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:12:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:13:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:13:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:13:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:13:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:14:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:14:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:14:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:14:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:15:01 lola-4 CROND[45910]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 12:15:01 lola-4 TIME: Time stamp for console Nov 5 12:15:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:15:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:15:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:15:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:16:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:16:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:16:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:16:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:17:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:17:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:17:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:17:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:18:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:18:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:18:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:18:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:19:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:19:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:19:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:19:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:20:01 lola-4 CROND[45944]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 12:20:01 lola-4 CROND[45945]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 12:20:01 lola-4 TIME: Time stamp for console Nov 5 12:20:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:20:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:20:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:20:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:21:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:21:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:21:27 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 12:21:27 lola-4 kernel: req@ffff8806a185b450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478373692 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 12:21:27 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 104 previous similar messages Nov 5 12:21:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:21:33 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 12:21:33 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 12:21:33 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 12:21:33 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 12:21:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:22:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:22:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:22:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:22:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:23:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:23:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:23:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:23:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:24:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:24:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:24:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:24:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:25:01 lola-4 CROND[45981]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 12:25:01 lola-4 TIME: Time stamp for console Nov 5 12:25:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:25:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:25:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:25:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:26:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:26:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:26:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:26:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:27:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:27:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:27:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:27:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:28:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:28:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:28:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:28:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:29:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:29:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:29:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:29:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:30:01 lola-4 CROND[46015]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 12:30:01 lola-4 CROND[46016]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 12:30:01 lola-4 TIME: Time stamp for console Nov 5 12:30:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:30:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:30:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:30:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:31:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:31:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:31:28 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 12:31:28 lola-4 kernel: req@ffff880812826450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478374293 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 12:31:28 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 12:31:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:31:34 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 12:31:34 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 12:31:34 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 12:31:34 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 12:31:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:32:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:32:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:32:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:32:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:33:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:33:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:33:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:33:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:34:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:34:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:34:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:34:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:35:01 lola-4 CROND[46052]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 12:35:01 lola-4 TIME: Time stamp for console Nov 5 12:35:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:35:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:35:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:35:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:36:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:36:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:36:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:36:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:37:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:37:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:37:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:37:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:38:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:38:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:38:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:38:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:39:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:39:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:39:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:39:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:40:01 lola-4 CROND[46087]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 12:40:01 lola-4 CROND[46088]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 12:40:01 lola-4 TIME: Time stamp for console Nov 5 12:40:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:40:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:40:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:40:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:41:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:41:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:41:29 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 12:41:29 lola-4 kernel: req@ffff880523b1b450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478374894 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 12:41:29 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 12:41:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:41:35 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 12:41:35 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 12:41:35 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 12:41:35 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 12:41:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:42:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:42:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:42:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:42:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:43:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:43:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:43:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:43:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:44:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:44:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:44:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:44:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:45:01 lola-4 CROND[46124]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 12:45:01 lola-4 TIME: Time stamp for console Nov 5 12:45:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:45:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:45:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:45:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:46:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:46:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:46:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:46:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:47:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:47:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:47:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:47:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:48:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:48:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:48:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:48:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:49:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:49:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:49:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:49:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:50:02 lola-4 CROND[46158]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 12:50:02 lola-4 CROND[46159]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 12:50:02 lola-4 TIME: Time stamp for console Nov 5 12:50:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:50:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:50:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:50:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:51:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:51:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:51:30 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 12:51:30 lola-4 kernel: req@ffff880655eb3450 x1550005401093744/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478375495 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 12:51:30 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 12:51:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:51:36 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 12:51:36 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 12:51:36 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 12:51:36 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 12:51:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:52:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:52:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:52:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:52:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:53:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:53:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:53:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:53:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:54:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:54:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:54:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:54:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:55:01 lola-4 CROND[46195]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 12:55:01 lola-4 TIME: Time stamp for console Nov 5 12:55:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:55:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:55:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:55:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:56:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:56:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:56:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:56:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:57:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:57:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:57:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 12:57:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:58:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:58:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:58:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:58:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:59:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:59:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:59:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 12:59:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:00:01 lola-4 CROND[46229]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 13:00:01 lola-4 CROND[46230]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 13:00:01 lola-4 TIME: Time stamp for console Nov 5 13:00:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:00:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:00:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 13:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 13:00:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:01:01 lola-4 CROND[46241]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 13:01:01 lola-4 run-parts(/etc/cron.hourly)[4624 starting 0anacron Nov 5 13:01:01 lola-4 run-parts(/etc/cron.hourly)[4625 finished 0anacron Nov 5 13:01:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:01:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:01:31 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 13:01:31 lola-4 kernel: req@ffff8807d5824450 x1550005401094688/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478376096 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 13:01:31 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 13:01:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:01:37 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 13:01:37 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 13:01:37 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 13:01:37 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 13:01:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:02:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:02:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:02:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:02:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:03:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:03:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:03:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:03:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:04:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:04:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:04:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:04:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:05:01 lola-4 CROND[46277]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 13:05:01 lola-4 TIME: Time stamp for console Nov 5 13:05:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:05:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:05:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:05:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:06:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:06:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:06:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:06:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:07:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:07:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:07:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:07:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:08:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:08:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:08:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:08:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:09:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:09:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:09:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:09:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:10:01 lola-4 CROND[46311]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 13:10:01 lola-4 CROND[46312]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 13:10:01 lola-4 TIME: Time stamp for console Nov 5 13:10:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:10:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:10:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:10:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:11:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:11:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:11:32 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 13:11:32 lola-4 kernel: req@ffff880519798450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478376697 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 13:11:32 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 5 13:11:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:11:38 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 13:11:38 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 13:11:38 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 13:11:38 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 13:11:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:12:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:12:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:12:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:12:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:13:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:13:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:13:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:13:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:14:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:14:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:14:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:14:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:15:01 lola-4 CROND[46349]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 13:15:01 lola-4 TIME: Time stamp for console Nov 5 13:15:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:15:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:15:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:15:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:16:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:16:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:16:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:16:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:17:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:17:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:17:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:17:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:18:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:18:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:18:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:18:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:19:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:19:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:19:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:19:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:20:01 lola-4 CROND[46383]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 13:20:01 lola-4 CROND[46384]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 13:20:01 lola-4 TIME: Time stamp for console Nov 5 13:20:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:20:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:20:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:20:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:21:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:21:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:21:33 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 13:21:33 lola-4 kernel: req@ffff88068d530850 x1550005401097056/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478377298 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 13:21:33 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 13:21:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:21:39 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 13:21:39 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 13:21:39 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 13:21:39 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 13:21:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:22:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:22:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:22:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:22:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:23:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:23:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:23:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:23:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:24:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:24:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:24:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:24:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:25:01 lola-4 CROND[46420]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 13:25:01 lola-4 TIME: Time stamp for console Nov 5 13:25:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:25:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:25:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:25:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:26:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:26:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:26:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:26:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:27:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:27:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:27:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:27:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:28:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:28:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:28:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:28:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:29:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:29:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:29:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:29:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:30:01 lola-4 CROND[46454]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 13:30:01 lola-4 CROND[46455]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 13:30:01 lola-4 TIME: Time stamp for console Nov 5 13:30:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:30:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:30:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:30:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:31:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:31:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:31:34 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 13:31:34 lola-4 kernel: req@ffff880837431450 x1550005401098800/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478377899 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 13:31:34 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 13:31:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:31:40 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 13:31:40 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 13:31:40 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 13:31:40 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 13:31:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:32:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:32:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:32:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:32:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:33:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:33:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:33:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:33:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:34:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:34:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:34:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:34:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:35:01 lola-4 CROND[46491]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 13:35:01 lola-4 TIME: Time stamp for console Nov 5 13:35:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:35:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:35:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:35:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:36:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:36:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:36:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:36:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:37:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:37:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:37:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:37:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:38:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:38:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:38:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:38:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:39:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:39:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:39:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:39:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:40:01 lola-4 CROND[46525]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 13:40:01 lola-4 CROND[46526]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 13:40:01 lola-4 TIME: Time stamp for console Nov 5 13:40:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:40:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:40:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:40:38 lola-4 sendmail[10625]: have been rejecting connections on daemon MTA for 21:01:45 Nov 5 13:40:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:41:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:41:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:41:35 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 13:41:35 lola-4 kernel: req@ffff8805e7b9a450 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478378500 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 13:41:35 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 13:41:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:41:41 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 13:41:41 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 13:41:41 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 13:41:41 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 13:41:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:42:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:42:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:42:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:42:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:43:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:43:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:43:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:43:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:44:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:44:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:44:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:44:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:45:01 lola-4 CROND[46563]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 13:45:01 lola-4 TIME: Time stamp for console Nov 5 13:45:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:45:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:45:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:45:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:46:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:46:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:46:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:46:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:47:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:47:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:47:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:47:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:48:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:48:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:48:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:48:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:49:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:49:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:49:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:49:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:50:01 lola-4 CROND[46597]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 13:50:01 lola-4 CROND[46598]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 13:50:01 lola-4 TIME: Time stamp for console Nov 5 13:50:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:50:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:50:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:50:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:51:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:51:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:51:36 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 13:51:36 lola-4 kernel: req@ffff88071a0dac50 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478379101 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 13:51:36 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 110 previous similar messages Nov 5 13:51:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:51:42 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 13:51:42 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 13:51:42 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 13:51:42 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 13:51:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:52:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:52:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:52:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:52:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:53:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:53:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:53:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:53:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:54:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:54:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:54:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:54:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:55:01 lola-4 CROND[46634]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 13:55:02 lola-4 TIME: Time stamp for console Nov 5 13:55:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:55:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:55:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:55:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:56:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:56:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:56:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:56:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:57:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:57:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:57:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:57:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 13:58:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:58:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:58:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:58:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:59:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:59:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:59:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 13:59:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:00:01 lola-4 CROND[46668]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 14:00:01 lola-4 CROND[46669]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 14:00:01 lola-4 TIME: Time stamp for console Nov 5 14:00:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:00:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:00:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 14:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 14:00:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:01:01 lola-4 CROND[46680]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 14:01:01 lola-4 run-parts(/etc/cron.hourly)[4668 starting 0anacron Nov 5 14:01:01 lola-4 run-parts(/etc/cron.hourly)[4668 finished 0anacron Nov 5 14:01:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:01:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:01:37 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 14:01:37 lola-4 kernel: req@ffff8807b0a81c50 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478379702 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 14:01:37 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 106 previous similar messages Nov 5 14:01:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:01:43 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 14:01:43 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 14:01:43 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 14:01:43 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 14:01:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:02:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:02:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:02:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:02:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:03:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:03:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:03:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:03:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:04:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:04:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:04:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:04:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:05:01 lola-4 CROND[46716]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 14:05:01 lola-4 TIME: Time stamp for console Nov 5 14:05:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:05:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:05:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:05:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:06:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:06:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:06:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:06:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:07:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:07:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:07:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:07:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:08:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:08:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:08:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:08:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:09:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:09:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:09:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:09:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:10:01 lola-4 CROND[46750]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 14:10:01 lola-4 CROND[46751]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 14:10:01 lola-4 TIME: Time stamp for console Nov 5 14:10:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:10:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:10:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:10:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:11:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:11:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:11:38 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 14:11:38 lola-4 kernel: req@ffff8804f7540450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478380303 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 14:11:38 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 14:11:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:11:44 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 14:11:44 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 14:11:44 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 14:11:44 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 14:11:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:12:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:12:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:12:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:12:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:13:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:13:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:13:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:13:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:14:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:14:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:14:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:14:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:15:01 lola-4 CROND[46787]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 14:15:01 lola-4 TIME: Time stamp for console Nov 5 14:15:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:15:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:15:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:15:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:16:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:16:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:16:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:16:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:17:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:17:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:17:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:17:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:18:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:18:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:18:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:18:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:19:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:19:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:19:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:19:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:20:01 lola-4 CROND[46822]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 14:20:01 lola-4 CROND[46823]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 14:20:01 lola-4 TIME: Time stamp for console Nov 5 14:20:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:20:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:20:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:20:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:21:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:21:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:21:39 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 14:21:39 lola-4 kernel: req@ffff88056d11a450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478380904 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 14:21:39 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 14:21:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:21:45 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 14:21:45 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 14:21:45 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 14:21:45 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 14:21:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:22:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:22:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:22:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:22:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:23:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:23:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:23:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:23:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:24:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:24:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:24:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:24:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:25:01 lola-4 CROND[46859]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 14:25:01 lola-4 TIME: Time stamp for console Nov 5 14:25:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:25:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:25:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:25:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:26:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:26:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:26:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:26:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:27:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:27:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:27:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:27:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:28:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:28:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:28:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:28:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:29:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:29:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:29:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:29:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:30:01 lola-4 CROND[46893]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 14:30:01 lola-4 CROND[46894]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 14:30:01 lola-4 TIME: Time stamp for console Nov 5 14:30:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:30:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:30:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:30:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:31:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:31:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:31:40 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 14:31:40 lola-4 kernel: req@ffff88074bd98450 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478381505 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 14:31:40 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 14:31:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:31:46 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 14:31:46 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 14:31:46 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 14:31:46 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 14:31:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:32:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:32:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:32:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:32:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:33:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:33:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:33:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:33:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:34:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:34:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:34:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:34:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:35:01 lola-4 CROND[46930]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 14:35:01 lola-4 TIME: Time stamp for console Nov 5 14:35:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:35:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:35:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:35:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:36:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:36:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:36:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:36:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:37:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:37:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:37:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:37:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:38:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:38:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:38:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:38:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:39:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:39:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:39:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:39:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:40:01 lola-4 CROND[46964]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 14:40:01 lola-4 CROND[46965]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 14:40:01 lola-4 TIME: Time stamp for console Nov 5 14:40:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:40:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:40:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:40:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:41:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:41:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:41:41 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 14:41:41 lola-4 kernel: req@ffff8806d8278850 x1550005401079776/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478382106 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 14:41:41 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 108 previous similar messages Nov 5 14:41:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:41:47 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 14:41:47 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 14:41:47 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 14:41:47 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 14:41:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:42:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:42:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:42:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:42:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:43:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:43:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:43:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:43:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:44:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:44:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:44:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:44:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:45:01 lola-4 CROND[47001]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 14:45:01 lola-4 TIME: Time stamp for console Nov 5 14:45:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:45:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:45:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:45:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:46:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:46:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:46:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:46:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:47:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:47:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:47:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:47:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:48:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:48:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:48:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:48:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:49:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:49:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:49:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:49:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:50:01 lola-4 CROND[47035]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 14:50:01 lola-4 CROND[47036]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 14:50:01 lola-4 TIME: Time stamp for console Nov 5 14:50:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:50:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:50:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:50:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:51:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:51:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:51:42 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 14:51:42 lola-4 kernel: req@ffff8807af297450 x1550005401098800/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478382707 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 14:51:42 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 108 previous similar messages Nov 5 14:51:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:51:48 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 14:51:48 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 14:51:48 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 14:51:48 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 14:51:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:52:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:52:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:52:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:52:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:53:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:53:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:53:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:53:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:54:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:54:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:54:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:54:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:55:01 lola-4 CROND[47073]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 14:55:01 lola-4 TIME: Time stamp for console Nov 5 14:55:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:55:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:55:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:55:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:56:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:56:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:56:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:56:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:57:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:57:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:57:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:57:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:58:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:58:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:58:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:58:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:59:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:59:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 14:59:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 14:59:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:00:01 lola-4 CROND[47107]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 15:00:01 lola-4 CROND[47108]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 15:00:01 lola-4 TIME: Time stamp for console Nov 5 15:00:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:00:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 15:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 15:00:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:01:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:01:01 lola-4 CROND[47119]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 15:01:01 lola-4 run-parts(/etc/cron.hourly)[4711 starting 0anacron Nov 5 15:01:01 lola-4 run-parts(/etc/cron.hourly)[4712 finished 0anacron Nov 5 15:01:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:01:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:01:43 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 15:01:43 lola-4 kernel: req@ffff8805c4fce050 x1550005401093744/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478383308 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 15:01:43 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 15:01:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:01:49 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 15:01:49 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 15:01:49 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 15:01:49 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 15:02:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:02:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:02:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:02:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:03:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:03:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:03:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:03:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:04:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:04:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:04:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:04:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:05:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:05:01 lola-4 CROND[47155]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 15:05:01 lola-4 TIME: Time stamp for console Nov 5 15:05:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:05:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:05:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:06:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:06:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:06:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:06:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:07:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:07:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:07:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:07:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:08:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:08:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:08:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:08:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:09:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:09:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:09:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:09:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:10:01 lola-4 CROND[47189]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 15:10:01 lola-4 CROND[47190]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 15:10:01 lola-4 TIME: Time stamp for console Nov 5 15:10:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:10:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:10:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:10:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:11:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:11:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:11:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:11:44 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 15:11:44 lola-4 kernel: req@ffff8804f06c8c50 x1550005401097056/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478383909 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 15:11:44 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 15:11:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:11:50 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 15:11:50 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 15:11:50 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 15:11:50 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 15:12:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:12:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:12:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:12:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:13:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:13:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:13:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:13:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:14:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:14:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:14:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:14:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:15:01 lola-4 CROND[47226]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 15:15:01 lola-4 TIME: Time stamp for console Nov 5 15:15:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:15:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:15:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:15:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:16:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:16:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:16:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:16:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:17:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:17:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:17:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:17:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:18:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:18:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:18:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:18:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:19:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:19:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:19:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:19:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:20:01 lola-4 CROND[47260]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 15:20:01 lola-4 CROND[47261]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 15:20:01 lola-4 TIME: Time stamp for console Nov 5 15:20:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:20:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:20:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:20:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:21:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:21:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:21:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:21:45 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 15:21:45 lola-4 kernel: req@ffff8805b3890c50 x1550005401098800/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478384510 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 15:21:45 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 15:21:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:21:51 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 15:21:51 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 15:21:51 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 15:21:51 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 15:22:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:22:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:22:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:22:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:23:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:23:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:23:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:23:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:24:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:24:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:24:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:24:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:25:01 lola-4 CROND[47298]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 15:25:01 lola-4 TIME: Time stamp for console Nov 5 15:25:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:25:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:25:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:25:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:26:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:26:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:26:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:26:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:27:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:27:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:27:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:27:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:28:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:28:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:28:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:28:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:29:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:29:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:29:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:29:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:30:01 lola-4 CROND[47332]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 15:30:01 lola-4 CROND[47333]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 15:30:01 lola-4 TIME: Time stamp for console Nov 5 15:30:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:30:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:30:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:30:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:31:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:31:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:31:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:31:45 lola-4 kernel: Lustre: 21683:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Nov 5 15:31:45 lola-4 kernel: req@ffff8805c3a61850 x1550005009576832/t0(0) o2->d9e9d3a3-2cdb-3592-6ba7-0fa4ebdd2aa8@192.168.1.117@o2ib100:-1/-1 lens 560/0 e 0 to 0 dl 1478385110 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 15:31:45 lola-4 kernel: Lustre: 21683:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 107 previous similar messages Nov 5 15:31:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:31:51 lola-4 kernel: Lustre: soaked-OST0008: Client d9e9d3a3-2cdb-3592-6ba7-0fa4ebdd2aa8 (at 192.168.1.117@o2ib100) reconnecting Nov 5 15:31:51 lola-4 kernel: Lustre: Skipped 24 previous similar messages Nov 5 15:31:51 lola-4 kernel: Lustre: soaked-OST0008: Connection restored to d9e9d3a3-2cdb-3592-6ba7-0fa4ebdd2aa8 (at 192.168.1.117@o2ib100) Nov 5 15:31:51 lola-4 kernel: Lustre: Skipped 24 previous similar messages Nov 5 15:32:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:32:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:32:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:32:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:33:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:33:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:33:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:33:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:34:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:34:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:34:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:34:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:35:01 lola-4 CROND[47369]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 15:35:01 lola-4 TIME: Time stamp for console Nov 5 15:35:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:35:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:35:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:35:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:36:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:36:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:36:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:36:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:37:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:37:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:37:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:37:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:38:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:38:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:38:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:38:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:39:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:39:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:39:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:39:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:40:01 lola-4 CROND[47403]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 15:40:01 lola-4 CROND[47404]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 15:40:01 lola-4 TIME: Time stamp for console Nov 5 15:40:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:40:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:40:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:40:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:41:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:41:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:41:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:41:47 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 15:41:47 lola-4 kernel: req@ffff880538993850 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478385712 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 15:41:47 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 109 previous similar messages Nov 5 15:41:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:41:53 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 15:41:53 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 15:41:53 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 15:41:53 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 15:42:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:42:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:42:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:42:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:43:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:43:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:43:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:43:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:44:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:44:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:44:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:44:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:45:01 lola-4 CROND[47440]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 15:45:01 lola-4 TIME: Time stamp for console Nov 5 15:45:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:45:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:45:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:45:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:46:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:46:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:46:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:46:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:47:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:47:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:47:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:47:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:48:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:48:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:48:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:48:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:49:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:49:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:49:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:49:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:50:01 lola-4 CROND[47474]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 15:50:01 lola-4 CROND[47475]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 15:50:01 lola-4 TIME: Time stamp for console Nov 5 15:50:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:50:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:50:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:50:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:51:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:51:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:51:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:51:48 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 15:51:48 lola-4 kernel: req@ffff880792e1bc50 x1550005401093744/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478386313 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 15:51:48 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 15:51:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:51:54 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 15:51:54 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 15:51:54 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 15:51:54 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 15:52:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:52:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:52:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:52:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:53:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:53:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:53:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:53:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:54:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:54:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:54:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:54:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:55:01 lola-4 CROND[47511]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 15:55:01 lola-4 TIME: Time stamp for console Nov 5 15:55:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:55:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:55:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:55:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:56:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:56:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:56:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:56:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:57:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 15:57:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:57:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:57:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:58:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:58:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:58:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:58:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:59:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:59:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:59:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 15:59:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:00:01 lola-4 CROND[47546]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 16:00:01 lola-4 CROND[47547]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 16:00:01 lola-4 TIME: Time stamp for console Nov 5 16:00:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:00:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:00:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 16:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 16:00:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:01:01 lola-4 CROND[47558]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 16:01:01 lola-4 run-parts(/etc/cron.hourly)[4755 starting 0anacron Nov 5 16:01:01 lola-4 run-parts(/etc/cron.hourly)[4756 finished 0anacron Nov 5 16:01:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:01:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:01:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:01:49 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 16:01:49 lola-4 kernel: req@ffff8804fa2b8c50 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478386914 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 16:01:49 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 16:01:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:01:55 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 16:01:55 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 16:01:55 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 16:01:55 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 16:02:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:02:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:02:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:02:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:03:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:03:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:03:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:03:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:04:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:04:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:04:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:04:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:05:01 lola-4 CROND[47594]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 16:05:01 lola-4 TIME: Time stamp for console Nov 5 16:05:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:05:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:05:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:05:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:06:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:06:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:06:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:06:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:07:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:07:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:07:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:07:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:08:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:08:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:08:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:08:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:09:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:09:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:09:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:09:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:10:01 lola-4 CROND[47628]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 16:10:01 lola-4 CROND[47629]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 16:10:01 lola-4 TIME: Time stamp for console Nov 5 16:10:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:10:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:10:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:10:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:11:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:11:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:11:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:11:50 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 16:11:50 lola-4 kernel: req@ffff88074b074050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478387515 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 16:11:50 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 16:11:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:11:56 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 16:11:56 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 16:11:56 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 16:11:56 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 16:12:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:12:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:12:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:12:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:13:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:13:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:13:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:13:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:14:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:14:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:14:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:14:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:15:01 lola-4 CROND[47665]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 16:15:01 lola-4 TIME: Time stamp for console Nov 5 16:15:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:15:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:15:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:15:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:16:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:16:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:16:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:16:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:17:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:17:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:17:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:17:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:18:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:18:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:18:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:18:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:19:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:19:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:19:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:19:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:20:01 lola-4 CROND[47699]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 16:20:01 lola-4 CROND[47700]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 16:20:01 lola-4 TIME: Time stamp for console Nov 5 16:20:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:20:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:20:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:20:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:21:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:21:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:21:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:21:51 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 16:21:51 lola-4 kernel: req@ffff8804ec846050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478388116 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 16:21:51 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 5 16:21:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:21:57 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 16:21:57 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 16:21:57 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 16:21:57 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 16:22:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:22:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:22:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:22:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:23:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:23:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:23:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:23:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:24:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:24:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:24:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:24:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:25:01 lola-4 CROND[47736]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 16:25:01 lola-4 TIME: Time stamp for console Nov 5 16:25:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:25:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:25:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:25:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:26:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:26:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:26:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:26:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:27:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:27:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:27:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:27:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:28:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:28:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:28:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:28:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:29:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:29:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:29:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:29:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:30:01 lola-4 CROND[47770]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 16:30:01 lola-4 CROND[47771]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 16:30:01 lola-4 TIME: Time stamp for console Nov 5 16:30:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:30:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:30:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:30:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:31:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:31:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:31:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:31:52 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 16:31:52 lola-4 kernel: req@ffff8805248c4450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478388717 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 16:31:52 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 16:31:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:31:58 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 16:31:58 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 16:31:58 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 16:31:58 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 16:32:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:32:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:32:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:32:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:33:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:33:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:33:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:33:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:34:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:34:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:34:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:34:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:35:01 lola-4 CROND[47808]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 16:35:01 lola-4 TIME: Time stamp for console Nov 5 16:35:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:35:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:35:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:35:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:36:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:36:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:36:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:36:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:37:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:37:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:37:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:37:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:38:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:38:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:38:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:38:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:39:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:39:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:39:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:39:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:40:01 lola-4 CROND[47842]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 16:40:01 lola-4 CROND[47843]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 16:40:01 lola-4 TIME: Time stamp for console Nov 5 16:40:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:40:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:40:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:40:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:40:53 lola-4 sendmail[10625]: have been rejecting connections on daemon MTA for 1+00:02:00 Nov 5 16:41:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:41:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:41:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:41:53 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 16:41:53 lola-4 kernel: req@ffff880812f34050 x1550005401079776/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478389318 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 16:41:53 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 16:41:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:41:59 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 16:41:59 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 16:41:59 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 16:41:59 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 16:42:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:42:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:42:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:42:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:43:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:43:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:43:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:43:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:44:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:44:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:44:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:44:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:45:01 lola-4 CROND[47879]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 16:45:01 lola-4 TIME: Time stamp for console Nov 5 16:45:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:45:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:45:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:45:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:46:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:46:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:46:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:46:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:47:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:47:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:47:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:47:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:48:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:48:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:48:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:48:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:49:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:49:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:49:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:49:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:50:01 lola-4 CROND[47913]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 16:50:01 lola-4 CROND[47914]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 16:50:01 lola-4 TIME: Time stamp for console Nov 5 16:50:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:50:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:50:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:50:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:51:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:51:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:51:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:51:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:51:54 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 16:51:54 lola-4 kernel: req@ffff88052a939c50 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478389919 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 16:51:54 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 16:52:00 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 16:52:00 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 16:52:00 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 16:52:00 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 16:52:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:52:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:52:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:52:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:53:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:53:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:53:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:53:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:54:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:54:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:54:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:54:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:55:01 lola-4 CROND[47950]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 16:55:01 lola-4 TIME: Time stamp for console Nov 5 16:55:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:55:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:55:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:55:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:56:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:56:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:56:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:56:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:57:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:57:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:57:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 16:57:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:58:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:58:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:58:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:58:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:59:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:59:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:59:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 16:59:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:00:01 lola-4 CROND[47984]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 17:00:01 lola-4 CROND[47985]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 17:00:01 lola-4 TIME: Time stamp for console Nov 5 17:00:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:00:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:00:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 17:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 17:00:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:01:01 lola-4 CROND[47996]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 17:01:01 lola-4 run-parts(/etc/cron.hourly)[4799 starting 0anacron Nov 5 17:01:01 lola-4 run-parts(/etc/cron.hourly)[4800 finished 0anacron Nov 5 17:01:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:01:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:01:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:01:55 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 17:01:55 lola-4 kernel: req@ffff88052794f050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478390520 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 17:01:55 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 17:01:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:02:01 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 17:02:01 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 17:02:01 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 17:02:01 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 17:02:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:02:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:02:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:02:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:03:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:03:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:03:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:03:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:04:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:04:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:04:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:04:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:05:01 lola-4 CROND[48033]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 17:05:01 lola-4 TIME: Time stamp for console Nov 5 17:05:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:05:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:05:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 489 Nov 5 17:05:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:06:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:06:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:06:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:06:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:07:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:07:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:07:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:07:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:08:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:08:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:08:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:08:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:09:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:09:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:09:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:09:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:10:01 lola-4 CROND[48067]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 17:10:01 lola-4 CROND[48068]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 17:10:01 lola-4 TIME: Time stamp for console Nov 5 17:10:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:10:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:10:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:10:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:11:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:11:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:11:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:11:56 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 17:11:56 lola-4 kernel: req@ffff8807c7ce2450 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478391121 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 17:11:56 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 5 17:11:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:12:02 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 17:12:02 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 17:12:02 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 17:12:02 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 17:12:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:12:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:12:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:12:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:13:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:13:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:13:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:13:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:14:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:14:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 489 Nov 5 17:14:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 489 Nov 5 17:14:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:15:01 lola-4 CROND[48104]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 17:15:01 lola-4 TIME: Time stamp for console Nov 5 17:15:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:15:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 489 Nov 5 17:15:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 489 Nov 5 17:15:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:16:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 489 Nov 5 17:16:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 489 Nov 5 17:16:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 489 Nov 5 17:16:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:17:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:17:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 489 Nov 5 17:17:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 489 Nov 5 17:17:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:18:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 489 Nov 5 17:18:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 489 Nov 5 17:18:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:18:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:19:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:19:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:19:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:19:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:20:01 lola-4 CROND[48138]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 17:20:01 lola-4 CROND[48139]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 17:20:01 lola-4 TIME: Time stamp for console Nov 5 17:20:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:20:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:20:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:20:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:21:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:21:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:21:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:21:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:21:57 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 17:21:57 lola-4 kernel: req@ffff88076eb1bc50 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478391722 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 17:21:57 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 17:22:03 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 17:22:03 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 17:22:03 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 17:22:03 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 17:22:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:22:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:22:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:22:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:23:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:23:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:23:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:23:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:24:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:24:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:24:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:24:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:25:01 lola-4 CROND[48175]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 17:25:01 lola-4 TIME: Time stamp for console Nov 5 17:25:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:25:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:25:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:25:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:26:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:26:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:26:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:26:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:27:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:27:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:27:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:27:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:28:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:28:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:28:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:28:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:29:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:29:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:29:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:29:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:30:01 lola-4 CROND[48209]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 17:30:01 lola-4 CROND[48210]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 17:30:01 lola-4 TIME: Time stamp for console Nov 5 17:30:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:30:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:30:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:30:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:31:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:31:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:31:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:31:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:31:58 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 17:31:58 lola-4 kernel: req@ffff8807418d8050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478392323 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 17:31:58 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 17:32:04 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 17:32:04 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 17:32:04 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 17:32:04 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 17:32:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:32:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:32:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:32:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:33:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:33:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:33:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:33:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:34:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:34:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:34:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:34:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:35:01 lola-4 CROND[48246]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 17:35:01 lola-4 TIME: Time stamp for console Nov 5 17:35:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:35:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:35:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:35:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:36:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:36:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:36:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:36:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:37:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:37:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:37:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:37:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:38:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:38:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:38:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:38:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:39:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:39:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:39:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:39:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:40:01 lola-4 CROND[48281]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 17:40:01 lola-4 CROND[48282]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 17:40:01 lola-4 TIME: Time stamp for console Nov 5 17:40:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:40:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:40:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:40:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:41:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:41:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:41:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:41:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:41:59 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 17:41:59 lola-4 kernel: req@ffff8805ce35a050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478392924 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 17:41:59 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 17:42:05 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 17:42:05 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 17:42:05 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 17:42:05 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 17:42:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:42:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:42:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:42:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:43:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:43:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:43:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:43:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:44:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:44:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:44:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:44:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:45:01 lola-4 CROND[48318]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 17:45:01 lola-4 TIME: Time stamp for console Nov 5 17:45:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:45:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:45:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:45:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:46:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:46:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:46:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:46:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:47:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:47:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:47:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:47:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:48:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:48:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:48:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:48:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:49:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:49:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:49:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:49:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:50:01 lola-4 CROND[48352]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 17:50:01 lola-4 CROND[48353]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 17:50:01 lola-4 TIME: Time stamp for console Nov 5 17:50:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:50:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:50:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:50:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:51:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:51:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:51:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:51:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:52:00 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 17:52:00 lola-4 kernel: req@ffff880826566c50 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478393525 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 17:52:00 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 17:52:06 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 17:52:06 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 17:52:06 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 17:52:06 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 17:52:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:52:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:52:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:52:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:53:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 17:53:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:53:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:53:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:54:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:54:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:54:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:54:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:55:01 lola-4 CROND[48389]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 17:55:01 lola-4 TIME: Time stamp for console Nov 5 17:55:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:55:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:55:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:55:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:56:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:56:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:56:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:56:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:57:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:57:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:57:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:57:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:58:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:58:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 17:58:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 17:58:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 17:59:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 17:59:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 17:59:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 17:59:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:00:01 lola-4 CROND[48423]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 18:00:01 lola-4 CROND[48424]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 18:00:01 lola-4 TIME: Time stamp for console Nov 5 18:00:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:00:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 18:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 18:00:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:01:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:01:01 lola-4 CROND[48435]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 18:01:01 lola-4 run-parts(/etc/cron.hourly)[4843 starting 0anacron Nov 5 18:01:01 lola-4 run-parts(/etc/cron.hourly)[4844 finished 0anacron Nov 5 18:01:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:01:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:01:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:02:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:02:01 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 18:02:01 lola-4 kernel: req@ffff8804cb6b2050 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478394126 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 18:02:01 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 5 18:02:07 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 18:02:07 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 18:02:07 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 18:02:07 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 18:02:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:02:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:02:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:03:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:03:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:03:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:03:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:04:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:04:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:04:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:04:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:05:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:05:01 lola-4 CROND[48471]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 18:05:01 lola-4 TIME: Time stamp for console Nov 5 18:05:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:05:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:05:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:06:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:06:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:06:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:06:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:07:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:07:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:07:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:07:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:08:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:08:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:08:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:08:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:09:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:09:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 18:09:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:09:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:10:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:10:01 lola-4 CROND[48505]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 18:10:01 lola-4 CROND[48506]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 18:10:01 lola-4 TIME: Time stamp for console Nov 5 18:10:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 18:10:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:10:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:11:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:11:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:11:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:11:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:12:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:12:02 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 18:12:02 lola-4 kernel: req@ffff880608399850 x1550005401093744/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478394727 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 18:12:02 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 18:12:08 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 18:12:08 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 18:12:08 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 18:12:08 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 18:12:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:12:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:12:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:13:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:13:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:13:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:13:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:14:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:14:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:14:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:14:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:15:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:15:01 lola-4 CROND[48543]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 18:15:01 lola-4 TIME: Time stamp for console Nov 5 18:15:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:15:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:15:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:16:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:16:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:16:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:16:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:17:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:17:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:17:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:17:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:18:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:18:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:18:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:18:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:19:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:19:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:19:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:19:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:20:01 lola-4 CROND[48577]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 18:20:01 lola-4 CROND[48578]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 18:20:01 lola-4 TIME: Time stamp for console Nov 5 18:20:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:20:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:20:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:20:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:21:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:21:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:21:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:21:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:22:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:22:03 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 18:22:03 lola-4 kernel: req@ffff88080fc0ac50 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478395328 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 18:22:03 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 18:22:09 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 18:22:09 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 18:22:09 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 18:22:09 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 18:22:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:22:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:22:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:23:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:23:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:23:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:23:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:24:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:24:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:24:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:24:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:25:01 lola-4 CROND[48614]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 18:25:01 lola-4 TIME: Time stamp for console Nov 5 18:25:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:25:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:25:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:25:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:26:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:26:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:26:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:26:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:27:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:27:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:27:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:27:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 18:28:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:28:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 18:28:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:28:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 18:29:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:29:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:29:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:29:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:30:01 lola-4 CROND[48648]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 18:30:01 lola-4 CROND[48649]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 18:30:01 lola-4 TIME: Time stamp for console Nov 5 18:30:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:30:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:30:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:30:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:31:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:31:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:31:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:31:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:32:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:32:04 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 18:32:04 lola-4 kernel: req@ffff88052f4df450 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478395929 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 18:32:04 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 18:32:10 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 18:32:10 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 18:32:10 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 18:32:10 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 18:32:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:32:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:32:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:33:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:33:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:33:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:33:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:34:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:34:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:34:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:34:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:35:01 lola-4 CROND[48685]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 18:35:01 lola-4 TIME: Time stamp for console Nov 5 18:35:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:35:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:35:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:35:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:36:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:36:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:36:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:36:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:37:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:37:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 18:37:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 18:37:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:38:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:38:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 18:38:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:38:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 18:39:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:39:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:39:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:39:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:40:01 lola-4 CROND[48719]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 18:40:01 lola-4 CROND[48720]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 18:40:01 lola-4 TIME: Time stamp for console Nov 5 18:40:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:40:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 18:40:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:40:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:41:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:41:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 18:41:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:41:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:42:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:42:05 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 18:42:05 lola-4 kernel: req@ffff88053dbdc850 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478396530 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 18:42:05 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 18:42:11 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 18:42:11 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 18:42:11 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 18:42:11 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 18:42:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:42:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:42:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:43:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:43:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:43:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:43:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:44:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:44:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:44:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:44:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:45:01 lola-4 CROND[48757]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 18:45:01 lola-4 TIME: Time stamp for console Nov 5 18:45:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:45:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:45:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:45:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:46:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:46:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:46:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:46:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:47:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:47:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:47:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:47:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:48:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:48:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:48:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:48:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:49:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:49:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:49:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:49:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:50:01 lola-4 CROND[48791]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 18:50:01 lola-4 CROND[48792]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 18:50:01 lola-4 TIME: Time stamp for console Nov 5 18:50:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:50:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:50:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:50:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:51:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:51:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:51:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:51:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:52:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:52:06 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 18:52:06 lola-4 kernel: req@ffff880784afd450 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478397131 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 18:52:06 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 104 previous similar messages Nov 5 18:52:12 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 18:52:12 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 18:52:12 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 18:52:12 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 18:52:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:52:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:52:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:53:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:53:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:53:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:53:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:54:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:54:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:54:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:54:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:55:01 lola-4 CROND[48828]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 18:55:01 lola-4 TIME: Time stamp for console Nov 5 18:55:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:55:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:55:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:55:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:56:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:56:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:56:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:56:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:57:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:57:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 18:57:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 18:57:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:58:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:58:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:58:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:58:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:59:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:59:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:59:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 18:59:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:00:01 lola-4 CROND[48862]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 19:00:01 lola-4 CROND[48863]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 19:00:01 lola-4 TIME: Time stamp for console Nov 5 19:00:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:00:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:00:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 19:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 19:00:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:01:01 lola-4 CROND[48874]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 19:01:01 lola-4 run-parts(/etc/cron.hourly)[4887 starting 0anacron Nov 5 19:01:01 lola-4 run-parts(/etc/cron.hourly)[4888 finished 0anacron Nov 5 19:01:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:01:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:01:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:01:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:02:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:02:07 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 19:02:07 lola-4 kernel: req@ffff8807040fec50 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478397732 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 19:02:07 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 19:02:13 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 19:02:13 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 19:02:13 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 19:02:13 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 19:02:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:02:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:02:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:03:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:03:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:03:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:03:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:04:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:04:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:04:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:04:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:05:01 lola-4 CROND[48910]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 19:05:01 lola-4 TIME: Time stamp for console Nov 5 19:05:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:05:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:05:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:05:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:06:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:06:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:06:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:06:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:07:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:07:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:07:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:07:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:08:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:08:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:08:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:08:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:09:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:09:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:09:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:09:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:10:01 lola-4 CROND[48944]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 19:10:01 lola-4 CROND[48945]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 19:10:01 lola-4 TIME: Time stamp for console Nov 5 19:10:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:10:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:10:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:10:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:11:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:11:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:11:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:11:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:12:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:12:08 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 19:12:08 lola-4 kernel: req@ffff8807f3d76050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478398333 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 19:12:08 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 19:12:14 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 19:12:14 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 19:12:14 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 19:12:14 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 19:12:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:12:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:12:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:13:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:13:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:13:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:13:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:14:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:14:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:14:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:14:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:15:01 lola-4 CROND[48981]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 19:15:01 lola-4 TIME: Time stamp for console Nov 5 19:15:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:15:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:15:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:15:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:16:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:16:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:16:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:16:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:17:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:17:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:17:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:17:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:18:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:18:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:18:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:18:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:19:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:19:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:19:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:19:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:20:01 lola-4 CROND[49016]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 19:20:01 lola-4 CROND[49017]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 19:20:01 lola-4 TIME: Time stamp for console Nov 5 19:20:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:20:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:20:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:20:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:21:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:21:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:21:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:21:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:22:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:22:09 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 19:22:09 lola-4 kernel: req@ffff88057bf16c50 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478398934 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 19:22:09 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 19:22:15 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 19:22:15 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 19:22:15 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 19:22:15 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 19:22:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:22:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:22:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:23:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:23:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:23:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:23:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:24:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:24:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:24:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:24:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:25:01 lola-4 CROND[49053]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 19:25:01 lola-4 TIME: Time stamp for console Nov 5 19:25:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:25:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:25:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:25:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:26:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:26:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:26:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:26:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:27:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:27:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:27:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:27:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:28:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:28:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:28:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:28:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:29:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:29:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:29:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:29:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:30:01 lola-4 CROND[49087]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 19:30:01 lola-4 CROND[49088]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 19:30:01 lola-4 TIME: Time stamp for console Nov 5 19:30:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:30:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:30:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:30:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:31:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:31:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:31:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:31:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:32:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:32:10 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 19:32:10 lola-4 kernel: req@ffff8807d5bdd450 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478399535 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 19:32:10 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 19:32:16 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 19:32:16 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 19:32:16 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 19:32:16 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 19:32:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:32:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:32:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:33:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:33:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:33:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:33:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:34:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:34:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:34:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:34:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:35:01 lola-4 CROND[49124]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 19:35:01 lola-4 TIME: Time stamp for console Nov 5 19:35:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:35:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:35:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:35:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:36:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:36:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:36:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:36:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:37:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:37:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:37:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:37:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:38:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:38:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:38:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:38:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:39:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:39:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:39:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:39:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:40:02 lola-4 CROND[49158]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 19:40:02 lola-4 CROND[49159]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 19:40:02 lola-4 TIME: Time stamp for console Nov 5 19:40:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:40:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:40:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:40:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:41:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:41:08 lola-4 sendmail[10625]: have been rejecting connections on daemon MTA for 1+03:02:15 Nov 5 19:41:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:41:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:41:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:42:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:42:11 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 19:42:11 lola-4 kernel: req@ffff88065e334c50 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478400136 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 19:42:11 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 5 19:42:17 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 19:42:17 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 19:42:17 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 19:42:17 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 19:42:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:42:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:42:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:43:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:43:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:43:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:43:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:44:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:44:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:44:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:44:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:45:01 lola-4 CROND[49195]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 19:45:01 lola-4 TIME: Time stamp for console Nov 5 19:45:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:45:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:45:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:45:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:46:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:46:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:46:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:46:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:47:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:47:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:47:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:47:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:48:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:48:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:48:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:48:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:49:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:49:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:49:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:49:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:50:01 lola-4 CROND[49229]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 19:50:01 lola-4 CROND[49230]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 19:50:01 lola-4 TIME: Time stamp for console Nov 5 19:50:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:50:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:50:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:50:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:51:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:51:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:51:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:51:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:52:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:52:12 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 19:52:12 lola-4 kernel: req@ffff880815527450 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478400737 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 19:52:12 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 19:52:18 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 19:52:18 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 19:52:18 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 19:52:18 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 19:52:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:52:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:52:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:53:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:53:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:53:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:53:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:54:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:54:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:54:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:54:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:55:01 lola-4 CROND[49267]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 19:55:01 lola-4 TIME: Time stamp for console Nov 5 19:55:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:55:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:55:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:55:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:56:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:56:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:56:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:56:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:57:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:57:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:57:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:57:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:58:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:58:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:58:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:58:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 19:59:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:59:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:59:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 19:59:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:00:01 lola-4 CROND[49301]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 20:00:01 lola-4 CROND[49302]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 20:00:01 lola-4 TIME: Time stamp for console Nov 5 20:00:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:00:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:00:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 20:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 20:00:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:01:01 lola-4 CROND[49313]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 20:01:01 lola-4 run-parts(/etc/cron.hourly)[4931 starting 0anacron Nov 5 20:01:01 lola-4 run-parts(/etc/cron.hourly)[4932 finished 0anacron Nov 5 20:01:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:01:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:01:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:01:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:02:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:02:13 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 20:02:13 lola-4 kernel: req@ffff8804eae38450 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478401338 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 20:02:13 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 20:02:19 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 20:02:19 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 20:02:19 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 20:02:19 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 20:02:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:02:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:02:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:03:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:03:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:03:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:03:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:04:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:04:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:04:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:04:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:05:01 lola-4 CROND[49349]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 20:05:01 lola-4 TIME: Time stamp for console Nov 5 20:05:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:05:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:05:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:05:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:06:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:06:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:06:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:06:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:07:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:07:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:07:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:07:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:08:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:08:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:08:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:08:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:09:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:09:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:09:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:09:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:10:01 lola-4 CROND[49383]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 20:10:01 lola-4 CROND[49384]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 20:10:01 lola-4 TIME: Time stamp for console Nov 5 20:10:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:10:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:10:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:10:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:11:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:11:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:11:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:11:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:12:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:12:14 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 20:12:14 lola-4 kernel: req@ffff8804d323bc50 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478401939 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 20:12:14 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 20:12:20 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 20:12:20 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 20:12:20 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 20:12:20 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 20:12:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:12:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:12:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:13:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:13:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:13:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:13:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:14:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:14:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:14:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:14:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:15:01 lola-4 CROND[49420]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 20:15:01 lola-4 TIME: Time stamp for console Nov 5 20:15:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:15:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:15:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:15:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:16:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:16:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:16:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:16:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:17:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:17:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:17:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:17:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:18:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:18:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:18:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:18:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:19:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:19:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:19:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:19:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:20:01 lola-4 CROND[49454]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 20:20:01 lola-4 CROND[49455]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 20:20:01 lola-4 TIME: Time stamp for console Nov 5 20:20:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:20:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:20:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:20:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:21:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:21:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:21:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:21:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:22:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:22:15 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 20:22:15 lola-4 kernel: req@ffff880525e31450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478402540 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 20:22:15 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 110 previous similar messages Nov 5 20:22:21 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 20:22:21 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 20:22:21 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 20:22:21 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 20:22:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:22:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:22:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:23:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:23:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:23:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:23:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:24:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:24:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:24:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:24:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:25:01 lola-4 CROND[49492]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 20:25:01 lola-4 TIME: Time stamp for console Nov 5 20:25:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:25:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:25:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:25:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:26:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:26:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:26:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:26:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:27:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:27:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:27:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:27:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:28:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:28:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:28:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:28:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:29:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:29:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:29:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:29:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:30:01 lola-4 CROND[49526]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 20:30:01 lola-4 CROND[49527]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 20:30:01 lola-4 TIME: Time stamp for console Nov 5 20:30:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:30:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:30:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:30:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:31:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:31:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:31:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:31:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:32:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:32:16 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 20:32:16 lola-4 kernel: req@ffff8805b9b9d050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478403141 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 20:32:16 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 106 previous similar messages Nov 5 20:32:22 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 20:32:22 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 20:32:22 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 20:32:22 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 20:32:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:32:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:32:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:33:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:33:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:33:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:33:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:34:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:34:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:34:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:34:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:35:01 lola-4 CROND[49563]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 20:35:01 lola-4 TIME: Time stamp for console Nov 5 20:35:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:35:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:35:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:35:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:36:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:36:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:36:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:36:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:37:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:37:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:37:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:37:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:38:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:38:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:38:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:38:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:39:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:39:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:39:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:39:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:40:01 lola-4 CROND[49597]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 20:40:01 lola-4 CROND[49598]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 20:40:01 lola-4 TIME: Time stamp for console Nov 5 20:40:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:40:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:40:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:40:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:41:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:41:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:41:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:41:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:42:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:42:17 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 20:42:17 lola-4 kernel: req@ffff88051a6dfc50 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478403742 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 20:42:17 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 20:42:23 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 20:42:23 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 20:42:23 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 20:42:23 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 20:42:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:42:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:42:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:43:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:43:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:43:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:43:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:44:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:44:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:44:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:44:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:45:01 lola-4 CROND[49634]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 20:45:01 lola-4 TIME: Time stamp for console Nov 5 20:45:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:45:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:45:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:45:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:46:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:46:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:46:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:46:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:47:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:47:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:47:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:47:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:48:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:48:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:48:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:48:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:49:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:49:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:49:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:49:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:50:01 lola-4 CROND[49668]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 20:50:01 lola-4 CROND[49669]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 20:50:01 lola-4 TIME: Time stamp for console Nov 5 20:50:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:50:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:50:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:50:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:51:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:51:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:51:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:51:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:52:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:52:18 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 20:52:18 lola-4 kernel: req@ffff8805d2296450 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478404343 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 20:52:18 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 20:52:24 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 20:52:24 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 20:52:24 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 20:52:24 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 20:52:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:52:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:52:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:53:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:53:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:53:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 20:53:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:54:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:54:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:54:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:54:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:55:01 lola-4 CROND[49705]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 20:55:01 lola-4 TIME: Time stamp for console Nov 5 20:55:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:55:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:55:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:55:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:56:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:56:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:56:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:56:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:57:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:57:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:57:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:57:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:58:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:58:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:58:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:58:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:59:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:59:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:59:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 20:59:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:00:01 lola-4 CROND[49740]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 21:00:01 lola-4 CROND[49741]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 21:00:01 lola-4 TIME: Time stamp for console Nov 5 21:00:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:00:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 21:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 21:00:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:01:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:01:01 lola-4 CROND[49752]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 21:01:01 lola-4 run-parts(/etc/cron.hourly)[4975 starting 0anacron Nov 5 21:01:01 lola-4 run-parts(/etc/cron.hourly)[4976 finished 0anacron Nov 5 21:01:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:01:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:01:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:02:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:02:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:02:19 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 21:02:19 lola-4 kernel: req@ffff880552e9dc50 x1550005401098800/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478404944 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 21:02:19 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 21:02:25 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 21:02:25 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 21:02:25 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 21:02:25 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 21:02:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:02:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:03:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:03:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:03:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:03:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:04:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:04:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:04:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:04:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:05:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:05:01 lola-4 CROND[49788]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 21:05:01 lola-4 TIME: Time stamp for console Nov 5 21:05:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:05:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:05:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:06:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:06:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:06:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:06:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:07:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:07:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:07:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:07:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:08:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:08:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:08:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:08:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:09:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:09:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:09:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:09:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:10:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:10:01 lola-4 CROND[49823]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 21:10:01 lola-4 CROND[49822]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 21:10:01 lola-4 TIME: Time stamp for console Nov 5 21:10:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:10:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:10:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:11:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:11:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:11:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:11:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:12:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:12:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:12:20 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 21:12:20 lola-4 kernel: req@ffff880674ab8450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478405545 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 21:12:20 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 108 previous similar messages Nov 5 21:12:26 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 21:12:26 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 21:12:26 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 21:12:26 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 21:12:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:12:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:13:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:13:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:13:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:13:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:14:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:14:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:14:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:14:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:15:01 lola-4 CROND[49859]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 21:15:01 lola-4 TIME: Time stamp for console Nov 5 21:15:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:15:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:15:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:15:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:16:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:16:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:16:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:16:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:17:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:17:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:17:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:17:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:18:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:18:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:18:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:18:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:19:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:19:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:19:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:19:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:20:01 lola-4 CROND[49893]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 21:20:01 lola-4 CROND[49894]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 21:20:01 lola-4 TIME: Time stamp for console Nov 5 21:20:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:20:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:20:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:20:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:21:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:21:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:21:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:21:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:22:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:22:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:22:21 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 21:22:21 lola-4 kernel: req@ffff88071ffc9450 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478406146 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 21:22:21 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 108 previous similar messages Nov 5 21:22:27 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 21:22:27 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 21:22:27 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 21:22:27 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 21:22:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:22:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:23:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:23:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:23:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:23:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:24:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:24:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:24:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:24:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:25:01 lola-4 CROND[49930]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 21:25:01 lola-4 TIME: Time stamp for console Nov 5 21:25:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:25:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:25:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:25:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:26:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:26:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:26:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:26:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:27:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:27:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:27:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:27:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:28:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:28:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:28:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:28:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:29:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:29:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:29:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:29:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:30:01 lola-4 CROND[49964]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 21:30:01 lola-4 CROND[49965]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 21:30:01 lola-4 TIME: Time stamp for console Nov 5 21:30:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:30:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:30:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:30:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:31:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:31:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:31:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:31:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:32:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:32:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:32:22 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 21:32:22 lola-4 kernel: req@ffff880748acd450 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478406747 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 21:32:22 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 21:32:28 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 21:32:28 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 21:32:28 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 21:32:28 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 21:32:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:32:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:33:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:33:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:33:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:33:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:34:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:34:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:34:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:34:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:35:01 lola-4 CROND[50002]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 21:35:01 lola-4 TIME: Time stamp for console Nov 5 21:35:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:35:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:35:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:35:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:36:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:36:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:36:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:36:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:37:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:37:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:37:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:37:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:38:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:38:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:38:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:38:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:39:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:39:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:39:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:39:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:40:01 lola-4 CROND[50036]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 21:40:01 lola-4 CROND[50037]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 21:40:01 lola-4 TIME: Time stamp for console Nov 5 21:40:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:40:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:40:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:40:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:41:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:41:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:41:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:41:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:42:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:42:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:42:23 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 21:42:23 lola-4 kernel: req@ffff88079527b450 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478407348 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 21:42:23 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 21:42:29 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 21:42:29 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 21:42:29 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 21:42:29 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 21:42:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:42:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:43:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:43:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:43:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:43:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:44:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:44:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:44:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:44:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:45:01 lola-4 CROND[50073]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 21:45:01 lola-4 TIME: Time stamp for console Nov 5 21:45:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:45:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:45:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:45:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:46:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:46:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:46:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:46:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:47:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:47:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:47:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:47:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:48:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:48:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:48:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:48:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:49:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:49:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:49:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:49:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:50:01 lola-4 CROND[50107]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 21:50:01 lola-4 CROND[50108]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 21:50:01 lola-4 TIME: Time stamp for console Nov 5 21:50:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:50:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:50:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:50:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:51:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:51:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:51:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:51:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:52:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:52:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:52:24 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 21:52:24 lola-4 kernel: req@ffff8804ecd5c850 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478407949 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 21:52:24 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 21:52:30 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 21:52:30 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 21:52:30 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 21:52:30 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 21:52:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:52:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:53:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:53:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:53:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:53:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:54:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:54:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:54:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:54:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:55:01 lola-4 CROND[50144]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 21:55:01 lola-4 TIME: Time stamp for console Nov 5 21:55:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:55:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:55:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:55:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:56:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:56:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:56:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:56:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:57:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:57:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:57:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:57:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:58:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 21:58:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:58:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:58:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:59:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:59:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:59:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 21:59:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:00:01 lola-4 CROND[50178]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 22:00:01 lola-4 CROND[50179]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 22:00:01 lola-4 TIME: Time stamp for console Nov 5 22:00:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:00:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:00:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 22:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 22:00:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:01:01 lola-4 CROND[50190]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 22:01:01 lola-4 run-parts(/etc/cron.hourly)[5019 starting 0anacron Nov 5 22:01:01 lola-4 run-parts(/etc/cron.hourly)[5019 finished 0anacron Nov 5 22:01:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:01:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:01:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:01:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:02:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:02:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:02:25 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 22:02:25 lola-4 kernel: req@ffff8805b5341850 x1550005401098800/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478408550 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 22:02:25 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 107 previous similar messages Nov 5 22:02:31 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 22:02:31 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 22:02:31 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 22:02:31 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 22:02:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:02:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:03:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:03:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:03:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:03:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:04:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:04:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:04:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:04:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:05:01 lola-4 CROND[50227]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 22:05:01 lola-4 TIME: Time stamp for console Nov 5 22:05:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:05:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:05:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:05:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:06:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:06:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:06:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:06:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:07:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:07:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:07:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:07:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:08:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:08:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:08:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:08:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:09:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:09:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:09:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:09:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:10:01 lola-4 CROND[50261]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 22:10:01 lola-4 CROND[50262]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 22:10:01 lola-4 TIME: Time stamp for console Nov 5 22:10:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:10:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:10:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:10:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:11:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:11:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:11:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:11:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:12:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:12:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:12:26 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 22:12:26 lola-4 kernel: req@ffff880501b81c50 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478409151 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 22:12:26 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 108 previous similar messages Nov 5 22:12:32 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 22:12:32 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 22:12:32 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 22:12:32 lola-4 kernel: Lustre: Skipped 25 previous similar messages Nov 5 22:12:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:12:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:13:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:13:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:13:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:13:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:14:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:14:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:14:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:14:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:15:01 lola-4 CROND[50298]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 22:15:01 lola-4 TIME: Time stamp for console Nov 5 22:15:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:15:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:15:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:15:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:16:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:16:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:16:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:16:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:17:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:17:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:17:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:17:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:18:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:18:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:18:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:18:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:19:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:19:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:19:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:19:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:20:01 lola-4 CROND[50332]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 22:20:01 lola-4 CROND[50333]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 22:20:01 lola-4 TIME: Time stamp for console Nov 5 22:20:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:20:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:20:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:20:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:21:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:21:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:21:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:21:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:22:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:22:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:22:27 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 22:22:27 lola-4 kernel: req@ffff880705080050 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478409752 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 22:22:27 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 22:22:33 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 22:22:33 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 22:22:33 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 22:22:33 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 22:22:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:22:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:23:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:23:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:23:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:23:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:24:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:24:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:24:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:24:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:25:01 lola-4 CROND[50369]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 22:25:01 lola-4 TIME: Time stamp for console Nov 5 22:25:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:25:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:25:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:25:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:26:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:26:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:26:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:26:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:27:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:27:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:27:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:27:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:28:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:28:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:28:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:28:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:29:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:29:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:29:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:29:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:30:01 lola-4 CROND[50403]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 22:30:01 lola-4 CROND[50404]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 22:30:01 lola-4 TIME: Time stamp for console Nov 5 22:30:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:30:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:30:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:30:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:31:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:31:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:31:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:31:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:32:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:32:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:32:28 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 22:32:28 lola-4 kernel: req@ffff880740d55050 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478410353 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 22:32:28 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 22:32:34 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 22:32:34 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 22:32:34 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 22:32:34 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 22:32:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:32:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:33:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:33:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:33:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:33:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:34:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:34:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:34:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:34:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:35:01 lola-4 CROND[50440]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 22:35:01 lola-4 TIME: Time stamp for console Nov 5 22:35:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:35:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:35:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:35:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:36:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:36:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:36:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:36:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:37:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:37:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:37:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:37:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:38:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:38:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:38:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:38:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:39:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:39:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:39:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:39:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:40:01 lola-4 CROND[50475]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 22:40:01 lola-4 CROND[50476]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 22:40:01 lola-4 TIME: Time stamp for console Nov 5 22:40:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:40:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:40:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:40:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:41:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:41:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:41:23 lola-4 sendmail[10625]: have been rejecting connections on daemon MTA for 1+06:02:30 Nov 5 22:41:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:41:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:42:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:42:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:42:29 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 22:42:29 lola-4 kernel: req@ffff8804e8fe3050 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478410954 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 22:42:29 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 22:42:35 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 22:42:35 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 22:42:35 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 22:42:35 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 22:42:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:42:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:43:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:43:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:43:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:43:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:44:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:44:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:44:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:44:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:45:01 lola-4 CROND[50512]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 22:45:01 lola-4 TIME: Time stamp for console Nov 5 22:45:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:45:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:45:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:45:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:46:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:46:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:46:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:46:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:47:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:47:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:47:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:47:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:48:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:48:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:48:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:48:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:49:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:49:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:49:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:49:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:50:01 lola-4 CROND[50546]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 22:50:01 lola-4 CROND[50547]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 22:50:01 lola-4 TIME: Time stamp for console Nov 5 22:50:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:50:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:50:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:50:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:51:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:51:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:51:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:51:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:52:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:52:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:52:30 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 22:52:30 lola-4 kernel: req@ffff880528e9d450 x1550005401093744/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478411555 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 22:52:30 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 5 22:52:36 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 22:52:36 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 22:52:36 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 22:52:36 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 22:52:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:52:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:53:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:53:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:53:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:53:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:54:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:54:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:54:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:54:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:55:01 lola-4 CROND[50583]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 22:55:01 lola-4 TIME: Time stamp for console Nov 5 22:55:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:55:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:55:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:55:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:56:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:56:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:56:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:56:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:57:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:57:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:57:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:57:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:58:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:58:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:58:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:58:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 22:59:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:59:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:59:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 22:59:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:00:02 lola-4 CROND[50617]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 23:00:02 lola-4 CROND[50618]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 23:00:02 lola-4 TIME: Time stamp for console Nov 5 23:00:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:00:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:00:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 5 23:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 5 23:00:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:01:01 lola-4 CROND[50629]: (root) CMD (run-parts /etc/cron.hourly) Nov 5 23:01:01 lola-4 run-parts(/etc/cron.hourly)[5062 starting 0anacron Nov 5 23:01:01 lola-4 run-parts(/etc/cron.hourly)[5063 finished 0anacron Nov 5 23:01:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:01:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:01:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:01:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:02:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:02:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:02:31 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 23:02:31 lola-4 kernel: req@ffff880524708c50 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478412156 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 23:02:31 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 23:02:37 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 23:02:37 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 23:02:37 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 23:02:37 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 23:02:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:02:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:03:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:03:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:03:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:03:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:04:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:04:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:04:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:04:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:05:01 lola-4 CROND[50665]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 23:05:01 lola-4 TIME: Time stamp for console Nov 5 23:05:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:05:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:05:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:05:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:06:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:06:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:06:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:06:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:07:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:07:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:07:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:07:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:08:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:08:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:08:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:08:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:09:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:09:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:09:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:09:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:10:01 lola-4 CROND[50699]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 23:10:01 lola-4 CROND[50700]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 23:10:01 lola-4 TIME: Time stamp for console Nov 5 23:10:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:10:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:10:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:10:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:11:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:11:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:11:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:11:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:12:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:12:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:12:32 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 23:12:32 lola-4 kernel: req@ffff8806b7ec2450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478412757 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 23:12:32 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 23:12:38 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 23:12:38 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 23:12:38 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 23:12:38 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 23:12:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:12:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:13:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:13:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:13:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:13:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:14:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:14:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:14:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:14:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:15:01 lola-4 CROND[50737]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 23:15:01 lola-4 TIME: Time stamp for console Nov 5 23:15:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:15:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:15:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:15:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:16:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:16:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:16:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:16:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:17:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:17:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:17:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:17:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:18:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:18:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:18:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:18:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:19:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:19:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:19:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:19:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:20:01 lola-4 CROND[50771]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 23:20:01 lola-4 CROND[50772]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 23:20:01 lola-4 TIME: Time stamp for console Nov 5 23:20:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:20:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:20:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:20:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:21:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:21:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:21:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:21:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:22:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:22:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:22:33 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 23:22:33 lola-4 kernel: req@ffff8804db456c50 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478413358 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 23:22:33 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 23:22:39 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 23:22:39 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 23:22:39 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 23:22:39 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 5 23:22:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:22:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:23:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:23:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:23:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:23:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:24:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:24:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:24:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:24:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:25:01 lola-4 CROND[50808]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 23:25:01 lola-4 TIME: Time stamp for console Nov 5 23:25:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:25:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:25:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:25:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:26:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:26:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:26:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:26:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:27:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:27:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:27:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:27:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:28:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:28:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:28:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:28:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:29:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:29:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:29:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:29:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:30:01 lola-4 CROND[50842]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 23:30:01 lola-4 CROND[50843]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 23:30:01 lola-4 TIME: Time stamp for console Nov 5 23:30:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:30:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:30:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:30:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:31:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:31:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:31:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:31:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:32:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:32:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:32:34 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 23:32:34 lola-4 kernel: req@ffff8804ff0a8450 x1550005401098800/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478413959 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 23:32:34 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 5 23:32:40 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 23:32:40 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 23:32:40 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 23:32:40 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 23:32:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:32:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:33:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:33:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:33:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:33:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:34:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:34:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:34:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:34:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:35:01 lola-4 CROND[50879]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 23:35:01 lola-4 TIME: Time stamp for console Nov 5 23:35:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:35:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:35:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:35:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:36:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:36:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:36:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:36:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:37:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:37:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:37:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:37:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:38:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:38:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:38:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:38:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:39:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:39:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:39:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:39:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:40:01 lola-4 CROND[50913]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 23:40:01 lola-4 CROND[50914]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 23:40:01 lola-4 TIME: Time stamp for console Nov 5 23:40:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:40:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:40:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:40:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:41:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:41:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:41:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:41:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:42:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:42:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:42:35 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 23:42:35 lola-4 kernel: req@ffff88052bae7450 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478414560 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 23:42:35 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 5 23:42:41 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 23:42:41 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 23:42:41 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 23:42:41 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 5 23:42:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:42:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:43:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:43:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:43:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:43:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:44:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:44:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:44:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:44:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:45:01 lola-4 CROND[50951]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 23:45:01 lola-4 TIME: Time stamp for console Nov 5 23:45:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:45:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:45:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:45:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:46:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:46:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:46:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:46:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:47:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:47:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:47:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:47:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:48:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:48:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:48:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:48:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:49:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:49:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:49:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:49:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:50:01 lola-4 CROND[50985]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 23:50:01 lola-4 CROND[50986]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 5 23:50:01 lola-4 TIME: Time stamp for console Nov 5 23:50:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:50:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:50:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:50:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:51:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:51:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:51:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:51:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:52:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:52:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 5 23:52:36 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 5 23:52:36 lola-4 kernel: req@ffff880808a37c50 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478415161 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 5 23:52:36 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 5 23:52:42 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 5 23:52:42 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 23:52:42 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 5 23:52:42 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 5 23:52:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:52:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:53:01 lola-4 CROND[51009]: (root) CMD (/usr/lib64/sa/sa2 -A) Nov 5 23:53:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:53:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:53:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:53:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:54:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:54:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:54:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:54:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:55:01 lola-4 CROND[51033]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 5 23:55:01 lola-4 TIME: Time stamp for console Nov 5 23:55:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:55:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:55:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:55:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:56:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:56:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:56:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:56:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:57:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:57:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:57:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:57:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:58:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:58:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:58:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:58:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:59:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:59:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:59:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 5 23:59:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:00:01 lola-4 CROND[51065]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 00:00:01 lola-4 CROND[51066]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 00:00:01 lola-4 TIME: Time stamp for console Nov 6 00:00:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:00:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 6 00:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 6 00:00:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:01:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:01:01 lola-4 CROND[51079]: (root) CMD (run-parts /etc/cron.hourly) Nov 6 00:01:01 lola-4 run-parts(/etc/cron.hourly)[5107 starting 0anacron Nov 6 00:01:01 lola-4 anacron[51090]: Anacron started on 2016-11-06 Nov 6 00:01:01 lola-4 anacron[51090]: Jobs will be executed sequentially Nov 6 00:01:01 lola-4 anacron[51090]: Normal exit (0 jobs run) Nov 6 00:01:01 lola-4 run-parts(/etc/cron.hourly)[5109 finished 0anacron Nov 6 00:01:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:01:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:01:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:02:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:02:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:02:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:02:37 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 00:02:37 lola-4 kernel: req@ffff8806b8caa050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478415762 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 00:02:37 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 6 00:02:43 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 00:02:43 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 00:02:43 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 00:02:43 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 00:02:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:03:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:03:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:03:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:03:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:04:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:04:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:04:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:04:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:05:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:05:01 lola-4 CROND[51119]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 00:05:01 lola-4 TIME: Time stamp for console Nov 6 00:05:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:05:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:05:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:06:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:06:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:06:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:06:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:07:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:07:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:07:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:07:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:08:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:08:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:08:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:08:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:09:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:09:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:09:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:09:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:10:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:10:01 lola-4 CROND[51153]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 00:10:01 lola-4 CROND[51154]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 00:10:01 lola-4 TIME: Time stamp for console Nov 6 00:10:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:10:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:10:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:11:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:11:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:11:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:11:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:12:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:12:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:12:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:12:38 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 00:12:38 lola-4 kernel: req@ffff88074288bc50 x1550005401079776/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478416363 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 00:12:38 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 6 00:12:44 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 00:12:44 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 6 00:12:44 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 00:12:44 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 6 00:12:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:13:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:13:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:13:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:13:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:14:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:14:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:14:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:14:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:15:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:15:01 lola-4 CROND[51190]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 00:15:01 lola-4 TIME: Time stamp for console Nov 6 00:15:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:15:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:15:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:16:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:16:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:16:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:16:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:17:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:17:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:17:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:17:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:18:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:18:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:18:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:18:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:19:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:19:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:19:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:19:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:20:01 lola-4 CROND[51225]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 00:20:01 lola-4 CROND[51226]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 00:20:01 lola-4 TIME: Time stamp for console Nov 6 00:20:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:20:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:20:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:20:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:21:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:21:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:21:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:21:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:22:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:22:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:22:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:22:39 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 00:22:39 lola-4 kernel: req@ffff88052f9e3850 x1550005401079776/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478416964 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 00:22:39 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 6 00:22:45 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 00:22:45 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 00:22:45 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 00:22:45 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 00:22:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:23:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:23:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:23:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:23:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:24:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:24:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:24:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:24:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:25:01 lola-4 CROND[51262]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 00:25:01 lola-4 TIME: Time stamp for console Nov 6 00:25:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:25:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:25:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:25:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:26:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:26:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:26:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:26:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:27:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:27:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:27:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:27:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:28:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:28:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:28:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:28:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:29:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:29:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:29:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:29:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:30:01 lola-4 CROND[51296]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 00:30:01 lola-4 CROND[51297]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 00:30:01 lola-4 TIME: Time stamp for console Nov 6 00:30:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:30:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:30:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:30:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:31:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:31:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:31:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:31:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:32:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:32:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:32:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:32:40 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 00:32:40 lola-4 kernel: req@ffff8806d110c850 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478417565 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 00:32:40 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 6 00:32:46 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 00:32:46 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 6 00:32:46 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 00:32:46 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 6 00:32:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:33:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:33:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:33:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:33:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:34:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:34:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:34:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:34:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:35:01 lola-4 CROND[51333]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 00:35:01 lola-4 TIME: Time stamp for console Nov 6 00:35:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:35:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:35:32 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:35:47 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:36:02 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:36:17 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:36:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:36:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:37:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:37:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:37:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:37:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:38:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:38:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:38:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:38:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:39:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:39:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:39:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:39:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:40:01 lola-4 CROND[51367]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 00:40:01 lola-4 TIME: Time stamp for console Nov 6 00:40:01 lola-4 CROND[51368]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 00:40:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:40:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:40:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:40:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:41:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:41:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:41:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:41:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:42:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:42:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:42:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:42:41 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 00:42:41 lola-4 kernel: req@ffff88073b415c50 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478418166 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 00:42:41 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 6 00:42:47 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 00:42:47 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 00:42:47 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 00:42:47 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 00:42:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:43:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:43:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:43:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:43:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:44:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:44:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:44:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:44:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:45:01 lola-4 CROND[51404]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 00:45:01 lola-4 TIME: Time stamp for console Nov 6 00:45:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:45:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:45:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:45:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:46:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:46:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:46:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:46:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:47:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:47:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:47:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:47:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:48:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:48:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:48:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:48:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:49:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:49:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:49:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:49:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:50:01 lola-4 CROND[51438]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 00:50:01 lola-4 CROND[51439]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 00:50:01 lola-4 TIME: Time stamp for console Nov 6 00:50:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:50:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:50:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:50:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:51:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:51:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:51:33 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:51:48 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:52:03 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:52:18 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:52:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:52:42 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 00:52:42 lola-4 kernel: req@ffff8807b5f8c050 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478418767 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 00:52:42 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 6 00:52:48 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 00:52:48 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 00:52:48 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 00:52:48 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 00:52:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:53:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:53:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 00:53:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 00:53:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:54:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:54:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 00:54:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:54:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 00:55:01 lola-4 CROND[51476]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 00:55:01 lola-4 TIME: Time stamp for console Nov 6 00:55:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:55:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 00:55:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:55:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:56:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:56:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:56:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:56:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:57:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:57:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:57:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:57:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:58:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:58:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:58:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:58:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:59:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:59:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:59:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 00:59:49 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:00:01 lola-4 CROND[51511]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:00:01 lola-4 TIME: Time stamp for console Nov 6 01:00:01 lola-4 CROND[51512]: (root) CMD (/usr/sbin/raid-check) Nov 6 01:00:01 lola-4 CROND[51513]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 01:00:04 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:00:19 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:00:34 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 6 01:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 6 01:00:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:01:01 lola-4 CROND[51527]: (root) CMD (run-parts /etc/cron.hourly) Nov 6 01:01:01 lola-4 run-parts(/etc/cron.hourly)[5152 starting 0anacron Nov 6 01:01:01 lola-4 anacron[51538]: Anacron started on 2016-11-06 Nov 6 01:01:01 lola-4 anacron[51538]: Jobs will be executed sequentially Nov 6 01:01:01 lola-4 anacron[51538]: Normal exit (0 jobs run) Nov 6 01:01:01 lola-4 run-parts(/etc/cron.hourly)[5154 finished 0anacron Nov 6 01:01:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:01:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:01:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:01:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:02:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:02:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:02:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:02:43 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 01:02:43 lola-4 kernel: req@ffff8805237c5450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478419368 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 01:02:43 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 6 01:02:49 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 01:02:49 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 6 01:02:49 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 01:02:49 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 6 01:02:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:03:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:03:20 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:03:35 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:03:50 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:04:05 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:04:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:04:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:04:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:05:01 lola-4 CROND[51567]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:05:01 lola-4 TIME: Time stamp for console Nov 6 01:05:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:05:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:05:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:05:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:06:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:06:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:06:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:06:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:07:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:07:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:07:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:07:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:08:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:08:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:08:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:08:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:09:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:09:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:09:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:09:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:10:01 lola-4 CROND[51601]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:10:01 lola-4 CROND[51602]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 01:10:01 lola-4 TIME: Time stamp for console Nov 6 01:10:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:10:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:10:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:10:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:11:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:11:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:11:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:11:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:12:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:12:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:12:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:12:44 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 01:12:44 lola-4 kernel: req@ffff88050c7f0450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478419969 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 01:12:44 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 6 01:12:50 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 01:12:50 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 01:12:50 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 01:12:50 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 01:12:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:13:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:13:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:13:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:13:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:14:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:14:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:14:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:14:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:15:01 lola-4 CROND[51638]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:15:01 lola-4 TIME: Time stamp for console Nov 6 01:15:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:15:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:15:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:15:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:16:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:16:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:16:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:16:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:17:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:17:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:17:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:17:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:18:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:18:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:18:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:18:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:19:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:19:21 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:19:36 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:19:51 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:20:01 lola-4 CROND[51672]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 01:20:01 lola-4 CROND[51673]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:20:01 lola-4 TIME: Time stamp for console Nov 6 01:20:06 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:20:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:20:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:20:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:21:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:21:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:21:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:21:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:22:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:22:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:22:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:22:45 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 01:22:45 lola-4 kernel: req@ffff88057521a450 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478420570 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 01:22:45 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 104 previous similar messages Nov 6 01:22:51 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 01:22:51 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 6 01:22:51 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 01:22:51 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 6 01:22:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:23:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:23:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:23:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:23:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:24:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:24:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:24:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:24:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:25:01 lola-4 CROND[51710]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:25:01 lola-4 TIME: Time stamp for console Nov 6 01:25:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:25:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:25:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:25:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:26:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:26:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:26:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:26:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:27:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:27:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:27:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:27:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:28:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:28:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:28:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:28:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:29:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:29:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:29:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:29:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:30:02 lola-4 CROND[51744]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:30:02 lola-4 CROND[51745]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 01:30:02 lola-4 TIME: Time stamp for console Nov 6 01:30:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:30:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:30:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:30:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:31:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:31:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:31:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:31:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:32:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:32:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:32:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:32:46 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 01:32:46 lola-4 kernel: req@ffff8804f2550450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478421171 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 01:32:46 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 6 01:32:52 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 01:32:52 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 01:32:52 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 01:32:52 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 01:32:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:33:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:33:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:33:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:33:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:34:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:34:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:34:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:34:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:35:01 lola-4 CROND[51781]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:35:01 lola-4 TIME: Time stamp for console Nov 6 01:35:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:35:22 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:35:37 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:35:52 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:36:07 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:36:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:36:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:36:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:37:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:37:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:37:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:37:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:38:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:38:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:38:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:38:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:39:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:39:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:39:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:39:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:40:01 lola-4 CROND[51815]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 01:40:01 lola-4 CROND[51816]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:40:01 lola-4 TIME: Time stamp for console Nov 6 01:40:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:40:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:40:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:40:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:41:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:41:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:41:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:41:38 lola-4 sendmail[10625]: have been rejecting connections on daemon MTA for 1+09:02:45 Nov 6 01:41:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:42:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:42:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:42:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:42:47 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 01:42:47 lola-4 kernel: req@ffff8805193c5850 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478421772 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 01:42:47 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 6 01:42:53 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 01:42:53 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 01:42:53 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 01:42:53 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 01:42:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:43:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:43:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:43:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:43:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:44:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:44:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:44:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:44:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:45:01 lola-4 CROND[51852]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:45:01 lola-4 TIME: Time stamp for console Nov 6 01:45:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:45:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:45:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:45:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:46:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:46:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:46:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:46:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:47:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:47:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:47:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:47:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:48:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:48:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:48:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:48:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:49:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:49:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:49:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:49:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:50:01 lola-4 CROND[51886]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:50:01 lola-4 CROND[51887]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 01:50:01 lola-4 TIME: Time stamp for console Nov 6 01:50:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:50:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:50:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:50:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:51:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:51:23 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:51:38 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:51:53 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:52:08 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:52:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:52:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:52:48 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 01:52:48 lola-4 kernel: req@ffff8804cd073450 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478422373 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 01:52:48 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 6 01:52:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:52:54 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 01:52:54 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 6 01:52:54 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 01:52:54 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 6 01:53:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:53:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:53:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:53:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:54:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:54:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:54:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:54:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:55:01 lola-4 CROND[51923]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:55:01 lola-4 TIME: Time stamp for console Nov 6 01:55:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:55:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:55:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:55:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:56:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:56:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:56:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:56:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:57:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:57:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:57:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:57:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:58:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:58:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:58:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:58:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:59:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:59:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:59:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:59:54 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:00:01 lola-4 CROND[51958]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:00:01 lola-4 CROND[51959]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 01:00:01 lola-4 TIME: Time stamp for console Nov 6 01:00:09 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:00:24 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:00:39 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 6 01:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 6 01:00:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:01:01 lola-4 CROND[51970]: (root) CMD (run-parts /etc/cron.hourly) Nov 6 01:01:01 lola-4 run-parts(/etc/cron.hourly)[5197 starting 0anacron Nov 6 01:01:01 lola-4 anacron[51981]: Anacron started on 2016-11-06 Nov 6 01:01:01 lola-4 anacron[51981]: Jobs will be executed sequentially Nov 6 01:01:01 lola-4 anacron[51981]: Normal exit (0 jobs run) Nov 6 01:01:01 lola-4 run-parts(/etc/cron.hourly)[5198 finished 0anacron Nov 6 01:01:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:01:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:01:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:01:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:02:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:02:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:02:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:02:49 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 01:02:49 lola-4 kernel: req@ffff8807f7a42450 x1550005401097664/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478422974 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 01:02:49 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 6 01:02:55 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 01:02:55 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 01:02:55 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 01:02:55 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 01:02:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:03:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:03:25 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:03:40 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:03:55 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:04:10 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:04:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:04:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:04:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:05:01 lola-4 CROND[52010]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:05:01 lola-4 TIME: Time stamp for console Nov 6 01:05:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:05:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:05:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:05:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:06:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:06:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:06:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:06:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:07:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:07:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:07:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:07:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:08:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:08:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:08:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:08:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:09:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:09:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:09:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:09:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:10:01 lola-4 CROND[52044]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 01:10:01 lola-4 CROND[52045]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:10:01 lola-4 TIME: Time stamp for console Nov 6 01:10:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:10:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:10:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:10:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:11:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:11:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:11:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:11:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:12:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:12:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:12:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:12:50 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 01:12:50 lola-4 kernel: req@ffff8804d0065c50 x1550005401079776/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478423575 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 01:12:50 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 105 previous similar messages Nov 6 01:12:56 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 01:12:56 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 6 01:12:56 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 01:12:56 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 6 01:12:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:13:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:13:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:13:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:13:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:14:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:14:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:14:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:14:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:15:01 lola-4 CROND[52081]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:15:01 lola-4 TIME: Time stamp for console Nov 6 01:15:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:15:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:15:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:15:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:16:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:16:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:16:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:16:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:17:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:17:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:17:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:17:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:18:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:18:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:18:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:18:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:19:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:19:26 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:19:41 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:19:56 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:20:01 lola-4 CROND[52115]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:20:01 lola-4 CROND[52116]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 01:20:01 lola-4 TIME: Time stamp for console Nov 6 01:20:11 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:20:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:20:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:20:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:21:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:21:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:21:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:21:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:22:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:22:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:22:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:22:51 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 01:22:51 lola-4 kernel: req@ffff880767e4a450 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478424176 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 01:22:51 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 6 01:22:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:22:57 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 01:22:57 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 01:22:57 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 01:22:57 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 01:23:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:23:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:23:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:23:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:24:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:24:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:24:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:24:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:25:01 lola-4 CROND[52152]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:25:01 lola-4 TIME: Time stamp for console Nov 6 01:25:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:25:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:25:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:25:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:26:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:26:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:26:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:26:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:27:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:27:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:27:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:27:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:28:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:28:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:28:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:28:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:29:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:29:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:29:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:29:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:30:01 lola-4 CROND[52186]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 01:30:01 lola-4 CROND[52187]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:30:01 lola-4 TIME: Time stamp for console Nov 6 01:30:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:30:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:30:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:30:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:31:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:31:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:31:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:31:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:32:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:32:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:32:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:32:52 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 01:32:52 lola-4 kernel: req@ffff88054485c450 x1550005401083984/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478424777 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 01:32:52 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 6 01:32:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:32:58 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 01:32:58 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 6 01:32:58 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 01:32:58 lola-4 kernel: Lustre: Skipped 26 previous similar messages Nov 6 01:33:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:33:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:33:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:33:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:34:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:34:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:34:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:34:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:35:01 lola-4 CROND[52224]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:35:01 lola-4 TIME: Time stamp for console Nov 6 01:35:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:35:27 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:35:42 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:35:57 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:36:12 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:36:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:36:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:36:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:37:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:37:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:37:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:37:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:38:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:38:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:38:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:38:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:39:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:39:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:39:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:39:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:40:01 lola-4 CROND[52258]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:40:01 lola-4 CROND[52259]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 01:40:01 lola-4 TIME: Time stamp for console Nov 6 01:40:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:40:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:40:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:40:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:41:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:41:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:41:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:41:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:42:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:42:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:42:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:42:53 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 01:42:53 lola-4 kernel: req@ffff8804e9880050 x1550005401082960/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478425378 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 01:42:53 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 112 previous similar messages Nov 6 01:42:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:42:59 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 01:42:59 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 01:42:59 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 01:42:59 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 01:43:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:43:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:43:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:43:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:44:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:44:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:44:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:44:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:45:01 lola-4 CROND[52295]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:45:01 lola-4 TIME: Time stamp for console Nov 6 01:45:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:45:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:45:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:45:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:46:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:46:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:46:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:46:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:47:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:47:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:47:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:47:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:48:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:48:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:48:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:48:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:49:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:49:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:49:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:49:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:50:01 lola-4 CROND[52329]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 01:50:01 lola-4 CROND[52330]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:50:01 lola-4 TIME: Time stamp for console Nov 6 01:50:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:50:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:50:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:50:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:51:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:51:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:51:43 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:51:58 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:52:13 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:52:28 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:52:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:52:54 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 01:52:54 lola-4 kernel: req@ffff880512d42450 x1550005401079776/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478425979 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 01:52:54 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 110 previous similar messages Nov 6 01:52:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:53:00 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 01:53:00 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 01:53:00 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 01:53:00 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 01:53:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:53:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:53:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:53:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:54:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:54:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:54:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:54:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:55:01 lola-4 CROND[52366]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 01:55:01 lola-4 TIME: Time stamp for console Nov 6 01:55:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:55:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:55:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:55:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:56:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:56:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 01:56:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:56:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:57:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:57:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:57:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:57:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:58:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:58:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:58:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:58:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:59:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:59:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:59:44 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 01:59:59 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:00:01 lola-4 CROND[52400]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 02:00:01 lola-4 CROND[52401]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 02:00:01 lola-4 TIME: Time stamp for console Nov 6 02:00:14 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:00:29 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:00:40 lola-4 sendmail[10625]: runqueue: Skipping queue run -- load average too high Nov 6 02:00:40 lola-4 sm-msp-queue[10634]: runqueue: Skipping queue run -- load average too high Nov 6 02:00:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:01:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:01:01 lola-4 CROND[52412]: (root) CMD (run-parts /etc/cron.hourly) Nov 6 02:01:01 lola-4 run-parts(/etc/cron.hourly)[5241 starting 0anacron Nov 6 02:01:01 lola-4 anacron[52423]: Anacron started on 2016-11-06 Nov 6 02:01:01 lola-4 anacron[52423]: Jobs will be executed sequentially Nov 6 02:01:01 lola-4 anacron[52423]: Normal exit (0 jobs run) Nov 6 02:01:01 lola-4 run-parts(/etc/cron.hourly)[5242 finished 0anacron Nov 6 02:01:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:01:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:01:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:02:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:02:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:02:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:02:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:02:55 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 02:02:55 lola-4 kernel: req@ffff88073caa6c50 x1550005401096640/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478426580 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 02:02:55 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 106 previous similar messages Nov 6 02:03:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:03:01 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 02:03:01 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 6 02:03:01 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 02:03:01 lola-4 kernel: Lustre: Skipped 23 previous similar messages Nov 6 02:03:15 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:03:30 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:03:45 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:04:00 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:04:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:04:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:04:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:05:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:05:01 lola-4 CROND[52453]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 02:05:01 lola-4 TIME: Time stamp for console Nov 6 02:05:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:05:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:05:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:06:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:06:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:06:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:06:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:07:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:07:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:07:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:07:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:08:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:08:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:08:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:08:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:09:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:09:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:09:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:09:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:10:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:10:01 lola-4 CROND[52487]: (root) CMD (/usr/lib64/sa/sa1 1 1) Nov 6 02:10:01 lola-4 CROND[52488]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 02:10:01 lola-4 TIME: Time stamp for console Nov 6 02:10:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:10:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:10:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:11:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:11:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:11:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:11:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:12:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:12:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:12:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:12:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:12:56 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Nov 6 02:12:56 lola-4 kernel: req@ffff8805fee5d450 x1550005401085472/t0(0) o1000->soaked-MDT0000-mdtlov_UUID@192.168.1.108@o2ib10:-1/-1 lens 520/0 e 0 to 0 dl 1478427181 ref 2 fl New:/2/ffffffff rc 0/-1 Nov 6 02:12:56 lola-4 kernel: Lustre: 33277:0:(service.c:1331:ptlrpc_at_send_early_reply()) Skipped 111 previous similar messages Nov 6 02:13:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:13:02 lola-4 kernel: Lustre: soaked-OST0002: Client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) reconnecting Nov 6 02:13:02 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 02:13:02 lola-4 kernel: Lustre: soaked-OST0002: Connection restored to soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib10) Nov 6 02:13:02 lola-4 kernel: Lustre: Skipped 27 previous similar messages Nov 6 02:13:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:13:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:13:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 487 Nov 6 02:14:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:14:16 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:14:31 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:14:46 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488 Nov 6 02:15:01 lola-4 CROND[52524]: (root) CMD (logger -p local0.notice -t TIME "Time stamp for console") Nov 6 02:15:01 lola-4 TIME: Time stamp for console Nov 6 02:15:01 lola-4 sendmail[10625]: rejecting connections on daemon MTA: load average: 488