Mar  7 12:30:19 sprig3 syslog-ng[2161]: Configuration reload request received, reloading configuration;
Mar  7 12:30:19 sprig3 syslog-ng[2161]: New configuration initialized;
Mar  7 12:34:52 sprig3 sshd[231906]: Authorized to user2, krb5 principal user2@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 12:34:52 sprig3 sshd[231906]: Accepted gssapi-with-mic for user2 from 111.222.34.217 port 57643 ssh2
Mar  7 12:35:01 sprig3 /usr/sbin/cron[231976]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 12:35:01 sprig3 /usr/sbin/cron[231977]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 12:35:01 sprig3 /usr/sbin/cron[231978]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 12:40:01 sprig3 /usr/sbin/cron[232025]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 12:40:01 sprig3 /usr/sbin/cron[232026]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 12:40:01 sprig3 /usr/sbin/cron[232027]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 12:45:01 sprig3 /usr/sbin/cron[232137]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 12:45:01 sprig3 /usr/sbin/cron[232138]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 12:45:01 sprig3 /usr/sbin/cron[232140]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 12:50:01 sprig3 /usr/sbin/cron[232234]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 12:50:01 sprig3 /usr/sbin/cron[232235]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 12:50:01 sprig3 /usr/sbin/cron[232236]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 12:55:01 sprig3 /usr/sbin/cron[232282]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 12:55:01 sprig3 /usr/sbin/cron[232283]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 12:55:01 sprig3 /usr/sbin/cron[232284]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 13:00:01 sprig3 /usr/sbin/cron[232341]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 13:00:01 sprig3 /usr/sbin/cron[232342]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 13:00:01 sprig3 /usr/sbin/cron[232344]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 13:05:02 sprig3 /usr/sbin/cron[232411]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 13:05:02 sprig3 /usr/sbin/cron[232412]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 13:05:02 sprig3 /usr/sbin/cron[232413]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 13:08:50 sprig3 sshd[232460]: Accepted publickey for root from 11.1.0.1 port 38827 ssh2
Mar  7 13:10:01 sprig3 /usr/sbin/cron[232509]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 13:10:01 sprig3 /usr/sbin/cron[232510]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 13:10:01 sprig3 /usr/sbin/cron[232511]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 13:15:01 sprig3 /usr/sbin/cron[232557]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 13:15:01 sprig3 /usr/sbin/cron[232559]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 13:15:01 sprig3 /usr/sbin/cron[232560]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 13:20:01 sprig3 /usr/sbin/cron[232625]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 13:20:01 sprig3 /usr/sbin/cron[232626]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 13:20:01 sprig3 /usr/sbin/cron[232627]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 13:25:01 sprig3 /usr/sbin/cron[232670]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 13:25:01 sprig3 /usr/sbin/cron[232671]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 13:25:01 sprig3 /usr/sbin/cron[232672]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 13:30:01 sprig3 /usr/sbin/cron[232722]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 13:30:01 sprig3 /usr/sbin/cron[232723]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 13:30:01 sprig3 /usr/sbin/cron[232724]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 13:30:19 sprig3 syslog-ng[2161]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=41', processed='center(received)=41', processed='destination(messages)=41', processed='destination(mailinfo)=0', processed='destination(mailwarn)=0', processed='destination(localmessages)=0', processed='destination(newserr)=0', processed='destination(mailerr)=0', processed='destination(netmgm)=0', processed='destination(warn)=0', processed='destination(console)=0', processed='destination(null)=0', processed='destination(mail)=0', processed='destination(xconsole)=0', processed='destination(firewall)=0', processed='destination(acpid)=0', processed='destination(newscrit)=0', processed='destination(newsnotice)=0', processed='source(src)=41'
Mar  7 13:35:01 sprig3 /usr/sbin/cron[232786]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 13:35:01 sprig3 /usr/sbin/cron[232787]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 13:35:01 sprig3 /usr/sbin/cron[232788]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 13:40:01 sprig3 /usr/sbin/cron[232836]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 13:40:01 sprig3 /usr/sbin/cron[232837]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 13:40:01 sprig3 /usr/sbin/cron[232838]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 13:44:38 sprig3 sshd[232879]: Connection closed by 111.33.10.15 [preauth]
Mar  7 13:44:47 sprig3 sshd[232881]: Connection closed by 111.33.10.15 [preauth]
Mar  7 13:44:57 sprig3 sshd[232883]: Connection closed by 111.33.10.15 [preauth]
Mar  7 13:45:01 sprig3 /usr/sbin/cron[232889]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 13:45:01 sprig3 /usr/sbin/cron[232892]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 13:45:01 sprig3 /usr/sbin/cron[232891]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 13:45:08 sprig3 sshd[232943]: Authorized to user1, krb5 principal user1@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 13:45:08 sprig3 sshd[232943]: Accepted gssapi-with-mic for user1 from 111.33.10.15 port 41121 ssh2
Mar  7 13:45:10 sprig3 sshd[232945]: Received disconnect from 111.33.10.15: 11: disconnected by user
Mar  7 13:45:15 sprig3 sshd[233007]: Connection closed by 111.33.10.15 [preauth]
Mar  7 13:47:05 sprig3 sshd[233017]: Authorized to user1, krb5 principal user1@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 13:47:05 sprig3 sshd[233017]: Accepted gssapi-with-mic for user1 from 111.33.10.15 port 41138 ssh2
Mar  7 13:47:07 sprig3 sshd[233019]: Received disconnect from 111.33.10.15: 11: disconnected by user
Mar  7 13:47:09 sprig3 sshd[233080]: Authorized to user1, krb5 principal user1@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 13:47:09 sprig3 sshd[233080]: Accepted gssapi-with-mic for user1 from 111.33.10.15 port 41139 ssh2
Mar  7 13:47:15 sprig3 sshd[233082]: Received disconnect from 111.33.10.15: 11: disconnected by user
Mar  7 13:47:17 sprig3 sshd[233147]: Authorized to user1, krb5 principal user1@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 13:47:17 sprig3 sshd[233147]: Accepted gssapi-with-mic for user1 from 111.33.10.15 port 41140 ssh2
Mar  7 13:47:18 sprig3 sshd[233149]: Received disconnect from 111.33.10.15: 11: disconnected by user
Mar  7 13:47:19 sprig3 sshd[233210]: Authorized to user1, krb5 principal user1@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 13:47:19 sprig3 sshd[233210]: Accepted gssapi-with-mic for user1 from 111.33.10.15 port 41141 ssh2
Mar  7 13:47:21 sprig3 sshd[233212]: Received disconnect from 111.33.10.15: 11: disconnected by user
Mar  7 13:47:22 sprig3 sshd[233273]: Authorized to user1, krb5 principal user1@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 13:47:22 sprig3 sshd[233273]: Accepted gssapi-with-mic for user1 from 111.33.10.15 port 41142 ssh2
Mar  7 13:47:23 sprig3 sshd[233275]: Received disconnect from 111.33.10.15: 11: disconnected by user
Mar  7 13:47:30 sprig3 sshd[233336]: Authorized to user1, krb5 principal user1@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 13:47:30 sprig3 sshd[233336]: Accepted gssapi-with-mic for user1 from 111.33.10.15 port 41148 ssh2
Mar  7 13:47:31 sprig3 sshd[233338]: Received disconnect from 111.33.10.15: 11: disconnected by user
Mar  7 13:47:33 sprig3 sshd[233399]: Authorized to user1, krb5 principal user1@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 13:47:33 sprig3 sshd[233399]: Accepted gssapi-with-mic for user1 from 111.33.10.15 port 41152 ssh2
Mar  7 13:47:34 sprig3 sshd[233401]: Received disconnect from 111.33.10.15: 11: disconnected by user
Mar  7 13:47:35 sprig3 sshd[233462]: Authorized to user1, krb5 principal user1@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 13:47:35 sprig3 sshd[233462]: Accepted gssapi-with-mic for user1 from 111.33.10.15 port 41153 ssh2
Mar  7 13:47:36 sprig3 sshd[233464]: Received disconnect from 111.33.10.15: 11: disconnected by user
Mar  7 13:47:36 sprig3 sshd[233525]: Authorized to user1, krb5 principal user1@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 13:47:36 sprig3 sshd[233525]: Accepted gssapi-with-mic for user1 from 111.33.10.15 port 41154 ssh2
Mar  7 13:47:37 sprig3 sshd[233527]: Received disconnect from 111.33.10.15: 11: disconnected by user
Mar  7 13:47:38 sprig3 sshd[233588]: Authorized to user1, krb5 principal user1@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 13:47:38 sprig3 sshd[233588]: Accepted gssapi-with-mic for user1 from 111.33.10.15 port 41155 ssh2
Mar  7 13:47:39 sprig3 sshd[233590]: Received disconnect from 111.33.10.15: 11: disconnected by user
Mar  7 13:48:33 sprig3 sshd[233654]: Authorized to user1, krb5 principal user1@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 13:48:33 sprig3 sshd[233654]: Accepted gssapi-with-mic for user1 from 111.33.10.15 port 41164 ssh2
Mar  7 13:50:01 sprig3 /usr/sbin/cron[233740]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 13:50:01 sprig3 /usr/sbin/cron[233741]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 13:50:01 sprig3 /usr/sbin/cron[233742]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 13:55:01 sprig3 /usr/sbin/cron[233786]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 13:55:01 sprig3 /usr/sbin/cron[233787]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 13:55:01 sprig3 /usr/sbin/cron[233788]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 14:00:01 sprig3 /usr/sbin/cron[233868]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 14:00:01 sprig3 /usr/sbin/cron[233869]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 14:00:01 sprig3 /usr/sbin/cron[233870]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 14:00:01 sprig3 sshd[233656]: Received disconnect from 111.33.10.15: 11: disconnected by user
Mar  7 14:00:03 sprig3 sshd[233921]: Authorized to user1, krb5 principal user1@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 14:00:03 sprig3 sshd[233921]: Accepted gssapi-with-mic for user1 from 111.33.10.15 port 41263 ssh2
Mar  7 14:00:05 sprig3 sshd[233923]: Received disconnect from 111.33.10.15: 11: disconnected by user
Mar  7 14:05:01 sprig3 /usr/sbin/cron[234005]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 14:05:01 sprig3 /usr/sbin/cron[234006]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 14:05:01 sprig3 /usr/sbin/cron[234007]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 14:05:23 sprig3 sshd[234040]: Accepted publickey for root from 11.1.0.1 port 40392 ssh2
Mar  7 14:05:23 sprig3 sshd[234040]: Received disconnect from 11.1.0.1: 11: disconnected by user
Mar  7 14:05:23 sprig3 sshd[234081]: Accepted publickey for root from 11.1.0.1 port 40396 ssh2
Mar  7 14:05:24 sprig3 sshd[234081]: Received disconnect from 11.1.0.1: 11: disconnected by user
Mar  7 14:10:01 sprig3 /usr/sbin/cron[234155]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 14:10:01 sprig3 /usr/sbin/cron[234156]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 14:10:01 sprig3 /usr/sbin/cron[234157]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 14:14:36 sprig3 intelremotemond[234349]: Main: daemon started
Mar  7 14:14:36 sprig3 intelremotemond[234349]: config file is /scratch/awe/opt/intel/ism/bin/intel64/../intelremotemond.conf
Mar  7 14:14:36 sprig3 intelremotemond[234349]: timeout value is 600
Mar  7 14:14:36 sprig3 intelremotemond[234349]: flushtime value is 10
Mar  7 14:14:36 sprig3 intelremotemond[234349]: limit value is 1000
Mar  7 14:14:36 sprig3 intelremotemond[234349]: stop value is 100000
Mar  7 14:14:36 sprig3 intelremotemond[234349]: Consumer: total events at start is 3
Mar  7 14:15:01 sprig3 /usr/sbin/cron[237191]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 14:15:01 sprig3 /usr/sbin/cron[237193]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 14:15:01 sprig3 /usr/sbin/cron[237194]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 14:16:35 sprig3 sshd[1345]: Accepted publickey for root from 11.1.0.1 port 40714 ssh2
Mar  7 14:20:01 sprig3 /usr/sbin/cron[4459]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 14:20:01 sprig3 /usr/sbin/cron[4460]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 14:20:01 sprig3 /usr/sbin/cron[4461]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 14:25:01 sprig3 /usr/sbin/cron[4514]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 14:25:01 sprig3 /usr/sbin/cron[4515]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 14:25:01 sprig3 /usr/sbin/cron[4516]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 14:27:36 sprig3 intelremotemond[234349]: Consumer: timeout, exiting...
Mar  7 14:27:36 sprig3 intelremotemond[4558]: SendDataToBackend: path to agent is /opt/intel/ism/bin/intel64/intelremotemonagent
Mar  7 14:27:36 sprig3 intelremotemond[234349]: Main: try to delete semaphore, status 0
Mar  7 14:27:36 sprig3 intelremotemond[234349]: Main: daemon stopped, force is 0
Mar  7 14:27:36 sprig3 intelremotemond[4558]: SendDataToBackend: start agent
Mar  7 14:29:19 sprig3 sshd[4563]: Authorized to user3, krb5 principal user3@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 14:29:19 sprig3 sshd[4563]: Accepted gssapi-with-mic for user3 from 111.222.39.189 port 34054 ssh2
Mar  7 14:30:01 sprig3 /usr/sbin/cron[4632]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 14:30:01 sprig3 /usr/sbin/cron[4634]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 14:30:01 sprig3 /usr/sbin/cron[4635]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 14:30:11 sprig3 intelremotemond[4788]: Main: daemon started
Mar  7 14:30:11 sprig3 intelremotemond[4788]: config file is /scratch/awe/opt/intel/ism/bin/intel64/../intelremotemond.conf
Mar  7 14:30:11 sprig3 intelremotemond[4788]: timeout value is 600
Mar  7 14:30:11 sprig3 intelremotemond[4788]: flushtime value is 10
Mar  7 14:30:11 sprig3 intelremotemond[4788]: limit value is 1000
Mar  7 14:30:11 sprig3 intelremotemond[4788]: stop value is 100000
Mar  7 14:30:11 sprig3 intelremotemond[4788]: Consumer: total events at start is 3
Mar  7 14:30:19 sprig3 syslog-ng[2161]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=155', processed='center(received)=147', processed='destination(messages)=147', processed='destination(mailinfo)=0', processed='destination(mailwarn)=0', processed='destination(localmessages)=0', processed='destination(newserr)=0', processed='destination(mailerr)=0', processed='destination(netmgm)=0', processed='destination(warn)=8', processed='destination(console)=0', processed='destination(null)=0', processed='destination(mail)=0', processed='destination(xconsole)=0', processed='destination(firewall)=0', processed='destination(acpid)=0', processed='destination(newscrit)=0', processed='destination(newsnotice)=0', processed='source(src)=147'
Mar  7 14:35:01 sprig3 /usr/sbin/cron[20295]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 14:35:01 sprig3 /usr/sbin/cron[20294]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 14:35:01 sprig3 /usr/sbin/cron[20296]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 14:36:16 sprig3 sshd[20333]: Authorized to user3, krb5 principal user3@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 14:36:16 sprig3 sshd[20333]: Accepted gssapi-with-mic for user3 from 111.222.39.189 port 50601 ssh2
Mar  7 14:40:01 sprig3 /usr/sbin/cron[20405]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 14:40:01 sprig3 /usr/sbin/cron[20406]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 14:40:01 sprig3 /usr/sbin/cron[20407]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 14:43:08 sprig3 intelremotemond[4788]: Consumer: timeout, exiting...
Mar  7 14:43:08 sprig3 intelremotemond[20448]: SendDataToBackend: path to agent is /opt/intel/ism/bin/intel64/intelremotemonagent
Mar  7 14:43:08 sprig3 intelremotemond[20448]: SendDataToBackend: start agent
Mar  7 14:43:08 sprig3 intelremotemond[4788]: Main: try to delete semaphore, status 0
Mar  7 14:43:08 sprig3 intelremotemond[4788]: Main: daemon stopped, force is 0
Mar  7 14:45:01 sprig3 /usr/sbin/cron[20465]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 14:45:01 sprig3 /usr/sbin/cron[20466]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 14:45:01 sprig3 /usr/sbin/cron[20467]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 14:47:20 sprig3 sshd[20534]: Connection closed by 111.222.16.12 [preauth]
Mar  7 14:50:01 sprig3 /usr/sbin/cron[20543]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 14:50:01 sprig3 /usr/sbin/cron[20544]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 14:50:01 sprig3 /usr/sbin/cron[20545]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 14:50:32 sprig3 sshd[20578]: Connection closed by 111.222.16.12 [preauth]
Mar  7 14:54:34 sprig3 sshd[20588]: Accepted publickey for root from 11.1.0.8 port 47079 ssh2
Mar  7 14:54:40 sprig3 sshd[20588]: Received disconnect from 11.1.0.8: 11: disconnected by user
Mar  7 14:55:01 sprig3 /usr/sbin/cron[20638]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 14:55:01 sprig3 /usr/sbin/cron[20639]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 14:55:01 sprig3 /usr/sbin/cron[20640]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 15:00:01 sprig3 /usr/sbin/cron[20691]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 15:00:01 sprig3 /usr/sbin/cron[20689]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 15:00:01 sprig3 /usr/sbin/cron[20692]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 15:05:01 sprig3 /usr/sbin/cron[20758]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 15:05:01 sprig3 /usr/sbin/cron[20757]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 15:05:01 sprig3 /usr/sbin/cron[20759]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 15:10:01 sprig3 /usr/sbin/cron[20808]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 15:10:01 sprig3 /usr/sbin/cron[20807]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 15:10:01 sprig3 /usr/sbin/cron[20809]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 15:13:40 sprig3 sshd[20851]: Authorized to user4, krb5 principal user4@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 15:13:40 sprig3 sshd[20851]: Accepted gssapi-with-mic for user4 from 111.222.38.165 port 34026 ssh2
Mar  7 15:13:44 sprig3 sshd[20853]: Received disconnect from 111.222.38.165: 11: disconnected by user
Mar  7 15:13:45 sprig3 sshd[20923]: Authorized to user4, krb5 principal user4@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 15:13:45 sprig3 sshd[20923]: Accepted gssapi-with-mic for user4 from 111.222.38.165 port 34027 ssh2
Mar  7 15:14:25 sprig3 sshd[232460]: Received disconnect from 11.1.0.1: 11: disconnected by user
Mar  7 15:14:59 sprig3 intelremotemond[21616]: Main: daemon started
Mar  7 15:14:59 sprig3 intelremotemond[21616]: config file is /scratch/awe/opt/intel/ism/bin/intel64/../intelremotemond.conf
Mar  7 15:14:59 sprig3 intelremotemond[21616]: timeout value is 600
Mar  7 15:14:59 sprig3 intelremotemond[21616]: flushtime value is 10
Mar  7 15:14:59 sprig3 intelremotemond[21616]: limit value is 1000
Mar  7 15:14:59 sprig3 intelremotemond[21616]: stop value is 100000
Mar  7 15:14:59 sprig3 intelremotemond[21616]: Consumer: total events at start is 0
Mar  7 15:15:01 sprig3 /usr/sbin/cron[21826]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 15:15:01 sprig3 /usr/sbin/cron[21828]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 15:15:01 sprig3 /usr/sbin/cron[21829]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 15:20:01 sprig3 /usr/sbin/cron[25862]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 15:20:01 sprig3 /usr/sbin/cron[25861]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 15:20:01 sprig3 /usr/sbin/cron[25863]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 15:25:01 sprig3 /usr/sbin/cron[27396]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 15:25:01 sprig3 /usr/sbin/cron[27397]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 15:25:01 sprig3 /usr/sbin/cron[27398]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 15:30:01 sprig3 /usr/sbin/cron[27485]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 15:30:01 sprig3 /usr/sbin/cron[27487]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 15:30:01 sprig3 /usr/sbin/cron[27488]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 15:30:19 sprig3 syslog-ng[2161]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=220', processed='center(received)=208', processed='destination(messages)=208', processed='destination(mailinfo)=0', processed='destination(mailwarn)=0', processed='destination(localmessages)=0', processed='destination(newserr)=0', processed='destination(mailerr)=0', processed='destination(netmgm)=0', processed='destination(warn)=12', processed='destination(console)=0', processed='destination(null)=0', processed='destination(mail)=0', processed='destination(xconsole)=0', processed='destination(firewall)=0', processed='destination(acpid)=0', processed='destination(newscrit)=0', processed='destination(newsnotice)=0', processed='source(src)=208'
Mar  7 15:30:56 sprig3 intelremotemond[21616]: Consumer: timeout, exiting...
Mar  7 15:30:56 sprig3 intelremotemond[27543]: SendDataToBackend: path to agent is /opt/intel/ism/bin/intel64/intelremotemonagent
Mar  7 15:30:56 sprig3 intelremotemond[21616]: Main: try to delete semaphore, status 0
Mar  7 15:30:56 sprig3 intelremotemond[27543]: SendDataToBackend: start agent
Mar  7 15:30:56 sprig3 intelremotemond[21616]: Main: daemon stopped, force is 0
Mar  7 15:30:58 sprig3 kernel: [1561313.377038] LustreError: 27353:0:(osc_cache.c:897:osc_extent_wait()) extent ffff881adab76e58@{[0 -> 1/255], [3|0|+|rpc|wihY|ffff881a6ecad888], [8192|2|+|-|ffff881b88745938|256|ffff8814ca670540]} scratch-OST0008-osc-ffff883f77c95800: wait ext to 0 timedout, recovery in progress?
Mar  7 15:30:58 sprig3 kernel: klogd 1.4.1, ---------- state change ----------
Mar  7 15:35:01 sprig3 /usr/sbin/cron[27558]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 15:35:01 sprig3 /usr/sbin/cron[27559]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 15:35:01 sprig3 /usr/sbin/cron[27560]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 15:37:38 sprig3 sshd[231908]: Received disconnect from 111.222.34.217: 11: disconnected by user
Mar  7 15:39:04 sprig3 intelremotemond[27606]: Main: daemon started
Mar  7 15:39:04 sprig3 intelremotemond[27606]: config file is /scratch/awe/opt/intel/ism/bin/intel64/../intelremotemond.conf
Mar  7 15:39:04 sprig3 intelremotemond[27606]: timeout value is 600
Mar  7 15:39:04 sprig3 intelremotemond[27606]: flushtime value is 10
Mar  7 15:39:04 sprig3 intelremotemond[27606]: limit value is 1000
Mar  7 15:39:04 sprig3 intelremotemond[27606]: stop value is 100000
Mar  7 15:39:05 sprig3 intelremotemond[27606]: Consumer: total events at start is 0
Mar  7 15:40:01 sprig3 /usr/sbin/cron[27791]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 15:40:01 sprig3 /usr/sbin/cron[27792]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 15:40:01 sprig3 /usr/sbin/cron[27793]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 15:41:06 sprig3 kernel: [1561920.703087] Lustre: scratch-OST0005-osc-ffff883f77c95800: Connection to scratch-OST0005 (at 11.3.0.14@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Mar  7 15:41:06 sprig3 kernel: [1561920.703179] LustreError: dumping log to /tmp/lustre-log.1394206866.9593
Mar  7 15:41:06 sprig3 kernel: [1561920.703489] LustreError: 167-0: scratch-OST0005-osc-ffff883f77c95800: This client was evicted by scratch-OST0005; in progress operations using this service will fail.
Mar  7 15:41:08 sprig3 kernel: [1561923.022524] LustreError: 11-0: scratch-OST0009-osc-ffff883f77c95800: Communicating with 11.3.0.15@o2ib, operation obd_ping failed with -107.
Mar  7 15:41:08 sprig3 kernel: [1561923.022539] Lustre: scratch-OST0009-osc-ffff883f77c95800: Connection to scratch-OST0009 (at 11.3.0.15@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Mar  7 15:41:08 sprig3 kernel: [1561923.022624] LustreError: dumping log to /tmp/lustre-log.1394206868.9598
Mar  7 15:41:08 sprig3 kernel: [1561923.022671] LustreError: dumping log to /tmp/lustre-log.1394206868.9596
Mar  7 15:41:08 sprig3 kernel: [1561923.022916] LustreError: 167-0: scratch-OST0009-osc-ffff883f77c95800: This client was evicted by scratch-OST0009; in progress operations using this service will fail.
Mar  7 15:41:09 sprig3 kernel: [1561923.604812] LustreError: can't open /tmp/lustre-log.1394206868.9596 for dump: rc -17
Mar  7 15:41:21 sprig3 kernel: [1561935.966008] LustreError: 11-0: scratch-OST0007-osc-ffff883f77c95800: Communicating with 11.3.0.15@o2ib, operation ldlm_enqueue failed with -107.
Mar  7 15:41:21 sprig3 kernel: [1561935.966017] Lustre: scratch-OST000a-osc-ffff883f77c95800: Connection to scratch-OST000a (at 11.3.0.15@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Mar  7 15:41:21 sprig3 kernel: [1561935.966023] LustreError: Skipped 2 previous similar messages
Mar  7 15:41:21 sprig3 kernel: [1561935.966042] Lustre: Skipped 1 previous similar message
Mar  7 15:41:21 sprig3 kernel: [1561935.966116] LustreError: dumping log to /tmp/lustre-log.1394206881.9570
Mar  7 15:41:21 sprig3 kernel: [1561935.966151] LustreError: dumping log to /tmp/lustre-log.1394206881.9573
Mar  7 15:41:21 sprig3 kernel: [1561935.966219] LustreError: dumping log to /tmp/lustre-log.1394206881.9573
Mar  7 15:41:21 sprig3 kernel: [1561935.966261] LustreError: dumping log to /tmp/lustre-log.1394206881.9576
Mar  7 15:41:21 sprig3 kernel: [1561935.966297] LustreError: dumping log to /tmp/lustre-log.1394206881.9577
Mar  7 15:41:21 sprig3 kernel: [1561935.966398] LustreError: 167-0: scratch-OST000a-osc-ffff883f77c95800: This client was evicted by scratch-OST000a; in progress operations using this service will fail.
Mar  7 15:41:21 sprig3 kernel: [1561935.966401] LustreError: Skipped 1 previous similar message
Mar  7 15:41:21 sprig3 kernel: [1561935.966491] LustreError: dumping log to /tmp/lustre-log.1394206881.9575
Mar  7 15:41:21 sprig3 kernel: [1561935.967108] Lustre: 9588:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) scratch: dirty page discard: 11.3.0.12@o2ib:11.3.0.13@o2ib:/scratch/fid: [0x5000014d4:0x19e3:0x0]//user5/Application/repositories/V5.4.1_machine/src/build/sprig_13/intelmpi/opt/QA/Results_260/settings.xml may get corrupted (rc -108)
Mar  7 15:41:22 sprig3 kernel: [1561936.194585] LustreError: dumping log to /tmp/lustre-log.1394206882.9571
Mar  7 15:41:22 sprig3 kernel: [1561936.522135] LustreError: can't open /tmp/lustre-log.1394206882.9571 for dump: rc -17
Mar  7 15:41:22 sprig3 kernel: [1561936.522155] LustreError: can't open /tmp/lustre-log.1394206882.9571 for dump: rc -17
Mar  7 15:41:22 sprig3 kernel: [1561936.522166] LustreError: can't open /tmp/lustre-log.1394206882.9571 for dump: rc -17
Mar  7 15:41:22 sprig3 kernel: [1561936.522172] LustreError: can't open /tmp/lustre-log.1394206882.9571 for dump: rc -17
Mar  7 15:41:28 sprig3 sshd[27872]: Authorized to user3, krb5 principal user3@DOMAIN.AWE.CO.UK (krb5_kuserok)
Mar  7 15:41:28 sprig3 sshd[27872]: Accepted gssapi-with-mic for user3 from 111.222.39.189 port 37150 ssh2
Mar  7 15:41:33 sprig3 kernel: [1561947.980963] LustreError: 11-0: scratch-OST0004-osc-ffff883f77c95800: Communicating with 11.3.0.14@o2ib, operation obd_ping failed with -107.
Mar  7 15:41:33 sprig3 kernel: [1561947.980971] LustreError: Skipped 5 previous similar messages
Mar  7 15:41:33 sprig3 kernel: [1561947.980979] Lustre: scratch-OST0004-osc-ffff883f77c95800: Connection to scratch-OST0004 (at 11.3.0.14@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Mar  7 15:41:33 sprig3 kernel: [1561947.980985] Lustre: Skipped 5 previous similar messages
Mar  7 15:41:33 sprig3 kernel: [1561947.981056] LustreError: dumping log to /tmp/lustre-log.1394206893.9588
Mar  7 15:41:33 sprig3 kernel: [1561947.981167] LustreError: dumping log to /tmp/lustre-log.1394206893.9588
Mar  7 15:41:33 sprig3 kernel: [1561947.981356] LustreError: 167-0: scratch-OST0004-osc-ffff883f77c95800: This client was evicted by scratch-OST0004; in progress operations using this service will fail.
Mar  7 15:41:33 sprig3 kernel: [1561947.981363] LustreError: Skipped 6 previous similar messages
Mar  7 15:41:34 sprig3 kernel: [1561948.108325] LustreError: can't open /tmp/lustre-log.1394206893.9588 for dump: rc -17
Mar  7 15:41:51 sprig3 kernel: [1561965.923917] LustreError: 27828:0:(cl_lock.c:1420:cl_unuse_try()) result = -108, this is unlikely!
Mar  7 15:41:51 sprig3 kernel: [1561965.923926] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) lock@ffff881978053cb8[3 0 0 1 0 00000000] P(0):[0, 18446744073709551615]@[0x5000014d5:0x7cae:0x0] {
Mar  7 15:41:51 sprig3 kernel: [1561965.923929] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     vvp@ffff881526cf0380: 
Mar  7 15:41:51 sprig3 kernel: [1561965.923932] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lov@ffff88151479cdc8: 4
Mar  7 15:41:51 sprig3 kernel: [1561965.923937] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     0 1: lock@ffff881acb5e1b38[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100020000:0xc95dc5:0x0] {
Mar  7 15:41:51 sprig3 kernel: [1561965.923941] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881ad5f17620: [0 ffff88151479cdc8 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7cae:0x0]] 
Mar  7 15:41:51 sprig3 kernel: [1561965.923947] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881ad555e800:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:41:51 sprig3 kernel: [1561965.923950] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881acb5e1b38
Mar  7 15:41:51 sprig3 kernel: [1561965.923953] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     1 0: ---
Mar  7 15:41:51 sprig3 kernel: [1561965.923955] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     2 0: ---
Mar  7 15:41:51 sprig3 kernel: [1561965.923959] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     3 1: lock@ffff8815142c0158[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100060000:0xc95c12:0x0] {
Mar  7 15:41:51 sprig3 kernel: [1561965.923963] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff88198f333b60: [3 ffff88151479cdc8 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7cae:0x0]] 
Mar  7 15:41:51 sprig3 kernel: [1561965.923968] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881cd5f53a40:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:41:51 sprig3 kernel: [1561965.923971] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff8815142c0158
Mar  7 15:41:51 sprig3 kernel: [1561965.923973] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) 
Mar  7 15:41:51 sprig3 kernel: [1561965.923975] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881978053cb8
Mar  7 15:41:51 sprig3 kernel: [1561965.923977] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) unuse return -108
Mar  7 15:42:21 sprig3 kernel: [1561995.882330] LustreError: 27828:0:(cl_lock.c:1420:cl_unuse_try()) result = -108, this is unlikely!
Mar  7 15:42:21 sprig3 kernel: [1561995.882357] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) lock@ffff881993eca698[3 0 0 1 0 00000000] P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c6e:0x0] {
Mar  7 15:42:21 sprig3 kernel: [1561995.882370] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     vvp@ffff881e953f71c8: 
Mar  7 15:42:21 sprig3 kernel: [1561995.882381] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lov@ffff881ae7c61a20: 4
Mar  7 15:42:21 sprig3 kernel: [1561995.882398] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     0 1: lock@ffff88196fa11198[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100090000:0xc95d86:0x0] {
Mar  7 15:42:21 sprig3 kernel: [1561995.882417] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881a08a0fae0: [0 ffff881ae7c61a20 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c6e:0x0]] 
Mar  7 15:42:21 sprig3 kernel: [1561995.882438] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b6a9e1140:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:21 sprig3 kernel: [1561995.882450] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff88196fa11198
Mar  7 15:42:21 sprig3 kernel: [1561995.882461] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     1 0: ---
Mar  7 15:42:21 sprig3 kernel: [1561995.882468] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     2 0: ---
Mar  7 15:42:21 sprig3 kernel: [1561995.882481] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     3 1: lock@ffff88197e7215f8[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100030000:0xc95e66:0x0] {
Mar  7 15:42:21 sprig3 kernel: [1561995.882497] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881e80054520: [3 ffff881ae7c61a20 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c6e:0x0]] 
Mar  7 15:42:21 sprig3 kernel: [1561995.882510] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881e7e916800:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:21 sprig3 kernel: [1561995.882517] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff88197e7215f8
Mar  7 15:42:21 sprig3 kernel: [1561995.882522] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) 
Mar  7 15:42:21 sprig3 kernel: [1561995.882526] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881993eca698
Mar  7 15:42:21 sprig3 kernel: [1561995.882531] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) unuse return -108
Mar  7 15:42:21 sprig3 kernel: [1561995.883875] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) lock@ffff881b8ee6b4d8[4 0 0 1 0 00000000] P(0):[0, 18446744073709551615]@[0x5000014d5:0x7cb6:0x0] {
Mar  7 15:42:21 sprig3 kernel: [1561995.883879] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     vvp@ffff881521a3e8a8: 
Mar  7 15:42:21 sprig3 kernel: [1561995.883882] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lov@ffff881511edb338: 4
Mar  7 15:42:21 sprig3 kernel: [1561995.883886] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     0 0: lock@ffff88196f38aa18[1 2 0 1 1 00000000] P(0):[0, 18446744073709551615]@[0x100090000:0xc95d9e:0x0] {
Mar  7 15:42:21 sprig3 kernel: [1561995.883891] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff8819785341a0: [0 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7cb6:0x0]] 
Mar  7 15:42:22 sprig3 kernel: [1561995.883896] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881ead3e3900:           (null)          0x41000 0x9f2b96c8143bb9a1 1           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:22 sprig3 kernel: [1561995.883899] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff88196f38aa18
Mar  7 15:42:22 sprig3 kernel: [1561995.883903] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     1 0: lock@ffff881adc938198[1 2 0 1 1 00000000] P(0):[0, 18446744073709551615]@[0x100020000:0xc95dc8:0x0] {
Mar  7 15:42:22 sprig3 kernel: [1561995.883907] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534ca0: [1 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7cb6:0x0]] 
Mar  7 15:42:22 sprig3 kernel: [1561995.883912] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881ead3e3c00:           (null)          0x41000 0x9f2b96c8143bb9a8 1           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:22 sprig3 kernel: [1561995.883915] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881adc938198
Mar  7 15:42:22 sprig3 kernel: [1561995.883917] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     2 0: ---
Mar  7 15:42:22 sprig3 kernel: [1561995.883921] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     3 0: lock@ffff881af269deb8[1 2 -108 1 0 00000000] P(0):[0, 18446744073709551615]@[0x100030000:0xc95e7e:0x0] {
Mar  7 15:42:22 sprig3 kernel: [1561995.883925] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534520: [3 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7cb6:0x0]] 
Mar  7 15:42:22 sprig3 kernel: [1561995.883929] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b91dc4180:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:22 sprig3 kernel: [1561995.883932] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269deb8
Mar  7 15:42:22 sprig3 kernel: [1561995.883934] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) 
Mar  7 15:42:22 sprig3 kernel: [1561995.883936] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881b8ee6b4d8
Mar  7 15:42:22 sprig3 kernel: [1561995.883938] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) unuse return -108
Mar  7 15:42:22 sprig3 kernel: [1561995.884289] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) lock@ffff881b8ee6b4d8[3 0 0 1 0 00000000] P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c81:0x0] {
Mar  7 15:42:22 sprig3 kernel: [1561995.884292] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     vvp@ffff881521a3e8a8: 
Mar  7 15:42:22 sprig3 kernel: [1561995.884295] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lov@ffff881511edb338: 4
Mar  7 15:42:22 sprig3 kernel: [1561995.884299] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     0 1: lock@ffff881af269deb8[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x1000b0000:0xc95c13:0x0] {
Mar  7 15:42:22 sprig3 kernel: [1561995.884303] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534520: [0 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c81:0x0]] 
Mar  7 15:42:22 sprig3 kernel: [1561995.884308] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b91dc4180:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:23 sprig3 kernel: [1561995.884311] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269deb8
Mar  7 15:42:23 sprig3 kernel: [1561995.884313] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     1 0: ---
Mar  7 15:42:23 sprig3 kernel: [1561995.884315] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     2 0: ---
Mar  7 15:42:23 sprig3 kernel: [1561995.884319] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     3 1: lock@ffff881af269d0b8[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100020000:0xc95db6:0x0] {
Mar  7 15:42:23 sprig3 kernel: [1561995.884333] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534a60: [3 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c81:0x0]] 
Mar  7 15:42:23 sprig3 kernel: [1561995.884338] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881ac9f77d40:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:23 sprig3 kernel: [1561995.884341] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269d0b8
Mar  7 15:42:23 sprig3 kernel: [1561995.884344] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) 
Mar  7 15:42:23 sprig3 kernel: [1561995.884347] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881b8ee6b4d8
Mar  7 15:42:23 sprig3 kernel: [1561995.884349] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) unuse return -108
Mar  7 15:42:23 sprig3 kernel: [1561995.884620] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) lock@ffff881b8ee6b4d8[4 0 0 1 0 00000000] P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c93:0x0] {
Mar  7 15:42:23 sprig3 kernel: [1561995.884624] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     vvp@ffff881521a3e8a8: 
Mar  7 15:42:23 sprig3 kernel: [1561995.884626] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lov@ffff881511edb338: 4
Mar  7 15:42:23 sprig3 kernel: [1561995.884630] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     0 1: lock@ffff881af269d0b8[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100060000:0xc95c09:0x0] {
Mar  7 15:42:23 sprig3 kernel: [1561995.884635] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534a60: [0 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c93:0x0]] 
Mar  7 15:42:23 sprig3 kernel: [1561995.884639] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881ac9f77d40:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:23 sprig3 kernel: [1561995.884642] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269d0b8
Mar  7 15:42:23 sprig3 kernel: [1561995.884646] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     1 1: lock@ffff881af269deb8[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100040000:0xc95927:0x0] {
Mar  7 15:42:23 sprig3 kernel: [1561995.884651] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534520: [1 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c93:0x0]] 
Mar  7 15:42:23 sprig3 kernel: [1561995.884655] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b91dc4180:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:23 sprig3 kernel: [1561995.884658] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269deb8
Mar  7 15:42:23 sprig3 kernel: [1561995.884662] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     2 0: lock@ffff881af269d898[1 2 0 1 1 00000000] P(0):[0, 18446744073709551615]@[0x100070000:0xc95cb5:0x0] {
Mar  7 15:42:23 sprig3 kernel: [1561995.884668] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534120: [2 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c93:0x0]] 
Mar  7 15:42:23 sprig3 kernel: [1561995.884674] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b8bc0d5c0:           (null)          0x41000 0x9f2b96c8143bb9f5 1           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:23 sprig3 kernel: [1561995.884677] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269d898
Mar  7 15:42:23 sprig3 kernel: [1561995.884679] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     3 0: ---
Mar  7 15:42:23 sprig3 kernel: [1561995.884681] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) 
Mar  7 15:42:23 sprig3 kernel: [1561995.884683] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881b8ee6b4d8
Mar  7 15:42:23 sprig3 kernel: [1561995.884685] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) unuse return -108
Mar  7 15:42:23 sprig3 kernel: [1561995.885065] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) lock@ffff881b8ee6b4d8[4 0 0 1 0 00000000] P(0):[0, 18446744073709551615]@[0x5000014d5:0x7cab:0x0] {
Mar  7 15:42:24 sprig3 kernel: [1561995.885070] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     vvp@ffff881521a3e8a8: 
Mar  7 15:42:24 sprig3 kernel: [1561995.885075] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lov@ffff881511edb338: 4
Mar  7 15:42:24 sprig3 kernel: [1561995.885082] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     0 1: lock@ffff881af269d898[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100090000:0xc95d9a:0x0] {
Mar  7 15:42:24 sprig3 kernel: [1561995.885091] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534120: [0 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7cab:0x0]] 
Mar  7 15:42:24 sprig3 kernel: [1561995.885099] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b8bc0d5c0:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:24 sprig3 kernel: [1561995.885104] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269d898
Mar  7 15:42:24 sprig3 kernel: [1561995.885110] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     1 1: lock@ffff881af269deb8[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100020000:0xc95dc4:0x0] {
Mar  7 15:42:24 sprig3 kernel: [1561995.885118] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534520: [1 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7cab:0x0]] 
Mar  7 15:42:24 sprig3 kernel: [1561995.885124] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b91dc4180:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:24 sprig3 kernel: [1561995.885128] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269deb8
Mar  7 15:42:24 sprig3 kernel: [1561995.885131] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     2 0: ---
Mar  7 15:42:24 sprig3 kernel: [1561995.885137] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     3 1: lock@ffff881ac18fe7b8[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100030000:0xc95e7a:0x0] {
Mar  7 15:42:24 sprig3 kernel: [1561995.885143] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff8819785347e0: [3 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7cab:0x0]] 
Mar  7 15:42:24 sprig3 kernel: [1561995.885148] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881ead3e3f00:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:24 sprig3 kernel: [1561995.885152] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881ac18fe7b8
Mar  7 15:42:24 sprig3 kernel: [1561995.885155] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) 
Mar  7 15:42:24 sprig3 kernel: [1561995.885158] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881b8ee6b4d8
Mar  7 15:42:24 sprig3 kernel: [1561995.885161] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) unuse return -108
Mar  7 15:42:24 sprig3 kernel: [1561995.885537] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) lock@ffff881b8ee6b4d8[4 0 0 1 0 00000000] P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c76:0x0] {
Mar  7 15:42:24 sprig3 kernel: [1561995.885543] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     vvp@ffff881521a3e8a8: 
Mar  7 15:42:25 sprig3 kernel: [1561995.885547] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lov@ffff881511edb338: 4
Mar  7 15:42:25 sprig3 kernel: [1561995.885553] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     0 1: lock@ffff881ac18fe7b8[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x1000b0000:0xc95c0f:0x0] {
Mar  7 15:42:25 sprig3 kernel: [1561995.885563] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff8819785347e0: [0 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c76:0x0]] 
Mar  7 15:42:25 sprig3 kernel: [1561995.885570] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881ead3e3f00:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:25 sprig3 kernel: [1561995.885574] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881ac18fe7b8
Mar  7 15:42:25 sprig3 kernel: [1561995.885579] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     1 0: lock@ffff881af269deb8[1 2 -108 1 0 00000000] P(0):[0, 18446744073709551615]@[0x100050000:0xc95923:0x0] {
Mar  7 15:42:25 sprig3 kernel: [1561995.885586] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534520: [1 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c76:0x0]] 
Mar  7 15:42:25 sprig3 kernel: [1561995.885594] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b91dc4180:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:25 sprig3 kernel: [1561995.885598] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269deb8
Mar  7 15:42:25 sprig3 kernel: [1561995.885602] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     2 0: ---
Mar  7 15:42:25 sprig3 kernel: [1561995.885609] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     3 0: lock@ffff881af269d0b8[1 2 -108 1 0 00000000] P(0):[0, 18446744073709551615]@[0x100020000:0xc95db2:0x0] {
Mar  7 15:42:25 sprig3 kernel: [1561995.885625] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534a60: [3 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c76:0x0]] 
Mar  7 15:42:25 sprig3 kernel: [1561995.885632] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881ac9f77d40:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:25 sprig3 kernel: [1561995.885636] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269d0b8
Mar  7 15:42:25 sprig3 kernel: [1561995.885638] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) 
Mar  7 15:42:25 sprig3 kernel: [1561995.885641] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881b8ee6b4d8
Mar  7 15:42:25 sprig3 kernel: [1561995.885644] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) unuse return -108
Mar  7 15:42:25 sprig3 kernel: [1561995.886055] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) lock@ffff881b8ee6b4d8[2 0 0 1 0 00000000] P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c96:0x0] {
Mar  7 15:42:25 sprig3 kernel: [1561995.886058] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     vvp@ffff881521a3e8a8: 
Mar  7 15:42:25 sprig3 kernel: [1561995.886061] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lov@ffff881511edb338: 4
Mar  7 15:42:26 sprig3 kernel: [1561995.886063] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     0 0: ---
Mar  7 15:42:26 sprig3 kernel: [1561995.886065] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     1 0: ---
Mar  7 15:42:26 sprig3 kernel: [1561995.886067] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     2 0: ---
Mar  7 15:42:26 sprig3 kernel: [1561995.886071] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     3 0: lock@ffff881af269d898[1 2 0 1 1 00000000] P(0):[0, 18446744073709551615]@[0x1000a0000:0xc95fcf:0x0] {
Mar  7 15:42:26 sprig3 kernel: [1561995.886075] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534120: [3 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c96:0x0]] 
Mar  7 15:42:26 sprig3 kernel: [1561995.886080] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b8bc0d5c0:           (null)          0x41000 0x9f2b96c8143bbaa4 1           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:26 sprig3 kernel: [1561995.886083] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269d898
Mar  7 15:42:26 sprig3 kernel: [1561995.886085] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) 
Mar  7 15:42:26 sprig3 kernel: [1561995.886087] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881b8ee6b4d8
Mar  7 15:42:26 sprig3 kernel: [1561995.886089] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) unuse return -108
Mar  7 15:42:26 sprig3 kernel: [1561995.886343] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) lock@ffff881b8ee6b4d8[3 0 0 1 0 00000000] P(0):[0, 18446744073709551615]@[0x5000014d5:0x7ca2:0x0] {
Mar  7 15:42:26 sprig3 kernel: [1561995.886346] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     vvp@ffff881521a3e8a8: 
Mar  7 15:42:26 sprig3 kernel: [1561995.886348] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lov@ffff881511edb338: 4
Mar  7 15:42:26 sprig3 kernel: [1561995.886352] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     0 1: lock@ffff881ac18fe7b8[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100000000:0xc95dc6:0x0] {
Mar  7 15:42:26 sprig3 kernel: [1561995.886357] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff8819785347e0: [0 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7ca2:0x0]] 
Mar  7 15:42:26 sprig3 kernel: [1561995.886361] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881ead3e3f00:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:26 sprig3 kernel: [1561995.886364] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881ac18fe7b8
Mar  7 15:42:26 sprig3 kernel: [1561995.886366] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     1 0: ---
Mar  7 15:42:26 sprig3 kernel: [1561995.886368] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     2 0: ---
Mar  7 15:42:26 sprig3 kernel: [1561995.886372] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     3 1: lock@ffff8819a7778a58[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100090000:0xc95d97:0x0] {
Mar  7 15:42:26 sprig3 kernel: [1561995.886376] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff8819632aa6a0: [3 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7ca2:0x0]] 
Mar  7 15:42:26 sprig3 kernel: [1561995.886384] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff8819781369c0:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:26 sprig3 kernel: [1561995.886387] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff8819a7778a58
Mar  7 15:42:26 sprig3 kernel: [1561995.886389] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) 
Mar  7 15:42:26 sprig3 kernel: [1561995.886391] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881b8ee6b4d8
Mar  7 15:42:26 sprig3 kernel: [1561995.886393] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) unuse return -108
Mar  7 15:42:26 sprig3 kernel: [1561995.886666] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) lock@ffff881b8ee6b4d8[3 0 0 1 0 00000000] P(0):[0, 18446744073709551615]@[0x5000014d5:0x7cac:0x0] {
Mar  7 15:42:26 sprig3 kernel: [1561995.886669] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     vvp@ffff881521a3e8a8: 
Mar  7 15:42:26 sprig3 kernel: [1561995.886672] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lov@ffff881511edb338: 4
Mar  7 15:42:26 sprig3 kernel: [1561995.886675] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     0 1: lock@ffff8819a7778a58[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100060000:0xc95c11:0x0] {
Mar  7 15:42:27 sprig3 kernel: [1561995.886680] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff8819632aa6a0: [0 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7cac:0x0]] 
Mar  7 15:42:27 sprig3 kernel: [1561995.886684] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff8819781369c0:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:27 sprig3 kernel: [1561995.886687] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff8819a7778a58
Mar  7 15:42:27 sprig3 kernel: [1561995.886689] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     1 0: ---
Mar  7 15:42:27 sprig3 kernel: [1561995.886691] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     2 0: ---
Mar  7 15:42:27 sprig3 kernel: [1561995.886695] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     3 1: lock@ffff881af269deb8[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100010000:0xc95da5:0x0] {
Mar  7 15:42:27 sprig3 kernel: [1561995.886700] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534520: [3 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7cac:0x0]] 
Mar  7 15:42:27 sprig3 kernel: [1561995.886705] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b91dc4180:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:27 sprig3 kernel: [1561995.886708] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269deb8
Mar  7 15:42:27 sprig3 kernel: [1561995.886709] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) 
Mar  7 15:42:27 sprig3 kernel: [1561995.886711] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881b8ee6b4d8
Mar  7 15:42:27 sprig3 kernel: [1561995.886713] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) unuse return -108
Mar  7 15:42:27 sprig3 kernel: [1561995.886969] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) lock@ffff881b8ee6b4d8[4 0 0 1 0 00000000] P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c9d:0x0] {
Mar  7 15:42:27 sprig3 kernel: [1561995.886972] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     vvp@ffff881521a3e8a8: 
Mar  7 15:42:27 sprig3 kernel: [1561995.886975] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lov@ffff881511edb338: 4
Mar  7 15:42:27 sprig3 kernel: [1561995.886979] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     0 1: lock@ffff881af269deb8[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100050000:0xc95930:0x0] {
Mar  7 15:42:27 sprig3 kernel: [1561995.886983] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534520: [0 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c9d:0x0]] 
Mar  7 15:42:27 sprig3 kernel: [1561995.886988] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b91dc4180:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:27 sprig3 kernel: [1561995.886991] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269deb8
Mar  7 15:42:27 sprig3 kernel: [1561995.886993] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     1 0: ---
Mar  7 15:42:28 sprig3 kernel: [1561995.886997] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     2 0: lock@ffff881af269d0b8[1 2 -108 1 0 00000000] P(0):[0, 18446744073709551615]@[0x100020000:0xc95dbf:0x0] {
Mar  7 15:42:28 sprig3 kernel: [1561995.887001] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534a60: [2 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c9d:0x0]] 
Mar  7 15:42:28 sprig3 kernel: [1561995.887005] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881ac9f77d40:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:28 sprig3 kernel: [1561995.887008] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269d0b8
Mar  7 15:42:28 sprig3 kernel: [1561995.887012] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     3 1: lock@ffff881ac18fe7b8[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100080000:0xc95f70:0x0] {
Mar  7 15:42:28 sprig3 kernel: [1561995.887017] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff8819785347e0: [3 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c9d:0x0]] 
Mar  7 15:42:28 sprig3 kernel: [1561995.887021] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881ead3e3f00:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:28 sprig3 kernel: [1561995.887024] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881ac18fe7b8
Mar  7 15:42:28 sprig3 kernel: [1561995.887026] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) 
Mar  7 15:42:28 sprig3 kernel: [1561995.887028] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881b8ee6b4d8
Mar  7 15:42:28 sprig3 kernel: [1561995.887030] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) unuse return -108
Mar  7 15:42:28 sprig3 kernel: [1561995.887295] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) lock@ffff881b8ee6b4d8[4 0 0 1 0 00000000] P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c86:0x0] {
Mar  7 15:42:28 sprig3 kernel: [1561995.887298] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     vvp@ffff881521a3e8a8: 
Mar  7 15:42:28 sprig3 kernel: [1561995.887301] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lov@ffff881511edb338: 4
Mar  7 15:42:28 sprig3 kernel: [1561995.887304] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     0 1: lock@ffff881ac18fe7b8[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x1000a0000:0xc95fca:0x0] {
Mar  7 15:42:28 sprig3 kernel: [1561995.887309] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff8819785347e0: [0 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c86:0x0]] 
Mar  7 15:42:28 sprig3 kernel: [1561995.887314] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881ead3e3f00:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:28 sprig3 kernel: [1561995.887316] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881ac18fe7b8
Mar  7 15:42:28 sprig3 kernel: [1561995.887319] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     1 0: ---
Mar  7 15:42:28 sprig3 kernel: [1561995.887323] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     2 0: lock@ffff881af269deb8[1 2 -108 1 0 00000000] P(0):[0, 18446744073709551615]@[0x1000b0000:0xc95c15:0x0] {
Mar  7 15:42:28 sprig3 kernel: [1561995.887327] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534520: [2 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c86:0x0]] 
Mar  7 15:42:28 sprig3 kernel: [1561995.887331] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b91dc4180:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:28 sprig3 kernel: [1561995.887334] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269deb8
Mar  7 15:42:28 sprig3 kernel: [1561995.887338] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     3 0: lock@ffff8819a7778a58[1 2 -108 1 0 00000000] P(0):[0, 18446744073709551615]@[0x100050000:0xc95929:0x0] {
Mar  7 15:42:28 sprig3 kernel: [1561995.887343] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff8819632aa6a0: [3 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c86:0x0]] 
Mar  7 15:42:28 sprig3 kernel: [1561995.887347] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff8819781369c0:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:28 sprig3 kernel: [1561995.887350] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff8819a7778a58
Mar  7 15:42:28 sprig3 kernel: [1561995.887352] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) 
Mar  7 15:42:28 sprig3 kernel: [1561995.887354] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881b8ee6b4d8
Mar  7 15:42:28 sprig3 kernel: [1561995.887356] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) unuse return -108
Mar  7 15:42:29 sprig3 kernel: [1561995.887732] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) lock@ffff881b8ee6b4d8[2 0 0 1 0 00000000] P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c74:0x0] {
Mar  7 15:42:29 sprig3 kernel: [1561995.887735] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     vvp@ffff881521a3e8a8: 
Mar  7 15:42:29 sprig3 kernel: [1561995.887738] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lov@ffff881511edb338: 4
Mar  7 15:42:29 sprig3 kernel: [1561995.887742] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     0 1: lock@ffff881af269deb8[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100080000:0xc95f63:0x0] {
Mar  7 15:42:29 sprig3 kernel: [1561995.887746] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534520: [0 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c74:0x0]] 
Mar  7 15:42:29 sprig3 kernel: [1561995.887751] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b91dc4180:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:29 sprig3 kernel: [1561995.887754] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269deb8
Mar  7 15:42:29 sprig3 kernel: [1561995.887756] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     1 0: ---
Mar  7 15:42:29 sprig3 kernel: [1561995.887758] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     2 0: ---
Mar  7 15:42:29 sprig3 kernel: [1561995.887760] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     3 0: ---
Mar  7 15:42:29 sprig3 kernel: [1561995.887762] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) 
Mar  7 15:42:29 sprig3 kernel: [1561995.887764] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881b8ee6b4d8
Mar  7 15:42:29 sprig3 kernel: [1561995.887766] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) unuse return -108
Mar  7 15:42:29 sprig3 kernel: [1561995.888041] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) lock@ffff881b8ee6b4d8[3 0 0 1 0 00000000] P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c92:0x0] {
Mar  7 15:42:29 sprig3 kernel: [1561995.888045] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     vvp@ffff881521a3e8a8: 
Mar  7 15:42:29 sprig3 kernel: [1561995.888049] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lov@ffff881511edb338: 4
Mar  7 15:42:29 sprig3 kernel: [1561995.888054] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     0 1: lock@ffff881af269deb8[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100090000:0xc95d92:0x0] {
Mar  7 15:42:29 sprig3 kernel: [1561995.888058] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534520: [0 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c92:0x0]] 
Mar  7 15:42:29 sprig3 kernel: [1561995.888062] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b91dc4180:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:29 sprig3 kernel: [1561995.888066] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269deb8
Mar  7 15:42:30 sprig3 kernel: [1561995.888069] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     1 1: lock@ffff881af269d518[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100020000:0xc95dbc:0x0] {
Mar  7 15:42:30 sprig3 kernel: [1561995.888074] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881b987d7b60: [1 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c92:0x0]] 
Mar  7 15:42:30 sprig3 kernel: [1561995.888078] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b932843c0:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:30 sprig3 kernel: [1561995.888080] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269d518
Mar  7 15:42:30 sprig3 kernel: [1561995.888083] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     2 0: ---
Mar  7 15:42:30 sprig3 kernel: [1561995.888085] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     3 0: ---
Mar  7 15:42:30 sprig3 kernel: [1561995.888087] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) 
Mar  7 15:42:30 sprig3 kernel: [1561995.888089] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881b8ee6b4d8
Mar  7 15:42:30 sprig3 kernel: [1561995.888091] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) unuse return -108
Mar  7 15:42:30 sprig3 kernel: [1561995.888331] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) lock@ffff881b8ee6b4d8[3 0 0 1 0 00000000] P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c87:0x0] {
Mar  7 15:42:30 sprig3 kernel: [1561995.888334] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     vvp@ffff881521a3e8a8: 
Mar  7 15:42:30 sprig3 kernel: [1561995.888337] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lov@ffff881511edb338: 4
Mar  7 15:42:30 sprig3 kernel: [1561995.888341] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     0 1: lock@ffff881af269d518[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100090000:0xc95d8e:0x0] {
Mar  7 15:42:30 sprig3 kernel: [1561995.888345] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881b987d7b60: [0 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c87:0x0]] 
Mar  7 15:42:30 sprig3 kernel: [1561995.888349] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b932843c0:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:30 sprig3 kernel: [1561995.888352] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269d518
Mar  7 15:42:30 sprig3 kernel: [1561995.888354] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     1 0: ---
Mar  7 15:42:30 sprig3 kernel: [1561995.888358] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     2 0: lock@ffff881ac18fe7b8[1 2 -108 1 0 00000000] P(0):[0, 18446744073709551615]@[0x100080000:0xc95f69:0x0] {
Mar  7 15:42:30 sprig3 kernel: [1561995.888362] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff8819785347e0: [2 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c87:0x0]] 
Mar  7 15:42:30 sprig3 kernel: [1561995.888366] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881ead3e3f00:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:31 sprig3 kernel: [1561995.888369] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881ac18fe7b8
Mar  7 15:42:31 sprig3 kernel: [1561995.888371] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     3 0: ---
Mar  7 15:42:31 sprig3 kernel: [1561995.888373] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) 
Mar  7 15:42:31 sprig3 kernel: [1561995.888375] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881b8ee6b4d8
Mar  7 15:42:31 sprig3 kernel: [1561995.888377] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) unuse return -108
Mar  7 15:42:31 sprig3 kernel: [1561995.888645] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) lock@ffff881b8ee6b4d8[4 0 0 1 0 00000000] P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c8c:0x0] {
Mar  7 15:42:31 sprig3 kernel: [1561995.888648] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     vvp@ffff881521a3e8a8: 
Mar  7 15:42:31 sprig3 kernel: [1561995.888651] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lov@ffff881511edb338: 4
Mar  7 15:42:31 sprig3 kernel: [1561995.888655] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     0 1: lock@ffff881af269d518[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x1000b0000:0xc95c17:0x0] {
Mar  7 15:42:31 sprig3 kernel: [1561995.888659] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881b987d7b60: [0 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c8c:0x0]] 
Mar  7 15:42:31 sprig3 kernel: [1561995.888663] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b932843c0:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:31 sprig3 kernel: [1561995.888666] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269d518
Mar  7 15:42:31 sprig3 kernel: [1561995.888668] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     1 0: ---
Mar  7 15:42:31 sprig3 kernel: [1561995.888672] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     2 1: lock@ffff881af269deb8[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100090000:0xc95d90:0x0] {
Mar  7 15:42:31 sprig3 kernel: [1561995.888677] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534520: [2 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c8c:0x0]] 
Mar  7 15:42:31 sprig3 kernel: [1561995.888681] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b91dc4180:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:31 sprig3 kernel: [1561995.888684] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269deb8
Mar  7 15:42:31 sprig3 kernel: [1561995.888688] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     3 1: lock@ffff881af269d7b8[1 2 -108 1 1 00000006] P(0):[0, 18446744073709551615]@[0x100020000:0xc95dba:0x0] {
Mar  7 15:42:31 sprig3 kernel: [1561995.888692] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881b987d7260: [3 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7c8c:0x0]] 
Mar  7 15:42:31 sprig3 kernel: [1561995.888697] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b93284b40:           (null)          0x41000 0x0 2           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:31 sprig3 kernel: [1561995.888700] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269d7b8
Mar  7 15:42:31 sprig3 kernel: [1561995.888702] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) 
Mar  7 15:42:31 sprig3 kernel: [1561995.888704] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881b8ee6b4d8
Mar  7 15:42:31 sprig3 kernel: [1561995.888706] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) unuse return -108
Mar  7 15:42:31 sprig3 kernel: [1561995.888977] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) lock@ffff881b8ee6b4d8[4 0 0 1 0 00000000] P(0):[0, 18446744073709551615]@[0x5000014d5:0x7cb2:0x0] {
Mar  7 15:42:31 sprig3 kernel: [1561995.888980] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     vvp@ffff881521a3e8a8: 
Mar  7 15:42:31 sprig3 kernel: [1561995.888983] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lov@ffff881511edb338: 4
Mar  7 15:42:31 sprig3 kernel: [1561995.888985] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     0 0: ---
Mar  7 15:42:31 sprig3 kernel: [1561995.888989] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     1 0: lock@ffff881af269deb8[1 2 0 1 1 00000000] P(0):[0, 18446744073709551615]@[0x100010000:0xc95da7:0x0] {
Mar  7 15:42:31 sprig3 kernel: [1561995.888993] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534520: [1 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7cb2:0x0]] 
Mar  7 15:42:32 sprig3 kernel: [1561995.888997] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b91dc4180:           (null)          0x41000 0x9f2b96c8143bbbc3 1           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:32 sprig3 kernel: [1561995.889000] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269deb8
Mar  7 15:42:32 sprig3 kernel: [1561995.889004] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     2 0: lock@ffff881af269d518[1 2 0 1 1 00000000] P(0):[0, 18446744073709551615]@[0x1000a0000:0xc95fd8:0x0] {
Mar  7 15:42:32 sprig3 kernel: [1561995.889008] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881b987d7b60: [2 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7cb2:0x0]] 
Mar  7 15:42:32 sprig3 kernel: [1561995.889012] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881b932843c0:           (null)          0x41000 0x9f2b96c8143bbbca 1           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:32 sprig3 kernel: [1561995.889015] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269d518
Mar  7 15:42:32 sprig3 kernel: [1561995.889019] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     3 0: lock@ffff881af269d0b8[1 2 0 1 1 00000000] P(0):[0, 18446744073709551615]@[0x100000000:0xc95dcb:0x0] {
Mar  7 15:42:32 sprig3 kernel: [1561995.889023] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     lovsub@ffff881978534a60: [3 ffff881511edb338 P(0):[0, 18446744073709551615]@[0x5000014d5:0x7cb2:0x0]] 
Mar  7 15:42:32 sprig3 kernel: [1561995.889028] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked())     osc@ffff881ac9f77d40:           (null)          0x41000 0x9f2b96c8143bbbd1 1           (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
Mar  7 15:42:32 sprig3 kernel: [1561995.889031] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881af269d0b8
Mar  7 15:42:32 sprig3 kernel: [1561995.889033] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) 
Mar  7 15:42:32 sprig3 kernel: [1561995.889035] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) } lock@ffff881b8ee6b4d8
Mar  7 15:42:32 sprig3 kernel: [1561995.889037] LustreError: 27828:0:(cl_lock.c:1435:cl_unuse_locked()) unuse return -108
Mar  7 15:42:38 sprig3 sshd[20925]: Received disconnect from 111.222.38.165: 11: disconnected by user
Mar  7 15:42:46 sprig3 kernel: [1562020.535941] LustreError: 27830:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0005_UUID: rc = -110 waiting for callback (2 != 0)
Mar  7 15:42:46 sprig3 kernel: [1562020.535961] LustreError: 27830:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff883ac3c15800 x1460291788607144/t0(0) o101->scratch-OST0005-osc-ffff883f77c95800@11.3.0.14@o2ib:28/4 lens 328/368 e 0 to 0 dl 1394205665 ref 1 fl Interpret:RE/0/0 rc -5/0
Mar  7 15:42:46 sprig3 kernel: [1562020.535979] LustreError: 27830:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0005_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Mar  7 15:42:48 sprig3 kernel: [1562022.856074] LustreError: 27834:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0002_UUID: rc = -110 waiting for callback (3 != 0)
Mar  7 15:42:48 sprig3 kernel: [1562022.856083] LustreError: 27834:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 1 previous similar message
Mar  7 15:42:48 sprig3 kernel: [1562022.856091] LustreError: 27833:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff8838cdbf6000 x1460291788607148/t0(0) o101->scratch-OST0009-osc-ffff883f77c95800@11.3.0.15@o2ib:28/4 lens 328/400 e 0 to 0 dl 1394205665 ref 1 fl Rpc:RE/0/ffffffff rc -5/-1
Mar  7 15:42:48 sprig3 kernel: [1562022.856101] LustreError: 27833:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 1 previous similar message
Mar  7 15:42:48 sprig3 kernel: [1562022.856118] LustreError: 27833:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0009_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Mar  7 15:43:01 sprig3 kernel: [1562035.798537] LustreError: 27852:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0008_UUID: rc = -110 waiting for callback (3 != 0)
Mar  7 15:43:01 sprig3 kernel: [1562035.798553] LustreError: 27850:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff881f4f781800 x1460291788619272/t0(0) o4->scratch-OST0003-osc-ffff883f77c95800@11.3.0.14@o2ib:6/4 lens 488/448 e 0 to 0 dl 1394206453 ref 2 fl Rpc:RE/0/ffffffff rc -5/-1
Mar  7 15:43:01 sprig3 kernel: [1562035.798576] LustreError: 27852:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 1 previous similar message
Mar  7 15:43:01 sprig3 kernel: [1562035.798583] LustreError: 27850:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 4 previous similar messages
Mar  7 15:43:01 sprig3 kernel: [1562035.798592] LustreError: 27850:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0003_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Mar  7 15:43:01 sprig3 kernel: [1562035.798602] LustreError: 27850:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 1 previous similar message
Mar  7 15:43:13 sprig3 kernel: [1562047.814544] LustreError: 27880:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0006_UUID: rc = -110 waiting for callback (1 != 0)
Mar  7 15:43:13 sprig3 kernel: [1562047.814562] LustreError: 27880:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 2 previous similar messages
Mar  7 15:43:13 sprig3 kernel: [1562047.814570] LustreError: 27879:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff8814c1498000 x1460291788674420/t0(0) o101->scratch-OST0004-osc-ffff883f77c95800@11.3.0.14@o2ib:28/4 lens 328/368 e 0 to 0 dl 1394206758 ref 1 fl Interpret:RE/0/0 rc -5/0
Mar  7 15:43:13 sprig3 kernel: [1562047.814575] LustreError: 27879:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 6 previous similar messages
Mar  7 15:43:13 sprig3 kernel: [1562047.814584] LustreError: 27880:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0006_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Mar  7 15:43:13 sprig3 kernel: [1562047.814595] LustreError: 27880:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 4 previous similar messages
Mar  7 15:43:56 sprig3 kernel: [1562090.422153] Lustre: 9588:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) scratch: dirty page discard: 11.3.0.12@o2ib:11.3.0.13@o2ib:/scratch/fid: [0x5000014cc:0xd640:0x0]//user5/Application/repositories/V5.4.1_machine/src/build/sprig_13/intelmpi/opt/include/zeus_igs.mod may get corrupted (rc -108)
Mar  7 15:43:56 sprig3 kernel: [1562090.422164] Lustre: 9588:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) Skipped 54 previous similar messages
Mar  7 15:44:06 sprig3 kernel: [1562100.646612] LustreError: 27551:0:(osc_cache.c:897:osc_extent_wait()) extent ffff881a6ccc7480@{[0 -> 2/255], [3|0|+|rpc|wihY|ffff881a4c09b8c8], [12288|3|+|-|ffff881b8bf6ad98|256|ffff8814c4fbc380]} scratch-OST0003-osc-ffff883f77c95800: wait ext to 0 timedout, recovery in progress?
Mar  7 15:44:26 sprig3 kernel: [1562120.369774] LustreError: 27830:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0005_UUID: rc = -110 waiting for callback (2 != 0)
Mar  7 15:44:26 sprig3 kernel: [1562120.369793] LustreError: 27830:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff883ac3c15800 x1460291788607144/t0(0) o101->scratch-OST0005-osc-ffff883f77c95800@11.3.0.14@o2ib:28/4 lens 328/368 e 0 to 0 dl 1394205665 ref 1 fl Interpret:RE/0/0 rc -5/0
Mar  7 15:44:26 sprig3 kernel: [1562120.369802] LustreError: 27830:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 1 previous similar message
Mar  7 15:44:26 sprig3 kernel: [1562120.369816] LustreError: 27830:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0005_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Mar  7 15:44:41 sprig3 kernel: [1562135.632376] LustreError: 27852:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0008_UUID: rc = -110 waiting for callback (3 != 0)
Mar  7 15:44:41 sprig3 kernel: [1562135.632392] LustreError: 27850:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff881f4f781800 x1460291788619272/t0(0) o4->scratch-OST0003-osc-ffff883f77c95800@11.3.0.14@o2ib:6/4 lens 488/448 e 0 to 0 dl 1394206453 ref 2 fl Rpc:RE/0/ffffffff rc -5/-1
Mar  7 15:44:41 sprig3 kernel: [1562135.632401] LustreError: 27852:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 3 previous similar messages
Mar  7 15:44:41 sprig3 kernel: [1562135.632411] LustreError: 27850:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 6 previous similar messages
Mar  7 15:44:41 sprig3 kernel: [1562135.632437] LustreError: 27850:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0003_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Mar  7 15:44:41 sprig3 kernel: [1562135.632450] LustreError: 27850:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 3 previous similar messages
Mar  7 15:44:53 sprig3 kernel: [1562147.648365] LustreError: 27880:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0006_UUID: rc = -110 waiting for callback (1 != 0)
Mar  7 15:44:53 sprig3 kernel: [1562147.648372] LustreError: 27879:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0004_UUID: rc = -110 waiting for callback (1 != 0)
Mar  7 15:44:53 sprig3 kernel: [1562147.648380] LustreError: 27879:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 2 previous similar messages
Mar  7 15:44:53 sprig3 kernel: [1562147.648399] LustreError: 27879:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff8814c1498000 x1460291788674420/t0(0) o101->scratch-OST0004-osc-ffff883f77c95800@11.3.0.14@o2ib:28/4 lens 328/368 e 0 to 0 dl 1394206758 ref 1 fl Interpret:RE/0/0 rc -5/0
Mar  7 15:44:53 sprig3 kernel: [1562147.648409] LustreError: 27880:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0006_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Mar  7 15:44:53 sprig3 kernel: [1562147.648416] LustreError: 27880:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 2 previous similar messages
Mar  7 15:44:53 sprig3 kernel: [1562147.648424] LustreError: 27879:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 7 previous similar messages
Mar  7 15:45:01 sprig3 /usr/sbin/cron[27902]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 15:45:01 sprig3 /usr/sbin/cron[27903]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 15:45:01 sprig3 /usr/sbin/cron[27904]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 15:46:06 sprig3 kernel: [1562220.203611] LustreError: 27830:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0005_UUID: rc = -110 waiting for callback (2 != 0)
Mar  7 15:46:06 sprig3 kernel: [1562220.203635] LustreError: 27830:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff883ac3c15800 x1460291788607144/t0(0) o101->scratch-OST0005-osc-ffff883f77c95800@11.3.0.14@o2ib:28/4 lens 328/368 e 0 to 0 dl 1394205665 ref 1 fl Interpret:RE/0/0 rc -5/0
Mar  7 15:46:06 sprig3 kernel: [1562220.203653] LustreError: 27830:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0005_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Mar  7 15:46:06 sprig3 kernel: [1562220.203678] LustreError: 27830:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 1 previous similar message
Mar  7 15:46:26 sprig3 kernel: [1562240.170746] Lustre: 9569:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) scratch: dirty page discard: 11.3.0.12@o2ib:11.3.0.13@o2ib:/scratch/fid: [0x5000014cc:0xd5b1:0x0]//user5/Application/repositories/V5.4.1_machine/src/build/sprig_13/intelmpi/opt/consts.o may get corrupted (rc -108)
Mar  7 15:46:26 sprig3 kernel: [1562240.170757] Lustre: 9569:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) Skipped 8 previous similar messages
Mar  7 15:46:33 sprig3 kernel: [1562247.482407] LustreError: 11-0: scratch-MDT0000-mdc-ffff883f77c95800: Communicating with 11.3.0.13@o2ib, operation obd_ping failed with -107.
Mar  7 15:46:33 sprig3 kernel: [1562247.482414] LustreError: Skipped 1 previous similar message
Mar  7 15:46:33 sprig3 kernel: [1562247.482424] Lustre: scratch-MDT0000-mdc-ffff883f77c95800: Connection to scratch-MDT0000 (at 11.3.0.13@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Mar  7 15:46:33 sprig3 kernel: [1562247.482429] Lustre: Skipped 1 previous similar message
Mar  7 15:46:33 sprig3 kernel: [1562247.482520] LustreError: dumping log to /tmp/lustre-log.1394207193.9573
Mar  7 15:46:33 sprig3 kernel: [1562247.482719] LustreError: 167-0: scratch-MDT0000-mdc-ffff883f77c95800: This client was evicted by scratch-MDT0000; in progress operations using this service will fail.
Mar  7 15:46:33 sprig3 kernel: [1562247.482726] LustreError: Skipped 1 previous similar message
Mar  7 15:46:33 sprig3 kernel: [1562247.492244] LustreError: 27954:0:(file.c:163:ll_close_inode_openhandle()) inode 360288059696092169 mdc close failed: rc = -108
Mar  7 15:46:33 sprig3 kernel: [1562247.504513] LustreError: 27954:0:(file.c:163:ll_close_inode_openhandle()) inode 360288059645856716 mdc close failed: rc = -108
Mar  7 15:46:33 sprig3 kernel: [1562247.504520] LustreError: 27954:0:(file.c:163:ll_close_inode_openhandle()) Skipped 2 previous similar messages
Mar  7 15:46:34 sprig3 kernel: [1562248.017850] LustreError: 27954:0:(file.c:163:ll_close_inode_openhandle()) inode 360288059645855501 mdc close failed: rc = -108
Mar  7 15:46:34 sprig3 kernel: [1562248.017859] LustreError: 27954:0:(file.c:163:ll_close_inode_openhandle()) Skipped 51 previous similar messages
Mar  7 15:46:35 sprig3 kernel: [1562249.025584] LustreError: 27954:0:(file.c:163:ll_close_inode_openhandle()) inode 360288059645855343 mdc close failed: rc = -108
Mar  7 15:46:35 sprig3 kernel: [1562249.025593] LustreError: 27954:0:(file.c:163:ll_close_inode_openhandle()) Skipped 89 previous similar messages
Mar  7 15:46:37 sprig3 kernel: [1562251.043794] LustreError: 27954:0:(file.c:163:ll_close_inode_openhandle()) inode 360288059645871028 mdc close failed: rc = -108
Mar  7 15:46:37 sprig3 kernel: [1562251.043804] LustreError: 27954:0:(file.c:163:ll_close_inode_openhandle()) Skipped 170 previous similar messages
Mar  7 15:46:41 sprig3 kernel: [1562255.073911] LustreError: 27954:0:(file.c:163:ll_close_inode_openhandle()) inode 360288059696092027 mdc close failed: rc = -108
Mar  7 15:46:41 sprig3 kernel: [1562255.073920] LustreError: 27954:0:(file.c:163:ll_close_inode_openhandle()) Skipped 335 previous similar messages
Mar  7 15:46:46 sprig3 kernel: [1562260.422664] LustreError: 27954:0:(ldlm_resource.c:804:ldlm_resource_complain()) scratch-MDT0000-mdc-ffff883f77c95800: namespace resource [0x5000014d9:0xca22:0x0].0 (ffff8819633745c0) refcount nonzero (8) after lock cleanup; forcing cleanup.
Mar  7 15:46:46 sprig3 kernel: [1562260.422677] LustreError: 27954:0:(ldlm_resource.c:1415:ldlm_resource_dump()) --- Resource: [0x5000014d9:0xca22:0x0].0 (ffff8819633745c0) refcount = 9
Mar  7 15:46:46 sprig3 kernel: [1562260.422787] LustreError: 27954:0:(import.c:1279:ptlrpc_invalidate_import_thread()) dump the log upon eviction
Mar  7 15:46:46 sprig3 kernel: [1562260.422874] LustreError: dumping log to /tmp/lustre-log.1394207206.27954
Mar  7 15:46:46 sprig3 kernel: [1562260.422880] Lustre: scratch-MDT0000-mdc-ffff883f77c95800: Connection restored to scratch-MDT0000 (at 11.3.0.13@o2ib)
Mar  7 15:47:46 sprig3 kernel: [1562320.037450] LustreError: 27830:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0005_UUID: rc = -110 waiting for callback (2 != 0)
Mar  7 15:47:46 sprig3 kernel: [1562320.037459] LustreError: 27830:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 8 previous similar messages
Mar  7 15:47:46 sprig3 kernel: [1562320.037486] LustreError: 27830:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff883ac3c15800 x1460291788607144/t0(0) o101->scratch-OST0005-osc-ffff883f77c95800@11.3.0.14@o2ib:28/4 lens 328/368 e 0 to 0 dl 1394205665 ref 1 fl Interpret:RE/0/0 rc -5/0
Mar  7 15:47:46 sprig3 kernel: [1562320.037490] LustreError: 27830:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 15 previous similar messages
Mar  7 15:47:46 sprig3 kernel: [1562320.037497] LustreError: 27830:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0005_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Mar  7 15:47:46 sprig3 kernel: [1562320.037499] LustreError: 27830:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 8 previous similar messages
Mar  7 15:49:10 sprig3 intelremotemond[27606]: Consumer: timeout, exiting...
Mar  7 15:49:11 sprig3 kernel: [1562405.179763] LustreError: 27781:0:(osc_cache.c:897:osc_extent_wait()) extent ffff881adab76330@{[0 -> 0/255], [3|0|+|rpc|wihY|ffff881a63f162a8], [4096|1|+|-|ffff881b96a10af8|256|ffff8836fad04500]} scratch-OST0007-osc-ffff883f77c95800: wait ext to 0 timedout, recovery in progress?
Mar  7 15:49:26 sprig3 kernel: [1562419.871299] LustreError: 27830:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0005_UUID: rc = -110 waiting for callback (2 != 0)
Mar  7 15:49:26 sprig3 kernel: [1562419.871319] LustreError: 27830:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 8 previous similar messages
Mar  7 15:49:26 sprig3 kernel: [1562419.871339] LustreError: 27830:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff883ac3c15800 x1460291788607144/t0(0) o101->scratch-OST0005-osc-ffff883f77c95800@11.3.0.14@o2ib:28/4 lens 328/368 e 0 to 0 dl 1394205665 ref 1 fl Interpret:RE/0/0 rc -5/0
Mar  7 15:49:26 sprig3 kernel: [1562419.871347] LustreError: 27830:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 15 previous similar messages
Mar  7 15:49:26 sprig3 kernel: [1562419.871362] LustreError: 27830:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0005_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Mar  7 15:49:26 sprig3 kernel: [1562419.871380] LustreError: 27830:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 8 previous similar messages
Mar  7 15:50:01 sprig3 /usr/sbin/cron[27970]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 15:50:01 sprig3 /usr/sbin/cron[27971]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 15:50:01 sprig3 /usr/sbin/cron[27972]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 15:50:04 sprig3 sshd[27983]: Accepted publickey for root from 11.1.0.8 port 47181 ssh2
Mar  7 15:50:15 sprig3 kernel: [1562468.234289] LustreError: 27985:0:(file.c:2716:ll_inode_revalidate_fini()) scratch: revalidate FID [0x200000007:0x1:0x0] error: rc = -4
Mar  7 15:50:15 sprig3 sshd[27983]: Received disconnect from 11.1.0.8: 11: disconnected by user
Mar  7 15:51:21 sprig3 kernel: [1562535.023660] LustreError: 27849:0:(osc_cache.c:897:osc_extent_wait()) extent ffff881a70a807c8@{[0 -> 0/255], [2|0|+|cache|wihY|ffff881954d31ba8], [4096|1|+|-|ffff88199967a278|256|          (null)]} scratch-OST0000-osc-ffff883f77c95800: wait ext to 0 timedout, recovery in progress?
Mar  7 15:51:22 sprig3 kernel: [1562535.039635] LustreError: 27846:0:(osc_cache.c:897:osc_extent_wait()) extent ffff881b20840e58@{[0 -> 1/255], [2|0|+|cache|wihY|ffff883b16123648], [8192|2|+|-|ffff881b8e211698|256|          (null)]} scratch-OST000a-osc-ffff883f77c95800: wait ext to 0 timedout, recovery in progress?
Mar  7 15:51:22 sprig3 kernel: [1562535.043634] LustreError: 27851:0:(osc_cache.c:897:osc_extent_wait()) extent ffff883b086f65d0@{[0 -> 0/255], [2|0|+|cache|wihY|ffff883b21e72188], [4096|1|+|-|ffff8836f8cb0cf8|256|          (null)]} scratch-OST0001-osc-ffff883f77c95800: wait ext to 0 timedout, recovery in progress?
Mar  7 15:51:46 sprig3 kernel: [1562559.639083] Lustre: 9571:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) scratch: dirty page discard: 11.3.0.12@o2ib:11.3.0.13@o2ib:/scratch/fid: [0x5000014cc:0xd5b2:0x0]/ may get corrupted (rc -108)
Mar  7 15:51:46 sprig3 kernel: [1562559.639092] Lustre: 9571:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) Skipped 31 previous similar messages
Mar  7 15:52:25 sprig3 kernel: [1562598.574056] Lustre: 9579:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) scratch: dirty page discard: 11.3.0.12@o2ib:11.3.0.13@o2ib:/scratch/fid: [0x5000014cc:0xd5aa:0x0]/ may get corrupted (rc -108)
Mar  7 15:52:25 sprig3 kernel: [1562598.574061] Lustre: 9579:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) Skipped 76 previous similar messages
Mar  7 15:52:46 sprig3 kernel: [1562619.538999] LustreError: 27830:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0005_UUID: rc = -110 waiting for callback (2 != 0)
Mar  7 15:52:46 sprig3 kernel: [1562619.539008] LustreError: 27830:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 17 previous similar messages
Mar  7 15:52:46 sprig3 kernel: [1562619.539042] LustreError: 27830:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff883ac3c15800 x1460291788607144/t0(0) o101->scratch-OST0005-osc-ffff883f77c95800@11.3.0.14@o2ib:28/4 lens 328/368 e 0 to 0 dl 1394205665 ref 1 fl Interpret:RE/0/0 rc -5/0
Mar  7 15:52:46 sprig3 kernel: [1562619.539046] LustreError: 27830:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 31 previous similar messages
Mar  7 15:52:46 sprig3 kernel: [1562619.539054] LustreError: 27830:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0005_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Mar  7 15:52:46 sprig3 kernel: [1562619.539057] LustreError: 27830:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 17 previous similar messages
Mar  7 15:53:56 sprig3 kernel: [1562689.422742] LustreError: 225684:0:(osc_cache.c:897:osc_extent_wait()) extent ffff881a6ccc7d08@{[0 -> 1/255], [3|0|+|rpc|wihY|ffff881a68605328], [8192|2|+|-|ffff881b96d7e238|256|ffff883a94464140]} scratch-OST0007-osc-ffff883f77c95800: wait ext to 0 timedout, recovery in progress?
Mar  7 15:54:26 sprig3 kernel: [1562719.372886] LustreError: 225692:0:(osc_cache.c:897:osc_extent_wait()) extent ffff88197296bb10@{[0 -> 0/255], [3|0|+|rpc|wiY|ffff881af64cc568], [4096|1|+|-|ffff881b74c42cb8|256|ffff883a94464140]} scratch-OST0007-osc-ffff883f77c95800: wait ext to 0 timedout, recovery in progress?
Mar  7 15:54:26 sprig3 kernel: [1562719.372971] Lustre: 9594:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) scratch: dirty page discard: 11.3.0.12@o2ib:11.3.0.13@o2ib:/scratch/fid: [0x5000014cc:0xd5ae:0x0]/ may get corrupted (rc -108)
Mar  7 15:54:26 sprig3 kernel: [1562719.372975] Lustre: 9594:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) Skipped 35 previous similar messages
Mar  7 15:54:56 sprig3 kernel: [1562749.323023] LustreError: 9676:0:(osc_cache.c:897:osc_extent_wait()) extent ffff881b9812a480@{[0 -> 0/255], [3|0|+|rpc|wiY|ffff883b226fc408], [4096|1|+|-|ffff883b81fbce78|256|ffff883a94464140]} scratch-OST0007-osc-ffff883f77c95800: wait ext to 0 timedout, recovery in progress?
Mar  7 15:55:01 sprig3 /usr/sbin/cron[28014]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 15:55:01 sprig3 /usr/sbin/cron[28015]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 15:55:01 sprig3 /usr/sbin/cron[28016]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 15:57:46 sprig3 kernel: [1562919.040578] LustreError: 27830:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0005_UUID: rc = -110 waiting for callback (2 != 0)
Mar  7 15:57:46 sprig3 kernel: [1562919.040587] LustreError: 27830:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 26 previous similar messages
Mar  7 15:57:46 sprig3 kernel: [1562919.040612] LustreError: 27830:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff883ac3c15800 x1460291788607144/t0(0) o101->scratch-OST0005-osc-ffff883f77c95800@11.3.0.14@o2ib:28/4 lens 328/368 e 0 to 0 dl 1394205665 ref 1 fl Interpret:RE/0/0 rc -5/0
Mar  7 15:57:46 sprig3 kernel: [1562919.040621] LustreError: 27830:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 47 previous similar messages
Mar  7 15:57:46 sprig3 kernel: [1562919.040628] LustreError: 27830:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0005_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Mar  7 15:57:46 sprig3 kernel: [1562919.040632] LustreError: 27830:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 26 previous similar messages
Mar  7 15:59:26 sprig3 kernel: [1563018.874733] Lustre: 9596:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) scratch: dirty page discard: 11.3.0.12@o2ib:11.3.0.13@o2ib:/scratch/fid: [0x5000014cc:0xd5ab:0x0]/ may get corrupted (rc -108)
Mar  7 15:59:26 sprig3 kernel: [1563018.874742] Lustre: 9596:0:(llite_lib.c:2506:ll_dirty_page_discard_warn()) Skipped 22 previous similar messages
Mar  7 15:59:51 sprig3 kernel: [1563043.537912] LustreError: 28028:0:(file.c:2716:ll_inode_revalidate_fini()) scratch: revalidate FID [0x200000007:0x1:0x0] error: rc = -4
Mar  7 16:00:01 sprig3 /usr/sbin/cron[28037]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 16:00:01 sprig3 /usr/sbin/cron[28038]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 16:00:01 sprig3 /usr/sbin/cron[28039]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 16:04:56 sprig3 kernel: [1563348.325614] LustreError: 225707:0:(osc_cache.c:897:osc_extent_wait()) extent ffff881adab76870@{[0 -> 0/255], [3|0|+|rpc|wiY|ffff881a6f31de68], [4096|1|+|-|ffff881b8689e438|256|ffff881f4fc82280]} scratch-OST0005-osc-ffff883f77c95800: wait ext to 0 timedout, recovery in progress?
Mar  7 16:05:01 sprig3 /usr/sbin/cron[28076]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 16:05:01 sprig3 /usr/sbin/cron[28075]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 16:05:01 sprig3 /usr/sbin/cron[28077]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 16:05:06 sprig3 kernel: [1563358.308966] LustreError: 9674:0:(osc_cache.c:897:osc_extent_wait()) extent ffff881a6ccc7b10@{[0 -> 6/255], [3|0|+|rpc|wiY|ffff881a4c09bd48], [28672|7|+|-|ffff881b8bf6a938|256|ffff881f4fc82280]} scratch-OST0005-osc-ffff883f77c95800: wait ext to 0 timedout, recovery in progress?
Mar  7 16:06:21 sprig3 kernel: [1563433.471743] LustreError: 27850:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0003_UUID: rc = -110 waiting for callback (2 != 0)
Mar  7 16:06:21 sprig3 kernel: [1563433.471762] LustreError: 27850:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 46 previous similar messages
Mar  7 16:06:21 sprig3 kernel: [1563433.471770] LustreError: 27850:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff881f4f781800 x1460291788619272/t0(0) o4->scratch-OST0003-osc-ffff883f77c95800@11.3.0.14@o2ib:6/4 lens 488/448 e 0 to 0 dl 1394206453 ref 2 fl Rpc:RE/0/ffffffff rc -5/-1
Mar  7 16:06:21 sprig3 kernel: [1563433.471774] LustreError: 27850:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 85 previous similar messages
Mar  7 16:06:21 sprig3 kernel: [1563433.471781] LustreError: 27850:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0003_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Mar  7 16:06:21 sprig3 kernel: [1563433.471784] LustreError: 27850:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 47 previous similar messages
Mar  7 16:09:27 sprig3 sshd[28089]: Accepted publickey for root from 11.1.0.8 port 47669 ssh2
Mar  7 16:09:28 sprig3 sshd[28089]: Received disconnect from 11.1.0.8: 11: disconnected by user
Mar  7 16:09:28 sprig3 kernel: [1563619.642571] LustreError: 28091:0:(file.c:2716:ll_inode_revalidate_fini()) scratch: revalidate FID [0x200000007:0x1:0x0] error: rc = -4
Mar  7 16:10:01 sprig3 /usr/sbin/cron[28118]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 16:10:01 sprig3 /usr/sbin/cron[28119]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 16:10:01 sprig3 /usr/sbin/cron[28120]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 16:15:01 sprig3 /usr/sbin/cron[28140]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 16:15:01 sprig3 /usr/sbin/cron[28143]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
Mar  7 16:15:01 sprig3 /usr/sbin/cron[28142]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 16:16:21 sprig3 kernel: [1564032.478143] LustreError: 27847:0:(import.c:324:ptlrpc_invalidate_import()) scratch-OST0007_UUID: rc = -110 waiting for callback (1 != 0)
Mar  7 16:16:21 sprig3 kernel: [1564032.478152] LustreError: 27847:0:(import.c:324:ptlrpc_invalidate_import()) Skipped 54 previous similar messages
Mar  7 16:16:21 sprig3 kernel: [1564032.478169] LustreError: 27847:0:(import.c:350:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff881f4f66e000 x1460291788674424/t0(0) o4->scratch-OST0007-osc-ffff883f77c95800@11.3.0.15@o2ib:6/4 lens 488/448 e 0 to 0 dl 1394206758 ref 2 fl Rpc:RE/0/ffffffff rc -5/-1
Mar  7 16:16:21 sprig3 kernel: [1564032.478177] LustreError: 27847:0:(import.c:350:ptlrpc_invalidate_import()) Skipped 99 previous similar messages
Mar  7 16:16:21 sprig3 kernel: [1564032.478183] LustreError: 27847:0:(import.c:366:ptlrpc_invalidate_import()) scratch-OST0007_UUID: RPCs in "Unregistering" phase found (0). Network is sluggish? Waiting them to error out.
Mar  7 16:16:21 sprig3 kernel: [1564032.478189] LustreError: 27847:0:(import.c:366:ptlrpc_invalidate_import()) Skipped 55 previous similar messages
Mar  7 16:20:01 sprig3 /usr/sbin/cron[28179]: (root) CMD ([ -x /awe/sbin/firewall/firewall-status.pl ] && /awe/sbin/firewall/firewall-status.pl > /dev/null 2>&1)
Mar  7 16:20:01 sprig3 /usr/sbin/cron[28180]: (root) CMD (   [ -x /awe/opt/ironside/run_ironside.sh ] && /awe/opt/ironside/run_ironside.sh > /dev/null 2>&1)
Mar  7 16:20:01 sprig3 /usr/sbin/cron[28178]: (root) CMD ([ -x /usr/local/sbin/awe.check_osts.sh ] && /usr/local/sbin/awe.check_osts.sh > /dev/null 2>&1)
