Jun  6 09:48:01 ehyperion348 Lustre: DEBUG MARKER: mds has failed over 6 times, and counting...
Jun  6 09:48:14 ehyperion348 LustreError: 10140:0:(client.c:2347:ptlrpc_replay_interpret()) @@@ status -116, old was 0  req@ffff81022af52400 x1403992854014310/t81604548746 o35->lustre-MDT0000_UUID@192.168.120.126@o2ib:23/10 lens 360/1352 e 0 to 1 dl 1339001315 ref 2 fl Interpret:R/4/0 rc -116/-116
Jun  6 09:48:14 ehyperion348 LustreError: 13572:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -11 from cancel RPC: canceling anyway
Jun  6 09:48:14 ehyperion348 LustreError: 13572:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -11
Jun  6 09:48:14 ehyperion348 Lustre: lustre-MDT0000-mdc-ffff810212362800: Connection restored to service lustre-MDT0000 using nid 192.168.120.126@o2ib.
Jun  6 09:50:44 ehyperion348 LustreError: 11-0: an error occurred while communicating with 192.168.120.126@o2ib. The ldlm_enqueue operation failed with -107
Jun  6 09:50:44 ehyperion348 LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
Jun  6 09:50:44 ehyperion348 LustreError: 25170:0:(mdc_locks.c:653:mdc_enqueue()) ldlm_cli_enqueue error: -4
Jun  6 09:50:44 ehyperion348 LustreError: 25170:0:(file.c:3331:ll_inode_revalidate_fini()) failure -4 inode 180355073
Jun  6 09:53:03 ehyperion348 LustreError: 17939:0:(o2iblnd_cb.c:2534:kiblnd_rejected()) 192.168.117.3@o2ib rejected: o2iblnd fatal error
Jun  6 09:53:03 ehyperion348 LustreError: 17939:0:(o2iblnd_cb.c:2534:kiblnd_rejected()) Skipped 39 previous similar messages
Jun  6 09:57:22 ehyperion348 mrshd[25363]: root@hyperion318.llnl.gov as root: cmd='PATH=/admin/scripts:/admin/bin:/bin:/usr/bin:/sbin:/usr/sbin;(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; sh -c "/usr/sbin/lctl mark Duration:               86400 Server failover period: 600 seconds Exited after:           3041 seconds Number of failovers before exit: mds: 6 times ost1: 0 times ost2: 0 times ost3: 0 times ost4: 0 times ost5: 0 times ost6: 0 times ost7: 0 times ost8: 0 times Status: FAIL: rc=1");echo XXRETCODE:$?'
Jun  6 09:57:22 ehyperion348 Lustre: DEBUG MARKER: Duration: 86400
Jun  6 09:57:23 ehyperion348 mrshd[25382]: root@hyperion318.llnl.gov as root: cmd='PATH=/admin/scripts:/admin/bin:/bin:/usr/bin:/sbin:/usr/sbin;(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; sh -c "test -f /tmp/client-load.pid &&         { kill -s TERM \$(cat /tmp/client-load.pid); rm -f /tmp/client-load.pid; }");echo XXRETCODE:$?'
Jun  6 10:00:17 ehyperion348 mrshd[25394]: root@ehyperion0 as root: cmd='rdistd -S'
Jun  6 10:03:12 ehyperion348 LustreError: 17939:0:(o2iblnd_cb.c:2534:kiblnd_rejected()) 192.168.117.3@o2ib rejected: o2iblnd fatal error
Jun  6 10:03:12 ehyperion348 LustreError: 17939:0:(o2iblnd_cb.c:2534:kiblnd_rejected()) Skipped 39 previous similar messages
Jun  6 10:13:22 ehyperion348 LustreError: 17939:0:(o2iblnd_cb.c:2534:kiblnd_rejected()) 192.168.117.3@o2ib rejected: o2iblnd fatal error
Jun  6 10:13:22 ehyperion348 LustreError: 17939:0:(o2iblnd_cb.c:2534:kiblnd_rejected()) Skipped 39 previous similar messages
