Jun  6 09:48:01 hyperion-mds3 Lustre: DEBUG MARKER: mds has failed over 6 times, and counting...
Jun  6 09:48:14 hyperion-mds3 Lustre: lustre-MDT0000: sending delayed replies to recovered clients
Jun  6 09:48:14 hyperion-mds3 Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0005_UUID now active, resetting orphans
Jun  6 09:48:14 hyperion-mds3 Lustre: Skipped 5 previous similar messages
Jun  6 09:50:44 hyperion-mds3 LustreError: 0:0:(ldlm_lockd.c:356:waiting_locks_callback()) ### lock callback timer expired after 150s: evicting client at 192.168.114.65@o2ib  ns: mdt-ffff8108340da000 lock: ffff810428824000/0xe77c1b87ad8303a9 lrc: 3/0,0 mode: PR/PR res: 180355073/3441793990 bits 0x3 rrc: 109 type: IBT flags: 0x20 remote: 0xc2ba3eafe5e6d3b6 expref: 8 pid: 18299 timeout: 4451957705
Jun  6 09:50:44 hyperion-mds3 LustreError: 0:0:(ldlm_lockd.c:356:waiting_locks_callback()) ### lock callback timer expired after 158s: evicting client at 192.168.114.42@o2ib  ns: mdt-ffff8108340da000 lock: ffff81012f8426c0/0xe77c1b87ad83036a lrc: 3/0,0 mode: PR/PR res: 180355073/3441793990 bits 0x3 rrc: 109 type: IBT flags: 0x20 remote: 0x89950a07d9564dab expref: 8 pid: 18299 timeout: 4451957705
Jun  6 09:50:44 hyperion-mds3 LustreError: 18470:0:(ldlm_lockd.c:1226:ldlm_handle_enqueue0()) ### lock on destroyed export ffff8101641d7800 ns: mdt-ffff8108340da000 lock: ffff810f937d7240/0xe77c1b87ad83071b lrc: 3/0,0 mode: PR/PR res: 180355073/3441793990 bits 0x3 rrc: 90 type: IBT flags: 0x4000000 remote: 0x908556673d49456d expref: 3 pid: 18470 timeout: 0
Jun  6 09:50:44 hyperion-mds3 LustreError: 18488:0:(ldlm_lockd.c:1226:ldlm_handle_enqueue0()) ### lock on destroyed export ffff810436a4c400 ns: mdt-ffff8108340da000 lock: ffff810837de36c0/0xe77c1b87ad8307df lrc: 3/0,0 mode: PR/PR res: 180355073/3441793990 bits 0x3 rrc: 92 type: IBT flags: 0x4000000 remote: 0x89950a07d9c11d8e expref: 3 pid: 18488 timeout: 0
Jun  6 09:57:22 hyperion-mds3 mrshd[18504]: root@ehyperion318 as root: cmd='PATH=/admin/scripts:/admin/bin:/bin:/usr/bin:/sbin:/usr/sbin;(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; sh -c "/usr/sbin/lctl mark Duration:               86400 Server failover period: 600 seconds Exited after:           3041 seconds Number of failovers before exit: mds: 6 times ost1: 0 times ost2: 0 times ost3: 0 times ost4: 0 times ost5: 0 times ost6: 0 times ost7: 0 times ost8: 0 times Status: FAIL: rc=1");echo XXRETCODE:$?'
Jun  6 09:57:22 hyperion-mds3 Lustre: DEBUG MARKER: Duration: 86400
Jun  6 09:57:23 hyperion-mds3 mrshd[18523]: root@ehyperion318 as root: cmd='PATH=/admin/scripts:/admin/bin:/bin:/usr/bin:/sbin:/usr/sbin;(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; sh -c "/usr/sbin/lctl dk > /home/white215/test_logs/2012-06-06/085655/recovery-mds-scale.test_failover_mds.debug_log.\$(hostname -s).1339001843.log;          dmesg > /home/white215/test_logs/2012-06-06/085655/recovery-mds-scale.test_failover_mds.dmesg.\$(hostname -s).1339001843.log");echo XXRETCODE:$?'
Jun  6 10:00:02 hyperion-mds3 mrshd[18543]: root@ehyperion0 as root: cmd='rdistd -S'
